A method for underwater hull surface defect detection based on lidar and camera fusion

Abstract

This invention provides a method for detecting underwater ship hull surface defects based on the fusion of lidar and camera technologies. The method includes: S1, acquiring and processing point cloud data and image data of weld crack defects on the underwater ship hull surface; S2, extracting features from the point cloud data and image data of the weld crack defects on the underwater ship hull surface, obtaining identical features, and achieving data fusion; S3, transforming the fused point cloud data of the weld crack defects on the underwater ship hull surface into an image coordinate system, and performing a completion algorithm, which will generate edge noise, resulting in the underwater ship hull surface defect detection result. This method combines the advantages of both image processing and lidar, obtaining more comprehensive and complete features after fusing image point cloud data to improve the accuracy of underwater target detection; and effectively solves the accuracy and stability issues of underwater ship hull surface defects in complex aquatic environments by fusing underwater spatial information and image semantic information.

Description

Technical Field

[0001] This invention relates to the field of underwater target detection technology, and in particular to a method for detecting defects on the surface of underwater ship hulls based on the fusion of lidar and camera. Background Technology

[0002] Welding defects, corrosion defects, and large-scale deformation are three common forms of surface damage to underwater hulls, affecting their structural integrity and watertightness. Welding defects, being the most difficult type of surface damage to detect, are characterized by cracks, porosity, and slag inclusions, making them extremely challenging to identify. When conducting underwater target detection, especially of underwater hulls, the influence of the water body means that information from a single sensor is insufficient for accurate detection of welding defects. LiDAR, as an active sensor, offers strong distance perception and complete 3D information, but its imaging details are sparse. Cameras, as passive (visible light) sensors, can acquire the shape and color of targets, suitable for objects with rich detail. However, they lack depth information and are significantly affected by external lighting conditions. The above methods only provide a 2D bounding box and cannot detect target distance information. Therefore, data fusion technology combining LiDAR and camera data is needed to improve the accuracy and reliability of measurements.

[0003] Current LiDAR camera data fusion technology can be divided into three levels: pixel-level LiDAR camera data fusion, feature-level LiDAR image fusion, and decision-level LiDAR image fusion. Current research mainly focuses on the lowest level, pixel-level LiDAR camera data fusion, which has a wide range of applications. Feature-level LiDAR camera data fusion, as a higher level, performs data fusion processing based on shape and texture features. However, there are no unified rules for feature selection and feature weights, limiting its application. Decision-level LiDAR camera data fusion is the highest level operation, requiring the user to set fusion rules, resulting in a relatively high degree of subjectivity and uncertainty in the fusion results.

[0004] Currently, multi-sensor fusion target detection algorithms are becoming increasingly common. For example, detection schemes based on LiDAR and cameras, such as the representative F-PointNet, employ a frustum approach. The main process first obtains 2D target bounding boxes using image detection methods; then, through sensor calibration, the 2D bounding boxes are projected onto the point cloud to generate the frustum; finally, the PointNet network generates accurate target bounding boxes, with the detection accuracy depending on the generation of the 2D bounding boxes. Other methods utilize depth information from point clouds and interpolation algorithms to obtain depth images (RGB-D), ultimately improving detection accuracy through feature-level fusion. A 3D object detection framework based on depth image data—Frustum PointNets—has also been proposed. This method operates directly on the original point cloud data, utilizing mature deep learning networks for 2D and 3D target detection respectively for localization, achieving good detection performance. However, most current target detection fusion algorithms do not consider underwater environments, making it difficult to perform target detection tasks in situations with weak visual perception, such as underwater environments. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an underwater hull surface defect detection method based on the fusion of lidar and camera. This method combines the advantages of both image processing and lidar, fusing image point cloud data to obtain more comprehensive and complete features to improve the accuracy of underwater target detection. By fusing underwater spatial information and image semantic information, it effectively solves the problem of accurate measurement and stable detection of underwater hull surface defects in complex aquatic environments.

[0006] To achieve the above objectives, the present invention discloses the following technical solution: A method for detecting surface defects on underwater ship hulls based on the fusion of lidar and camera is provided, comprising:

[0007] S1: Acquire and process point cloud data and image data of welding crack defects on the surface of underwater hull;

[0008] S11: Use an underwater lidar based on line structured light to scan the welding cracks and defects on the surface of the underwater hull to obtain point cloud data; use an underwater camera to capture images of the welding cracks and defects on the surface of the underwater hull to obtain image data.

[0009] S12: Use point cloud data normal vector estimation, Gaussian filtering to remove high-frequency information from point cloud data, and SE-SSD to enhance point cloud data to obtain the contour information of cracks on the hull surface, thus realizing the preprocessing operation of point cloud data.

[0010] S13: Preprocessing of underwater ship hull surface welding crack defect image data is achieved by using homomorphic filtering, histogram-based image enhancement, and a repair algorithm based on light attenuation prior and background light fusion.

[0011] S2: Extract features from the point cloud data and image data of welding crack defects on the underwater hull surface, obtain the same features, and achieve data fusion. This includes the following sub-steps:

[0012] S21: Perform grayscale processing on the underwater hull surface welding crack defect image data after preprocessing in step S1, calculate the average value of the three components R, G and B of each pixel in the image data point by point, and then assign the average value to the three components R, G and B of the pixel.

[0013] S22: Extract features from the processed point cloud data and image data of welding crack defects on the underwater hull surface;

[0014] S23: Obtain the common features of the point cloud data and image data of welding crack defects on the underwater hull surface, and use the ICP point cloud matching algorithm to calculate the transformation relationship between the underwater lidar coordinate system and the underwater camera coordinate system to achieve data fusion; the ICP point cloud matching algorithm calculates the rotation matrix R and translation matrix t of the underwater camera, and the calculation expression for transforming from the underwater camera coordinate system P to the underwater lidar coordinate system Q is as follows:

[0015] q i =Rp i +t;

[0016] Where, q i Let R be the coordinates of the i-th underwater lidar; R is the rotation matrix of the underwater camera; p i Let be the coordinates of the i-th underwater camera; t be the translation matrix of the underwater camera; and i be the coordinate number.

[0017] Simultaneously, the objective function M for minimizing the optimization process of the ICP point cloud matching algorithm is constructed. ICP As shown below:

[0018]

[0019] Among them, M ICP The objective function to be minimized in the optimization process of the ICP point cloud matching algorithm; n is the total number of coordinates;

[0020] Match the closest point pair (P) in the underwater camera coordinate system P and the underwater lidar coordinate system Q. i Q i The rotation matrix R and translation matrix t of the underwater camera were optimized using the ICP point cloud matching algorithm.

[0021] S3: The point cloud data of welding crack defects on the underwater hull surface, fused in step S2, is transformed into an image coordinate system to obtain a sparse depth map. The sparse depth map is then completed by densifying it. After the completion operation using the completion algorithm, edge noise will be generated. Guided filtering is used to smooth it. The output function of the pixel is shown below:

[0022]

[0023] Where Q is the output RGB-D image; I is the guide image; k is the corresponding pixel index; a is the first coefficient of the linear function when the observation window is at k; b is the second coefficient of the linear function when the observation window is at k; w k For guiding the set of image pixels;

[0024] After correcting the information distortion under stateless guidance, a depth image that meets the requirements is obtained, which is then completed to form an RGB-D image, thereby obtaining the detection results of underwater ship hull surface defects.

[0025] Preferably, step S12 uses point cloud data normal vector estimation, specifically as follows:

[0026] The normal vector of the point cloud data is estimated using the principle of point cloud normal vector estimation. The point cloud data of the welding crack defect on the surface of the underwater ship hull is a set of fixed-point sample point cloud datasets on the surface of the real object. Normal vector estimation is performed on each point in the point cloud dataset as an approximate inference of the surface normal. The problem of estimating the normal vector of a point on the surface is the problem of approximating the normal vector of a tangent plane on the surface. Normal vector estimation is essentially estimating a plane in the neighborhood of each point. Estimating a plane requires a point and a normal vector. The normal vector is the normal vector to be estimated. This point is the average value of this group of neighborhood points. A direction l is sought such that the distribution of the projection points of all neighborhood points on direction l is most concentrated. The specific method is as follows.

[0027]

[0028] Where, x i Let c be the j-th point cloud data; c is the average point value of the object surface point cloud data; l f is the normal vector of the point cloud data on the object surface; j is the point cloud dataset number; m is the total number of point cloud datasets; min is the function for finding the minimum value.

[0029] Preferably, in step S12, Gaussian filtering is used to remove high-frequency information from the point cloud data of welding crack defects on the underwater hull surface, specifically as follows:

[0030] Gaussian filtering uses a Gaussian function to smooth point clouds. It takes the average pixel value within the filtering window, and the weights of pixels at different positions within the window are calculated using the Gaussian function. This weight window is called the Gaussian kernel function. The method for obtaining the Gaussian kernel function is shown below:

[0031]

[0032] Wherein G(x) d ,y d ,σ) is the Gaussian kernel function; x d y is the x-coordinate of the pixels within the window; d σ represents the ordinate of the pixels within the window; σ is the standard deviation of the Gaussian function.

[0033] Using the obtained Gaussian circle symmetric function G(x) d ,y d The Gaussian filter is performed by convolving the point cloud data (σ) with the data to be processed. The calculation expression is shown below:

[0034] F(x d ,y d )=G(x d ,y d ,σ)*f(x d ,y d );

[0035] Where F(x) d ,y d f(x) represents the result of convolution calculation on point cloud data; d ,y d () represents unprocessed point cloud data.

[0036] Preferably, in step S12, SE-SSD is used to enhance the point cloud data of welding crack defects on the underwater hull surface, specifically as follows:

[0037] Based on the data in the point cloud data, determine its ground truth bounding box center, and connect the ground truth bounding box center to the vertex of the ground truth bounding box to form a cone. Divide the point cloud data into six subsets, and perform augmentation operations on the six subsets of point cloud data. Specifically, this includes the following three sub-steps:

[0038] S121: Random deletion sub-step: Randomly delete all points in a randomly selected pyramid to simulate an object being partially occluded by obstacles, in order to help the network infer the complete shape from the remaining points;

[0039] S122: Random swap sub-step: Randomly select an input target in the current scene and swap the point set with the point set of another input target, thereby increasing the diversity of target samples by utilizing the surface similarity between targets;

[0040] S123: Farthest sampling sub-step: Randomly sample sub-sample points in a randomly selected pyramid to simulate the sparsity variation of points caused by the change in the distance between the underwater lidar and the location of the defect to be detected on the hull.

[0041] Preferably, in step S13, homomorphic filtering is used to process the underwater hull surface welding crack defect image data, specifically as follows:

[0042] The high and low frequencies of the image data are processed separately to achieve high-frequency enhancement and low-frequency suppression. To achieve low-frequency suppression and high-frequency enhancement, the sigmoid function is introduced into homomorphic filtering, resulting in a high-pass filtering model, as shown below:

[0043]

[0044] Where H(u,v) is the high-pass filtering model; r H For high-frequency gain; r L is the low-frequency gain; k is the high-pass filter constant, used to control the slope of the transition surface between high and low frequencies of the filter; D(u,v) represents the distance from the frequency coordinate (u,v) to the center frequency origin coordinate (u0,v0); D0 is the cutoff frequency;

[0045] The expression for calculating the distance from the frequency coordinates (u, v) to the center frequency origin coordinates (u0, v0) is as follows:

[0046]

[0047] Where u is the x-axis of the center frequency; v is the y-axis of the center frequency; u0 is the x-axis of the center frequency origin; and v0 is the y-axis of the center frequency origin.

[0048] Preferably, the underwater hull surface welding crack defect image enhancement based on histogram in step S13 specifically includes:

[0049] Using the HE algorithm, histogram-based image enhancement is performed, transforming the image histogram from a narrow, unimodal histogram to a more uniformly distributed histogram. Let l... h With w h These are the original image data and the histogram-equalized image gray levels L, respectively. h and w h Normalize so that l h and w h If l remains in the interval [0,1], then for l h The transformation is performed using the following formula, as shown below:

[0050]

[0051] Among them, l hThe image grayscale level of the original image data; w h The image gray levels after histogram equalization; T(l h ) is the transformation function; P(l) h ) is the probability density function;

[0052] The transformation function is the cumulative distribution function of l, as shown below.

[0053]

[0054] If the probability density function P(l) h ) and transformation function w h =T(l h If the probability density function of the gray levels of the transformed image is P(w), then... h ), as shown below:

[0055]

[0056] Among them, P(w h ) represents the image grayscale level w h Probability density function.

[0057] Preferably, in step S13, underwater hull surface image restoration is performed based on a light attenuation prior and a background light fusion algorithm, specifically as follows:

[0058] Histogram correction technology transforms the cumulative distribution function of input image data into a transformation function. By transforming pixel values, the original image data is transformed to produce an output image with approximately the same number of pixels at most gray levels. Based on the attenuation characteristics of background light and water light and the correlation between image regions, two background light weighted fusion methods are used to improve the accuracy of background light estimation.

[0059] The maximum intensity prior needs to consider the attenuation characteristics of different light rays in water, avoiding the influence of bright objects in the foreground area on the background light estimation. Using the assumption that the light attenuation in the red channel of an underwater image is much greater than that in the green and blue channels to estimate the background light, then... The method for obtaining it is as follows:

[0060]

[0061] in, As the primary background light; The light intensity of the dark channel R channel; The light intensity of the dark channel G channel; The light intensity of the dark channel B channel; I c is the background light transformation function; p is the pixel position of the background light obtained from the input image; max is the maximization function; argmin is the minimization function;

[0062] The difference between the light intensity of the red channel and the maximum intensity of the blue-green channel is calculated using the above two formulas. The larger the value, the closer the camera is to the scene point, and the lower the attenuation rate of the red light is compared to distant scene points. The smaller the value, the more distant the pixel is. The first background light is determined by the pixel position of the minimum value.

[0063] The background light is determined by the pixel position of the minimum value. The image is evenly divided into 4 blocks using a quadtree partitioning method, and the light intensity variance of these blocks is calculated. The block with the smallest variance is then further divided until the fifth layer is reached. The average light intensity of the image block with the smallest variance in that layer is then determined and used as the second background light, as shown below:

[0064]

[0065] in, As the second background light; The average light intensity; I c (x) represents the light intensity of the block;

[0066] The two background lights are compared using the following expression:

[0067]

[0068] in, Maximum background light; Minimum background light;

[0069] Set the linear weighted blending function for the background light as follows:

[0070]

[0071] Where λ(α,β) is the linear weighted fusion function of the background light; α is the proportion of pixels with values ​​greater than half in each channel of the unrestored underwater image; β is a fixed threshold; m λ These are fixed parameters that are set.

[0072] The linear weighted fusion function for the background light is the Sigmoid function, with an output value ranging from (0,1).

[0073] Preferably, in step S22, feature extraction is performed on the processed underwater hull surface welding crack defect point cloud data and image data using a scale-invariant feature transformation method, specifically:

[0074] S221: Scale-space peak selection. Keypoints that potentially satisfy scale invariance and rotation invariance are selected in the scale space. A Laplacian of Gaussian is used to construct the image's scale space, and Gaussian kernels with different variances are filtered, as shown below:

[0075] Lm(x,y,σ1)=G1(x,y,σ1)*Im(x,y);

[0076] Where Lm(x,y,σ1) is the filtering function of the homoscedastic Gaussian kernel, Im(x,y) is the original image data, G1(x,y,σ1) is the three-dimensional Gaussian kernel function, σ1 is the scale space standard deviation, x is the abscissa of the original image, and y is the ordinate of the original image.

[0077] Since G1 is a three-dimensional Gaussian kernel function in scale space, as shown below:

[0078]

[0079] S222: Keypoint localization, accurately locating the positions of feature keypoints, involving the removal of false keypoints; low-contrast candidate keypoints are removed, and a Taylor series expansion is performed in scale space, as shown below:

[0080]

[0081] in, is the Taylor expansion; D is the Taylor expansion in DoG scale space;

[0082] Taking the derivative of the Taylor expansion D over the DoG scale space yields a matrix; taking the derivative again and setting the function value to 0 gives the extrema of the key point's exact location.

[0083] The Hessian matrix of the DoG surface D is calculated as follows:

[0084]

[0085] Where H is the Hessian matrix; D xx The gradient change in the x-direction on the DoG; D xy The product of Dx and Dy; D yy Let represent the gradient change along the y-direction on the DoG matrix; Tr(H) is the trace of the H matrix; λ1 is the first eigenvalue; λ2 is the second eigenvalue; and Det(H) is the determinant of the H matrix.

[0086] Non-critical points are removed, as shown below:

[0087]

[0088] Where r is the ratio of eigenvalues;

[0089] S223: Orientation assignment. Based on the local gradient direction of the keypoint, an orientation is assigned to each corresponding keypoint. For an image Lm smoothed at a certain scale, the central derivative of Lm at each keypoint is calculated, and then the gradient and orientation information at each keypoint are calculated, as shown below:

[0090]

[0091] Where m(x,y) is the position function of the keypoint; θ(x,y) is the direction function of the keypoint; and Lm is the scale-smoothed image.

[0092] S224: Keypoint description. For each keypoint, a high-dimensional vector is used to describe each keypoint.

[0093] Compared with the prior art, the present invention has the following beneficial effects:

[0094] This invention provides a method for detecting underwater ship surface defects based on the fusion of lidar and camera. Compared with traditional methods, this method combines the advantages of both image and lidar, and obtains more comprehensive and complete features after fusing image point cloud data to improve the accuracy of underwater target detection. By fusing underwater spatial information and image semantic information, this method effectively solves the problem of accurately measuring underwater ship surface defects in complex aquatic environments. Attached Figure Description

[0095] Figure 1 This is a flowchart of the underwater hull surface defect detection method based on lidar and camera fusion according to the present invention;

[0096] Figure 2 This is a schematic diagram of a data preprocessing module according to an embodiment of the present invention;

[0097] Figure 3 These are comparison images of the Gaussian filtering effect on the point cloud of the target component before and after according to an embodiment of the present invention;

[0098] Figure 4 This is a schematic diagram of the feature extraction module according to an embodiment of the present invention;

[0099] Figure 5 This is a SIFT key point detection map of the target part point cloud according to an embodiment of the present invention;

[0100] Figure 6 This is a boundary extraction map of the target component defect point cloud according to an embodiment of the present invention;

[0101] Figure 7This is a flowchart of the joint calibration module according to an embodiment of the present invention;

[0102] Figure 8 This is a schematic diagram of the data fusion processing module according to an embodiment of the present invention. Detailed Implementation

[0103] Exemplary embodiments, features, and aspects of the present invention will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0104] Welding defects, corrosion defects, and large-size deformations are three common forms of surface damage to underwater hulls, affecting the structural integrity and watertightness of the hull. Welding defects, as the most difficult form of surface damage to be detected, are characterized by cracks, porosity, and slag inclusions, and are usually difficult to detect and identify.

[0105] The present invention provides a method for detecting underwater ship hull surface defects based on the fusion of lidar and camera, such as... Figure 1 As shown, it includes:

[0106] S1: Acquire and process point cloud data and image data of welding crack defects on the surface of the underwater hull; use an underwater camera to take pictures of the underwater submarine to acquire image data. The point cloud data is in PCD format and the image data is in JPG format.

[0107] S11: Use an underwater lidar based on line structured light to scan the welded crack defects on the surface of the underwater hull to obtain point cloud data; use an underwater camera to capture images of the welded crack defects on the surface of the underwater hull to obtain image data.

[0108] S12: Use the POINTXYZ structure in the POINT class of the PCL 3D point cloud library to read PCD format point cloud data. Member variables contain 3D coordinate values ​​and floating-point numbers for alignment; for example... Figure 2 The diagram shown is a schematic of the data preprocessing module according to an embodiment of the present invention. It uses point cloud data normal vector estimation, Gaussian filtering to remove high-frequency information from the point cloud data, and SE-SSD to enhance the point cloud data, thereby obtaining the contour information of the cracks on the hull surface and realizing the preprocessing operation of the point cloud data.

[0109] Using point cloud data normal vector estimation, specifically:

[0110] The normal vector of point cloud data is estimated using the principle of point cloud normal vector estimation. Point cloud data of welding crack defects on the surface of an underwater ship hull is a set of fixed-point sample point cloud datasets on the surface of a real object. Normal vector estimation is performed on each point in the point cloud dataset as an approximate inference of the surface normal. The problem of estimating the normal vector of a tangent plane on a surface from the problem of estimating the normal vector of a single point on the surface is essentially the estimation of a plane in the neighborhood of each point. Estimating a plane requires a point and a normal vector, and the normal vector is the normal vector to be estimated. This point is the average value of the neighborhood points. A direction l is sought such that the distribution of the projection points of all neighborhood points on direction l is most concentrated. The specific method is as follows:

[0111]

[0112] Where, x i Let c be the j-th point cloud data; c is the average point value of the object surface point cloud data; l f is the normal vector of the point cloud data on the object surface; j is the point cloud dataset number; m is the total number of point cloud datasets; min is the function for finding the minimum value.

[0113] Gaussian filtering is used to remove high-frequency information from point cloud data of welding crack defects on the surface of underwater ship hulls, specifically as follows:

[0114] Gaussian filtering uses a Gaussian function to smooth point clouds. It takes the average pixel value within the filtering window, and the weights of pixels at different positions within the window are calculated using the Gaussian function. This weight window is called the Gaussian kernel function. The method for obtaining the Gaussian kernel function is shown below:

[0115]

[0116] Wherein G(x) d ,y d ,σ) is the Gaussian kernel function; x d y is the x-coordinate of the pixels within the window; d σ represents the ordinate of the pixels within the window; σ is the standard deviation of the Gaussian function.

[0117] Using the obtained Gaussian circle symmetric function G(x) d ,y d The Gaussian filter is performed by convolving the point cloud data (σ) with the data to be processed. The calculation expression is shown below:

[0118] F(x d ,y d )=G(x d ,y d ,σ)*f(x d ,y d );

[0119] Where F(x) d,y d f(x) represents the result of convolution calculation on point cloud data; d ,y d () represents unprocessed point cloud data.

[0120] like Figure 3 The image shown is a comparison of the Gaussian filtering effect on the point cloud of the target component according to an embodiment of the present invention. After Gaussian filtering, stray point clouds in the area to be detected of the target component are effectively removed. In a preferred embodiment of the present invention, the target component is a part of the surface of an underwater hull, such as the sidewall of the hull.

[0121] SE-SSD was used to enhance the point cloud data of welding crack defects on the surface of underwater ship hulls, specifically as follows:

[0122] Based on the data in the point cloud, determine its ground truth bounding box center and connect the ground truth bounding box center with the ground truth edge. After correcting the information distortion under stateless guidance, obtain a depth image that meets the requirements, and complete it to form an RGB-D image; connect the bounding box vertices to form a cone, divide the point cloud data into six subsets, and perform augmentation operations on the six subsets of point cloud data, specifically including the following three sub-steps:

[0123] S121: Random deletion sub-step: Randomly delete all points in a randomly selected pyramid to simulate an object being partially obscured by obstacles, in order to help the network infer the complete shape from the remaining points.

[0124] S122: Random swap sub-step: Randomly select an input target in the current scene and swap the point set with the point set of another input target, thereby increasing the diversity of target samples by utilizing the surface similarity between targets.

[0125] S123: Farthest sampling sub-step: Randomly sample sub-sample points in a randomly selected pyramid to simulate the sparsity variation of points caused by the change in the distance between the underwater lidar and the location of the defect to be detected on the hull.

[0126] S13: Use the Mat class in the OpenCV library to read JPG format image data, and use homomorphic filtering, histogram-based image enhancement, and a light attenuation prior and background light fusion algorithm to perform preprocessing operations on underwater ship hull surface welding crack defect image data.

[0127] Homomorphic filtering is used to process underwater ship hull surface welding crack defect image data, specifically as follows:

[0128] The high and low frequencies of the image data are processed separately to achieve high-frequency enhancement and low-frequency suppression. To achieve low-frequency suppression and high-frequency enhancement, the sigmoid function is introduced into homomorphic filtering, resulting in a high-pass filtering model, as shown below:

[0129]

[0130] Where H(u,v) is the high-pass filtering model; r H For high-frequency gain; r L is the low-frequency gain; k is the high-pass filter constant, used to control the slope of the transition surface between high and low frequencies of the filter; D(u,v) represents the distance from the frequency coordinate (u,v) to the center frequency origin coordinate (u0,v0); D0 is the cutoff frequency.

[0131] The expression for calculating the distance from the frequency coordinates (u, v) to the center frequency origin coordinates (u0, v0) is as follows:

[0132]

[0133] Where u is the x-axis of the center frequency; v is the y-axis of the center frequency; u0 is the x-axis of the center frequency origin; and v0 is the y-axis of the center frequency origin.

[0134] Image enhancement of underwater hull surface welding crack defects based on histograms, specifically:

[0135] Using the HE algorithm, histogram-based image enhancement is performed, transforming the image histogram from a narrow, unimodal histogram to a more uniformly distributed histogram. Let l... h With w h These are the original image data and the histogram-equalized image gray levels L, respectively. h and w h Normalize so that l h and w h If l remains in the interval [0,1], then for l h The transformation is performed using the following formula, as shown below:

[0136]

[0137] Among them, l h The image grayscale level of the original image data; w h The image gray levels after histogram equalization; T(l h ) is the transformation function; P(l) h ) is the probability density function.

[0138] The transformation function is the cumulative distribution function of l, as shown below.

[0139]

[0140] If the probability density function P(l) h ) and transformation function w h =T(l hIf the probability density function of the gray levels of the transformed image is P(w), then... h ), as shown below:

[0141]

[0142] Among them, P(w h ) represents the image grayscale level w h Probability density function.

[0143] The image restoration of welding crack defects on the surface of underwater ship hulls is based on an algorithm that combines optical attenuation priors and background light fusion.

[0144] Histogram correction technology transforms the cumulative distribution function of input image data into a transformation function. By transforming pixel values, the original image data transformation produces an output image with approximately the same number of pixels at most gray levels. Based on the attenuation characteristics of background light and water light and the correlation between image regions, two background light weighted fusion methods are used to improve the accuracy of background light estimation.

[0145] The maximum intensity prior needs to consider the attenuation characteristics of different light rays in water, avoiding the influence of bright objects in the foreground area on the background light estimation. Using the assumption that the light attenuation in the red channel of an underwater image is much greater than that in the green and blue channels to estimate the background light, then... The method for obtaining it is as follows:

[0146]

[0147] in, As the primary background light; The light intensity of the dark channel R channel; The light intensity of the dark channel G channel; The light intensity of the dark channel B channel; I c is the background light transformation function; p is the pixel position of the background light obtained from the input image; max is the maximization function; argmin is the minimization function.

[0148] The difference between the light intensity of the red channel and the maximum intensity of the blue-green channel is calculated using the above two formulas. The larger the value, the closer the camera is to the scene point, and the lower the attenuation rate of the red light is compared to that of distant scene points. The smaller the value, the more distant the pixel is. The first background light is determined by the pixel position of the minimum value.

[0149] The background light is determined by the pixel position of the minimum value. The image is evenly divided into 4 blocks using a quadtree partitioning method, and the light intensity variance of these blocks is calculated. The block with the smallest variance is then further divided until the fifth layer is reached. The average light intensity of the image block with the smallest variance in that layer is then determined and used as the second background light, as shown below:

[0150]

[0151] in, As the second background light; The average light intensity; I c (x) represents the light intensity of the block.

[0152] The two background lights are compared using the following expression:

[0153]

[0154] in, Maximum background light; Minimum background light.

[0155] Set the linear weighted blending function for the background light as follows:

[0156]

[0157] Where λ(α,β) is the linear weighted fusion function of the background light; α is the proportion of pixels with values ​​greater than half in each channel of the unrestored underwater image; β is a fixed threshold; m λ These are fixed parameters that are set.

[0158] The linear weighted fusion function for the background light is the Sigmoid function, with an output value ranging from (0,1).

[0159] S2: Perform feature extraction on point cloud data and image data, obtain identical features, and achieve data fusion. The input point cloud data is a POINTXYZ structure from the POINT class in the PCL library, and the input image data is a Mat class from the OpenCV library, such as... Figure 4 The diagram shown is a schematic diagram of a feature extraction module according to an embodiment of the present invention, which specifically includes the following sub-steps:

[0160] S21: Perform grayscale processing on the underwater hull surface welding crack defect image data after preprocessing in step S1, calculate the average value of the R, G and B components of each pixel in the image data point by point, and then assign the average value to the R, G and B components of this pixel.

[0161] S22: Feature extraction is performed on the processed point cloud data and image data of welding crack defects on the underwater hull surface; in step S22, feature extraction is performed on the processed point cloud data and image data of welding crack defects on the hull surface using the scale-invariant feature transformation method, specifically:

[0162] S221: Scale-space peak selection. Keypoints that potentially satisfy scale invariance and rotation invariance are selected in the scale space. A Laplacian of Gaussian is used to construct the image's scale space, and Gaussian kernels with different variances are filtered, as shown below:

[0163] Lm(x,y,σ1)=G1(x,y,σ1)*Im(x,y);

[0164] Where Lm(x,y,σ1) is the filtering function of the homoscedastic Gaussian kernel, Im(x,y) is the original image data, G1(x,y,σ1) is the three-dimensional Gaussian kernel function, σ1 is the standard deviation in scale space, x is the abscissa of the original image, and y is the ordinate of the original image.

[0165] Since G1 is a three-dimensional Gaussian kernel function in scale space, as shown below:

[0166]

[0167] S222: Keypoint localization, accurately locating the positions of feature keypoints, involving the removal of false keypoints; low-contrast candidate keypoints are removed, and a Taylor series expansion is performed in scale space, as shown below:

[0168]

[0169] in, is the Taylor expansion; D is the Taylor expansion in DoG scale space.

[0170] Taking the derivative of the Taylor expansion D over the DoG scale space yields a matrix; taking the derivative again and setting the function value to 0 gives the extrema of the key point's exact location.

[0171] The Hessian matrix of the DoG surface D is calculated as follows:

[0172]

[0173] Where H is the Hessian matrix; D xx The gradient change in the x-direction on the DoG; D xy The product of Dx and Dy; D yy Let λ be the gradient change in the y direction on DoG; Tr(H) be the trace of the H matrix; λ1 be the first eigenvalue; λ2 be the second eigenvalue; and Det(H) be the determinant of the H matrix.

[0174] Non-critical points are removed, as shown below:

[0175]

[0176] Where r is the ratio of eigenvalues.

[0177] S223: Orientation assignment. Based on the local gradient direction of the keypoint, an orientation is assigned to each corresponding keypoint. For an image Lm smoothed at a certain scale, the central derivative of Lm at each keypoint is calculated, and then the gradient and orientation information at each keypoint are calculated, as shown below:

[0178]

[0179] Where m(x,y) is the position function of the keypoint; θ(x,y) is the direction function of the keypoint; and Lm is the scale-smoothed image.

[0180] S224: Keypoint Description. For each keypoint, a high-dimensional vector is used to describe it. For example... Figure 5 The image shown is a SIFT key point detection map of a target part according to an embodiment of the present invention. As can be seen from the image, the locations of key points near the defect features of the target part are marked. Figure 6 The image shown is a boundary extraction map of the point cloud of the target part defect according to an embodiment of the present invention, which effectively extracts the boundary contour point cloud of the target part defect and effectively removes other background information.

[0181] S23: Obtain the common features of point cloud data and image data of welding crack defects on the surface of underwater hulls; the input point cloud data is a POINTXYZ structure in the POINT class of the PCL library, and the input image data is a Mat class in the OpenCV library. Obtain the common features of the point cloud data and image data, such as... Figure 7 The diagram shows a flowchart of the joint calibration module according to an embodiment of the present invention; the transformation relationship between the underwater lidar coordinate system and the underwater camera coordinate system is calculated using the ICP point cloud matching algorithm to achieve data fusion; the ICP point cloud matching algorithm calculates the rotation matrix R and translation matrix t of the underwater camera, and the calculation expression for transforming from the underwater camera coordinate system P to the underwater lidar coordinate system Q is shown below:

[0182] q i =Rp i +t;

[0183] Where, q i Let R be the coordinates of the i-th underwater lidar; R is the rotation matrix of the underwater camera; p i Let t be the coordinates of the i-th underwater camera; t be the translation matrix of the underwater camera; and i be the coordinate number.

[0184] Simultaneously, the objective function M for minimizing the optimization process of the ICP point cloud matching algorithm is constructed. ICP As shown below:

[0185]

[0186] Among them, M ICP The objective function to be minimized in the optimization process of the ICP point cloud matching algorithm is denoted as n, which is the total number of coordinates.

[0187] Match the closest point pair (P) in the underwater camera coordinate system P and the underwater lidar coordinate system Q. i Q i The rotation matrix R and translation matrix t of the underwater camera were optimized using the ICP point cloud matching algorithm.

[0188] S3: The point cloud data of welding crack defects on the underwater hull surface, fused in step S2, is transformed into an image coordinate system to obtain a sparse depth map. The sparse depth map is then completed by densifying it. After the completion operation using the IP_basic completion algorithm, edge noise is generated. Guided filtering is used to smooth it. The output function of the pixel is shown below:

[0189]

[0190] Where Q is the output RGB-D image; I is the guide image; k is the corresponding pixel index; a is the first coefficient of the linear function when the observation window is at k; b is the second coefficient of the linear function when the observation window is at k; w k This is the set of pixels in the guiding image.

[0191] like Figure 8 The diagram shown is a schematic of the data fusion processing module according to an embodiment of the present invention. After correcting information distortion under stateless guidance, a depth image that meets the requirements is obtained, which is then completed to form an RGB-D image, thereby obtaining the underwater hull surface defect detection results.

[0192] This invention provides an underwater object measurement method based on the fusion of LiDAR and camera data. The embodiments of this invention combine the advantages of both image processing and LiDAR, fusing image point cloud data to obtain more comprehensive and complete features to improve the accuracy of underwater target detection. By fusing underwater spatial information and image semantic information, the accuracy and stability of underwater vessels in complex aquatic environments are effectively improved. The method of this invention preprocesses underwater image data based on a priori light attenuation and background light fusion algorithms. According to the attenuation characteristics of background light and water light and the correlation between image regions, multiple background light weighted fusion methods are used to improve the accuracy of background light estimation, avoiding the problem of severe image degradation in complex underwater imaging environments and solving the subsequent difficulties in underwater feature recognition.

[0193] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for detecting defects on the surface of a ship hull underwater based on fusion of lidar and camera, characterized in that: It comprises the following steps: S1: obtaining and processing point cloud data and image data of underwater hull surface welding crack defects; S11: scanning the underwater hull surface welding crack defects by underwater laser radar based on line structured light to obtain point cloud data; using an underwater camera to take pictures of the underwater hull surface welding crack defects to obtain image data; S12: using point cloud data normal vector estimation, removing high-frequency information from the point cloud data using Gaussian filtering, and enhancing the point cloud data using SE-SSD, obtaining the contour information of the hull surface crack, and realizing the preprocessing operation of the point cloud data; S13: using homomorphic filtering processing, histogram-based image enhancement, and background light fusion algorithm based on light attenuation priori to realize the preprocessing operation of the underwater hull surface welding crack defect image data; S2: extracting features from the underwater hull surface welding crack defect point cloud data and image data, obtaining the same features, and realizing data fusion, specifically including the following sub-steps: S21: performing gray processing on the preprocessed underwater hull surface welding crack defect image data in step S1, calculating the average value of R, G and B components of each pixel point in the image data point by point, and then assigning the average value to the R, G and B components of the pixel point; S22: extracting features from the processed underwater hull surface welding crack defect point cloud data and image data; S23: obtaining the same features of the underwater hull surface welding crack defect point cloud data and image data, calculating the conversion relationship between the underwater laser radar coordinate system and the underwater camera coordinate system by using the ICP point cloud matching algorithm, and realizing data fusion; the ICP point cloud matching algorithm converts from the underwater camera coordinate system P to the underwater laser radar coordinate system Q by calculating the rotation matrix R and the translation matrix t, and the calculation expression is as follows: ； wherein, is the i-th underwater laser radar coordinate; is the rotation matrix of the underwater camera; is the i-th underwater camera coordinate; is the translation matrix of the underwater camera; i is the coordinate number; At the same time, the minimization objective function of the ICP point cloud matching algorithm optimization process is constructed As follows: ； wherein, is the minimization objective function for the ICP point cloud matching algorithm optimization process; n is the total number of coordinates; Match the closest point pair (P i , Q i ) in underwater camera coordinate system P and underwater lidar coordinate system Q, and use ICP point cloud matching algorithm to optimize and calculate the rotation matrix R and translation matrix t of underwater camera; S3: converting the underwater hull surface welding crack defect point cloud data fused in step S2 into an image coordinate system to obtain a sparse depth map; the sparse depth map is completed by densifying the depth map, and the output result function of the pixel point is as follows: ； wherein, is an output RGB-D image; I is a guide image; k is a corresponding pixel index; a is a first coefficient of a linear function when the observation window is located at k; b is a second coefficient of the linear function when the observation window is located at k; is a set of guide image pixel points; After correcting the information distortion under the guidance of stateless, the required depth image is obtained, the depth image is completed to form an RGB-D image, and thus the underwater hull surface defect detection result is obtained.

2. The method for detecting underwater hull surface defects based on fusion of lidar and camera according to claim 1, characterized in that: In step S12, the point cloud data normal vector estimation is used, specifically as follows: Estimate the normal vector of the point cloud data using the normal vector estimation principle of the point cloud method; the underwater hull surface welding crack defect point cloud data is a set of fixed point sample point cloud data set on the surface of a real object; the normal of each point of the point cloud data set is estimated as an approximate inference of the surface normal; the normal of a point on the surface is an approximate estimation of the normal of a tangent plane of the surface, and the normal estimation is essentially estimating a plane in the neighborhood of each point, which requires a point and a normal vector, and the normal vector is the normal vector to be estimated, and the point is the average of the neighborhood points; a direction l is sought so that the distribution of the projection points of all neighborhood points in the direction l is the most concentrated, and the specific method is as follows; ； wherein, is the jth point cloud data; is the average point of the object surface point cloud data; is the normal vector of the object surface point cloud data; j is the point cloud data set number; is the total number of point cloud data sets; is the minimum value function.

3. The method for detecting underwater hull surface defects based on fusion of lidar and camera according to claim 1, characterized in that: In the step S12, Gaussian filtering is used to remove high-frequency information from the underwater hull surface welding crack defect point cloud data, specifically as follows: Gaussian filtering is a smoothing process for point cloud using Gaussian function, taking the average value of pixels in the filter window, and the weight of pixels at different positions in the window is obtained by Gaussian function calculation, and the weight window is called the acquisition method of Gaussian kernel function, as follows: ； wherein, is a Gaussian kernel function; is a horizontal pixel coordinate within the window; is a vertical pixel coordinate within the window; is a standard deviation of the Gaussian function; Using the obtained Gaussian circular symmetric function The convolution calculation is performed with the point cloud data to be processed, and the Gaussian filtering is completed, and the calculation expression is as follows: ； wherein, is the result of a convolutional computation for point cloud data; is the unprocessed point cloud data.

4. The method for detecting underwater hull surface defects based on fusion of lidar and camera according to claim 1, characterized in that: In the step S12, SE-SSD is used to enhance the underwater hull surface welding crack defect point cloud data, specifically as follows: According to the data in the point cloud data, the true value boundary box center is determined, and the true value boundary box center and the true value boundary are corrected under the guidance of the stateless guide, and the required depth image is obtained, and the RGB-D image is completed; the frame vertices are connected to form a cone, and the point cloud data is divided into six subsets, and the six subset point cloud data is operated to enhance the operation, which specifically includes the following three sub-steps: S121: Randomly delete the sub-step: randomly delete all points in the randomly selected pyramid, simulate the object being partially blocked by the obstacle, and help the network infer the complete shape from the remaining points; S122: Randomly exchange the sub-step: randomly select an input target in the current scene, and exchange the point set to the point set in another input target, so as to increase the diversity of target samples by using the surface similarity between targets; S123: The farthest end sampling sub-step: randomly sample the sample points in the randomly selected pyramid, simulate the sparse change of points caused by the change of the distance between the underwater laser radar and the hull defect position to be detected.

5. The method for detecting underwater hull surface defects based on fusion of lidar and camera according to claim 1, characterized in that: In the step S13, homomorphic filtering is used to process the underwater hull surface welding crack defect image data, specifically as follows: The high and low frequencies of the image data are processed respectively to achieve the purpose of high frequency enhancement and low frequency suppression; in order to realize low frequency suppression and high frequency enhancement, sigmoid function is introduced into homomorphic filtering, and high pass filtering processing model is obtained, as follows: ； wherein, is a high pass filter model; is a high frequency gain; is a low frequency gain; k is a high pass filter constant to control the slope of the high-low frequency transition of the filter; represents the frequency coordinate to the center frequency origin coordinate distance; is a cut-off frequency; frequency coordinate to the center frequency origin coordinate The calculation expression of the distance is shown as follows: ； wherein is a center frequency abscissa; is a center frequency ordinate; is a center frequency origin abscissa; is a center frequency origin ordinate.

6. The method for detecting underwater hull surface defects based on fusion of lidar and camera according to claim 1, characterized in that: In the step S13, the underwater hull surface welding crack defect image is enhanced based on histogram, specifically as follows: Using HE algorithm, histogram-based image enhancement is completed, and the image histogram is transformed from a narrow unimodal histogram to an evenly distributed histogram. Set l h w h respectively are the original image data and the gray level of the histogram equalized image. Normalize l h and w h , so that l h and w h remain in the interval [0, 1]. If l h is transformed according to the following formula: ； wherein, l h is the image gray level of the original image data; w h is the image gray level of the histogram equalized image; T(l h ) is the transform function; P(l h ) is the probability density function; The transformation function is the cumulative distribution function of l, as follows ； If the probability density function P(l h ) and the transform function w h = T(l h ), then the probability density function P(w h ) of the transformed image gray levels is given by: ； wherein for image gray level w h Probability density function.

7. The method for detecting underwater hull surface defects based on fusion of lidar and camera according to claim 1, characterized in that: In the step S13, the underwater hull surface welding crack defect image is repaired based on the light attenuation prior and background light fusion algorithm, specifically as follows: The histogram correction technique of converting the cumulative distribution function of input image data into a transform function makes the original image data transformed by pixel value transformation to have a certain same number of pixels on most gray levels; according to the correlation between the background light and the light attenuation characteristics of the water body and the image region, two background light weighted fusion methods are adopted to improve the accuracy of the background light estimation; The maximum intensity prior needs to avoid the influence of the existence of brighter objects in the foreground region on the background light estimation according to the attenuation characteristics of different lights in the water body. The light attenuation degree of the underwater image in the red channel is much greater than that in the green and blue channels, and the background light is estimated by using the assumption that The acquisition method is as follows: ； wherein, is the first background light; is the light intensity of the dark channel R channel; is the light intensity of the dark channel G channel; is the light intensity of the dark channel B channel; is the background light transfer function; p is the pixel position from which the background light is obtained from the input image; is the max function; is the min function; According to the above two formulas, the difference between the light intensity of the red channel and the maximum intensity of the blue-green channel is calculated, the larger the value, the closer the camera distance to the scene point, and the red light attenuation rate is smaller than the far scene point; the smaller the value, the pixel point is a far scene point, and the minimum value of the pixel position is determined as the first background light; The background light is determined by the minimum value of the pixel position, and the image is uniformly divided into four blocks by the quadtree division method, and the light intensity variance of the blocks is calculated, the block with the minimum variance is determined, and then it is further divided, until it is divided into the fifth layer, and the light intensity mean value of the block with the minimum variance in the layer is determined as the second background light, as follows: ； wherein, is a second background light; is a mean light intensity; is a block light intensity; The two background lights are compared, and the expression is as follows: ； wherein is the maximum background light; is the minimum background light; The linear weighted fusion function of the background light is set as follows: ； wherein, is a linearly weighted fusion function of the background light; a is the proportion of the number of pixels whose values are greater than half of the total number of pixels in each channel of the un-restored underwater image; and β is a fixed threshold value set; is a fixed parameter set The linear weighted fusion function of the background light is a Sigmoid function, and the output value range is (0, 1).

8. The method for detecting underwater hull surface defects based on fusion of lidar and camera according to claim 1, characterized in that: The feature extraction of the processed ship surface welding crack defect point cloud data and image data in step S22 is carried out by using the scale invariant feature transformation method, specifically: S221: Scale space peak selection, select potential key points that meet the scale invariance and rotation invariance in the scale space; adopt Laplacian of Gaussian to construct the scale space of the image, and filter different variance Gaussian kernels, as follows: ； wherein, Filtering function for homoscedastic Gaussian kernel is the original image data; is a three-dimensional Gaussian kernel function; is the scale space standard deviation; is the original image abscissa; is the original image ordinate; In the scale space, G1 is a three-dimensional Gaussian kernel function, as follows: ； S222: Key point positioning, accurately positioning the position of the feature key point, involving the elimination of false key points; the low-contrast candidate key points are removed, and the Taylor series expansion of the scale space is carried out, as follows: ； wherein, is the Taylor expansion; is the Taylor expansion of the DoG scale space; The derivative of the Taylor expansion D of the DoG scale space is a matrix; the second derivative is obtained, and the extreme value is the accurate position of the key point when the function value is equal to 0; The Hessian matrix of the DoG surface D is calculated, as follows: ； wherein, is the Hessian matrix matrix; is the gradient change in the x direction on the DoG; is the product of Dx and Dy; is the gradient change in the y direction on the DoG; is the trace of the H matrix; is the first eigenvalue; is the second eigenvalue; is the determinant of the H matrix; Non-key points are removed, as follows: ； wherein is the ratio of eigenvalues; S223: Direction assignment, assign a direction to each corresponding key point based on the local gradient direction of the key point; for the smoothed image Lm at a certain scale, the central derivative of Lm at each key point is calculated, and then the gradient and direction information at each key point are calculated, as follows: ； wherein, a position function for the keypoint; a direction function for the keypoint; an image smoothed for scale; S224: Key point description, for each key point, a high-dimensional vector is used to describe each key point.

Images