Method of ROV inspection of underwater submerged structures

Abstract

The application discloses a ROV inspection method for underwater submerged structures, and is specifically implemented according to the following steps: step 1, underwater deployment of an inspection robot and attitude adjustment; step 2, filtering of underwater interference information, and then adoption of a color cast correction algorithm to realize clear processing; step 3, fusion of visual and inertial navigation sensor data to estimate the position and attitude of the ROV; step 4, detection of a target of a detection structure entering the field of view of a camera, and then adoption of a target tracking algorithm based on deep learning to enable the robot to approach the underwater submerged structure; and step 5, after close-range arrival at the target, adoption of a crack detection method based on image processing to extract geometric and texture information of the underwater submerged structure with defect characteristics. The application can independently perform underwater inspection operations only by using sensor data acquired by an IMU and a camera carried by the ROV, so as to detect surface cracks, holes and other hidden dangers of the underwater submerged structure, and effectively improve work efficiency and reduce costs.

Description

Technical Field

[0001] This invention belongs to the technical field of underwater robot inspection methods, specifically relating to an ROV inspection method for underwater submerged structures. Background Technology

[0002] With the continuous enhancement of the Haihe River Engineering's capabilities in recent decades, numerous underwater submerged structures, including bridge supports, dams, and other large-scale projects of varying sizes, have been constructed. Currently, these underwater structures are gradually transitioning from the construction phase to the maintenance phase, and many structures are already showing signs of performance degradation, placing enormous pressure on maintenance work.

[0003] These structures exist in an underwater environment, subjected to erosion and wear from river currents and sediment, as well as various forms of corrosion. They are also susceptible to impacts from ships and floating objects, and the effects of aquatic organisms. These factors make underwater structures prone to various defects; therefore, effective underwater structure inspection is a prerequisite for structural assessment and repair.

[0004] Traditional inspection methods primarily involve underwater visual observation or palpation by divers, or specialized divers using underwater cameras to photograph the structure before submitting the images to experts for inspection. However, the complex and ever-changing underwater environment presents various adverse effects on divers' work. The inability to maintain effective real-time communication between divers and shore-based personnel leads to low work efficiency, and distraction poses a significant risk. Furthermore, these methods are time-consuming, require multiple personnel, and are expensive. The high costs in terms of human resources, financial resources, and time present numerous challenges to inspection work. Summary of the Invention

[0005] The purpose of this invention is to provide an ROV inspection method for underwater submersible structures, which can achieve high-efficiency and high-precision autonomous inspection.

[0006] The technical solution adopted in this invention is an ROV inspection method for underwater submersible structures, which is implemented according to the following steps:

[0007] Step 1: Deploy the inspection robot underwater and adjust its attitude;

[0008] Step 2: Filter underwater interference information, and then use a color cast correction algorithm to achieve sharpening.

[0009] Step 3: Estimate the position and attitude of the ROV by fusing visual and inertial sensor data;

[0010] Step 4: For the detected structural targets that enter the camera's field of view, use a deep learning-based target tracking algorithm to bring the robot closer to the underwater submerged structure;

[0011] Step 5: After reaching the target location at close range, extract the geometric and texture information of the underwater submerged structure with defect features using an image processing-based crack detection method.

[0012] The invention is further characterized by:

[0013] Step 2 is implemented in the following steps:

[0014] Step 2.1: Use guided filtering to denoise the underwater image while preserving the edge details in the original image;

[0015] Step 2.2: Use the color cast factor value to judge and distinguish between images without color cast and images with color cast in the underwater images;

[0016] Step 2.3: Perform color compensation on different images with color cast to obtain color-compensated images;

[0017] Step 2.4: Use the contrast-limited adaptive histogram equalization (CLAHE) algorithm to perform global contrast enhancement on the color-compensated image.

[0018] Step 3 is implemented in the following steps:

[0019] Step 3.1: Using the Raspberry Pi clock as a reference, use C++ library functions to calculate the timestamps of the camera and IMU respectively. Finally, use the positioning algorithm to estimate the time deviation to achieve timestamp synchronization between the IMU and the camera.

[0020] Step 3.2: Use a camera to acquire visual images, and use the IMU's accelerometer and gyroscope to measure acceleration and angular velocity at their respective operating frequencies.

[0021] Step 3.3: According to the multi-state constrained Kalman filter (MSCKF) algorithm, define the system state vector as follows: Where x I For the IMU state vector, Indicates camera status;

[0022] Step 3.4: Integrate the acceleration and angular velocity obtained by IMU measurement based on kinematic formulas to obtain the nominal state and attitude. The velocity is calculated using the zeroth-order integral from formula (1). and location The fourth-order Runge-Kutta method integral update is calculated using formulas (2) and (3) respectively:

[0023]

[0024]

[0025]

[0026] Where Ω(ω) is a matrix with respect to angular velocity ω, and k is the Runge-Kutta coefficient;

[0027] Step 3.5: The error state is predicted based on the noise measured by the IMU using the extended Kalman prediction equations (4) and (5):

[0028]

[0029]

[0030] in, Let F be the system state at time k. k Let P be the state transition matrix. k Q represents the covariance of the state at time k. k The covariance represents the effect of the IMU error on the state from time k-1 to k.

[0031] Step 3.6: Perform FAST feature extraction and LK optical flow feature tracking on the visual images acquired in Step 3.2. After obtaining visual observations, the system updates the error state based on the extended Kalman update equation:

[0032]

[0033]

[0034] P′ k =P k -K′H k P k =IK′H k )P k (8)

[0035] Where K′ is the Kalman gain, H k R represents the Jacobian matrix of the observation relative to the state at time k. k Represents the observed covariance matrix. It is the observation at time k;

[0036] Step 3.7: The mean of the error state is incorporated into the nominal state to obtain the true state, and then it is reset to zero. The covariance matrix of the error state is continuously updated to reflect this state update and reset process.

[0037] Step 3.8: When the accelerometer and gyroscope of the inertial measurement unit collect new acceleration and angular velocity data, return to step 3.4.

[0038] Step 4 is implemented in the following steps:

[0039] Step 4.1: Initialize the bounding box in the first frame;

[0040] Step 4.2: Assume that in frame t-1, the tracker previously predicted the target to be located at c = (c x c y The image has a width of w and a height of h. The network crops and scales the previous frame of the image to center the tracked object in the image. Frame t-1 is selected with a width of w and a height of h. x c y The clipping is centered on ), with width and height k1w and k1h respectively;

[0041] Step 4.3, cropping the current frame t with c′=(c′ x , c′ y ), where c′ is the expected average position of the target object. Let c′ = c, which is equivalent to a constant position motion model. The clipping width and height of the current frame are k2w and k2h, respectively, where w and h are the width and height of the predicted bounding box in the previous frame, and k2 is defined as the search radius of the target object.

[0042] Step 4.4: Using the two cropped frames as input, train a convolutional neural network (CNN) to predict the position of the bounding box in the current frame; both frames are passed through a set of convolutional layers, which are just the first five convolutional layers of the CaffeNet architecture. The outputs of these convolutional layers are concatenated into a vector of length 4096, which is input to three fully connected layers. The last fully connected layer is finally connected to the output layer, which contains four nodes representing the top and bottom of the bounding box;

[0043] Step 4.5: In each subsequent frame, the cropped frames t-1 and t are input into the network to predict the position of the object in frame t. The remaining parts of the video are then re-cropped and the frames are fed into the network. The network will track the motion of the target object throughout the sequence.

[0044] Step 5 is implemented in the following steps:

[0045] Step 5.1: Convert the image to grayscale;

[0046] Step 5.2: Convert the image into a format suitable for analysis and processing, and increase the contrast of the image;

[0047] Step 5.3: Canny edge detection to obtain relevant crack information;

[0048] Step 5.4: Use morphological connection to connect adjacent cracks, use seed filling method to find all connected components, select the current pixel as seed and assign a new label, and then add the positions of all its neighboring pixels with the same pixel value to the stack.

[0049] Step 5.5: Pop the top element from the stack, assign it the same label, and add the positions of all its neighboring pixels with the same pixel value to the stack. Repeat this operation until the stack is empty. At this point, a connected region is found and marked with the same label.

[0050] Step 5.6: Traverse the image, repeating steps 5.4 and 5.5 until all connected regions label_1, label_2, ..., label_n are found;

[0051] Step 5.7: Delete the noise areas that are not cracks, and finally extract the skeleton for each connected component and measure its length and width to obtain the final crack information.

[0052] The beneficial effects of this invention are: the ROV inspection method for underwater submerged structures of this invention, through four parts—image enhancement, underwater positioning, target tracking, and crack detection—can inspect surface cracks, holes, etc. of underwater submerged structures, achieving high-precision, high-efficiency, and long-distance autonomous underwater robot inspection operations, and fully leveraging the advantages of unmanned remote-controlled underwater robots for long-term operation and real-time video transmission. Detailed Implementation

[0053] The present invention will now be described in detail with reference to specific embodiments.

[0054] The ROV inspection method for underwater submersible structures shall be implemented in the following steps:

[0055] Step 1: Deploy the underwater inspection robot (BlueROV2 Heavy) and adjust its attitude;

[0056] Step 2: Filter underwater interference information, and then use a color cast correction algorithm to achieve sharpening.

[0057] Step 3: Estimate the position and attitude of the ROV by fusing visual and inertial sensor data;

[0058] Step 4: For the detected structural targets that enter the camera's field of view, use a deep learning-based target tracking algorithm to bring the robot closer to the underwater submerged structure;

[0059] Step 5: After reaching the target location at close range, extract the geometric and texture information of the underwater submerged structure with defect features using an image processing-based crack detection method.

[0060] Example 1

[0061] Deployment: Connect the underwater inspection robot to the onshore computer via a 100-meter mooring cable, adjust the mooring cable, and place the underwater robot into the waters to be inspected;

[0062] Underwater movement: The underwater inspection robot adjusts the rotation of its eight thrusters clockwise and counterclockwise via remote control, achieving six degrees of freedom in attitude adjustment.

[0063] Image enhancement: The image enhancement algorithm is activated to filter out interference information such as water droplets and ripples that may appear in the underwater image. Then, a color cast correction algorithm is used to achieve sharpness. The specific steps are as follows:

[0064] Step 2.1: Use guided filtering to denoise the underwater image while preserving the edge details in the original image;

[0065] Step 2.2: Use the color cast factor value to judge and distinguish underwater images from those without color cast and those with color cast. The images with color cast are divided into green color cast images, blue color cast images, and yellow color cast images.

[0066] Step 2.3: Perform color compensation on different images with color cast to obtain color-compensated images;

[0067] Step 2.4: Use the contrast-limited adaptive histogram equalization (CLAHE) algorithm to perform global contrast enhancement on the color-compensated image.

[0068] Example 2

[0069] Underwater positioning: The MSCKF algorithm based on filtering is used to fuse visual and inertial navigation data to estimate the position and attitude of the ROV, so as to realize the monitoring of the true pose of the ROV. The specific implementation is carried out according to the following steps:

[0070] Step 3.1: The BlueROV platform does not implement hardware synchronization between the IMU and the camera, so a software synchronization method is adopted: using the Raspberry Pi clock as a reference, the timestamps of the camera and the IMU are calculated separately using C++ library functions, and finally the time deviation is estimated by the positioning algorithm to achieve timestamp synchronization between the IMU and the camera.

[0071] Step 3.2: The camera acquires visual images according to its operating frequency, and the IMU's accelerometer and gyroscope measure acceleration and angular velocity according to their respective operating frequencies.

[0072] Step 3.3: According to the multi-state constrained Kalman filter (MSCKF) algorithm, define the system state vector as follows: Where x I For the IMU state vector, Indicates camera status;

[0073] Step 3.4: Integrate the acceleration and angular velocity obtained by IMU measurement based on kinematic formulas to obtain the nominal state and attitude. The velocity is calculated using the zeroth-order integral from formula (1). and location The fourth-order Runge-Kutta method integral update is calculated using formulas (2) and (3) respectively:

[0074]

[0075]

[0076]

[0077] Where Ω(ω) is a matrix with respect to angular velocity ω, and k is the Runge-Kutta coefficient;

[0078] Step 3.5: The error state is predicted based on the noise measured by the IMU using the extended Kalman prediction equations (4) and (5):

[0079]

[0080]

[0081] in, Let F be the system state at time k. k Let P be the state transition matrix. k Q represents the covariance of the state at time k. k The covariance represents the effect of the IMU error on the state from time k-1 to k.

[0082] Step 3.6: Perform FAST feature extraction and LK optical flow feature tracking on the visual images acquired in Step 3.2. After obtaining visual observations, the system updates the error state based on the extended Kalman update equation:

[0083]

[0084]

[0085] P′ k =P k -K′H k P k =(IK′H k )P k (8)

[0086] Where K′ is the Kalman gain, H k R represents the Jacobian matrix of the observation relative to the state at time k. k Represents the observed covariance matrix. It is the observation at time k;

[0087] Step 3.7: The mean of the error state is incorporated into the nominal state to obtain the true state, and then it is reset to zero. The covariance matrix of the error state is continuously updated to reflect this state update and reset process.

[0088] Step 3.8: When the accelerometer and gyroscope of the inertial measurement unit collect new acceleration and angular velocity data, return to step 3.4.

[0089] Example 3

[0090] Target tracking: When the target structure enters the camera's field of view, the target is selected on the host computer, and a deep learning-based target tracking algorithm guides the robot to approach the underwater submerged structure to be detected. This is implemented according to the following steps:

[0091] Step 4.1: Initialize the bounding box in the first frame, i.e., select the object to be tracked;

[0092] Step 4.2, Determining the Tracking Object: Assume that in frame t-1, the target previously predicted by the tracker (the target initially selected in the first frame) is located at c = (c x c y The image has a width of w and a height of h. The network crops and scales the previous frame of the image to center the tracked object in the image. Frame t-1 is selected with a width of w and a height of h. x c y The cropping centered on ) has a width of k1w and a height of k1h. This cropping tells the network which object is being tracked.

[0093] Step 4.3, Determining the tracking range: The current frame t is clipped with c′=(c′ x , c′ y ), where c′ is the expected average position of the target object. Let c′ = c, which is equivalent to a constant position motion model. The clipping width and height of the current frame are k2w and k2h, respectively, where w and h are the width and height of the predicted bounding box in the previous frame, and k2 is defined as the search radius of the target object. As long as the target object is not occluded and the movement speed is not too fast, the target will be located in this area. For fast-moving objects, the size of the search area can be increased, but at the cost of increasing the complexity of the network.

[0094] Step 4.4: Using the two cropped frames as input, train a convolutional neural network (CNN) to predict the position of the bounding box in the current frame; both frames are passed through a set of convolutional layers, which are just the first five convolutional layers of the CaffeNet architecture. The outputs of these convolutional layers are concatenated into a vector of length 4096, which is input to three fully connected layers. The last fully connected layer is finally connected to the output layer, which contains four nodes representing the top and bottom of the bounding box;

[0095] Step 4.5: In each subsequent frame, the cropped frames t-1 and t are input into the network to predict the position of the object in frame t. The remaining parts of the video are then re-cropped and the frames are fed into the network. The network will track the motion of the target object throughout the sequence.

[0096] Example 4

[0097] Crack detection: After approaching the target location at close range, an image processing-based crack detection method is used to extract the geometric and texture information of the underwater submerged structure that contains defect features. The detection results are then displayed and saved. The specific implementation steps are as follows:

[0098] Step 5.1: Image grayscale processing. Image grayscale processing can be used as a preprocessing step for crack detection, preparing for subsequent higher-level operations such as image segmentation, image recognition and image analysis. Grayscale images have smaller memory footprints and faster processing speeds, and can also increase visual contrast and highlight the target area.

[0099] Step 5.2: In order to improve the quality of the image and enhance its visual effect, the image is transformed into a form suitable for analysis and processing, highlighting information that is meaningful for human or machine analysis, increasing the contrast of the image, and providing better preconditions for subsequent image processing.

[0100] Step 5.3, Canny edge detection, mainly consists of image denoising, image gradient calculation, non-maximum suppression, and threshold filtering to obtain relevant crack information;

[0101] Step 5.4: Use morphological connection to connect adjacent cracks, use seed filling method to find all connected components, select the current pixel (pixel value greater than 1) as seed and assign a new label, and then add the positions of all its neighboring pixels with the same pixel value to the stack.

[0102] Step 5.5: Pop the top element from the stack, assign it the same label, and add the positions of all its neighboring pixels with the same pixel value to the stack. Repeat this operation until the stack is empty. At this point, a connected region is found and marked with the same label.

[0103] Step 5.6: Traverse the image, repeating steps 5.4 and 5.5 until all connected regions label_1, label_2, ..., label_n are found;

[0104] Step 5.7: Delete the noise areas that are not cracks, and finally extract the skeleton for each connected component and measure its length and width to obtain the final crack information.

[0105] The ROV inspection method for underwater submerged structures of the present invention can autonomously carry out underwater inspection operations using only sensor data acquired by the IMU and camera carried by the ROV, so as to detect potential hazards such as surface cracks and holes in the underwater submerged structure, effectively improving work efficiency and reducing costs.

Claims

1. A method for inspecting ROVs with underwater submersible structures, characterized in that, The specific steps are as follows: Step 1: Deploy the inspection robot underwater and adjust its attitude; Step 2: Filter underwater interference information, and then use a color cast correction algorithm to achieve sharpening. Step 3: Estimate the position and attitude of the ROV by fusing visual and inertial sensor data; Step 4: For the detected structural targets that enter the camera's field of view, use a deep learning-based target tracking algorithm to bring the robot closer to the underwater submerged structure; Step 5: After reaching the target location at close range, extract the geometric and texture information of the underwater submerged structure with defect features using an image processing-based crack detection method; Step 4 is implemented in the following steps: Step 4.1: Initialize the bounding box in the first frame; Step 4.2, assuming in In frame -1, the tracker previously predicted the target to be located in Width is The height is The network crops and scales the previous frame of the image to center the tracked object within the image, capturing the frame. -1 in China The center is the crop, and the width and height are respectively and ; Step 4.3, Current Frame The cutting ,in Let the expected average position of the target object be... This is equivalent to a constant position motion model, where the cropping width and height of the current frame are respectively... and ,in and It is the width and height of the bounding box predicted in the previous frame. Defined as the search radius of the target object; Step 4.4: Using the two cropped frames as input, train a convolutional neural network to predict the position of the bounding box in the current frame; both frames are passed through a set of convolutional layers, which are just the first five convolutional layers of the CaffeNet architecture. The outputs of these convolutional layers are concatenated into a vector of length 4096, which is input to three fully connected layers. The last fully connected layer is finally connected to the output layer, which contains four nodes representing the top and bottom of the bounding box. Step 4.5: In each subsequent frame, the frame... -1 and frames The cropped input is fed into the network to predict the object in the frame. The remaining portion of the video is then re-cropped and the frames are fed into the network, which will track the motion of the target object throughout the sequence. Step 5 is implemented in the following steps: Step 5.1: Convert the image to grayscale; Step 5.2: Convert the image into a format suitable for analysis and processing, and increase the contrast of the image; Step 5.3: Canny edge detection to obtain relevant crack information; Step 5.4: Use morphological connection to connect adjacent cracks, use seed filling method to find all connected components, select the current pixel as seed and assign a new label, and then add the positions of all its neighboring pixels with the same pixel value to the stack. Step 5.5: Pop the top element of the stack, assign it the same label, and add the positions of all its neighboring pixels with the same pixel value to the stack. Repeat the operation until the stack is empty. At this point, a connected region is found and marked with the same label. Step 5.6: Traverse the image, repeating steps 5.4 and 5.5 until all connected regions label_1, label_2, ..., label_n are found; Step 5.7: Delete the noise areas that are not cracks, and finally extract the skeleton for each connected component and measure its length and width to obtain the final crack information.

2. The ROV inspection method for underwater submersible structures according to claim 1, characterized in that, Step 2 is implemented in the following steps: Step 2.1: Use guided filtering to denoise the underwater image while preserving the edge details in the original image; Step 2.2: Use the color cast factor value to judge and distinguish between images without color cast and images with color cast in the underwater images; Step 2.3: Perform color compensation on different images with color cast to obtain color-compensated images; Step 2.4: Use the contrast-limited adaptive histogram equalization (CLAHE) algorithm to perform global contrast enhancement on the color-compensated image.

3. The ROV inspection method for underwater submersible structures according to claim 1, characterized in that, Step 3 is implemented in the following steps: Step 3.1: Using the Raspberry Pi clock as a reference, use C++ library functions to calculate the timestamps of the camera and IMU respectively. Finally, use the positioning algorithm to estimate the time deviation to achieve timestamp synchronization between the IMU and the camera. Step 3.2: Use a camera to acquire visual images, and use the IMU's accelerometer and gyroscope to measure acceleration and angular velocity at their respective operating frequencies. Step 3.3: According to the multi-state constrained Kalman filter (MSCKF) algorithm, define the system state vector as follows: ,in For the IMU state vector, Indicates camera status; Step 3.4: Integrate the acceleration and angular velocity obtained by IMU measurement based on kinematic formulas to obtain the nominal state and attitude. The velocity is calculated using the zeroth-order integral from formula (1). and location The fourth-order Runge-Kutta method integral update is calculated using formulas (2) and (3) respectively: （1） （2） （3） in, For angular velocity The matrix, Runge-Kutta coefficient; Step 3.5: The error state is predicted based on the noise measured by the IMU using the extended Kalman prediction equations (4) and (5): （4） （5） in, For the first System status at all times Here is the state transition matrix. express The covariance of the state at time step. express arrive The covariance of the effect of the IMU error at time step on the state; Step 3.6: Perform FAST feature extraction and LK optical flow feature tracking on the visual images acquired in Step 3.

2. After obtaining visual observations, the system updates the error state based on the extended Kalman update equation: （6） （7） （8） in, For Kalman gain, Indicates the observation relative to the state. Jacobian matrix at time, Represents the observed covariance matrix. It is the first Observation of time; Step 3.7: The mean of the error state is incorporated into the nominal state to obtain the true state, and then it is reset to zero. The covariance matrix of the error state is continuously updated to reflect this state update and reset process. Step 3.8: When the accelerometer and gyroscope of the inertial measurement unit collect new acceleration and angular velocity data, return to step 3.4.

Images