Three-Dimensional Reconstruction Method for Underwater Damage of Marine Equipment Based on Vision and IMU Fusion

Abstract

The present invention provides a three-dimensional reconstruction method for underwater damage of marine equipment based on the fusion of vision and IMU, comprising the following steps: calibrating the internal parameters of the left and right cameras of a binocular camera and the external parameters between the left and right cameras of the binocular camera and the platform IMU in an underwater environment; calibrating the external parameter matrix of the driving IMU coordinate system and the laser sensor coordinate system on each axis of the driving system; identifying and roughly positioning the damage area based on underwater vision three-dimensional reconstruction; planning the optimal path for the underwater mobile platform and performing local obstacle avoidance based on the three-dimensional reconstruction point cloud to control the underwater mobile platform to move near the damage area; planning the trajectory of the drainage system and draining water by the drainage system; determining the position of the laser of the laser sensor using the driving IMU data to achieve three-dimensional reconstruction of the fine laser sensor data of the damage area. This method can provide high-precision three-dimensional reconstruction results of the damage area, can assist other devices for autonomous repair, and improve the operation efficiency of marine equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater three-dimensional reconstruction, and more specifically, to a method for three-dimensional reconstruction of underwater damage of marine equipment based on the fusion of vision and IMU. Background Art

[0002] Marine equipment such as ships, offshore oil and gas platforms, offshore wind power equipment, etc. are easily damaged due to long-term exposure to adverse factors such as huge waves, humid environments, seawater erosion, and collisions. Traditional methods require returning to port for repair or manual underwater inspection, which not only consumes a large amount of time and economic costs but also generates many safety hazards. Using an underwater mobile platform for damage location can well solve the above problems. Through the autonomous positioning and three-dimensional reconstruction technology of the underwater mobile platform, a clear and accurate underwater damage model can be established, and the above repair work can be completed in cooperation with an autonomous repair system.

[0003] Currently, the commonly used underwater positioning systems include underwater acoustic positioning systems and underwater SLAM methods. Underwater acoustic positioning systems include ultra-short baseline, short baseline, and long baseline positioning, which are expensive and difficult to install. Common SLAM methods include methods based on sonar and cameras. Sonar equipment is expensive and an acoustic method with low resolution, which is more suitable for deep-sea positioning. While the camera-based method needs to overcome underwater optical refraction and is sensitive to light, and is prone to positioning failure when feature points are not obvious.

[0004] The steps of the current camera three-dimensional reconstruction system include camera distortion correction and three-dimensional reconstruction. Camera distortion correction includes methods based on a single-viewpoint model and methods based on a calibration board or auxiliary hardware. The method based on a single-viewpoint model has low accuracy because it only considers the perspective model without considering the underwater refraction model. The method based on a calibration board and auxiliary hardware considers the underwater refraction model, so it has high accuracy. Three-dimensional reconstruction mainly directly or indirectly obtains three-dimensional point clouds based on camera parameters, and then superimposes the three-dimensional point clouds through positioning data. However, the above method of simply using a camera for underwater positioning has low accuracy, resulting in a reduction in the accuracy of three-dimensional reconstruction. Summary of the Invention

[0005] To overcome the deficiencies in the prior art, the purpose of the present invention is to provide a method for three-dimensional reconstruction of underwater damage of marine equipment based on the fusion of vision and IMU; this method can provide high-precision three-dimensional reconstruction results of the damaged area, can assist other devices for autonomous repair, and improve the operation efficiency of marine equipment.

[0006] To achieve the above object, the present invention is realized by the following technical solutions: A three-dimensional reconstruction method for underwater damage of marine equipment based on the fusion of vision and IMU, characterized in that: it is realized through an underwater damage three-dimensional reconstruction system; the underwater damage three-dimensional reconstruction system includes an underwater mobile platform and a computing host; the underwater mobile platform includes an underwater mobile platform body, and a binocular camera, a platform IMU, a laser sensor, a laser drive system, a communication system and a drainage system mounted on the underwater mobile platform body; drive IMUs are respectively arranged on each axis of the laser drive system; the communication system is used for communication between the underwater mobile platform and the computing host;

[0007] The three-dimensional reconstruction method for underwater damage of marine equipment includes the following steps:

[0008] S1. Fix the binocular camera and the platform IMU to the underwater mobile platform respectively; perform internal parameter calibration of the left and right cameras of the binocular camera and external parameter calibration of the left and right cameras of the binocular camera and the platform IMU in the underwater environment;

[0009] S2. Fix the laser sensor at the end of the drive system; calibrate the external parameter matrix of the coordinate systems of the drive IMUs on each axis of the drive system and the coordinate system of the laser sensor;

[0010] S3. Collect binocular camera image data and platform IMU data; integrate the acceleration and angular velocity data in the platform IMU data to obtain the position and attitude observations in the platform IMU coordinate system; for a frame corresponding to the binocular camera image data, perform pose observations, damage detection, and three-dimensional point cloud generation in the binocular camera coordinate system respectively; fuse the pose observation results and superimpose the continuous three-dimensional point clouds to obtain the underwater three-dimensional reconstruction point cloud, and use it to verify the damage detection results; according to the pose observations and damage detection results, plan the optimal path for the underwater mobile platform, and perform local obstacle avoidance based on the three-dimensional reconstruction point cloud to control the underwater mobile platform to move near the damage area;

[0011] S4. Plan the trajectory of the drainage system, and the drainage system drains the damage area; according to the external parameter matrix obtained in S2, use the drive IMU data to determine the position of the laser of the laser sensor, so as to realize the three-dimensional reconstruction of the fine laser sensor data of the damage area.

[0012] Preferably, S1 includes the following steps:

[0013] S11. The binocular camera is fixed at the front end of the underwater mobile platform body, and the visual direction is between 10° and 30° obliquely downward; the platform IMU is fixed in the middle of the underwater mobile platform body, which is equivalent to the centroid position of the underwater mobile platform;

[0014] S12. Place the calibration board and the underwater mobile platform underwater at the same time; the calibration board appears in the fields of view of the left and right cameras of the binocular camera at the same time;

[0015] Move the underwater mobile platform so that the calibration plates are distributed at various positions in the left and right camera fields of view of the binocular camera; record multiple sets of binocular camera image data; the communication system transmits the multiple sets of binocular camera image data to the computing host; the computing host performs relevant calibration calculations, including the internal parameter calibration of the left and right cameras of the binocular camera and the external parameter calibration of the left and right cameras of the binocular camera and the platform IMU.

[0016] Preferably, in step S12,

[0017] The internal parameter calibration of the left and right cameras of the binocular camera refers to:

[0018]

[0019] where l represents the left camera; r represents the right camera; K l , K r respectively represent the internal parameter matrices of the left and right cameras; f xl , f yl , f xr , f yr respectively represent the lengths of the focal lengths of the left and right cameras in the x-axis and y-axis directions expressed in pixels; (u 0l , v 0l ), (u 0r , v 0r ) respectively represent the actual pixel coordinates of the principal points of the left and right camera image plane coordinate systems;

[0020] The external parameter calibration of the left and right cameras of the binocular camera and the platform IMU refers to:

[0021] Set the platform IMU coordinate system as the world coordinate system, then the conversion relationship from the left and right camera image points of the binocular camera to the platform IMU coordinate system is:

[0022]

[0023]

[0024] where, are the two-dimensional coordinates in the left and right camera coordinate systems respectively; is the three-dimensional coordinate in the platform IMU coordinate system; R lr , R ri respectively represent the 3*3 rotation matrices from the right camera to the left camera and from the left camera to the platform IMU coordinate system; T lr , T ri respectively represent the 1*3 translation vectors from the right camera to the left camera and from the left camera to the platform IMU coordinate system.

[0025] Preferably, in step S2, aligning the drive IMU coordinate system and the laser sensor coordinate system on each axis of the drive system means:

[0026] According to the positional relationships of the binocular camera, the drainage system, and the laser driving system, the conversion relationship from the centroid coordinate system of the underwater mobile platform to the drainage system coordinate system and the conversion relationship between the drainage system coordinate system and the laser driving system coordinate system are obtained;

[0027] By controlling the movement of the laser point of the laser sensor on a calibration board with known parameters; the communication system connects the laser sensor and the driving IMU to acquire data and send it to the computing host, and the computing host completes the calibration calculation to obtain the conversion relationship between the laser driving system coordinate system and the laser sensor coordinate system;

[0028] Align the four coordinate systems of the laser sensor, the laser driving system, the drainage system, and the centroid of the underwater mobile platform.

[0029] Preferably, the method for aligning the four coordinate systems of the laser sensor, the laser driving system, the drainage system, and the centroid of the underwater mobile platform is:

[0030] Calibrate the extrinsic parameter matrix of any two of the coordinate systems of the laser sensor, the laser driving system, the drainage system, and the centroid of the underwater mobile platform, including the rotation matrix and the translation vector:

[0031]

[0032] where A and B respectively represent two coordinate systems, X represents a 4×4 extrinsic parameter matrix, R represents a 3×3 rotation matrix, and T represents a 1×3 translation vector.

[0033] Preferably, in S3, the attitude observation in the platform IMU coordinate system refers to:

[0034] The velocity V, translation vector T, and rotation matrix R obtained by integrating the platform IMU data from time k to time k + 1 are respectively expressed as:

[0035] V k+1 =V k +a△t

[0036]

[0037]

[0038] where V k ,V k+1 are the velocities at time k and time k + 1 respectively; a is the acceleration; △t is the time interval; T k ,T k+1 are the translation vectors at time k and time k + 1 respectively; R k ,R k+1 are the rotation matrices at time k and time k + 1 respectively; ω is the angular velocity; is the Kronecker product.

[0039] Preferably, in the above S3, the attitude observation in the binocular camera coordinate system means:

[0040] Extract feature points from the binocular camera image data, and construct a circular area with the feature points as the center:

[0041]

[0042] θ = arctan(m 01 / m 10 )

[0043] where C represents the centroid of the circular area, θ represents the direction vector of the feature point, and m pq represents the moment of the circular area, defined as:

[0044]

[0045] where R represents the radius of the circular area; x, y represent the x-axis coordinate and the y-axis coordinate; I(x, y) represents the grayscale equation;

[0046] By extracting and matching the feature points of consecutive multi-frame binocular camera image data, and using the matched pixel points to establish a PnP solution problem, the rotation matrix R and translation vector T of the binocular camera are obtained.

[0047] Preferably, in the above S3, the generation of the three-dimensional point cloud in the binocular camera coordinate system means:

[0048] Perform the above-mentioned feature point extraction and matching on the left and right camera images of the same frame of binocular camera, and calculate the disparity based on the sum of squared grayscale error algorithm:

[0049]

[0050] where x, y, and d are the x-axis coordinate, y-axis coordinate, and disparity respectively; i, j are the change values in the x-axis and y-axis directions respectively; m, n are the maximum values in the x-axis and y-axis directions respectively; I1(x, y), I2(x, y) represent the grayscale equation;

[0051] Generate three-dimensional point cloud data through the disparity and the original coordinates. The three-dimensional coordinates are expressed as:

[0052]

[0053] where x l , x r are the abscissa values corresponding to the left and right cameras respectively; y l , y r are the ordinate values of the left and right cameras respectively; f x , fy They are the corresponding focal lengths in the left and right camera intrinsics respectively; X, Y, and Z are three-dimensional coordinates respectively; D is the depth value, which can be calculated by the following formula:

[0054] D = Bf / d

[0055] Wherein, B is the baseline length, f is the camera focal length, and d is the left and right image disparity.

[0056] Preferably, the S4 refers to: according to the position of the damaged area, planning a motion trajectory for the underwater mobile platform so that the drainage system covers the damaged area and drains the water to form a dry space; using the laser drive system to control the laser sensor to perform three-dimensional scanning in the dry space; transmitting the laser sensor data and the drive IMU data to the computing host through the communication system; the computing host uses the drive IMU data to obtain the attitude of the laser drive system according to the extrinsic matrix obtained in S2, transforms to obtain the position of the laser sensor, and obtains a fine three-dimensional reconstruction of the laser sensor data through the position of the laser sensor and the point cloud data; detecting the damage position based on the three-dimensional reconstruction result.

[0057] Preferably, in the S4, the three-dimensional reconstruction of the laser sensor data refers to:

[0058] The laser sensor emits laser pulses at a fixed frequency, judges the distance by receiving the returned reflected light through the receiver, and roughly distinguishes the target material according to the reflection intensity at the same time. The ranging formula is:

[0059] L = tc / 2

[0060] Wherein, L is the target distance, t is the return time, and c is the speed of light;

[0061] Using the drive IMU to predict the pose of the laser sensor, and then obtaining the three-dimensional reconstruction result of the laser sensor through the rotation matrix R and the translation vector T.

[0062] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0063] 1. The present invention can independently detect and three-dimensionally reconstruct underwater damages of marine equipment, solve the labor cost and economic cost, and improve the safety at the same time;

[0064] 2. The present invention improves the positioning accuracy and underwater three-dimensional reconstruction accuracy through the fusion of vision and IMU. The damage detection method based on the fusion and verification of images and point clouds can more accurately locate the damage position of underwater marine equipment;

[0065] 3. In the present invention, the water near the damaged area can be accurately drained, so as to realize high-precision laser three-dimensional reconstruction and damage identification, and also provide convenience for other autonomous repair equipment. Description of the Drawings

[0066] Figure 1 It is a schematic flowchart of the three-dimensional reconstruction method for underwater damage of marine equipment based on the fusion of vision and IMU in the present invention;

[0067] Figure 2 It is a schematic structural diagram of the underwater damage three-dimensional reconstruction system adopted in the present invention;

[0068] Figure 3 It is a communication schematic diagram of the three-dimensional reconstruction method for underwater damage of marine equipment based on the fusion of vision and IMU in the present invention;

[0069] Figure 4 It is a schematic diagram of coordinate system conversion in the three-dimensional reconstruction method for underwater damage of marine equipment based on the fusion of vision and IMU in the present invention. Specific embodiments

[0070] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0071] Embodiment

[0072] A three-dimensional reconstruction method for underwater damage of marine equipment based on the fusion of vision and IMU in this embodiment is specifically as follows Figure 1 shown, and it is implemented through an underwater damage three-dimensional reconstruction system.

[0073] The underwater damage three-dimensional reconstruction system includes an underwater mobile platform and a computing host, as Figure 2 shown; the underwater mobile platform includes an underwater mobile platform body 1, and a binocular camera 4, a platform IMU 6, a laser sensor 5, a laser driving system 3, a communication system, and a drainage system 2 mounted on the underwater mobile platform body 1.

[0074] Drive IMUs are respectively arranged on each axis of the laser driving system 3; the communication system is used for communication between the underwater mobile platform and the computing host; specifically, the communication system is fixed in the middle of the underwater mobile platform body, and is used for collecting binocular image data and IMU data and sending them to the computing host, and at the same time receiving relevant control instructions from the computing host to drive the underwater mobile platform.

[0075] The three-dimensional reconstruction method for underwater damage of marine equipment includes the following steps:

[0076] S1. Fix the binocular camera and the platform IMU to the underwater mobile platform respectively; perform internal parameter calibration of the left and right cameras of the binocular camera and external parameter calibration of the left and right cameras of the binocular camera and the platform IMU in the underwater environment.

[0077] S1 includes the following steps:

[0078] S11. The binocular camera is fixed at the front end of the underwater mobile platform body, and the visual direction is between 10° and 30° obliquely downward; the platform IMU is fixed in the middle of the underwater mobile platform body, corresponding to the centroid position of the underwater mobile platform.

[0079] S12. Place the calibration board and the underwater mobile platform underwater at the same time; the calibration board appears in the fields of view of the left and right cameras of the binocular camera at the same time; while ensuring that the fields of view of the binocular camera can completely include the calibration board, rotate in all directions as much as possible to ensure that the calibration of the three axes of the driving IMU can be completed. The data recording time in this step does not need to be very long, more than 15 frames per second for the binocular camera and more than 100 frames per second for the driving IMU.

[0080] Move the underwater mobile platform so that the calibration board is distributed at various positions in the fields of view of the left and right cameras of the binocular camera; record multiple groups of binocular camera image data; the communication system transmits the multiple groups of binocular camera image data to the computing host; the computing host performs relevant calibration calculations, including the internal parameter calibration of the left and right cameras of the binocular camera and the external parameter calibration of the left and right cameras of the binocular camera and the platform IMU.

[0081] S12. The internal parameter calibration of the left and right cameras of the binocular camera refers to:

[0082]

[0083] Among them, l represents the left camera; r represents the right camera; K l , K r respectively represent the internal parameter matrices of the left and right cameras; f xl , f yl , f xr , f yr respectively represent the lengths of the focal lengths of the left and right cameras in the x-axis and y-axis directions expressed in pixels; (u 0l , v 0l ), (u 0r , v 0r ) respectively represent the actual pixel coordinates of the principal points of the image plane coordinate systems of the left and right cameras.

[0084] The external parameter calibration of the left and right cameras of the binocular camera and the platform IMU refers to:

[0085] Set the platform IMU coordinate system as the world coordinate system, then the conversion relationship from the image points of the left and right cameras of the binocular camera to the platform IMU coordinate system is:

[0086]

[0087]

[0088] Among them, are the two-dimensional coordinates in the left and right camera coordinate systems respectively. are the three-dimensional coordinates in the platform IMU coordinate system; R lr , R ri are respectively the 3×3 rotation matrices from the right camera to the left camera and from the left camera to the platform IMU coordinate system; T lr , T ri are respectively the 1×3 translation vectors from the right camera to the left camera and from the left camera to the platform IMU coordinate system.

[0089] S2. Fix the laser sensor at the end of the drive system; calibrate the extrinsic parameter matrix of the drive IMU coordinate system and the laser sensor coordinate system on each axis of the drive system.

[0090] Specifically, as Figure 4 shown, according to the positional relationships of the binocular camera, the drainage system, and the laser drive system, obtain the conversion relationships from the underwater mobile platform centroid coordinate system to the drainage system coordinate system and from the drainage system coordinate system to the laser drive system coordinate system;

[0091] Control the movement of the laser point of the laser sensor on a calibration board with known parameters; the communication system connects the laser sensor and the drive IMU to obtain data and send it to the computing host, and the computing host completes the calibration calculation to obtain the conversion relationship between the laser drive system coordinate system and the laser sensor coordinate system;

[0092] Align the four coordinate systems of the laser sensor, the laser drive system, the drainage system, and the underwater mobile platform centroid. After the offline calibration is completed, Figure 4 all the coordinate system conversion relationships in

[0093] The method to align the four coordinate systems of the laser sensor, the laser drive system, the drainage system, and the underwater mobile platform centroid is:

[0094] Calibrate the extrinsic parameter matrix of any two of the coordinate systems of the laser sensor, the laser drive system, the drainage system, and the underwater mobile platform centroid, including the rotation matrix and the translation vector:

[0095]

[0096] Among them, A and B respectively represent two coordinate systems, X represents the 4×4 extrinsic parameter matrix, R represents the 3×3 rotation matrix, and T represents the 1×3 translation vector.

[0097] S3. The host computer first collects the binocular camera image data and the platform IMU data; reads the previous internal and external parameter calibration results; then fuses the platform IMU data and the binocular camera image data to obtain the positioning result in the left camera coordinate system. At the same time, according to the binocular detection principle, the damage information is detected in the left camera coordinate system and the three-dimensional point cloud of the current frame image is generated; the positioning result and the three-dimensional point cloud information are fused to filter and superimpose the point cloud of each frame to generate a continuous three-dimensional reconstruction result, and the damage position detected by binocular is verified according to the point cloud of the three-dimensional reconstruction; then, according to the positioning result and the damage area position in the left camera coordinate system, the global movement path of the underwater mobile platform is planned, and it is transformed into the centroid coordinate system of the underwater mobile platform, and the control signal is sent to the communication system of the underwater mobile platform through the communication bus; during the movement of the underwater mobile platform, local obstacle avoidance is carried out according to the three-dimensional information saved in the real-time three-dimensional reconstruction result until the underwater mobile platform moves near the damage area.

[0098] Among them, the attitude observation in the platform IMU coordinate system means:

[0099] The velocity V, translation vector T, and rotation matrix R obtained by integrating the platform IMU data from time k to time k + 1 are respectively expressed as:

[0100] V k+1 = V k + a△t

[0101]

[0102]

[0103] Among them, V k , V k+1 are the velocities at time k and time k + 1 respectively; a is the acceleration; △t is the time interval; T k , T k+1 are the translation vectors at time k and time k + 1 respectively; R k , R k+1 are the rotation matrices at time k and time k + 1 respectively; ω is the angular velocity; is the Kronecker product.

[0104] The attitude observation in the binocular camera coordinate system means:

[0105] Extract feature points from the binocular camera image data and construct a circular area with the feature points as the center:

[0106]

[0107] θ = arctan(m 01 / m 10 )

[0108] Among them, C represents the centroid of the circular area, θ represents the direction vector of the feature point, and m pq represents the moment of the circular area, which is defined as:

[0109]

[0110] where R represents the radius of the circular area; x and y represent the x-axis coordinate and y-axis coordinate respectively; I(x, y) represents the grayscale equation;

[0111] By extracting and matching the feature points of the continuous multi-frame binocular camera image data, and using the matched pixel points to establish a PnP solution problem, the rotation matrix R and translation vector T of the binocular camera are obtained.

[0112] The generation of the three-dimensional point cloud in the binocular camera coordinate system means:

[0113] Performing the above-mentioned feature point extraction and matching on the left and right camera images of the same frame of the binocular camera, and calculating the disparity based on the grayscale error square accumulation algorithm:

[0114]

[0115] where x, y, and d are the x-axis coordinate, y-axis coordinate, and disparity respectively; i and j are the change values in the x-axis and y-axis directions respectively; m and n are the maximum values in the x-axis and y-axis directions respectively; I1(x, y) and I2(x, y) represent the grayscale equation;

[0116] Generating three-dimensional point cloud data through the disparity and the original coordinates, and the three-dimensional coordinates are expressed as:

[0117]

[0118] where x l and x r are the abscissa values corresponding to the left and right cameras respectively; y l and y r are the ordinate values of the left and right cameras respectively; f x , f y are the corresponding focal lengths in the internal parameters of the left and right cameras respectively; X, Y, and Z are the three-dimensional coordinates; D is the depth value, which can be calculated by the following formula:

[0119] D = Bf / d

[0120] where B is the baseline length, f is the camera focal length, and d is the disparity between the left and right images.

[0121] S4. Plan the trajectory of the drainage system and control the drainage of the drainage system; according to the external parameter matrix obtained in S2, use the driving IMU data to determine the position of the laser of the laser sensor, so as to realize the three-dimensional reconstruction of the fine laser sensor data in the damaged area.

[0122] Specifically, according to the position of the damaged area, a motion trajectory is planned for the underwater mobile platform so that the drainage system covers the damaged area and drains the water to form a dry space; the laser driving system is used to control the laser sensor to perform three-dimensional scanning in the dry space; the laser sensor data and the driving IMU data are transmitted to the computing host through the communication system; the computing host uses the driving IMU data to obtain the attitude of the laser driving system according to the external parameter matrix obtained in S2, transforms to obtain the position of the laser sensor, and obtains a fine three-dimensional reconstruction of the laser sensor data through the position of the laser sensor and the point cloud data; the damaged position is detected based on the three-dimensional reconstruction result.

[0123] The three-dimensional reconstruction of the laser sensor data means that:

[0124] The laser sensor emits laser pulses at a fixed frequency, judges the distance by receiving the returned reflected light through the receiver, and roughly distinguishes the target material according to the reflection intensity at the same time. The ranging formula is:

[0125] L = tc / 2

[0126] where L is the target distance, t is the return time, and c is the speed of light;

[0127] The pose of the laser sensor is predicted by using the driving IMU, and then the three-dimensional reconstruction result of the laser sensor is obtained through the rotation matrix R and the translation vector T. At this time, fine detection of the damaged position can be realized, and the error can be controlled within 0.2 mm, providing a high-precision positioning result for other autonomous repair devices.

[0128] The above embodiments are the preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and shall be included in the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method for underwater damage of marine equipment based on the fusion of vision and IMU, characterized in that: It is realized by an underwater damage three-dimensional reconstruction system; the underwater damage three-dimensional reconstruction system includes an underwater mobile platform and a computing host; the underwater mobile platform includes an underwater mobile platform body, and a binocular camera, a platform IMU, a laser sensor, a laser driving system, a communication system and a drainage system mounted on the underwater mobile platform body; driving IMUs are respectively arranged on each axis of the laser driving system; the communication system is used for communication between the underwater mobile platform and the computing host; The method for three-dimensional reconstruction of underwater damage of marine equipment includes the following steps: S1. Fix the binocular camera and the platform IMU to the underwater mobile platform respectively; perform internal parameter calibration of the left and right cameras of the binocular camera and external parameter calibration of the left and right cameras of the binocular camera and the platform IMU in the underwater environment; S2. Fix the laser sensor at the end of the driving system; Calibrate the external parameter matrix of the driving IMU coordinate system and the laser sensor coordinate system on each axis of the driving system, and align the driving IMU coordinate system and the laser sensor coordinate system on each axis of the driving system; S3. Collect binocular camera image data and platform IMU data; Integrate the acceleration and angular velocity data in the platform IMU data to obtain the position and attitude observations in the platform IMU coordinate system; for a frame corresponding to the binocular camera image data, perform pose observations, damage detection and three-dimensional point cloud generation in the binocular camera coordinate system respectively; fuse the pose observation results and superimpose the continuous three-dimensional point clouds to obtain the underwater three-dimensional reconstruction point cloud, and use it to verify the damage detection results; according to the pose observations and damage detection results, plan the optimal path for the underwater mobile platform, and perform local obstacle avoidance based on the three-dimensional reconstruction point cloud to control the underwater mobile platform to move near the damage area; S4. Plan the trajectory of the drainage system, and the drainage system drains the damage area; according to the external parameter matrix obtained in S2, use the driving IMU data to determine the position of the laser of the laser sensor, so as to realize the three-dimensional reconstruction of the fine laser sensor data of the damage area.

2. The three-dimensional reconstruction method for underwater damage of marine equipment based on the fusion of vision and IMU according to claim 1, characterized in that: The said S1 includes the following steps: S11. The binocular camera is fixed at the front end of the underwater mobile platform body, and the visual direction is between 10° and 30° obliquely downward; the platform IMU is fixed in the middle of the underwater mobile platform body, which is equivalent to the centroid position of the underwater mobile platform; S12. Place the calibration board and the underwater mobile platform underwater at the same time; the calibration board appears in the fields of view of the left and right cameras of the binocular camera at the same time; Move the underwater mobile platform so that the calibration board is distributed at various positions in the fields of view of the left and right cameras of the binocular camera; Record multiple groups of binocular camera image data; The communication system transmits multiple groups of binocular camera image data to the computing host; The computing host performs relevant calibration calculations, including: internal parameter calibration of the left and right cameras of the binocular camera, and external parameter calibration of the left and right cameras of the binocular camera and the platform IMU.

3. The three-dimensional reconstruction method for underwater damage of marine equipment based on the fusion of vision and IMU according to claim 2, characterized in that: The said S12, The internal parameter calibration of the left and right cameras of the binocular camera means: Among them, l represents the left camera; r represents the right camera; K l , K r respectively represent the intrinsic parameter matrices of the left and right cameras; f xl , f yl , f xr , f yr respectively represent the lengths of the focal lengths of the left and right cameras in the x-axis and y-axis directions expressed in pixels; (u 0l , v 0l ), (u 0r , v 0r ) respectively represent the actual pixel coordinates of the principal points of the image plane coordinate systems of the left and right cameras; The external parameter calibration of the left and right cameras of the binocular camera and the platform IMU means: Set the platform IMU coordinate system as the world coordinate system, then the conversion relationship between the image points of the left and right cameras of the binocular camera and the platform IMU coordinate system is: wherein, are two-dimensional coordinates in the left and right camera coordinate systems respectively; is the three-dimensional coordinate in the platform IMU coordinate system; R lr , R ri are the 3×3 rotation matrices from the right camera to the left camera and from the left camera to the platform IMU coordinate system respectively; T lr , T ri are the 1×3 translation vectors from the right camera to the left camera and from the left camera to the platform IMU coordinate system respectively.

4. The three-dimensional reconstruction method for underwater damage of marine equipment based on the fusion of vision and IMU according to claim 1, characterized in that: The S2, aligning the driving IMU coordinate system and the laser sensor coordinate system on each axis of the driving system, means: Based on the positional relationships of the binocular camera, the drainage system, and the laser driving system, obtain the conversion relationships from the underwater mobile platform centroid coordinate system to the drainage system coordinate system and from the drainage system coordinate system to the laser driving system coordinate system; Control the movement of the laser points of the laser sensor on a calibration board with known parameters; The communication system connects the laser sensor and the driving IMU to acquire data and send it to the computing host, and the computing host completes the calibration calculation to obtain the conversion relationship between the laser driving system coordinate system and the laser sensor coordinate system; Align the four coordinate systems of the laser sensor, the laser driving system, the drainage system, and the underwater mobile platform centroid.

5. The three-dimensional reconstruction method for underwater damage of marine equipment based on the fusion of vision and IMU according to claim 4, characterized in that: The method for aligning the four coordinate systems of the laser sensor, the laser driving system, the drainage system, and the underwater mobile platform centroid is: Calibrate the extrinsic parameter matrices of any two of the coordinate systems of the laser sensor, the laser driving system, the drainage system, and the underwater mobile platform centroid, including the rotation matrix and the translation vector: Among them, A and B respectively represent two coordinate systems, X represents a 4×4 extrinsic parameter matrix, R represents a 3×3 rotation matrix, and T represents a 1×3 translation vector.

6. The three-dimensional reconstruction method of underwater damage of marine equipment based on vision and IMU fusion according to claim 1, wherein: In the S3, the attitude observation in the platform IMU coordinate system means: The velocity V, translation vector T, and rotation matrix R obtained by integrating the platform IMU data from the k-th moment to the (k + 1)-th moment are respectively expressed as: V k+1 = V k + aΔt where V k , V k+1 are the velocities at time k and time k + 1 respectively; a is the acceleration; △t is the time interval; T k , T k+1 are the translation vectors at time k and time k + 1 respectively; R k , R k+1 are the rotation matrices at time k and time k + 1 respectively; ω is the angular velocity; is the Kronecker product.

7. The three-dimensional reconstruction method of underwater damage of marine equipment based on vision and IMU fusion according to claim 6, wherein: In the S3, the pose observation in the binocular camera coordinate system means: Extract feature points from the binocular camera image data and construct a circular region with the feature points as the center: θ = arctan(m 01 / m 10 ) Among them, C represents the centroid of the circular region, θ represents the direction vector of the feature point, and m pq represents the moment of the circular region, which is defined as: Among them, R represents the radius of the circular region; x, y represent the x-axis coordinate and the y-axis coordinate; I(x, y) represents the gray level equation; Through the extraction and matching of feature points from consecutive multiple frames of binocular camera image data, use the matched pixel points to establish a PnP solution problem to obtain the rotation matrix R and translation vector T of the binocular camera.

8. The three-dimensional reconstruction method of underwater damage of marine equipment based on vision and IMU fusion according to claim 7, wherein: In the S3, the generation of the three-dimensional point cloud in the binocular camera coordinate system means: Extract and match feature points from the left and right camera images of the same frame of the binocular camera, and perform disparity calculation based on the gray error square accumulation algorithm: Among them, x, y, d are respectively the x-axis coordinate, the y-axis coordinate, and the disparity; i, j are respectively the change values in the x-axis and y-axis directions; m, n are respectively the maximum values in the x-axis and y-axis directions; I1(x, y), I2(x, y) represent the gray level equation; Generate three-dimensional point cloud data through the disparity and the original coordinates, and the three-dimensional coordinates are expressed as: where x l and x r are the abscissa values corresponding to the left and right cameras respectively; y l and y r are the ordinate values of the left and right cameras respectively; f x and f y are the corresponding focal lengths in the internal parameters of the left and right cameras respectively; X, Y, and Z are three-dimensional coordinates; D is the depth value, which can be calculated by the following formula: D = Bf / d Among them, B is the baseline length, f is the camera focal length, and d is the disparity between the left and right images.

9. The three-dimensional reconstruction method of underwater damage of marine equipment based on vision and IMU fusion according to claim 1, wherein: S4 means: According to the position of the damage area, plan the movement trajectory for the underwater mobile platform so that the drainage system covers the damage area and drains the water to form a dry space; use the laser drive system to control the laser sensor to perform three-dimensional scanning in the dry space; transmit the laser sensor data and the driven IMU data to the computing host through the communication system; according to the external parameter matrix obtained in S2, the computing host uses the driven IMU data to obtain the attitude of the laser drive system, transforms to obtain the position of the laser sensor, and through the position of the laser sensor and the point cloud data, obtains a fine three-dimensional reconstruction of the laser sensor data; detect the damage position based on the three-dimensional reconstruction result.

10. The three-dimensional reconstruction method of underwater damage of marine equipment based on vision and IMU fusion according to claim 9, wherein: In S4, the three-dimensional reconstruction of the laser sensor data means: The laser sensor emits laser pulses at a fixed frequency, judges the distance by receiving the returned reflected light through the receiver, and roughly distinguishes the target material according to the reflection intensity at the same time. The ranging formula is: L = tc / 2 where L is the target distance, t is the return time, and c is the speed of light; Use the driven IMU to predict the pose of the laser sensor, and then obtain the three-dimensional reconstruction result of the laser sensor through the rotation matrix R and the translation vector T.

Images