A 3D reconstruction method for underwater damage to marine equipment based on the integration of vision and IMU.

Abstract

The present invention provides a 3D reconstruction method for underwater damage of marine equipment based on vision and IMU fusion. The method includes the steps of performing, in an underwater environment, internal parameter calibration of the left and right cameras of a stereo camera and extrinsic parameter calibration between the left and right cameras of the stereo camera and a platform IMU; calibrating the extrinsic parameter matrices of the drive IMU coordinate system and the laser sensor coordinate system on each axis of the drive system; identifying and roughly locating the damaged area based on the 3D reconstruction of the underwater vision; planning an optimal path for the underwater mobile platform, performing local obstacle avoidance based on the 3D reconstructed point cloud, and controlling the underwater mobile platform to move near the damaged area; planning a trajectory for a drainage system and draining water using the drainage system; and determining the laser position of a laser sensor using drive IMU data to realize a precise 3D reconstruction of the damaged area based on the laser sensor data. This method provides a highly accurate 3D reconstruction of the damaged area, assisting other equipment in autonomous repair and improving the work efficiency of the marine equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater three-dimensional reconstruction, and specifically, to a method for three-dimensional reconstruction of underwater damage of marine equipment based on the fusion of vision and IMU.

Background Art

[0002] Marine equipment such as ships, offshore oil and gas platforms, and offshore wind power facilities are affected by adverse factors such as huge waves, humid environments, seawater corrosion, and collisions over a long period of time, and structural damage is likely to occur. Conventional methods require returning to port for repair or manually inspecting underwater, which not only incurs a great deal of time and economic costs but also poses many safety risks. By using an underwater mobile platform to perform damage positioning, the above problems can be effectively solved. With the autonomous positioning and three-dimensional reconstruction technology of the underwater mobile platform, a clear and accurate underwater damage model can be established and combined with an autonomous repair system to complete the above repair work.

[0003] Currently, commonly used underwater positioning systems include underwater acoustic positioning systems and underwater SLAM methods. Underwater acoustic positioning systems include ultra-short baseline positioning, short baseline positioning, and long baseline positioning, which are expensive and difficult to install. Common SLAM methods include methods using sonar and methods using cameras. Sonar equipment is expensive and has low resolution due to its acoustic method, so it is suitable for deep-sea positioning. However, the method using a camera needs to overcome the refraction of light in water and is sensitive to light rays, so when feature points are unclear, the problem of positioning failure is likely to occur.

[0004] Conventional camera-based 3D reconstruction systems involve two steps: camera distortion correction and 3D reconstruction. Camera distortion correction includes methods based on a single-viewpoint model and methods based on a calibration board or auxiliary hardware. Methods based on a single-viewpoint model only consider the transparency model and do not consider the refraction model underwater, resulting in lower accuracy. Methods based on a calibration board and auxiliary hardware consider the refraction model underwater, resulting in higher accuracy. 3D reconstruction mainly involves acquiring a 3D point cloud directly or indirectly based on camera parameters and then superimposing the 3D point cloud with positioning data. However, as mentioned above, the accuracy of underwater positioning using only a camera is low, which also reduces the accuracy of 3D reconstruction. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] To overcome the shortcomings and deficiencies of conventional technologies, the object of the present invention is to provide a method for three-dimensional reconstruction of underwater damage to marine equipment based on the integration of vision and IMU. This method can provide highly accurate three-dimensional reconstruction results of the damaged area, support autonomous repair by other equipment, and improve the operational efficiency of marine equipment. [Means for solving the problem]

[0006] To achieve the above objective, the present invention is realized by the following technical means. A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU is realized by a three-dimensional reconstruction system for underwater damage, the three-dimensional reconstruction system for underwater damage comprises an underwater mobile platform and a host computer, the underwater mobile platform comprising an underwater mobile platform body and a stereo camera mounted on the underwater mobile platform body, a platform IMU, a laser sensor, a laser drive system, a communication system and a drainage system, each axis of the laser drive system is provided with a drive IMU, and the communication system is used for communication between the underwater mobile platform and the host computer. A method for 3D reconstruction of underwater damage to marine equipment is: S1, the stereo camera and platform IMU are fixed to the underwater mobile platform, and the internal parameter calibration of the left and right cameras of the stereo camera and the external parameter calibration of the left and right cameras of the stereo camera and the platform IMU are performed in the underwater environment. S2, a step of fixing the laser sensor to the tip of the drive system and calibrating the external parameter matrices of the drive IMU coordinate system and the laser sensor coordinate system on each axis of the drive system, Steps include: S3, collecting stereo camera image data and platform IMU data; integrating acceleration data and angular velocity data in the platform IMU data to obtain position and attitude observations in the platform IMU coordinate system; performing position and attitude observations, damage detection, and 3D point cloud generation for frames corresponding to the stereo camera image data in the stereo camera coordinate system; fusing the position and attitude observation results and superimposing the continuous 3D point clouds to obtain an underwater 3D reconstructed point cloud; using this to verify the damage detection results; planning the optimal path for the underwater mobile platform based on the position and attitude observations and damage detection results; performing local obstacle avoidance based on the 3D reconstructed point cloud; and controlling the underwater mobile platform to move near the damaged area. S4 is characterized by including the step of planning the trajectory of the drainage system, draining the damaged area using the drainage system, and determining the laser position of the laser sensor using the drive IMU data based on the external parameter matrix obtained in S2, thereby realizing a three-dimensional reconstruction of detailed laser sensor data of the damaged area.

[0007] Preferably, S1 is S11, the steps include fixing the stereo camera to the front end of the underwater mobile platform body, with the viewing direction angled diagonally downwards between 10° and 30°, and fixing the platform IMU to the middle of the underwater mobile platform body corresponding to the center of mass of the underwater mobile platform, S12. Place the calibration board and the underwater moving platform in water simultaneously, such that the calibration board appears in the fields of view of the left and right cameras of the stereo camera at the same time. Move the underwater moving platform so that the calibration board is distributed at each position in the fields of view of the left and right cameras of the stereo camera, record multiple sets of stereo camera image data, transmit the multiple sets of stereo camera image data to the host computer by means of a communication system, and perform the following steps by the host computer: related calibration calculations, internal parameter calibration of the left and right cameras of the stereo camera, and external parameter calibration of the left and right cameras of the stereo camera and the IMU for the platform.

[0008] Preferably, in S12, The internal parameter calibration of the left and right cameras of the stereo camera is

Equation

Equation

number

number

number

[0009] Preferably, in S2, the drive IMU coordinate system and the laser sensor coordinate system on each axis of the drive system are aligned as follows: Based on the positional relationship between the stereo camera, the drainage system, and the laser drive system, the transformation relationship from the underwater mobile platform's mass center coordinate system to the drainage system coordinate system, and the transformation relationship between the drainage system coordinate system and the laser drive system coordinate system are obtained. The laser point of the laser sensor is controlled to move across a calibration board with known parameters, the laser sensor and the drive IMU are connected via a communication system to acquire data, which is then transmitted to a host computer. The host computer performs calibration calculations and obtains the transformation relationship between the laser drive system coordinate system and the laser sensor coordinate system. Align the four coordinate systems: the laser sensor, the laser drive system, the drainage system, and the center of mass of the underwater mobile platform.

[0010] Preferably, a method for aligning the four coordinate systems of the laser sensor, laser drive system, drainage system, and the center of mass of the underwater moving platform is: Calibrate the external parameter matrix, which includes the rotation matrix and translation vectors of any two coordinate systems of the laser sensor, laser drive system, drainage system, and underwater mobile platform center of mass.

number

[0011] Preferably, in S3, attitude observation in the platform IMU coordinate system is performed. The velocity V, translation vector T, and rotation matrix R obtained by integrating the platform IMU data from time k to time k+1 are respectively

number

number

number

number

[0012] Preferably, in S3, attitude observation in the stereo camera coordinate system is performed. Feature points are extracted from stereo camera image data, and circular regions are constructed around the feature points.

number

number

[0013] Preferably, in S3, the generation of a 3D point cloud in the stereo camera coordinate system is performed. The above feature points are extracted and matched from the left and right camera images of a stereo camera in the same frame, and disparity is calculated based on the grayscale sum of squared errors algorithm.

number

number

[0014] Preferably, S4 plans the movement trajectory of the underwater mobile platform based on the location of the damaged area so that the drainage system covers and drains the damaged area to form a dry space, controls the laser sensor to perform a three-dimensional scan in the dry space using a laser drive system, transmits the laser sensor data and drive IMU data to the host computer via a communication system, the host computer obtains the attitude of the laser drive system using the drive IMU data based on the external parameter matrix obtained in S2, converts it to obtain the position of the laser sensor, obtains a detailed three-dimensional reconstruction of the laser sensor data based on the laser sensor position and point cloud data, and detects the damaged location based on the three-dimensional reconstruction result.

[0015] Preferably, in S4, the three-dimensional reconstruction of the laser sensor data is performed. A laser sensor emits laser pulses at a fixed frequency, receives the reflected light back by a receiver to determine the distance, and roughly distinguishes the material of the object based on the reflection intensity. The distance measurement formula is as follows: L = tc / 2 (In the equation, L is the target distance, t is the return time, and c is the speed of light.) The position and orientation of the laser sensor are predicted using a drive IMU, and the 3D reconstruction result of the laser sensor is obtained based on the rotation matrix R and translation vector T. [Effects of the Invention]

[0016] The present invention has the following advantages and beneficial effects compared to the prior art. 1. The present invention enables autonomous detection and 3D reconstruction of underwater damage to marine equipment, thereby reducing labor and economic costs while improving safety. 2. The present invention improves positioning accuracy and underwater 3D reconstruction accuracy by fusing vision and IMU, and enables more accurate positioning of damaged underwater marine equipment by using a damage detection method based on fused verification of images and point clouds. 3. The present invention enables accurate drainage of water near the damaged area, thereby realizing high-precision laser-based 3D reconstruction and damage identification, and also provides convenience to other autonomous repair devices. [Brief explanation of the drawing]

[0017] [Figure 1] This is a flowchart of the three-dimensional reconstruction method for underwater damage to marine equipment based on the fusion of vision and IMU according to the present invention. [Figure 2] This is a schematic diagram of the structure of the three-dimensional reconstruction system for underwater damage according to the present invention. [Figure 3] This figure shows the communication of a 3D reconstruction method for underwater damage to marine equipment based on the fusion of vision and IMU according to the present invention. [Figure 4] This figure shows the coordinate system transformation in the three-dimensional reconstruction method of underwater damage to marine equipment based on the fusion of vision and IMU according to the present invention. [Modes for carrying out the invention]

[0018] The present invention will be described in detail below with reference to the drawings, with reference to specific embodiments.

[0019] (Examples) The three-dimensional reconstruction method of underwater damage to marine equipment based on the fusion of vision and IMU according to this embodiment is realized by a three-dimensional underwater damage reconstruction system, with the specific process shown in Figure 1.

[0020] The underwater damage 3D reconstruction system comprises an underwater mobile platform and a host computer. As shown in Figure 2, the underwater mobile platform includes an underwater mobile platform body 1, a stereo camera 4 mounted on the underwater mobile platform body 1, a platform IMU 6, a laser sensor 5, a laser drive system 3, a communication system, and a drainage system 2.

[0021] Each axis of the laser drive system 3 is equipped with a drive IMU, and a communication system is used for communication between the underwater mobile platform and the host computer. Specifically, the communication system is fixed to the middle of the underwater mobile platform body and is used to collect stereo camera image data and IMU data and transmit them to the host computer, as well as to receive relevant control commands from the host computer to drive the underwater mobile platform.

[0022] The method for 3D reconstruction of underwater damage to marine equipment includes the following steps S1 to S4.

[0023] S1: The stereo camera and the platform IMU are fixed to the underwater mobile platform, and the internal parameters of the left and right cameras of the stereo camera, as well as the external parameters of the left and right cameras of the stereo camera and the platform IMU are calibrated in the underwater environment.

[0024] S1 is S11, the steps include fixing the stereo camera to the front end of the underwater mobile platform body, with the viewing direction angled diagonally downwards between 10° and 30°, and fixing the IMU for the platform to the middle of the underwater mobile platform body corresponding to the center of mass of the underwater mobile platform, S12, the calibration board and underwater moving platform are simultaneously placed underwater, and the calibration board is made to appear simultaneously in the field of view of the left and right cameras of the stereo camera. While ensuring that the calibration board is fully included in the field of view of the stereo camera at the same time, the platform is rotated as far as possible in each direction to ensure calibration with the 3 axes of the drive IMU. The data recording time in this step does not need to be long, with the stereo camera recording at 15 frames / second or more and the drive IMU recording at 100 frames / second or more. The procedure includes the steps of moving an underwater mobile platform so that the calibration boards are distributed at each position in the field of view of the left and right cameras of the stereo camera, recording multiple sets of stereo camera image data, transmitting the multiple sets of stereo camera image data to a host computer via a communication system, and having the host computer perform the relevant calibration calculations, internal parameter calibration of the left and right cameras of the stereo camera, and external parameter calibration of the left and right cameras of the stereo camera and the IMU for the platform.

[0025] In S12, the internal parameter calibration of the left and right cameras of the stereo camera is performed.

number

number

number

number

number

[0026] S2: The laser sensor is fixed to the tip of the drive system, and the external parameter matrices of the drive IMU coordinate system and the laser sensor coordinate system on each axis of the drive system are calibrated.

[0027] Specifically, as shown in Figure 4, based on the positional relationship between the stereo camera, the drainage system, and the laser drive system, the transformation relationship from the underwater mobile platform's mass center coordinate system to the drainage system coordinate system, and the transformation relationship between the drainage system coordinate system and the laser drive system coordinate system are obtained. The laser point of the laser sensor is controlled to move across a calibration board with known parameters, the laser sensor and the drive IMU are connected via a communication system to acquire data, which is then transmitted to a host computer. The host computer performs calibration calculations and obtains the transformation relationship between the laser drive system coordinate system and the laser sensor coordinate system. The four coordinate systems—the laser sensor, the laser drive system, the drainage system, and the center of mass of the underwater mobile platform—are aligned. Once offline calibration is complete, the transformation relationships of all coordinate systems in Figure 4 are known.

[0028] The method for aligning the four coordinate systems of the laser sensor, laser drive system, drainage system, and the center of mass of the underwater mobile platform is as follows: Calibrate the external parameter matrix, which includes the rotation matrix and translation vectors of any two coordinate systems of the laser sensor, laser drive system, drainage system, and underwater mobile platform center of mass.

number

[0029] S3: Stereo camera image data and platform IMU data are collected by the host computer, the previous internal parameter calibration results and external parameter calibration results are read, then the platform IMU data and stereo camera image data are fused to obtain positioning results in the left camera coordinate system, damage information is detected in the left camera coordinate system based on the stereo camera's detection principle and a 3D point cloud of the current frame image is generated, the positioning results and 3D point cloud information are fused and the point clouds for each frame are filtered and superimposed to generate a continuous 3D reconstruction result, the damage location detected by the stereo camera is verified based on the 3D reconstruction point cloud, then the overall movement path of the underwater mobile platform is planned based on the positioning results and the location of the damaged area in the left camera coordinate system, converted to the underwater mobile platform's mass center coordinate system, and a control signal is sent to the underwater mobile platform's communication system via the communication bus, and as the underwater mobile platform moves, local obstacle avoidance is performed based on the 3D information stored by the real-time 3D reconstruction result until the underwater mobile platform moves near the damaged area.

[0030] Attitude observation in the platform's IMU coordinate system is, The velocity V, translation vector T, and rotation matrix R obtained by integrating the platform IMU data from time k to time k+1 are respectively

number

number

number

number

[0031] Attitude observation in a stereo camera coordinate system is, Feature points are extracted from stereo camera image data, and circular regions are constructed around the feature points.

number

number

[0032] 3D point cloud generation in a stereo camera coordinate system is The above feature points are extracted and matched from the left and right camera images of a stereo camera in the same frame, and disparity is calculated based on the grayscale sum of squared errors algorithm.

number

number

[0033] S4: The trajectory of the drainage system is planned, the drainage system is controlled to drain, and the laser position of the laser sensor is determined using the drive IMU data based on the external parameter matrix obtained in S2, thereby achieving a three-dimensional reconstruction of the laser sensor data with high precision in the damaged area.

[0034] Specifically, based on the location of the damaged area, the movement trajectory of the underwater mobile platform is planned so that the drainage system covers and drains the damaged area to form a dry space. A laser drive system is used to control the laser sensor to perform a 3D scan in the dry space. Laser sensor data and drive IMU data are transmitted to the host computer via a communication system. The host computer, based on the external parameter matrix obtained in S2, uses the drive IMU data to acquire the attitude of the laser drive system, converts it to acquire the position of the laser sensor, obtains a detailed 3D reconstruction of the laser sensor data based on the laser sensor position and point cloud data, and detects the damaged location based on the 3D reconstruction result.

[0035] Three-dimensional reconstruction of laser sensor data is A laser sensor emits laser pulses at a fixed frequency, receives the reflected light back by a receiver to determine the distance, and roughly distinguishes the material of the object based on the reflection intensity. The distance measurement formula is as follows: L = tc / 2 (In the equation, L is the target distance, t is the return time, and c is the speed of light.) The position and orientation of the laser sensor are predicted using a drive IMU, and the 3D reconstruction result of the laser sensor is obtained based on the rotation matrix R and translation vector T. At this time, precise detection of the damage location is achieved, the error is reduced to within 0.2 mm, and high-precision positioning results can be provided to other autonomous repair equipment.

[0036] While the above embodiments are preferred embodiments of the present invention, the embodiments of the present invention are not limited to the above embodiments, and any other changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principles of the present invention should all be equivalent substitutions and shall all be within the scope of protection of the present invention.

[0037] (Note) (Note 1) A method for three-dimensional reconstruction of underwater damage to marine equipment based on the integration of vision and IMU, This is realized by a 3D reconstruction system for underwater damage, which comprises an underwater mobile platform and a host computer. The underwater mobile platform comprises an underwater mobile platform body and a stereo camera mounted on the underwater mobile platform body, a platform IMU, a laser sensor, a laser drive system, a communication system, and a drainage system. Each axis of the laser drive system is provided with a drive IMU, and the communication system is used for communication between the underwater mobile platform and the host computer. A method for 3D reconstruction of underwater damage to marine equipment is: S1, the stereo camera and platform IMU are fixed to the underwater mobile platform, and the internal parameter calibration of the left and right cameras of the stereo camera and the external parameter calibration of the left and right cameras of the stereo camera and the platform IMU are performed in the underwater environment. S2, a step of fixing the laser sensor to the tip of the drive system and calibrating the external parameter matrices of the drive IMU coordinate system and the laser sensor coordinate system on each axis of the drive system, Steps include: S3, collecting stereo camera image data and platform IMU data; integrating acceleration data and angular velocity data in the platform IMU data to obtain position and attitude observations in the platform IMU coordinate system; performing position and attitude observations, damage detection, and 3D point cloud generation for frames corresponding to the stereo camera image data in the stereo camera coordinate system; fusing the position and attitude observation results and superimposing the continuous 3D point clouds to obtain an underwater 3D reconstructed point cloud; using this to verify the damage detection results; planning the optimal path for the underwater mobile platform based on the position and attitude observations and damage detection results; performing local obstacle avoidance based on the 3D reconstructed point cloud; and controlling the underwater mobile platform to move near the damaged area. A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, comprising the steps of: S4, planning the trajectory of the drainage system, draining the damaged area using the drainage system, and determining the laser position of the laser sensor using the drive IMU data based on the external parameter matrix obtained in S2, thereby realizing three-dimensional reconstruction of detailed laser sensor data of the damaged area.

[0038] (Note 2) The aforementioned S1 is, S11, the steps include fixing the stereo camera to the front end of the underwater mobile platform body, with the viewing direction angled diagonally downwards between 10° and 30°, and fixing the IMU for the platform to the middle of the underwater mobile platform body corresponding to the center of mass of the underwater mobile platform, S12, the calibration board and the underwater mobile platform are simultaneously placed underwater so that the calibration board appears simultaneously in the field of view of the left and right cameras of the stereo camera. A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, as described in Appendix 1, comprising the steps of: moving an underwater mobile platform so that calibration boards are distributed at each position in the field of view of the left and right cameras of a stereo camera; recording multiple sets of stereo camera image data; transmitting multiple sets of stereo camera image data to a host computer via a communication system; and having the host computer perform relevant calibration calculations, internal parameter calibration of the left and right cameras of the stereo camera, and external parameter calibration of the left and right cameras of the stereo camera and the IMU for the platform.

[0039] (Note 3) In S12, The internal parameter calibration of the left and right cameras of a stereo camera is as follows:

number

number

number

number

number

[0040] (Note 4) In S2, the alignment of the drive IMU coordinate system and the laser sensor coordinate system on each axis of the drive system is performed as follows: Based on the positional relationship between the stereo camera, the drainage system, and the laser drive system, the transformation relationship from the underwater mobile platform's mass center coordinate system to the drainage system coordinate system, and the transformation relationship between the drainage system coordinate system and the laser drive system coordinate system are obtained. The laser point of the laser sensor is controlled to move across a calibration board with known parameters, the laser sensor and the drive IMU are connected via a communication system to acquire data, which is then transmitted to a host computer. The host computer performs calibration calculations and obtains the transformation relationship between the laser drive system coordinate system and the laser sensor coordinate system. A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU as described in Appendix 1, characterized by aligning four coordinate systems: a laser sensor, a laser drive system, a drainage system, and the center of mass of an underwater mobile platform.

[0041] (Note 5) The method for aligning the four coordinate systems of the laser sensor, laser drive system, drainage system, and the center of mass of the underwater mobile platform is as follows: A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, as described in Appendix 4, characterized by calibrating an external parameter matrix including rotation matrices and translation vectors of any two coordinate systems of a laser sensor, a laser drive system, a drainage system, and the center of mass of an underwater mobile platform.

number

[0042] (Note 6) In S3, attitude observation in the platform IMU coordinate system is performed as follows: The velocity V, translation vector T, and rotation matrix R obtained by integrating the platform IMU data from time k to time k+1 are respectively

number

number

number

number

[0043] (Note 7) In S3, attitude observation in the stereo camera coordinate system is performed as follows: Feature points are extracted from stereo camera image data, and circular regions are constructed around the feature points.

number

number

[0044] (Note 8) In S3, the 3D point cloud generation in the stereo camera coordinate system is performed as follows: The above feature points are extracted and matched from the left and right camera images of a stereo camera in the same frame, and disparity is calculated based on the grayscale sum of squared errors algorithm.

number

number

[0045] (Note 9) The method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, as described in Appendix 1, is characterized in that S4 plans the movement trajectory of the underwater mobile platform so that the drainage system covers and drains the damaged area to form a dry space based on the location of the damaged area, controls the laser sensor to perform three-dimensional scanning in the dry space using a laser drive system, transmits the laser sensor data and drive IMU data to a host computer via a communication system, the host computer acquires the attitude of the laser drive system using the drive IMU data based on the external parameter matrix obtained in S2, converts it to acquire the position of the laser sensor, obtains a detailed three-dimensional reconstruction of the laser sensor data based on the position of the laser sensor and point cloud data, and detects the damaged location based on the three-dimensional reconstruction result.

[0046] (Note 10) In S4, the 3D reconstruction of the laser sensor data is performed. A laser sensor emits laser pulses at a fixed frequency, receives the reflected light back by a receiver to determine the distance, and roughly distinguishes the material of the object based on the reflection intensity. The distance measurement formula is as follows: L = tc / 2 (In the equation, L is the target distance, t is the return time, and c is the speed of light.) A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, as described in Appendix 9, characterized by using a drive IMU to predict the position and attitude of a laser sensor and obtaining a three-dimensional reconstruction result of the laser sensor based on a rotation matrix R and a translation vector T.

Claims

1. A method for three-dimensional reconstruction of underwater damage to marine equipment based on the integration of vision and IMU, This is realized by a three-dimensional reconstruction system for underwater damage, which comprises an underwater mobile platform and a host computer. The underwater mobile platform comprises an underwater mobile platform body, a stereo camera mounted on the underwater mobile platform body, a platform IMU, a laser sensor, a laser drive system, a communication system, and a drainage system. Each axis of the laser drive system is provided with a drive IMU, and the communication system is used for communication between the underwater mobile platform and the host computer. A method for 3D reconstruction of underwater damage to marine equipment is: S1, The steps include fixing the stereo camera and the IMU for the platform to the underwater mobile platform, performing internal parameter calibration of the left and right cameras of the stereo camera, and external parameter calibration of the left and right cameras of the stereo camera and the IMU for the platform in an underwater environment, S2, a step of fixing the laser sensor to the tip of the drive system and calibrating the external parameter matrices of the drive IMU coordinate system and the laser sensor coordinate system on each axis of the drive system, S3. Stereo camera image data and platform IMU data are collected, acceleration data and angular velocity data in the platform IMU data are integrated to obtain position and attitude observations in the platform IMU coordinate system, position and attitude observations, damage detection, and 3D point cloud generation are performed for frames corresponding to the stereo camera image data in the stereo camera coordinate system, the position and attitude observation results are merged and the continuous 3D point clouds are superimposed to obtain an underwater 3D reconstructed point cloud, which is used to verify the damage detection results, the optimal path of the underwater mobile platform is planned based on the position and attitude observations and damage detection results, local obstacle avoidance is performed based on the 3D reconstructed point cloud, and the underwater mobile platform is controlled to move near the damaged area. A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, comprising the steps of: S4, planning the trajectory of the drainage system, draining the damaged area using the drainage system, and determining the laser position of the laser sensor using the drive IMU data based on the external parameter matrix obtained in S2, thereby realizing three-dimensional reconstruction of detailed laser sensor data of the damaged area.

2. The aforementioned S1 is, S11, The steps include fixing the stereo camera to the front end of the underwater mobile platform body, setting the viewing direction to be diagonally downward between 10° and 30°, and fixing the IMU for the platform to the middle part of the underwater mobile platform body corresponding to the center of mass of the underwater mobile platform, S12, The calibration board and the underwater mobile platform are simultaneously placed underwater so that the calibration board appears simultaneously in the field of view of the left and right cameras of the stereo camera. A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, according to claim 1, comprising the steps of: moving an underwater mobile platform so that calibration boards are distributed at each position in the field of view of the left and right cameras of a stereo camera; recording multiple sets of stereo camera image data; transmitting multiple sets of stereo camera image data to a host computer via a communication system; and having the host computer perform relevant calibration calculations, internal parameter calibration of the left and right cameras of the stereo camera, and external parameter calibration of the left and right cameras of the stereo camera and the IMU for the platform.

3. In S12, The internal parameter calibration of the left and right cameras of a stereo camera is as follows: [Math 1] (where l represents the left camera, r represents the right camera, K ｌ , K ｒ respectively represent the internal parameter matrices of the left and right cameras, f ｘｌ , f ｙｌ , f ｘｒ , f ｙｒ respectively represent the lengths of the focal lengths in the x-axis and y-axis directions of the left and right cameras in pixel units, (u ０ｌ , v ｏｌ ), (u ０ｒ , v ０ｒ ) respectively represent the actual pixel coordinates of the principal points in the image plane coordinate systems of the left and right cameras.) And, External parameter calibration between the left and right cameras of the stereo camera and the IMU for the platform is performed. If the platform's IMU coordinate system is considered the world coordinate system, then the transformation relationship from the image points of the left and right cameras of the stereo camera to the platform's IMU coordinate system is: [Math 2] [Math 3] (In the formula, [Math 4] These are 2D coordinates in the left and right camera coordinate systems, [Math 5] R is a 3D coordinate in the platform's IMU coordinate system. ｌｒ , R ｒｉ These are 3x3 rotation matrices, respectively, from the right camera to the left camera and from the left camera to the platform IMU coordinate system, and T ｌｒ , T ｒｉ The method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, as described in claim 2, wherein the vectors are 1*3 translation vectors from the right camera to the left camera and from the left camera to the platform IMU coordinate system.

4. In S2, the alignment of the drive IMU coordinate system and the laser sensor coordinate system on each axis of the drive system is performed as follows: Based on the positional relationship between the stereo camera, the drainage system, and the laser drive system, the transformation relationship from the underwater mobile platform's mass center coordinate system to the drainage system coordinate system, and the transformation relationship between the drainage system coordinate system and the laser drive system coordinate system are obtained. The laser point of the laser sensor is controlled to move across a calibration board with known parameters, the laser sensor and the drive IMU are connected via a communication system to acquire data, which is then transmitted to a host computer. The host computer performs calibration calculations and obtains the transformation relationship between the laser drive system coordinate system and the laser sensor coordinate system. A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, characterized by aligning four coordinate systems: a laser sensor, a laser drive system, a drainage system, and the center of mass of an underwater mobile platform, as described in claim 1.

5. The method for aligning the four coordinate systems of the laser sensor, laser drive system, drainage system, and the center of mass of the underwater mobile platform is as follows: A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, as described in claim 4, characterized by calibrating an external parameter matrix including rotation matrices and translation vectors of any two coordinate systems of a laser sensor, a laser drive system, a drainage system, and the center of mass of an underwater mobile platform. [Math 6] (In the equation, A and B represent two coordinate systems, X represents a 4x4 external parameter matrix, R represents a 3x3 rotation matrix, and T represents a 1x3 translation vector.)

6. In S3, attitude observation in the platform IMU coordinate system is performed as follows: The velocity V, translation vector T, and rotation matrix R obtained by integrating the platform IMU data from time k to time k+1 are respectively [Number 7] [Number 8] [Number 9] (In the formula, V ｋ , V ｋ＋１ Δt is the velocity at time k and time k+1, respectively, a is the acceleration, Δt is the time interval, and T ｋ , T ｋ＋１ These are the translation vectors at time k and time k+1, respectively, and R ｋ , R ｋ＋１ These are the rotation matrices at time k and time k+1, respectively, and ω is the angular velocity. [Number 10] A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU according to claim 1, characterized in that it is expressed as the Kronecker product.

7. In S3, the attitude observation in the stereo camera coordinate system is performed as follows: Feature points are extracted from stereo camera image data, and circular regions are constructed around the feature points. [Math 11] θ=arctan(m ０１ / m １０ ) [In the formula, C represents the center of mass of the circular region, θ represents the direction vector of the feature point, and m ｐｑ This represents the moment in a circular region, [Math 12] (In the formula, R represents the radius of the circular region, x and y represent the x-axis and y-axis coordinates, and I(x,y) represents the grayscale equation.) A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, characterized in that feature points are extracted from stereo camera image data of multiple consecutive frames and matched, a PnP problem is set using the matched pixel points, and a rotation matrix R and translation vector T of the stereo camera are obtained, as described in claim 6.

8. In S3, the generation of a 3D point cloud in the stereo camera coordinate system is performed as follows: The above feature points are extracted and matched from the left and right camera images of a stereo camera in the same frame, and disparity is calculated based on the grayscale sum of squared errors algorithm. [Number 13] (In the formula, x, y, and d are the x-axis coordinate, y-axis coordinate, and parallax, respectively; i and j are the changes in the x-axis and y-axis directions, respectively; m and n are the maximum values ​​in the x-axis and y-axis directions, respectively. １ (x, y), I ２ (x, y represent the grayscale equation) Three-dimensional point cloud data is generated using parallax and the original coordinates, and the three-dimensional coordinates are, [Number 14] [In the formula, x ｌ , x ｒ These are the horizontal coordinate values ​​corresponding to the left and right cameras, respectively, and y ｌ , y ｒ These are the vertical coordinate values ​​of the left and right cameras, respectively, and f ｘ , f ｙ ∫, ∫, and ∫ are the corresponding focal lengths of the intrinsic parameters of the left and right cameras, X, Y, and Z are the three-dimensional coordinates, D is the depth value, and the following formula applies: D = Bf / d A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, as described in claim 7, characterized in that it is expressed as [wherein B is the baseline length, f is the focal length of the camera, and d is the parallax between the left and right images].

9. The method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, as described in claim 1, is characterized in that S4 plans the movement trajectory of the underwater mobile platform so that the drainage system covers and drains the damaged area to form a dry space based on the location of the damaged area, controls the laser sensor to perform three-dimensional scanning in the dry space using a laser drive system, transmits the laser sensor data and drive IMU data to a host computer via a communication system, the host computer acquires the attitude of the laser drive system using the drive IMU data based on the external parameter matrix obtained in S2, converts it to acquire the position of the laser sensor, obtains a detailed three-dimensional reconstruction of the laser sensor data based on the position of the laser sensor and point cloud data, and detects the damaged location based on the three-dimensional reconstruction result.

10. In S4, the three-dimensional reconstruction of the laser sensor data is performed. A laser sensor emits laser pulses at a fixed frequency, receives the reflected light back from the receiver to determine the distance, and roughly distinguishes the material of the object based on the reflection intensity. The distance measurement formula is as follows: L = tc / 2 (In the equation, L is the target distance, t is the return time, and c is the speed of light.) A method for three-dimensional reconstruction of underwater damage to marine equipment based on the fusion of vision and IMU, characterized in that a drive IMU is used to predict the position and attitude of a laser sensor, and a three-dimensional reconstruction result of the laser sensor is obtained based on a rotation matrix R and a translation vector T, as described in claim 9.

Images