Depth camera-based mechanical arm hull curved plate three-dimensional point cloud acquisition device and method

Abstract

The invention provides a mechanical arm hull curved plate three-dimensional point cloud collecting device and method based on a depth camera, the device comprises a supporting frame, a sliding table, a six-axis mechanical arm, a sliding driving part, the depth camera and an ultrasonic distance measuring sensor, the sliding table is installed on the supporting frame in a sliding mode, the six-axis mechanical arm is installed on the sliding table, the sliding driving part is connected with the sliding table, and the depth camera is installed on the sliding table. The depth camera and the ultrasonic distance measuring sensor are arranged at the tail end of the six-axis mechanical arm; the ultrasonic distance measuring sensor is used for detecting the distance between the tail end of the six-axis mechanical arm and the hull curved plate, the six-axis mechanical arm adjusts the shooting height of the depth camera according to the distance detected by the ultrasonic distance measuring sensor, and the six-axis mechanical arm can further adjust the shooting angle of the depth camera. Therefore, the depth camera is always perpendicular to the local curved surface of the hull curved plate for shooting. According to the invention, full-automatic three-dimensional point cloud acquisition of the hull curved plate can be realized, and the accuracy of the acquired point cloud can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of non-contact acquisition, in particular to a mechanical arm ship body curved plate three-dimensional point cloud acquisition device and method based on a depth camera. BACKGROUND

[0002] At present, most ship body curved plates have longitudinal (along the length direction of the ship) and transverse (along the width or depth direction of the ship) curvatures, forming a complex three-dimensional curved surface. Ship body curved plate measurement is a key link in the ship body curved plate forming process, and the three-dimensional geometric precision directly determines the ship body assembly quality. At present, most shipyards detect the processing precision of curved plates using a sample plate sample box manual detection. This measurement method detects the geometric precision of the curved plate by manufacturing a fixed wooden model, which consumes a large amount of wood and has the problems of low detection efficiency and accuracy. With the development of neural networks, the curved plate detection method is developing towards generating ship body curved plate three-dimensional point clouds based on deep learning models, but to evaluate the generated three-dimensional point clouds, some standard ship body curved plate three-dimensional point clouds need to be collected for comparative analysis. Therefore, the collection of ship body curved plate three-dimensional point clouds is in urgent need, and the current ship body curved plate three-dimensional point cloud acquisition device has some shortcomings.

[0003] At present, commonly used ship body curved plate three-dimensional point cloud acquisition equipment includes a handheld laser scanner, a gantry with a three-dimensional displacement sensor device, and a photogrammetry system. The handheld laser scanner can realize flexible three-dimensional point cloud acquisition, but the automatic program is low. The gantry with a three-dimensional displacement sensor device can acquire three-dimensional point clouds of ship body curved plates with sufficient length, but the setting of the gantry limits the range of the width direction of the ship body curved plate, and the device can only linearly acquire three-dimensional point clouds, which has poor flexibility and is easy to lose details of the ship body curved plate. The photogrammetry system has the advantage of low cost, only requiring a camera and a calibration tool, but the precision of the acquired point clouds is low. Therefore, it is necessary to develop a new type of ship body curved plate three-dimensional point cloud acquisition equipment. SUMMARY

[0004] The present application aims to solve at least one of the technical problems raised in the background, and provides a mechanical arm ship body curved plate three-dimensional point cloud acquisition device based on a depth camera, which can realize fully automatic ship body curved plate three-dimensional point cloud acquisition and improve the precision of the acquired point clouds.

[0005] To achieve the above-mentioned purpose, the technical solution adopted by the present application is:

[0006] A kind of mechanical arm ship body curved plate three-dimensional point cloud acquisition device based on depth camera, including support frame, sliding table, six-axis mechanical arm, sliding drive piece, depth camera and ultrasonic ranging sensor, the sliding table is slidably installed on the support frame, the six-axis mechanical arm is installed on the sliding table, the sliding drive piece is connected with the sliding table, to drive the sliding table moves, the depth camera and the ultrasonic ranging sensor are all installed on the end of the six-axis mechanical arm;The ultrasonic ranging sensor is used to detect the distance between the end of the six-axis mechanical arm and the ship body curved plate, the six-axis mechanical arm adjusts the shooting height of the depth camera according to the distance detected by the ultrasonic ranging sensor, the six-axis mechanical arm can also adjust the shooting angle of the depth camera, to ensure that the depth camera always vertically local curved surface of ship body curved plate is photographed.

[0007] Further, the support frame is provided with a plurality of guide rails, and the plurality of guide rails are arranged in parallel and at intervals;A plurality of sliding grooves are arranged at intervals on the bottom surface of the sliding table corresponding to the plurality of guide rails, and the plurality of sliding grooves are respectively slidably connected to the plurality of guide rails.

[0008] Further, the sliding drive piece includes a motor, a gear and a rack, the motor is fixed on the sliding table and connected to the gear, the rack is fixed on the support frame and arranged in parallel and at intervals with the guide rails, and the rack is engaged with the gear.

[0009] Further, the sliding table is provided with a through hole;The motor body of the motor is fixed on the top surface of the sliding table, and the output shaft of the motor passes through the through hole movably;The gear is located below the sliding table, and the gear and the rack are both located between the two sliding grooves.

[0010] Further, the ultrasonic ranging sensor is connected to the end of the six-axis mechanical arm by a connecting piece.

[0011] Further, the connecting piece includes a clamp and a fixing ring, the clamp is detachably sleeved on the six-axis mechanical arm, and the fixing ring is fixed on the outer circumferential surface of the clamp;The ultrasonic ranging sensor is inserted into the fixing ring and threadedly connected with the fixing ring, and the detection end of the ultrasonic ranging sensor faces the front of the end of the six-axis mechanical arm.

[0012] Further, the ultrasonic ranging sensor and the depth camera are arranged in front of the end of the six-axis mechanical arm at intervals, and the ultrasonic ranging sensor is located above the depth camera.

[0013] The application also provides a kind of mechanical arm ship body curved plate three-dimensional point cloud acquisition method based on depth camera, including the following steps:

[0014] The deep camera-based mechanical arm ship hull curved plate three-dimensional point cloud acquisition device is provided;

[0015] The ship hull curved plate to be measured is placed on one side of the support frame and below the ultrasonic ranging sensor and the depth camera;

[0016] The six-axis mechanical arm is driven by the sliding drive to move on the sliding table, and meanwhile, the six-axis mechanical arm drives the end of the six-axis mechanical arm to move according to a predetermined path, so as to drive the depth camera and the ultrasonic ranging sensor to move above the ship hull curved plate according to the predetermined path;

[0017] During the movement of the depth camera and the ultrasonic ranging sensor above the ship hull curved plate, the ultrasonic ranging sensor detects the distance between the end of the six-axis mechanical arm and the ship hull curved plate, and the six-axis mechanical arm adjusts the shooting height of the depth camera according to the distance detected by the ultrasonic ranging sensor; the six-axis mechanical arm also adjusts the shooting angle of the depth camera, so as to ensure that the depth camera always shoots vertically to the local curved surface of the ship hull curved plate.

[0018] Further, the movement path of the end of the six-axis mechanical arm is composed of discrete points, and a smooth trajectory is generated by using cubic spline interpolation, and the parameter equation of the i-th path P i (t) of the cubic spline curve is as follows:

[0019] P i (t)=b i +c i t+d i t 2 +e i t 3 (t∈[0,1])

[0020] In the formula, b i represents the starting position parameter of the i-th path, which determines the position of the trajectory at the starting point, c i represents the first term parameter of the i-th path, which is directly related to the starting speed of the trajectory; d i represents the second term coefficient of the i-th path, which is related to the acceleration change of the trajectory; e i represents the third term coefficient of the i-th path, which is related to the rate of change of the acceleration of the trajectory; t represents the time variable, and the value range is [0, 1], which represents the normalized time process of the path, when t=0, it represents the starting moment of the motion, and when t=1, it represents the ending moment of the motion;

[0021] The planning of the instantaneous speed v(t) of the end of the six-axis mechanical arm is determined by the following formula:

[0022]

[0023] Where t represents the global physical time (in seconds), t a , t d They represent the acceleration and deceleration time of the end of the six-axis robot arm, v max is the preset maximum speed, t total is the total exercise time.

[0024] Due to the adoption of the above technical solution, the present invention has the following beneficial effects:

[0025] The above-mentioned three-dimensional point cloud acquisition device and method of the hull curved plate based on the depth camera can make the depth camera move according to the set path through the cooperation of the sliding drive part and the six-axis robotic arm, thereby realizing fully automated three-dimensional point cloud acquisition of the hull curved plate; in addition, the above-mentioned three-dimensional point cloud acquisition device of the hull curved plate based on the depth camera can be set on one side of the hull curved plate. Compared with the method of combining the gantry with the three-dimensional displacement sensor equipment, the placement of the hull curved plate will not be restricted by the width of the gantry, so it is suitable for large-scale acquisition of the three-dimensional point cloud of the hull curved plate, and with the cooperation of the sliding drive part and the six-axis robotic arm, the movement position and angle of the depth camera can be adjusted to better acquire high-precision three-dimensional point clouds of the hull curved plate, and ensure that the depth camera is always perpendicular to the local curved surface of the hull curved plate for shooting, avoiding point cloud distortion caused by angle deviation, thereby improving the accuracy of the acquired point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a schematic structural diagram of a three-dimensional point cloud acquisition device for a hull curved plate using a robotic arm and based on a depth camera according to a preferred embodiment of the present invention;

[0027] Figure 2 for Figure 1 The diagram shows the structure of the six-axis manipulator of the hull curved plate three-dimensional point cloud acquisition device based on the depth camera, the depth camera and the ultrasonic ranging sensor;

[0028] Figure 3 for Figure 1 The schematic diagram of the structure of the sliding drive component of the 3D point cloud acquisition device of the hull curved plate of the manipulator arm based on the depth camera is shown;

[0029] Figure 4 This is a schematic diagram of the planned path of the six-axis robotic arm driving the depth camera to photograph the curved plate of the hull according to the intelligent path planning algorithm in a preferred embodiment of the present invention.

[0030] Main element symbol explanation: 100, depth camera based mechanical arm ship body curved plate three-dimensional point cloud acquisition device; 10, support frame; 11, guide rail; 20, sliding table; 21, sliding groove; 30, six-axis mechanical arm; 40, sliding drive; 41, motor; 411, motor body; 413, output shaft; 43, gear; 45, rack; 50, depth camera; 51, left camera; 52, right camera; 60, ultrasonic ranging sensor; 61, external thread; 70, connecting piece; 71, clamp; 73, fixing ring; 200, ship body curved plate. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0032] It should be noted that when a component is referred to as being "fixed" to another component, it can be directly on the other component or there can be intervening components. When a component is referred to as being "connected" to another component, it can be directly connected to the other component or there can be intervening components. When a component is referred to as being "disposed" on another component, it can be directly on the other component or there can be intervening components. The terms "vertical", "horizontal", "left", "right", and similar terms as used herein are for purposes of description only.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0034] Please also refer to Figures 1 to 3 A depth camera based mechanical arm ship body curved plate three-dimensional point cloud acquisition device 100 provided by a preferred embodiment of the present application comprises a support frame 10, a sliding table 20, a six-axis mechanical arm 30, a sliding drive 40, a depth camera 50, and an ultrasonic ranging sensor 60.

[0035] The sliding table 20 is slidably arranged on the support frame 10. Specifically, a plurality of guide rails 11 are arranged on the top surface of the support frame 10 in parallel and at intervals. A plurality of sliding grooves 21 are arranged on the bottom surface of the sliding table 20 at intervals corresponding to the plurality of guide rails 11, and the plurality of sliding grooves 21 are respectively slidably connected to the plurality of guide rails 11. In the embodiment, the number of the guide rails 11 and the sliding grooves 21 is two, but it can be understood that the number of the guide rails 11 and the sliding grooves 21 can also be other numbers according to needs, and the present application does not limit the number.

[0036] The six-axis robot arm 30 is arranged on the sliding table 20. In the embodiment, the six-axis robot arm 30 is fixedly connected to the top surface of the sliding table 20. The six-axis robot arm 30 is an industrial robot with six degrees of freedom, which has a structure similar to a human arm. Through the cooperative work of the controller, the teach pendant and other hardware and the motion control algorithm and other software, the precise, flexible and efficient motion control of the six-axis robot arm 30 is realized. The structure of the six-axis robot arm 30 is a prior art, and here will not be described in detail to save space.

[0037] The sliding driving member 40 is connected with the sliding table 20 to drive the guide rail 11 of the sliding table 20 to move. In the embodiment, the sliding driving member 40 includes a motor 41, a gear 43 and a rack 45. The motor 41 is fixed on the sliding table 20 and connected with the gear 43. The rack 45 is fixed on the support frame 10 and arranged in parallel and at intervals with the guide rail 11. The rack 45 is engaged with the gear 43. Specifically, the sliding table 20 is provided with a through hole (not shown in the figure). The motor body 411 of the motor 41 is fixed on the top surface of the sliding table 20, and the output shaft 413 of the motor 41 is movably arranged through the through hole. The gear 43 is located below the sliding table 20, and the gear 43 and the rack 45 are both located between the two sliding grooves 21. The gear 43 is connected with the output shaft 413 of the motor 41 by a key connection or other means to rotate synchronously with the output shaft 413 of the motor 41. When the motor 41 works, the gear 43 is driven to move along the rack 45, so that the sliding table 20 and the six-axis robot arm 30 fixedly connected with the sliding table 20 move along the guide rail 11 together, thereby the lateral working range of the six-axis robot arm 30 can be expanded.

[0038] The depth camera 50 and the ultrasonic ranging sensor 60 are both arranged at the end of the six-axis robot arm 30. The ultrasonic ranging sensor 60 is used to detect the distance between the end of the six-axis robot arm 30 and the hull curved plate 200. The six-axis robot arm 30 is used to adjust the shooting angle of the depth camera 50, so as to ensure that the depth camera 50 always shoots perpendicularly to the local curved surface of the hull curved plate 200, and adjusts the shooting height of the depth camera 50 according to the distance detected by the ultrasonic ranging sensor 60.

[0039] In the embodiment, the ultrasonic ranging sensor 60 and the depth camera 50 are arranged in front of the end of the six-axis robot arm 30 at intervals, and the ultrasonic ranging sensor 60 is located above the depth camera 50. The ultrasonic ranging sensor 60 is mounted on the end of the six-axis robot arm 30 through a connecting piece 70. The connecting piece 70 includes a clamp 71 and a fixing ring 73. The clamp 71 is detachably sleeved on the six-axis robot arm 30, and the structure of the clamp 71 is a prior art, which will not be described here to save space. The fixing ring 73 is fixed on the outer circumferential surface of the clamp 71, and the inner circumferential surface of the fixing ring 73 is provided with an internal thread (not shown). The ultrasonic ranging sensor 60 is inserted into the fixing ring 73, and the outer circumferential surface of the ultrasonic ranging sensor 60 is provided with an external thread 61. The ultrasonic ranging sensor 60 is threadedly connected with the internal thread of the fixing ring 73 through the external thread 61, and the detection end of the ultrasonic ranging sensor 60 faces the front of the end of the six-axis robot arm 30. The depth camera 50 has a left camera 51 and a right camera 52 arranged opposite to each other at intervals. The structures of the ultrasonic ranging sensor 60 and the depth camera 50 are prior arts, which will not be described here to save space.

[0040] The embodiment of the present application also provides a ship body curved plate three-dimensional point cloud acquisition method based on a depth camera.

[0041] S1, providing the ship body curved plate three-dimensional point cloud acquisition device 100 based on a depth camera.

[0042] S2, placing the ship body curved plate 200 to be measured on one side of the support frame 10 and below the ultrasonic ranging sensor 60 and the depth camera 50;

[0043] S3, driving the six-axis robot arm 30 to move on the sliding table 20 through the sliding driving piece 40, and simultaneously, the six-axis robot arm 30 drives the end of the six-axis robot arm 30 to move according to a predetermined path, so as to drive the depth camera 50 and the ultrasonic ranging sensor 60 to move above the ship body curved plate 200 according to the predetermined path;

[0044] The ultrasonic ranging sensor 60 detects the distance between the end of the six-axis robot arm 30 and the hull curved plate 200, and the six-axis robot arm 30 adjusts the shooting height of the depth camera 50 according to the distance detected by the ultrasonic ranging sensor 60. The shooting height determines the measurement accuracy of the depth camera 50. The distance between the end of the six-axis robot arm 30 and the hull curved plate 200 can be monitored in real time by the ultrasonic ranging sensor. When the three-dimensional modeling requirement requires high accuracy, the depth camera 50 will automatically adjust to the required height position according to the feedback of the ultrasonic ranging sensor 60; when the three-dimensional modeling requirement is high efficiency, the depth camera 60 will appropriately increase the distance between the depth camera 50 and the hull curved plate 200 according to the distance feedback of the ultrasonic ranging sensor 60. By adjusting the height of the depth camera 50, the accuracy of the three-dimensional point cloud acquisition device for the hull curved plate 200 is controlled. Therefore, when the accuracy requirement of three-dimensional modeling is set, the six-axis robot arm 30 will automatically adjust the horizontal height of the depth camera 50 and keep it fixed to shoot the hull curved plate 200.

[0045] The six-axis robot arm 30 also adjusts the shooting angle of the depth camera 50 to ensure that the depth camera 50 always shoots vertically to the local curved surface of the hull curved plate 200. That is, when the model established by the three-dimensional modeling according to the image shot by the depth camera 50 is not a plane, the six-axis robot arm 30 also adjusts the shooting angle of the depth camera 50 until the model established by the three-dimensional modeling is a plane, that is, the depth camera 50 can shoot vertically to the local curved surface of the hull curved plate 200.

[0046] The contour of the hull curved plate 200 is recognized by the depth camera 50, and the hull curved plate 200 is path planned according to an intelligent path planning algorithm. The general path planned is as shown in Figure 4 The circle in Figure 4 represents the actual shooting range of the depth camera 50, Figure 4 The rectangle in Figure 4 represents the actual collection range. The depth camera starts shooting from point A, walks along the "S" type route to collect three-dimensional point clouds of the hull curved plate, and completes the collection at point B. During the entire collection process, the depth camera 50 always walks along the "S" type route along the contour edge of the hull curved plate 200 to collect three-dimensional point clouds.

[0047] When the six-axis robot arm 30 is working, the end pose of the six-axis robot arm 30 can be obtained by the forward kinematics of the robot arm. The end pose of the six-axis robot arm 30 is described by a function of the joint angles of the six-axis robot arm 30. The Denavit-Hartenberg (D-H) parameters are used to establish the coordinate system transformation, and the homogeneous transformation matrix (the transformation between adjacent joint coordinate systems) is as follows:

[0048]

[0049] θi, di, ai, τi, respectively represent the angle, offset, length, twist angle of the joint, i i i 、 represents the homogeneous transformation matrix from the i-th joint coordinate system to the i-1-th joint coordinate system; the homogeneous transformation matrix of each joint coordinate system is obtained, and the end pose of the robot arm can be calculated by the following formula

[0050]

[0051] The depth camera is installed at the end of the robot arm, and the pose of the depth camera can be calculated by the following formula:

[0052]

[0053] In the formula, is the fixed transformation matrix from the camera coordinate system (C) to the end coordinate system (E) of the six-axis robot arm, which is determined by hand-eye calibration.

[0054] At the same time, the inverse kinematics of the robot arm can be inversely deduced from the end pose of the six-axis robot arm 30, and this process usually needs a numerical solution (such as Jacobian matrix iteration method), and the joint angle can be obtained by the following formula:

[0055]

[0056] In the formula, v is the end linear velocity and angular velocity of the six-axis robot arm, J is the Jacobian matrix, and q is the joint angle; represents the time derivative of the joint angle, i.e. the joint angular velocity.

[0057] The moving path of the end of the six-axis robot arm 30 is composed of discrete points, and in the embodiment of the application, a smooth trajectory is generated by using cubic spline interpolation, and the parametric equation of the i-th path P i (t) of the cubic spline curve is as follows:

[0058] P i (t)=b i +c i t+d i t 2 +e i t 3 (t∈[0,1])

[0059] In the formula, b i , c i , d i , e​​​i Determined by the continuity constraints of the path point position, velocity and acceleration; where b i Indicates the starting position parameter of the i-th path, which determines the position of the trajectory at the starting point, c i represents the linear parameter of the i-th path, which is directly related to the starting velocity of the trajectory; d i represents the quadratic term coefficient of the i-th path, which is related to the acceleration change of the trajectory; e i It represents the cubic coefficient of the i-th path segment, which is related to the rate of change of the acceleration of the trajectory; t represents the time variable, with a value range of [0,1], representing the normalized time process of the path segment. When t = 0, it indicates the start time of the movement, and t = 1 indicates the end time of the movement.

[0060] The planning of the instantaneous velocity v(t) of the end of the six-axis robot arm is determined by the following formula:

[0061]

[0062] Where, t a , t d They represent the acceleration and deceleration time of the end of the six-axis robot arm, v max is the preset maximum speed, t total is the total exercise time.

[0063] When the motor 41 is working, the motor 41 drives the gear 43 to move through the key connection, and the gear 43 engages with the rack 45, thereby realizing the movement of the slide 20 along the guide rail 11. The specific distance moved by the slide 20 (that is, the lateral movement distance S of the six-axis robot arm 30) is calculated by the following formula:

[0064] S=N×C=N×D×π

[0065] Where N is the number of revolutions of the motor 41, C is the circumference of the gear pitch circle, and D is the pitch circle diameter of the gear. The pitch circle diameter D can be calculated using the following formula:

[0066] D=m×z

[0067] Where m is the gear module and z is the number of gear teeth.

[0068] The calculation formula for the number of rotations N of the motor 41 is as follows:

[0069]

[0070] Where θ is the motor rotation angle. For example, when the gear module is 2mm, the number of teeth is 20, and the motor rotates 10 times, the calculated gear pitch circle diameter, gear pitch circle circumference, and movement distance S of the slide 20 are:

[0071] D = 2 x 20 = 40 mm

[0072] C = π x 40 ≈ 125.66 mm

[0073] S = 10 x 125.66 ≈ 1256.6 mm

[0074] When collecting the three-dimensional point cloud, the depth camera 50 also needs to be connected to the computer (not shown in the figure) through the USB data line, and the official code provided on the computer is run to obtain the RGB image, depth image and point cloud image of the photographed object, thereby realizing three-dimensional modeling. By starting the motor 41, the motor 41 is reversed to drive the gear 43 to rotate, and the gear 43 moves along the rack 45 to drive the six-axis mechanical arm 30 to move along the guide rail 11. In order to obtain clearer RGB pictures and accurate point cloud images, the six-axis mechanical arm 30 needs to be close enough to the measured object. The distance between the end of the six-axis mechanical arm 30 and the curved plate 200 of the ship body is obtained by the ultrasonic ranging sensor 60, and then the height of the end of the six-axis mechanical arm 30 is adjusted. Generally, the size of the curved plate 200 of the ship body is large, and the reach of the six-axis mechanical arm 30 on the market is rarely so long. Even if there is a six-axis mechanical arm 30 as long as that, its positioning accuracy will inevitably be low, which will affect the collection of the point cloud. Therefore, the sliding driving part 40 is needed to drive the six-axis mechanical arm 30 to move along the guide rail 11 to increase the collection range of the point cloud. According to the point cloud of each part of the curved plate 200 of the ship body collected, the point cloud is spliced by using the iterative closest point algorithm (ICP), so as to obtain the three-dimensional point cloud of the entire curved plate 200 of the ship body. The iterative closest point algorithm is one of the most classic and most widely used algorithms in three-dimensional point cloud registration. The goal is to find the best rigid body transformation between two points through iterative optimization, so that their overlapping areas are aligned, thereby realizing three-dimensional point cloud splicing.

[0075] The embodiment of the present application uses a ZED2i binocular stereo depth camera. The depth camera 50 is connected to the computer through the USB3.0 data line, and the corresponding python script is run on the computer. The RGB image, depth image and point cloud image of the object can be obtained on the computer. The specific principle of conversion from the RGB image to the depth image and finally to the point cloud image is as follows:

[0076] (1) The first step is image acquisition: the left and right cameras 51, 52 of the ZED2i binocular stereo depth camera simultaneously shoot the RGB image of the curved plate 200 of the ship body, and then the image is preprocessed (eliminate lens distortion, based on the internal parameter matrix K and distortion coefficient of factory calibration), and finally the left and right RGB images of the left and right camera outputs 51, 52 are obtained;

[0077] (2) The second step is depth image generation: using a semi-global matching algorithm to find corresponding points between left and right RGB images, and calculating a parallax map. The key formula for calculating the parallax map is:

[0078] d = u left - u right

[0079] where d is the disparity, which is inversely proportional to the depth value Z; u left and u right respectively represent the horizontal pixel coordinates of a point in the left and right RGB images obtained by the left and right cameras 51, 52, which can be converted into a depth value through binocular geometry, thereby generating an initial depth map, and the conversion formula is as follows:

[0080]

[0081] where f is the focal length and b is the binocular baseline distance. The initial depth map is processed through post-processing (hole filling, filtering) to generate a final depth map.

[0082] (3) Third step, parameter calibration and alignment: the intrinsic matrix (focal length f x , f y , optical center c x , c y ) and distortion coefficient of the depth camera 50 are determined through intrinsic calibration, the rotation and translation matrix (R, t) of the depth camera 50 is determined through extrinsic calibration, and the final depth map obtained in the second step is mapped to the RGB coordinate system through the calibration parameters to ensure one-to-one correspondence of pixels;

[0083] (4) Fourth step, conversion between pixel coordinate system and camera coordinate system: the 3D coordinates (Xc, Yc, Zc) of each pixel in the camera coordinate system are calculated using each pixel coordinate (u, v) and its depth d, and the calculation formula is as follows:

[0084]

[0085] (5) Fifth step, point cloud generation: the 3D coordinates Xc, Yc, Zc of each pixel are combined into a 3D point, and the RGB color (the corresponding pixel value is obtained from the aligned RGB image) is added to form the final three-dimensional point cloud.

[0086] The above-mentioned depth camera-based robotic arm hull curved plate three-dimensional point cloud acquisition device 100 and method, through the cooperation of the sliding drive member 40 and the six-axis robotic arm 30, can enable the depth camera 50 to move according to the set path, thereby realizing fully automated hull curved plate three-dimensional point cloud acquisition; in addition, the above-mentioned depth camera-based robotic arm hull curved plate three-dimensional point cloud acquisition device 100 can be set on one side of the hull curved plate 200. Compared with the gantry with a three-dimensional displacement sensor device, the placement of the hull curved plate 200 will not be restricted by the width of the gantry, so it is suitable for large-scale acquisition of hull curved plate three-dimensional point clouds, and with the cooperation of the sliding drive member 40 and the six-axis robotic arm 30, the movement position and angle of the depth camera 50 can be adjusted to better acquire high-precision hull curved plate three-dimensional point clouds, and the depth camera 50 always shoots perpendicular to the local curved surface of the hull curved plate, avoiding point cloud distortion caused by angle deviation, thereby improving the accuracy of the acquired point cloud.

[0087] In addition, in this embodiment, the intelligent path planning algorithm of the six-axis robotic arm 30 can also be obtained through the imitation learning strategy, that is, the depth camera 50 is used to identify the outline of the hull curved plate 200, and the six-axis robotic arm 30 is allowed to perform three-dimensional point cloud collection on various types of hull curved plates according to the appropriate path through manual operation of the teach pendant. Then, the neural network is trained using the data from the teach pendant to obtain the intelligent path planning algorithm of the six-axis robotic arm 30, so that the six-axis robotic arm 30 can automatically plan the collection path when facing a new hull curved plate, thereby realizing fully automatic three-dimensional point cloud collection of the hull curved plate.

[0088] Therefore, the hull curved plate three-dimensional point cloud acquisition device and method based on the depth camera 50 of the present invention can realize the high degree of automation, large-scale, accurate and efficient acquisition of the hull curved plate three-dimensional point cloud, so as to be used for producing the hull curved plate image data set and evaluating the processing accuracy of the hull curved plate.

[0089] The above description is a detailed description of the preferred embodiments of the present invention, but the embodiments are not intended to limit the scope of the patent application of the present invention. Any equivalent changes or modifications completed under the technical spirit suggested by the present invention should fall within the patent scope covered by the present invention.

Claims

1. A three-dimensional point cloud acquisition device for a hull curved plate using a robotic arm based on a depth camera, characterized in that: It includes a support frame, a slide, a six-axis robotic arm, a sliding drive, a depth camera and an ultrasonic ranging sensor, the slide is slidably mounted on the support frame, the six-axis robotic arm is mounted on the slide, the sliding drive is connected to the slide to drive the slide to move, the depth camera and the ultrasonic ranging sensor are both mounted on the end of the six-axis robotic arm; the ultrasonic ranging sensor is used to detect the distance between the end of the six-axis robotic arm and the hull curved plate, the six-axis robotic arm adjusts the shooting height of the depth camera according to the distance detected by the ultrasonic ranging sensor, and the six-axis robotic arm can also adjust the shooting angle of the depth camera to ensure that the depth camera always shoots perpendicular to the local curved surface of the hull curved plate.

2. The 3D point cloud acquisition device for a hull curved plate using a robotic arm and a depth camera according to claim 1, wherein: The support frame is provided with a plurality of guide rails, which are arranged in parallel and at intervals; the bottom surface of the slide is provided with a plurality of sliding grooves corresponding to the plurality of guide rails, and the plurality of sliding grooves are respectively and slidably connected to the plurality of guide rails.

3. The 3D point cloud acquisition device for a hull curved plate using a robotic arm and a depth camera according to claim 2, wherein: The sliding drive component includes a motor, a gear and a rack. The motor is fixed on the slide and connected to the gear. The rack is fixed on the support frame and is arranged parallel to the guide rail and spaced apart. The rack is engaged with the gear.

4. The 3D point cloud acquisition device for a hull curved plate using a robotic arm and a depth camera according to claim 3, wherein: The slide is provided with a through hole; the motor body of the motor is fixed on the top surface of the slide, and the output shaft of the motor movably passes through the through hole; the gear is located below the slide, and the gear and the rack are both located between the two slide grooves.

5. The 3D point cloud acquisition device for a hull curved plate using a robotic arm and a depth camera according to claim 1, wherein: The ultrasonic distance measuring sensor is installed at the end of the six-axis robotic arm through a connecting piece.

6. The 3D point cloud acquisition device for a hull curved plate using a robotic arm and a depth camera according to claim 5, wherein: The connecting part includes a clamp and a fixing ring, the clamp is detachably mounted on the six-axis robotic arm, and the fixing ring is fixed to the outer circumferential surface of the clamp; the ultrasonic ranging sensor is inserted into the fixing ring and threadedly connected to the fixing ring, and the detection end of the ultrasonic ranging sensor is facing directly in front of the end of the six-axis robotic arm.

7. The 3D point cloud acquisition device for a hull curved plate using a robotic arm and a depth camera according to claim 1, wherein: The ultrasonic ranging sensor and the depth camera are arranged at intervals in front of the end of the six-axis robotic arm, and the ultrasonic ranging sensor is located above the depth camera.

8. A method for collecting three-dimensional point clouds of a curved plate of a hull using a robotic arm based on a depth camera, characterized in that: The following steps are involved: Provide a three-dimensional point cloud acquisition device for a hull curved plate of a robotic arm based on a depth camera as claimed in claim 1; Placing the hull curved plate to be measured on one side of the support frame and below the ultrasonic ranging sensor and the depth camera; The six-axis robotic arm is driven to move on the slide by the sliding drive member, and at the same time, the six-axis robotic arm drives the end of the six-axis robotic arm to move along a predetermined path, so as to drive the depth camera and the ultrasonic ranging sensor to move above the hull curved plate along the predetermined path; During the movement of the depth camera and the ultrasonic ranging sensor above the hull curved plate, the ultrasonic ranging sensor detects the distance between the end of the six-axis robotic arm and the hull curved plate, and the six-axis robotic arm adjusts the shooting height of the depth camera according to the distance detected by the ultrasonic ranging sensor; the six-axis robotic arm also adjusts the shooting angle of the depth camera to ensure that the depth camera always shoots perpendicular to the local curved surface of the hull curved plate.

9. The method for collecting three-dimensional point clouds of a curved plate of a hull using a manipulator arm based on a depth camera according to claim 8, wherein: The moving path of the end of the six-axis robot arm is composed of discrete points, and a smooth trajectory is generated by cubic spline interpolation. The i-th segment path P of the cubic spline curve is i The parametric equation for (t) is as follows: P i (t)=b i +c i t+d i t 2 +e i t 3 (t∈[0,1]) Where b i Indicates the starting position parameter of the i-th path, which determines the position of the trajectory at the starting point, c i represents the linear parameter of the i-th path, which is directly related to the starting velocity of the trajectory; d i represents the quadratic term coefficient of the i-th path, which is related to the acceleration change of the trajectory; e i represents the cubic coefficient of the i-th path segment, which is related to the rate of change of the acceleration of the trajectory; t represents the time variable, with a value range of [0,1], representing the normalized time process of the path segment. When t = 0, it indicates the start time of the movement, and t = 1 indicates the end time of the movement; The planning of the instantaneous velocity v(t) of the end of the six-axis robot arm is determined by the following formula: Where t represents the global physical time, t a , t d They represent the acceleration and deceleration time of the end of the six-axis robot arm, v max is the preset maximum speed, t total is the total exercise time.

Images