A mechanical arm vision-guided alignment device based on line laser profile sensor

Abstract

The application provides a mechanical arm visual guidance alignment device, comprising: a target object in a disc shape; a plurality of line laser profile sensors installed at the end of the mechanical arm and used for emitting a strip-shaped laser beam to the target object, the plurality of strip-shaped laser beams emitted by the plurality of line laser profile sensors form an included angle, and the plurality of line laser profile sensors are used for pose perception of the target object and generation of pose perception data; a visual guidance system comprising a data processing unit, the data processing unit is used for receiving and processing the pose perception data generated by the plurality of line laser profile sensors to obtain a desired pose of the end of the mechanical arm; and a trajectory planning system used for planning a continuous motion trajectory of the end of the mechanical arm from a current pose to the desired pose according to the desired pose of the end of the mechanical arm, so as to guide the tool at the end of the mechanical arm to align with the target object.

Description

Technical Field

[0001] This invention relates to a vision-guided alignment method for robotic arms based on a line laser profile sensor, used to guide the end effector of the robotic arm to automatically align with a target object. Background Technology

[0002] Visual guidance of robotic arms refers to using visual sensors to perceive the target object and guide the robotic arm to move in a desired posture to the desired position of the target object, so that the end effector of the robotic arm can perform operations such as alignment, gripping, welding, and spraying on the target object. It is also known as visual servoing of robotic arms.

[0003] Commonly used vision sensors for visual servoing include ordinary optical cameras, stereo cameras, and LiDAR. Ordinary optical cameras, unable to directly acquire depth information of the target object, are mostly used for image-based visual servoing. Stereo cameras and LiDAR, which can directly acquire the three-dimensional position of the target object, are mostly used for position-based visual servoing. Depending on the installation location of the vision sensor, visual servoing can be divided into eye-in-hand and eye-to-hand types. Eye-in-hand refers to mounting the vision sensor on the robotic arm, which moves with it. Eye-to-hand refers to mounting the vision sensor outside the robotic arm, simultaneously observing both the end effector and the target object. Position-based eye-in-hand visual servoing solutions are widely used due to their safety, efficiency, flexibility, and ease of deployment.

[0004] In visual sensors, ordinary light cameras cannot directly acquire the depth information of the target object. When implementing image-based visual guidance schemes, it is necessary to calculate the image Jacobian matrix and its inverse matrix in real time. The multiple solutions problem and the singularity problem of the image Jacobian matrix are prone to occur when solving parameters, resulting in poor global performance and low reliability.

[0005] Position-based vision guidance solutions commonly use stereo cameras, including binocular cameras and structured light cameras. Binocular cameras recover depth based on parallax, achieving 3D position measurement accuracy from centimeters to sub-millimeter levels. Their measurement frequency also reaches the level of ordinary optical cameras. However, they are prone to failure with textured targets, and measurements are easily affected by external lighting conditions. Structured light cameras calculate the shape and depth information of an object's surface by projecting structured light and utilizing its reflection or deformation. They are divided into monocular and binocular structured light cameras. The measurement accuracy of structured light cameras is greatly affected by the measurement distance, typically ranging from millimeters to sub-millimeters. Structured light cameras also provide good 3D measurement results for weakly textured targets and have strong resistance to environmental influences; however, their measurement frequency is relatively low, generally not exceeding 5 Hz.

[0006] LiDAR measures the distance to a target using time-of-flight measurement and then calculates its three-dimensional position based on the horizontal and vertical angles of the laser beam. Multi-line LiDAR is well-suited for long-distance, large-area three-dimensional scene scanning, with accuracy typically ranging from centimeters to millimeters and scanning frequencies between 10 and 20 Hz. It is widely used in surveying and autonomous driving, but less so in the vision guidance of robotic arms.

[0007] The effectiveness of vision guidance for robotic arms depends on the performance of the vision sensor. The higher the measurement accuracy of the vision sensor, the higher the accuracy of the calculated desired pose of the robotic arm, and the more accurate the vision guidance. The higher the measurement frequency of the vision sensor, the higher the feedback frequency in the closed-loop control circuit, and the more efficient the vision guidance.

[0008] However, the highest measurement accuracy of commonly used sensors for visual guidance of robotic arms can only reach the sub-millimeter level, and the measurement frequency is usually in the range of several to tens of hertz, which limits the visual guidance effect. For robotic arm visual guidance scenarios with ultra-high precision and ultra-high response requirements, such as surgical robots, existing sensors and methods are often unable to meet the requirements. Summary of the Invention

[0009] To address the aforementioned problems, this invention proposes a robotic arm vision-guided alignment device based on a line laser contour sensor.

[0010] This application provides a vision-guided alignment device for a robotic arm, comprising:

[0011] The target object is disc-shaped;

[0012] Multiple line laser profile sensors are installed at the end of a robotic arm to emit strip-shaped laser beams toward the target object. The multiple strip-shaped laser beams emitted by the multiple line laser profile sensors are at an angle to each other. The multiple line laser profile sensors are used to perceive the pose of the target object and generate pose perception data.

[0013] The vision guidance system includes a data processing unit, which receives and processes pose perception data generated by multiple line laser contour sensors to obtain the desired pose of the robotic arm end effector.

[0014] A trajectory planning system is used to plan the continuous motion trajectory of the robotic arm end effector from its current pose to the desired pose, based on the desired pose of the robotic arm end effector, so as to guide the robotic arm end effector tool to align with the target object.

[0015] According to at least one embodiment of this application, the number of the plurality of line laser profile sensors is three.

[0016] According to at least one embodiment of this application, the plurality of strip-shaped laser beams are uniformly distributed in a direction surrounding the center of the target object.

[0017] According to at least one embodiment of this application, the processing procedure of the data processing unit of the visual guidance system includes:

[0018] Extract the edge points of the target object;

[0019] The edge circle of the target object is determined by fitting a spatial circle based on the edge points of the target object.

[0020] Calculate the desired pose of the edge circle in the visual sensor coordinate system;

[0021] Calculate the desired pose of the robotic arm.

[0022] According to at least one embodiment of this application, each line laser profile sensor includes a laser receiving device and a laser emitting device, wherein the laser receiving device is used to receive changes in the reflected light from the target object and acquire a linear laser point cloud on the surface of the target object.

[0023] This application also provides an operation method for a robotic arm vision-guided alignment device, including:

[0024] S1: Multiple line laser contour sensors emit strip-shaped laser beams to scan the disk surface of a disk-shaped target object, obtaining multiple line-shaped laser point clouds. The line-shaped laser point clouds intersect with the edge of the target object to form intersection points.

[0025] S2: Extract all suspected edge points of the disk-shaped target from multiple linear laser point clouds by using the roughness characteristics of the intersection points;

[0026] S3: Determine the edge circle of the target object by fitting a spatial circle based on the edge points of the target object;

[0027] S4: Calculate the desired pose of the edge circle in the visual sensor coordinate system, and then calculate the desired pose of the robotic arm end effector in the base coordinate system.

[0028] According to at least one embodiment of this application, in step S3, the edge points of the target object are determined to be correctly extracted by judging whether the calculated radius of the spatial circle is equal to the manufacturing radius of the target object.

[0029] According to at least one embodiment of this application, the above steps S1-S4 are continuously performed, the vision guidance system continuously calculates the desired pose of the end effector of the robotic arm, and continuously transmits it to the position control loop of the robotic arm through trajectory planning, and the robotic arm continuously transmits the latest pose of the end effector to the vision guidance system.

[0030] According to at least one embodiment of this application, when it is necessary for the end effector of a robotic arm to be precisely aligned with a certain position on an object of arbitrary shape, a disc-shaped target object is mounted at that position.

[0031] This application also provides a computer-readable storage medium having software instructions stored thereon, which, when executed, perform the above-described method.

[0032] The robotic arm vision guidance alignment device provided in this application solves the problems of limited accuracy and low data acquisition frequency of commonly used vision servo sensors such as stereo cameras and lidar. It achieves micron-level vision guidance alignment accuracy for robotic arms and a pose feedback frequency of thousands of hertz in the closed-loop control loop. It can be widely used in robotic arm vision guidance alignment scenarios with ultra-high precision and ultra-high response speed requirements. Attached Figure Description

[0033] Figure 1a This is a side view of a line laser profile sensor.

[0034] Figure 1b A three-dimensional schematic diagram of a line laser profile sensor;

[0035] Figure 1c A bottom view of a line laser profile sensor;

[0036] Figure 2 A schematic diagram of a line laser contour sensor scanning a target object;

[0037] Figure 3 A schematic diagram showing the installation positions of the three line laser profile sensors;

[0038] Figure 4 A schematic diagram of three line laser profilometers scanning a disk target.

[0039] Figure 5 A schematic diagram of three line laser profilometers scanning a target object;

[0040] Figure 6 This demonstrates the extraction of edge points from a linear point cloud;

[0041] Figure 7 This illustrates the process of finding the circumcircle of a spatial triangle.

[0042] Figure 8 A schematic diagram showing the establishment of the tool coordinate system and the disk coordinate system is shown;

[0043] Figure 9 A visual servoing system based on a line laser contour sensor is shown. Detailed Implementation

[0044] The basic measurement principle of a line laser profile sensor (also known as a line laser displacement sensor) is similar to that of a lidar sensor, but its design is specifically for short-range, small-area, and high-precision 3D scene scanning. The structure of the line laser profile sensor 10 is as follows... Figures 1a-1cAs shown, it includes a power interface 1, a network port 2, a laser receiving window 3, and a laser emitting window 4.

[0045] like Figure 2 As shown, the line laser profile sensor 10 illuminates the surface of the target object 5 with a strip laser 6 at a fixed angle. A CMOS (Complementary Metal-Oxide-Semiconductor) sensor receives the changes in the reflected light, acquiring a high-precision (micrometer-level) linear three-dimensional point cloud 7 on the target object's surface. The frequency can reach thousands of hertz, and it is widely used for precise measurement of the three-dimensional contours of objects. The performance of the line laser profile sensor is shown in the table below:

[0046] Because line laser profile sensors are designed for measuring the profiles of small objects, a single line laser profile sensor can only scan a line of point cloud on the surface of the target object instantaneously, and its sensing range is very limited. Therefore, it is difficult to use it as a vision sensor in a robotic arm vision guidance system to perceive the pose of the target object.

[0047] This invention innovatively applies line laser contour sensors to the visual guidance of robotic arms. By simultaneously using multiple line laser contour sensors mounted at a fixed angle at the end of the robotic arm, and combining them with a special target object shape—a disk—and data processing algorithms tailored to the characteristics of line laser contour sensors, the line laser contour sensors can perceive the pose of the disk-shaped target object. This solves the problems of limited scanning frequency and measurement accuracy of traditional structured light or binocular sensors and lidar, achieving micron-level visual guidance alignment accuracy for robotic arms and a pose feedback frequency of over kilohertz in the closed-loop control loop. It can be widely applied to visual guidance alignment scenarios for robotic arms requiring micron-level accuracy and millisecond-level response speed.

[0048] Hereinafter, embodiments according to this application will be described in detail with reference to the accompanying drawings. One embodiment of this application provides a robotic arm vision-guided alignment device, such as... Figures 3-4 As shown, it includes:

[0049] Target object 15 is disc-shaped;

[0050] Three line laser profile sensors 10 are mounted at the end of the robotic arm and can simultaneously emit strip-shaped laser beams toward the target object 15. The layout of the three line laser profile sensors 10 is as follows: Figure 3 As shown, the layout causes the angle between the three strip-shaped laser beams emitted to be approximately 60 degrees.

[0051] Visual guidance systems (such as) Figure 9 As shown, it includes a data processing unit, which is used to receive and process pose perception data generated by three line laser contour sensors to obtain the desired pose of the robotic arm end effector.

[0052] The trajectory planning system is used to plan the continuous motion trajectory of the robotic arm end effector from the current pose to the desired pose based on the desired pose of the robotic arm end effector, so as to guide the robotic arm end effector tool to align with the target object 15.

[0053] like Figure 4 As shown, the working method of the robotic arm vision-guided alignment device provided in this embodiment includes the following steps:

[0054] S1: As Figure 4 As shown, three linear laser contour sensors 10 emit strip-shaped laser beams to scan the disk surface of the disk-shaped target 15, obtaining three linear laser point clouds 17 (e.g., ...). Figure 5 As shown), the three linear laser point clouds 17 intersect with the edge of the disk-shaped target object 15 to form intersection points p1, p2, and p3;

[0055] S2: Based on the three linear laser point clouds 17, extract all suspected edge points of the disk-shaped target object 15 from the three linear laser point clouds 17 by using the roughness characteristics of the intersection points p1, p2, and p3.

[0056] Based on point cloud data obtained by line laser profile sensor 10, the roughness of a point is defined by calculating the distance difference between adjacent points along the Z-axis. Points with lower roughness are considered planar points, points with higher roughness are considered edge points, and points with roughness in the middle are considered non-target points. Figure 6 As shown.

[0057] rougnness(i) = |2×point z (i)-point z (i+1)-point z (i-1)|

[0058] Three line laser profile sensors can measure three linear laser point clouds of the disk-shaped target 15. By calibrating the extrinsic parameters between the three sensors, the data of the three linear point clouds are transformed into a unified coordinate system. By using the roughness characteristics of the points, all suspected edge points of the disk-shaped target 15 in the three linear point clouds are extracted.

[0059] S3: Calculate and determine a spatial circle based on every three suspected edge points. Determine whether the calculated spatial circle radius r is equal to the actual manufacturing radius of the disk-shaped target object 15, and whether the edge points of the disk-shaped target object 15 have been extracted correctly.

[0060] Among all suspected edge points, a spatial circle can be uniquely determined by every three edge points: Where A, B, C, and D are plane parameters, p0(x0,y0,z0) is the center of the circle, and r is the radius of the circle.

[0061] Since the manufacturing radius of the target object 15 is known, and the radius size will not change with the rigid transformation, the edge points of the target object 15 can be determined by judging whether the calculated circle radius r is equal to the actual radius, thus determining the edge circle of the disk-shaped target object 15.

[0062] There are many methods for fitting a spatial circle from three spatial points. This embodiment uses the method of solving for the circumcircle of a spatial triangle, such as... Figure 7 As shown. The main process for solving the circumcircle of a spatial triangle is as follows:

[0063] 1) Determine the midpoint p of p1p2 12 = (p1+p2) / 2, the midpoint of p1p3 is p. 13 = (p1+p3) / 2; Determine the vector

[0064] 2) Let the center of the circle be p0(x0,y0,z0), and determine the vector.

[0065] The equation can be obtained as follows:

[0066] 3) Based on the plane conditions, the equation can be obtained:

[0067] 4) Solve the equations in 2) and 3) simultaneously to obtain the plane parameters A, B, C, D and the center p0(x0,y0,z0), where the vector (A,B,C) is the normal direction of the circle.

[0068] 5) According to ||p i -p0‖ 2 =r 2 Determine the radius of the spatial circle.

[0069] S4: Calculate the equation of the edge circle of the disk-shaped target 15 in the coordinate system of the robot arm vision sensor. Based on the external parameter calibration parameters of the robot arm end tool coordinate system, calculate the change of the robot arm end tool coordinate system from the current pose to the desired pose. Then calculate the desired pose of the robot arm end in the base coordinate system.

[0070] After obtaining the equation of the edge circle of the disk-shaped target 15 in the vision sensor coordinate system, the change in the tool coordinate system from the current pose to the desired pose can be calculated based on the extrinsic parameter calibration parameters of the tool coordinate system at the end of the robotic arm.

[0071] like Figure 8As shown, firstly, the Z-axis of the coordinate system 201 for the robotic arm end-effector tool is defined as the orientation of the end of the robotic arm end-effector tool 20, with the origin at the end point of the robotic arm end-effector tool 20. The X and Y axes are arbitrary and conform to the right-hand rule. Secondly, the origin of the coordinate system 151 for the disk-shaped target object 15 is defined as the center of the upper surface circle, with the Z-axis pointing downwards along the normal direction of the upper surface of the disk. The X and Y axes are arbitrary and conform to the right-hand rule.

[0072] When the end-effector tool 20 of the robotic arm is aligned with the disk-shaped target object 15, the Z-axis of the coordinate system 151 of the disk-shaped target object 15 and the Z-axis of the coordinate system 201 of the end-effector tool of the robotic arm are oriented in the same direction, that is, the direction vector of the Z-axis of the disk-shaped target object 15 in the desired tool coordinate system. Furthermore, the coordinates of the origin of the disk-shaped target object coordinate system 151 in the desired tool coordinate system are... d is the distance from the end point of the desired tool to the center of the disc-shaped target 15.

[0073] Based on the external parameters between the line laser profile sensor 10 and the tool coordinate system 201 Transform the instantaneous equation of the edge circle of the disk-shaped target 15 in the current sensor coordinate system to the current tool coordinate system 201, and obtain the direction vector of the Z-axis of the disk-shaped target coordinate system 151 in the current tool coordinate system 201. The origin of the disk coordinate system under the current tool coordinate system 201 The angle θ between the current Z-axis direction and the desired Z-axis direction in the end-effector coordinate system 201 can be expressed by the formula... The calculated direction of rotation Can be used The calculation yields the rotation matrix of tool coordinate system 201 from the current attitude to the desired attitude. Translation of tool coordinate system 201 from the current position to the desired position Can be used The calculation yields the pose change of the tool coordinate system from the current pose to the desired pose based on the rotation and translation amounts.

[0074] Finally, based on the extrinsic parameters between the tool coordinate system and the robotic arm end effector coordinate system... Through The change in the robotic arm's end effector from its current pose to its desired pose can be calculated. Current pose of the robotic arm end effector in the robotic arm base coordinate system Given, through the formula This allows us to obtain the pose of the robot arm's end effector coordinate system in the robot arm's base coordinate system.

[0075] The calculated desired pose of the robotic arm end effector The data is transmitted to the trajectory planning system, which plans a continuous motion trajectory of the robotic arm's end effector from the current pose to the desired pose. This trajectory is then transmitted to the robotic arm position control loop, which controls the robotic arm's end effector to move smoothly along the planned trajectory to the desired pose, thus completing the visual guidance alignment between the robotic arm's end effector and the disc-shaped target object 15.

[0076] Visual servoing systems based on line laser contour sensors, such as Figure 9 As shown. Steps S1-S4 above are continuously performed. The vision guidance system continuously calculates the desired pose of the robotic arm's end effector and transmits it to the robotic arm's position control loop via trajectory planning. The robotic arm also continuously transmits its latest end effector pose to the vision guidance system, forming a closed loop. This reduces the impact of the robotic arm's absolute positioning accuracy on the final vision guidance accuracy, thereby improving the vision guidance accuracy. Furthermore, because the line laser contour sensor has an extremely high measurement frequency (tens of thousands of hertz), the pose feedback frequency in the entire closed-loop control loop also reaches tens of thousands of hertz, resulting in a high response speed for the entire system. When the disc-shaped target 15 moves, the robotic arm's end effector can efficiently track it.

[0077] The above describes how three line laser contour sensors are used to perceive the pose of the disc-shaped target 15, guiding the robotic arm's end effector to precisely align with the disc-shaped target 15. When the robotic arm's end effector needs to precisely align with a specific position on an object of arbitrary shape, simply installing the disc-shaped target 15 in the corresponding position will achieve position-based, visually guided alignment.

[0078] This invention proposes a visual guidance alignment method for robotic arms based on line laser profile sensors. This method guides the end effector of the robotic arm to automatically align with a target object. By applying line laser profile sensors to the visual guidance of the robotic arm, at least three line laser profile sensors are installed at a fixed angle at the end effector of the robotic arm. Combined with a special target object shape—a disk—and a data processing algorithm tailored to the characteristics of line laser profile sensors, the method enables the line laser profile sensors to perceive the pose of the disk-shaped target object.

Claims

1. A vision-guided alignment device for a robotic arm, comprising: The target object is disc-shaped; Multiple line laser profile sensors are installed at the end of a robotic arm to emit strip-shaped laser beams toward the target object. The multiple strip-shaped laser beams emitted by the multiple line laser profile sensors are at an angle to each other. The multiple line laser profile sensors are used to perceive the pose of the target object and generate pose perception data. The vision guidance system includes a data processing unit, which receives and processes pose perception data generated by multiple line laser contour sensors to obtain the desired pose of the robotic arm end effector. A trajectory planning system is used to plan a continuous motion trajectory of the robotic arm's end effector from its current pose to the desired pose, based on the desired pose of the end effector, in order to guide the robotic arm's end effector tool to align with the target object. The number of the plurality of line laser contour sensors is three; The plurality of strip-shaped laser beams are uniformly distributed in the direction surrounding the center of the target object; The data processing unit of the visual guidance system includes the following processing steps: Extract the edge points of the target object; The edge circle of the target object is determined by fitting a spatial circle based on the edge points of the target object. Calculate the desired pose of the edge circle in the visual sensor coordinate system; Calculate the desired pose of the robotic arm; When the robotic arm's end effector needs to be precisely aligned with a specific location on an object of arbitrary shape, the disc-shaped target object is installed at that location.

2. The apparatus according to claim 1, wherein, Each line laser profile sensor includes a laser receiver and a laser emitter. The laser receiver is used to receive changes in the reflected light from the target object and acquire a linear laser point cloud on the surface of the target object.

3. A method for operating the robotic arm vision-guided alignment device according to claim 1, comprising: S1: Multiple line laser contour sensors emit strip-shaped laser beams to scan the disk surface of a disk-shaped target object, obtaining multiple line-shaped laser point clouds. The line-shaped laser point clouds intersect with the edge of the target object to form intersection points. S2: Extract all suspected edge points of the disk-shaped target from multiple linear laser point clouds by using the roughness characteristics of the intersection points; S3: Determine the edge circle of the target object by fitting a spatial circle based on the edge points of the target object; S4: Calculate the desired pose of the edge circle in the visual sensor coordinate system, and then calculate the desired pose of the robotic arm end effector in the base coordinate system.

4. The method according to claim 3, wherein, In step S3, the edge points of the target object are correctly extracted by determining whether the calculated radius of the spatial circle is equal to the manufacturing radius of the target object.

5. The method according to claim 3, wherein, The above steps S1-S4 continue to be performed. The vision guidance system continuously calculates the desired pose of the robotic arm's end effector and transmits it to the robotic arm's position control loop through trajectory planning. The robotic arm continuously transmits the latest pose of its end effector to the vision guidance system.

6. The method according to claim 3, wherein, When the end effector of a robotic arm needs to be precisely aligned with a specific location on an object of arbitrary shape, the disc-shaped target object is mounted at that location.

7. A computer-readable storage medium having software instructions stored thereon, which, when executed, perform the method according to any one of claims 3-6.

Images