A visual servoing compensation method, system, device and storage medium

Abstract

This invention relates to the field of automatic screw fastening technology, specifically a visual servo compensation method, system, device, and storage medium. The method includes: acquiring the 6D pose of a target workpiece as the robot's guiding coordinates to guide the end effector of the robotic arm to connect with the clamping fixture of the target workpiece; the robot performing mechanical compensation through a redundant mechanism; visually tracking and identifying the clamping fixture of the target workpiece to determine its 6D pose; reading the robot's 6D pose and calculating the offset error of the robot relative to the clamping fixture in various directions; and obtaining the control parameters of the robot's end effector based on the offset error using a position compensation algorithm to achieve visual compensation for the robotic arm. By acquiring the 6D pose of the target workpiece, real-time positioning and matching between the robot and the target fixture are achieved. The robotic arm can adaptively adjust according to the 6D pose offset of the target workpiece through mechanical and visual compensation.

Description

Technical Field

[0001] This invention relates to the field of automatic screw fastening technology, specifically to a visual servo compensation method, system, device, and storage medium. Background Technology

[0002] Assembly is a major component of the entire production system and an indispensable part of mechanical manufacturing. With the development of automation in mechanical manufacturing, the market demands higher product output efficiency in the assembly process. Screw fastening is a common assembly method in manufacturing and an important step in the assembly process.

[0003] Traditional screw fastening relies primarily on manual labor, which is inefficient, dependent on operator experience, and subject to fatigue from prolonged work, impacting installation efficiency. Therefore, while automated screw fastening devices can replace manual labor to improve efficiency and automate assembly of workpieces on production lines, significantly reducing worker workload, automated screw fastening devices often require pre-programming of each threaded hole position, making operation cumbersome. They also cannot directly identify and locate threaded holes. Furthermore, as workpieces constantly shift along the automated production line, the automated screw fastening device cannot accurately follow the screw-tightening action of the electric screwdriver bit due to workpiece displacement, resulting in unsatisfactory assembly quality, low yield rates in manufacturing, and severely impacting product quality. Summary of the Invention

[0004] To address the aforementioned problems, the first aspect of this invention provides a visual servoing compensation method, the steps of which are as follows: S1. Obtain the 6D pose of the target workpiece as the robot's guiding coordinates, and guide the tooling of the robot's end effector to connect with the clamping tooling of the target workpiece. S2. The robot performs mechanical compensation through the redundant mechanism of the tooling of the fixture; S3. Take a picture of the clamping fixture for the target workpiece and perform visual tracking and recognition to determine the 6D pose of the clamping fixture; S4. Based on the 6D pose of the clamping fixture, read the 6D pose of the robot and calculate the offset error of the robot relative to the clamping fixture in each direction. S5. Based on the offset error, the control parameters of the robot end effector are obtained through the position compensation algorithm to achieve visual compensation for the robotic arm.

[0005] In some implementations of the first aspect, step S3 involves photographing the clamping fixture of the target workpiece and performing visual tracking and recognition. The specific method is as follows: S31. Take a picture of the clamping fixture and determine the initial position of the fixture by visually recognizing the positioning hole at the rear end of the clamping fixture. S32. Visually track the positioning hole area of ​​the clamping fixture and cut out the key area where the positioning hole of the clamping fixture is located. S33. Extract and identify the positioning holes based on the key area where the positioning holes are located, obtain the mask information and edge feature points of the positioning holes, and determine the 6D pose of the positioning holes.

[0006] In some implementations of the first aspect, the 6D pose of the positioning hole is:

[0007] in, x servo , y servo , z servo To hold the tooling in Cartesian coordinates in the robot coordinate system, a servo , b servo , c servo For along xyz The rotation angles of the three axes.

[0008] In some implementations of the first aspect, the 6D pose of the robot read in S4 is as follows: , in, For the robot's Cartesian coordinates, For along xyz Rotation angles of the three axes, The offset errors in each direction are obtained based on the 6D pose of the clamping fixture and the 6D pose of the robot. The calculation formula is as follows: , in, Err servo Indicates offset error. X servo This represents the 6D pose of the positioning hole. This represents the robot's 6D pose.

[0009] S4 further includes tracking the error based on the calculated offset error of the robot relative to the clamping fixture in each direction. If the obtained offset error in the x and y directions is greater than or equal to the offset of the clamping fixture along the x and y directions, then step S3 is repeated to reconfirm the 6D pose of the clamping fixture.

[0010] In some implementations of the first aspect, the control parameters of the robot end effector obtained through the position compensation algorithm in S5 are as follows: , in, Err servo Indicates offset error. For PID control parameters, The new 6D pose of the robot is obtained after calculation based on offset error and PID control algorithm.

[0011] The visual servo compensation method also includes S6, which controls the fixed electric screwdriver of the robotic arm to press down and perform a wire-tightening action based on the control parameters obtained at the robot end.

[0012] The second aspect provides a visual servoing compensation system, including: The guidance module is used to obtain the 6D pose of the target workpiece as the robot's guidance coordinates, and guides the tooling of the robot's end effector to connect with the clamping tooling of the target workpiece. The mechanical compensation module is used for mechanical compensation of the robot through the redundant mechanism of the tooling of the fixture; The 6D pose confirmation module is used to photograph the clamping fixture of the target workpiece and perform visual tracking and recognition to determine the 6D pose of the clamping fixture. The offset error calculation module is used to read the robot's 6D pose based on the 6D pose of the clamping fixture, and calculate the offset error of the robot relative to the clamping fixture in each direction. The vision compensation module is used to obtain the control parameters of the robot end effector based on the offset error through a position compensation algorithm, thereby realizing visual compensation for the robotic arm.

[0013] A visual servo compensation device includes a processor and a memory, wherein the processor implements a visual servo compensation method as described above when executing program data stored in the memory.

[0014] A computer-readable storage medium for storing control program data, wherein the control program data, when executed by a processor, implements a visual servo compensation method as described above.

[0015] The beneficial effects are: This application utilizes visual recognition, tracking, and positioning technologies for the target workpiece and tooling. By acquiring the 6D pose of the target workpiece, it achieves real-time positioning and matching between the robot and the target tooling. Simultaneously, the robot arm can adaptively adjust according to the 6D pose offset of the target workpiece through mechanical and visual compensation, thereby improving the production efficiency of screw fastening. Visual compensation better guides the robot arm to align with the threaded hole of the target workpiece, thus improving the success rate of screw fastening and achieving a high yield rate for manufacturing high-quality products. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a visual servoing compensation method. Figure 2 A schematic diagram of the automatic screw fastening device; 1. Robotic arm; 2. Fixture tooling; 3. Redundancy mechanism; 4. Vision module; 5. Electromagnet; 6. Fixed electric screwdriver; 7. Target workpiece; 8. Clamping fixture. Detailed Implementation

[0017] Exemplary embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings.

[0018] See Figure 1 This application provides a visual servo compensation method applied to an automatic screw fastening device. The automatic screw fastening device includes a robotic arm 1, a fixture 2, a redundancy mechanism 3, a vision module 4, a fixed electric screwdriver 6, and an electromagnet 5. The specific steps of the visual servo compensation method are as follows: S1. Obtain the 6D pose of the target workpiece 7 as the robot's guiding coordinates, and guide the tooling 2 of the end effector of the robot's robotic arm 1 to connect with the clamping tooling 8 of the target workpiece 7. First, the robot's vision module 4 locates the position of the target workpiece 7 in real time and sends the position information to the robotic arm 1 to control the posture of the robotic arm 1, so that the robotic arm 1 can follow the target workpiece 7 to move on the assembly line.

[0019] Simultaneously, the vision module 4 performs visual recognition on the threaded holes of the target workpiece 7, obtains the mask information of each threaded hole by cropping the main distribution area of ​​the threaded holes, and extracts the edge feature points of each threaded hole; the 6D pose of the target workpiece 7 is solved by using the edge feature points of the threaded holes of the target workpiece 7, and the 6D pose of the target workpiece 7 is obtained as follows: , in, x , y , z Let the Cartesian coordinates of the target workpiece 7 in the robot coordinate system be given. a , b , cFor along xyz Rotation angles of the three axes; According to the Levenberg-Marquarelt optimization algorithm, the pose of the target workpiece 7 can be solved as follows: , in X For the target 6D pose, λ The introduced coefficients are in the range of 0 and 1. I It is the identity matrix. J For Jacobi matrix, ● T This represents the transpose of a vector or matrix. Err The detection error is due to the screw hole edge point. J Specifically, it can be expressed as , in, For effective feature points on the edge of the threaded hole, the subscript is... i Represents the first point on the edge of the screw hole. i One point, superscript k Representing the k One screw hole.

[0020] The 6D pose of the target workpiece 7 obtained by the solution is used as the guiding coordinate of the robot. The tooling 2 of the end clamp of the downward guiding robot arm 1 is activated to achieve handshake with the clamping tooling 8 of the target workpiece 7, and the tooling 2 of the clamping tooling and the clamping tooling 8 are fixed by the electromagnet 5.

[0021] S2. The robot performs mechanical compensation through the redundant mechanism 3 of the fixture tooling 2; The clamping fixture 8 is provided with two electromagnet 5 connection holes. A circular iron block is installed at the bottom of the electromagnet 5 connection hole. When the electromagnet 5 is energized, the iron block is attracted and fixed in the hole. When the fixture 2 is fixedly connected with the clamping fixture 8, it will shake or vibrate. The robot realizes mechanical compensation for the fixture 2 through the redundancy mechanism 3. The redundancy mechanism 3 includes a slider and a guide rail, which enables the fixture 2 of the end of the robotic arm 1 to move in space in two mutually perpendicular directions. This helps the fixture 2 to adjust its position according to the clamping fixture 8 of the target workpiece 7, so as to realize the fixed connection of the two fixtures.

[0022] S3. Take a picture of the clamping fixture 8 of the target workpiece 7 and perform visual tracking and recognition to determine the 6D pose of the clamping fixture 8. When the clamping fixture 8 becomes redundant with the fixture 2, the clamping fixture 8 may be affected by vibration and shift along the x and y directions. To enable the robot to identify and detect this shift, the tail plate of the clamping fixture 8 of the target workpiece 7 is provided with a rear positioning hole. The vision module 4 of the robotic arm 1 performs dynamic real-time imaging of the positioning hole for identification and tracking. The specific method is as follows: S31. Take a picture of the clamping fixture 8, and determine the initial position of the fixture by visually recognizing the positioning hole at the rear end of the clamping fixture 8. S32. Visually track the positioning hole area of ​​the clamping fixture 8 and cut out the key area where the positioning hole of the clamping fixture 8 is located. S33. Extract and identify the positioning holes based on the key area where the positioning holes are located, obtain the mask information and edge feature points of the positioning holes, and determine the 6D pose of the positioning holes. By cropping the key area where the positioning holes of the clamping fixture 8 are located, and performing binarization and opening / closing operations on the cropped image, the mask information of each positioning hole is obtained; by performing gradient processing on the area of ​​each positioning hole covered by the mask, the edge feature points of the positioning holes are obtained using the corner point extraction method. Similar to the 6D pose determination method for the target workpiece 7 mentioned above, the 6D pose of the positioning hole is determined by the edge feature points of the positioning hole as follows:

[0023] in, x servo , y servo , z servo To determine the Cartesian coordinates of the clamping fixture 8 in the robot coordinate system, a servo , b servo , c servo For along xyz The rotation angles of the three axes.

[0024] S4. Based on the 6D pose of the clamping fixture 8, read the 6D pose of the robot and calculate the offset error of the robot relative to the clamping fixture 8 in each direction. After the robot obtains the 6D pose information of the positioning hole, the host computer reads the robot's 6D pose in the current state as follows:

[0025] in, For the robot's Cartesian coordinates, For along xyzThe rotation angles of the three axes.

[0026] The offset errors in each direction are obtained based on the 6D pose of the clamping fixture 8 and the 6D pose of the robot. The calculation formula is as follows: , in, Err servo Indicates offset error. X servo This represents the 6D pose of the positioning hole. This represents the robot's 6D pose.

[0027] After the tooling 2 of the robotic arm 1 is fixedly connected to the clamping tooling 8 of the target workpiece 7, the offset of the clamping tooling 8 caused by vibration or shaking is small. In order to ensure the accuracy of the offset error result, step S4 also includes error verification based on the calculated offset error of the robot relative to the clamping tooling 8 in each direction. If the offset error in the x and y directions is greater than or equal to the offset of the clamping tooling along the x and y directions, then step S3 is repeated to reconfirm the 6D pose of the clamping tooling 8.

[0028] S5. Based on the calculated offset error, the control parameters of the robot end are obtained through the position compensation algorithm to achieve visual compensation of the fixed electric screwdriver 6 at the end of the robotic arm 1.

[0029] The visual servoing compensation method obtains the following control parameters for the robot's end effector: , in, Err servo Indicates offset error. For PID control parameters, The new 6D pose of the robot is obtained after calculation based on offset error and PID control algorithm.

[0030] Meanwhile, the visual servo compensation includes step S6, which, based on the control parameters obtained at the end of the robot, aligns the fixed electric screwdriver 6 of the robotic arm 1 with the threaded hole of the target workpiece 7. The robot powers on the electric screwdriver, and the ball screw structure controls the fixed electric screwdriver 6 of the robotic arm 1 to press down and perform a screw tightening action, thereby completing the screw fastening of the target workpiece 7.

[0031] Furthermore, the present invention also provides a visual servoing compensation system, comprising: The guidance module is used to obtain the 6D pose of the target workpiece 7 as the robot's guidance coordinates, and guide the tooling 2 of the end effector of the robotic arm 1 to connect with the clamping tooling 8 of the target workpiece 7. The robot's vision module 4 locates the position of the target workpiece 7 in real time and sends the position information to the robotic arm 1 to control the posture of the robotic arm 1, so that the robotic arm 1 can follow the target workpiece 7 to move on the production line. The obtained 6D pose of the target workpiece 7 is used as the robot's guidance coordinates, and the tooling 2 of the end effector of the robotic arm 1 is pressed down to achieve a handshake with the clamping tooling 8 of the target workpiece 7. The tooling 2 of the tooling and the clamping tooling 8 are fixed by the electromagnet 5. The mechanical compensation module is used for the robot to perform mechanical compensation through the redundant mechanism 3 of the tooling 2 of the fixture; The 6D pose confirmation module is used to photograph and visually track the clamping fixture 8 of the target workpiece 7 to determine the 6D pose of the clamping fixture 8. The offset error calculation module is used to read the robot's 6D pose based on the 6D pose of the clamping fixture 8 and calculate the offset error of the robot relative to the clamping fixture 8 in each direction. The vision compensation module is used to obtain the control parameters of the robot end effector based on the offset error through the position compensation algorithm, so as to realize the vision compensation of the end effector of the robotic arm 1.

[0032] Furthermore, the present invention provides a visual servo compensation device, including a processor and a memory, wherein the processor executes program data stored in the memory to implement a visual servo compensation method as described above.

[0033] Finally, the present invention provides a computer-readable storage medium for storing control program data, wherein the control program data, when executed by a processor, implements a visual servo compensation method as described above.

Claims

1. A visual servoing compensation method, characterized in that, Includes the following steps: S1. Obtain the 6D pose of the target workpiece as the robot's guiding coordinates, and guide the tooling of the robot's end effector to connect with the clamping tooling of the target workpiece. S2. The robot performs mechanical compensation through the redundant mechanism of the fixture tooling. The redundant mechanism includes a slider and a guide rail, which enables the fixture tooling at the end of the robotic arm to move in space in two mutually perpendicular directions. This helps the fixture tooling to adjust its position according to the clamping fixture of the target workpiece, thereby achieving a fixed connection between the two fixtures. S3. Take a picture of the clamping fixture for the target workpiece and perform visual tracking and recognition to determine the 6D pose of the clamping fixture; S4. Based on the 6D pose of the clamping fixture, read the 6D pose of the robot and calculate the offset error of the robot relative to the clamping fixture in each direction. S5. Based on the offset error, the control parameters of the robot end are obtained through position compensation to achieve visual compensation of the robotic arm.

2. The visual servoing compensation method of claim 1, wherein, In step S3, the clamping fixture for the target workpiece is photographed and visually tracked and identified. The specific method is as follows: S31. Take a picture of the clamping fixture, and determine the initial position of the clamping fixture by visually recognizing the positioning hole at the rear end of the clamping fixture. S32. Visually track the positioning hole area of ​​the clamping fixture and cut out the key area where the positioning hole of the clamping fixture is located. S33. Extract and identify the positioning holes based on the key area where the positioning holes are located, obtain the mask information and edge feature points of the positioning holes, and determine the 6D pose of the positioning holes.

3. The visual servoing compensation method of claim 2, wherein, The 6D pose of the positioning hole is: ， wherein x servo , y servo , z servo Cartesian coordinates of the clamping tool in the robot coordinate system, a servo , b servo , c servo rotation angles along xyz three axes.

4. The visual servoing compensation method of claim 2, wherein, The 6D pose of the robot read in S4 is as follows: ， wherein, is the Cartesian coordinate of the robot, is the rotation angle along xyz three axes, The offset errors in each direction are obtained based on the 6D pose of the clamping fixture and the 6D pose of the robot. The calculation formula is as follows: ， wherein, Err servo denotes the offset error, X servo denotes the 6D pose of the positioning hole, denotes the 6D pose of the robot.

5. The visual servoing compensation method of claim 4, wherein, In step S5, the control parameters of the robot end effector are obtained through a position compensation algorithm as follows: ， wherein, Err servo denotes the offset error, are PID control parameters, is a new 6D pose of the robot calculated based on the offset error and the PID control algorithm.

6. The visual servoing compensation method of claim 1, wherein, It also includes S6, which controls the fixed electric screwdriver of the robotic arm to press down and perform a screw-tightening action based on the control parameters obtained at the end of the robot.

7. The visual servoing compensation method of claim 1, wherein, S4 further includes performing error verification based on the calculated offset errors of the robot relative to the clamping fixture in each direction. If the obtained offset errors in the x and y directions are greater than or equal to the offsets of the clamping fixture along the x and y directions, then step S3 is repeated to reconfirm the 6D pose of the clamping fixture.

8. A visual servoing compensation system implementing the visual servoing compensation method of claim 1, characterized in that, include: The guidance module is used to obtain the 6D pose of the target workpiece as the robot's guidance coordinates, and guides the tooling of the robot's end effector to connect with the clamping tooling of the target workpiece. The mechanical compensation module is used for mechanical compensation of the robot through the redundant mechanism of the tooling of the fixture; The 6D pose confirmation module is used to photograph the clamping fixture of the target workpiece and perform visual tracking and recognition to determine the 6D pose of the clamping fixture. The offset error calculation module is used to read the robot's 6D pose based on the 6D pose of the clamping fixture, and calculate the offset error of the robot relative to the clamping fixture in each direction. The vision compensation module is used to obtain the control parameters of the robot end effector based on the offset error through a position compensation algorithm, thereby realizing visual compensation for the robotic arm.

9. A visual servoing compensation device, characterized in that, It includes a processor and a memory, wherein the processor implements a visual servo compensation method as described in any one of claims 1-7 when executing program data stored in the memory.

10. A computer-readable storage medium, characterized in that, Used to store control program data, wherein the control program data, when executed by a processor, implements a visual servo compensation method as described in any one of claims 1-7.

Images