A method and system for a humanoid two-armed robot to tighten valves

Abstract

The present invention belongs to the technical field of humanoid robot control, and particularly relates to a method and system for tightening valves of a humanoid dual-arm robot, including: obtaining a valve image dataset based on a binocular vision camera; annotating the valve image dataset to generate an annotated dataset; performing enhancement processing on the annotated data; identifying and positioning the valve based on a preset binocular vision algorithm and YOLO, and identifying the position, border, and center point of the valve; moving the dual-arm robot to both sides of the valve, and keeping the positions and postures of the ends of the two arms consistent in all directions except the clamping direction; performing clamping control based on an adaptive variable admittance controller; driving the rotation of the valve through the rotation of the joints at the ends of the two arms to achieve valve tightening. The present invention realizes fast and accurate valve recognition, greatly enhancing the reliability and flexibility of the system in practical applications.

Description

Technical Field

[0001] The present invention belongs to the technical field of humanoid robot control, and particularly relates to a method and system for tightening valves of a humanoid dual-arm robot. Background Technique

[0002] Screw-type valve devices are crucial in many industrial scenarios, such as petrochemical plants, nuclear power plants, and hydropower stations. The valves in these places usually involve high-pressure and high-temperature environments, requiring high reliability and high-precision operations. Currently, most valve-switching devices adopt dedicated mechanical structures. Although they can complete specific valve-tightening actions, their limitations become particularly prominent in emergency situations. For example, when an emergency occurs and the valve needs to be quickly closed, the operator must first replace the end effector. This process not only takes time but also may increase safety risks. In addition, existing electric grippers and pneumatic grippers have deficiencies in clamping force and cannot effectively meet the operation requirements of high-torque valves. In high-torque environments, these grippers often cannot provide sufficient clamping force, resulting in unstable operations or the inability to complete tasks. This makes it urgent to develop a more powerful clamping system to ensure that valves can be quickly and safely operated at critical moments.

[0003] In terms of valve positioning, YOLO (You Only Look Once) is currently commonly used for recognition, combined with a point cloud library (PCL) for target feature extraction. Although this is an advanced technical means, the complexity of this process may lead to insufficient real-time performance and low recognition efficiency. In rapidly changing working conditions, this cumbersome process affects the system response speed, thereby affecting the safety and effectiveness of the entire operation. Therefore, simplifying the recognition process and improving the recognition accuracy will be an important direction for enhancing the application of this technology. Summary of the Invention

[0004] Object of the Invention: The object of the present invention is to provide a method and system for tightening valves of a humanoid dual-arm robot in view of the deficiencies of the prior art, realizing rapid and accurate valve recognition, and greatly enhancing the reliability and flexibility of the system in practical applications.

[0005] Technical Solution: The method for tightening valves of the humanoid dual-arm robot described in the present invention includes the following steps:

[0006] S101. Obtain a valve image dataset based on a binocular vision camera;

[0007] S102. Label the valve image dataset to generate a labeled dataset, where the labeled dataset includes the bounding box of the valve and the central point position of the valve;

[0008] S103. Perform enhancement processing on the labeled dataset to expand the dataset;

[0009] S104. Identify and locate the valve based on a preset binocular vision algorithm and YOLO model, and identify the position, border, and center point of the valve;

[0010] S201. Identify and locate the valve through steps S101 to S104 to obtain the border position and center point position of the valve;

[0011] S202. Move the dual-arm robot to both sides of the valve, and keep the positions and poses of the ends of the two arms consistent in all directions except the clamping direction;

[0012] S203. Perform clamping control based on an adaptive variable admittance controller;

[0013] S204. Drive the rotation of the valve by rotating the joints at the ends of the dual-arm robot to tighten the valve.

[0014] To further improve the above technical solution, the dual-arm robot includes a left arm and a right arm. Actuators are provided at the ends of both the left arm and the right arm. The actuator includes a flange, a rubber-coated wheel, a six-axis force sensor, and an operating part. After the six-axis force sensor, the rubber-coated wheel, and the flange center are aligned, they are connected in sequence. After connection, the six-axis force sensor is accommodated in the rubber-coated wheel, the flange is fixed outside the rubber-coated wheel, the six-axis force sensor is connected to the end of the left arm or the right arm, and the flange is connected to the operating part.

[0015] Furthermore, the operating part is a two-finger gripper.

[0016] Furthermore, the steps of identifying and locating the valve by the binocular vision algorithm and YOLO model are as follows:

[0017] (1) Use the trained YOLOv8 model to detect different types of valves to obtain the prediction boxes of valve targets: where, , where, and respectively represent the upper left and lower right coordinates of the prediction box;

[0018] (2) Use image alignment to extract the point cloud information in the depth image that aligns with the prediction box: , where each point cloud data is represented as a three-dimensional coordinate with the camera coordinate system as the origin ;

[0019] (3) Perform centering processing on the point cloud data, and calculate the centroid of the point cloud corresponding within the prediction box and the centered point cloud data . The calculation formulas are as follows: , ;

[0020] (4) Calculate the covariance matrix of the centered point cloud data , and the calculation formula is as follows: , where is the transpose of the centered point cloud data;

[0021] (5) Solve the eigenvalues and eigenvectors of the covariance matrix, and the calculation formula is as follows: ;

[0022] (6) Calculate the point cloud normal vector . Assume that the eigenvector corresponding to the largest eigenvalue is , then this eigenvector is the normal vector of the point cloud data ;

[0023] (7) Transform the point cloud normal vector and the center of the prediction box from the camera coordinate system to the dual-arm robot world coordinate system. The transformation formula is: , , where is the world coordinate system, is the global camera coordinate system, represents the rotation transformation matrix from the coordinate system to the coordinate system, represents the offset vector from the

[0024] (8) Determine the pre-grasping position of the valve, and set the rough positioning result as the target position of the robotic arm , which is obtained by moving 40 cm along the point cloud normal vector of this area from the center of the prediction box . The calculation formula is as follows: ,

[0025] (9) After obtaining the target position, use the inverse kinematics of the robotic arm to calculate the joint angles corresponding to the valve grasping points: .

[0026] Further, the adaptive variable admittance controller performs clamping control including: reading the force data obtained by the six-axis force sensor , and performing gravity compensation and coordinate transformation; calculating the error between the desired force and the actual force ​​​, and bring it into the adaptive variable admittance controller to control the two arms to maintain a constant force in the clamping direction; the adaptive variable admittance controller dynamically adjusts the control parameters through an adaptive control algorithm, including adjusting the damping coefficient matrix b, the stiffness coefficient matrix k, and the mass coefficient matrix m.

[0027] Furthermore, the control formula for the control parameters of the adaptive variable admittance controller is:

[0028]

[0029] where m is the mass coefficient matrix, b is the damping coefficient matrix, and k is the stiffness coefficient matrix. is the position error, representing the difference between the actual position and the desired position at time t. is the velocity error, representing the difference between the actual velocity and the desired velocity at time t. is the acceleration error, representing the difference between the actual acceleration and the desired acceleration at time t. is the position error at time t + 1. is the velocity error at time t + 1. is the position acceleration error at time t + 1, and T is the time step.

[0030] ∆b is the adaptive damping adjustment parameter, and its update rate is: , where , is the update rate, represents the adaptive parameter at the current time t, represents the adaptive parameter at time t + 1.

[0031] Add the desired trajectory to the output by the adaptive admittance controller to obtain the actual motion trajectories of the left and right arms.

[0032] A system for implementing the above-mentioned humanoid double-arm robot valve tightening method includes:

[0033] A binocular vision camera for obtaining a valve image dataset.

[0034] A binocular vision algorithm and a YOLO model for obtaining the image dataset and performing valve recognition and positioning.

[0035] A double-arm robot, including a left arm and a right arm. Actuators are provided at the ends of the left arm and the right arm for clamping the valve and performing a tightening operation under adaptive variable admittance control. The actuator has a six-dimensional force sensor for real-time detection of the valve clamping force and transmitting the force data to the adaptive variable admittance controller.

[0036] An adaptive variable admittance controller is used to adjust the clamping force of the two arms according to the feedback of the force sensor, ensuring a constant valve clamping force and controlling the rotation of the two-arm manipulator to tighten the valve.

[0037] Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows: In the valve recognition process of the present invention, the YOLO object detection algorithm and the binocular vision algorithm are combined to realize the automatic recognition and positioning of the valve; the binocular vision algorithm improves the valve recognition accuracy through stereo vision and depth calculation technology, and can recognize and position the valve in a complex industrial environment; binocular vision can realize the RGB recognition of images, and depth vision obtains depth information. Based on the fusion of the two pieces of information, the spatial pose information and edge detection information of the valve can be recognized, so the accuracy and speed of target recognition are improved. Based on the adaptive variable admittance controller of the present invention, the clamping force and movement trajectory of the two arms are adjusted in real time to meet the clamping and tightening requirements of different valve types.

[0038] The valve-tightening structure of the present invention is simple and relatively low in cost; compared with the traditional use of a dedicated valve-tightening device, it is obviously a waste of time to replace the end effector in some emergency places. This structure can be equipped with a five-finger / two-finger tool at the end to quickly perform other operations; there is no angle limit, the operation is simple and efficient, and it will not require changing hands like a person turning a steering wheel when there is an angle limit. It has more advantages than a person tightening a valve; visual control is simple and easy to implement, and clamping can be achieved without passing through other complex algorithms; the high-friction coefficient rubber-coated wheel can also protect the six-axis force sensor from being easily damaged in a high-risk environment, and this solution can provide a relatively large torque with the high-friction coefficient rubber-coated wheel. If two fingers are used to clamp and rotate, there is often a risk of slipping, and the torque that can be provided is also relatively small, making it difficult to twist relatively large valves. Description of the Drawings

[0039] Figure 1 Schematic diagram of the end effector connection device of the embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the end structure of the embodiment device of the present invention;

[0041] Figure 3 Flowchart of valve recognition based on YOLO in the embodiment of the present invention;

[0042] Figure 4 Effect diagram of valve recognition using YOLO in the embodiment of the present invention;

[0043] Figure 5 Flowchart of two-arm valve tightening based on YOLO recognition and positioning in the embodiment of the present invention;

[0044] Figure 6 Effect diagram of valve tightening in the embodiment of the present invention;

[0045] Figure 7 This is the adaptive variable admittance control of the embodiment of the present invention.

[0046] In the figure: 1: End effector flange; 2: High-friction coefficient rubber-coated wheel; 3: Six-axis force sensor; 4: Two-finger gripper. Specific implementation manners

[0047] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the described embodiments.

[0048] Embodiment 1: As Figure 1 、 Figure 2 shown, the end effector connection device includes an end effector flange 1, a high-friction coefficient rubber-coated wheel 2, a six-axis force sensor 3, and a two-finger gripper 4. The six-axis force sensor 3 is connected to the end of the robotic arm. The high-friction coefficient rubber-coated wheel 2 wraps the six-axis force sensor 3. The end effector flange 1, the high-friction coefficient rubber-coated wheel 2, and the six-axis force sensor 3 align the hole positions and can be connected by internal hexagon bolts. The two-finger gripper 4 is connected to the end effector flange 1, and the two-finger gripper 4 at the end can be changed according to actual operation requirements.

[0049] As Figure 3 shown, the valve recognition flowchart based on YOLO is as follows:

[0050] S101. Obtain a valve image dataset based on a binocular vision camera, and specifically select the source of the dataset, such as valves in industrial environments such as hydropower stations and petrochemical plants;

[0051] S102. Annotate the collected valve dataset to generate an annotated dataset, and use the LabelImg tool for manual annotation to mark the position of the bounding box and its center point of the valve;

[0052] S103. Perform enhancement processing on the annotated data, formulate a specific enhancement strategy, and process the image through various methods such as rotation, flipping, scaling, and brightness adjustment to expand the dataset;

[0053] S104. Identify and locate the valve based on a preset binocular vision algorithm and a YOLO model, and combine the preset binocular vision algorithm, basic parameters, and the target YOLO model to identify and locate the valve.

[0054] By combining the target YOLO model with the binocular vision algorithm, the automatic identification and location of the valve are realized, and the specific steps are as follows:

[0055] 1. Use the trained YOLOv8 model to detect different types of valves to obtain the prediction boxes of valve targets:

[0056]

[0057] Among them, and respectively represent the upper left and lower right coordinates of the prediction box;

[0058] 2. Obtain the RGB image and depth image captured by the binocular camera, and use image alignment to extract the point cloud data in the depth image that is aligned with the prediction box:

[0059]

[0060] Among them, each point cloud data is represented as a three-dimensional coordinate with the camera coordinate system as the origin ;

[0061] 3. Perform centering processing on the point cloud data, and calculate the centroid of the point cloud corresponding within the prediction box and the centered point cloud data , and the calculation formulas are as follows:

[0062]

[0063]

[0064] 4. Calculate the covariance matrix of the centered point cloud data , and the calculation formula is as follows

[0065]

[0066] Among them, is the transpose of the centered point cloud data;

[0067] 5. Solve the eigenvalues and eigenvectors of the covariance matrix, and the calculation formulas are as follows:

[0068]

[0069] 6. Calculate the point cloud normal vector Assume that the eigenvector corresponding to the largest eigenvalue is , then this eigenvector is the normal vector of this point cloud data ;

[0070] 7. Transform the point cloud normal vector and the prediction box center from the camera coordinate system to the dual-arm robot world coordinate system, and the transformation formula is:

[0071]

[0072]

[0073] Among them, is the world coordinate system, is the global camera coordinate system, represents the rotation transformation matrix from the coordinate system to the coordinate system, represents the offset vector from the

[0074] 8. Determine the pre-grasping position of the valve and set the rough positioning result as the target position of the robotic arm , and there is the center of the prediction box at this position Move 40 cm along the normal vector of the point cloud in this area direction to obtain, and the calculation formula is as follows:

[0075]

[0076] 9. After obtaining the target position, use the inverse kinematics of the robotic arm to calculate the joint angles corresponding to the valve grasping points:

[0077]

[0078] As Figure 4 shown, it is the effect diagram of YOLO recognizing the valve. Based on YOLO, the valve is recognized and positioned to obtain its center position and the position of the valve border.

[0079] As Figure 5 shown, it is the flow chart of the two-arm valve turning based on YOLO recognition and positioning. The specific clamping process is as follows:

[0080] S201. Valve recognition and positioning. According to the above valve recognition process, the valve can be recognized and positioned, so as to read the positions of the upper left corner and the upper right corner of the valve border, and read the position of the valve center;

[0081] S202. Move the left and right arms to both sides of the valve. Figure 6 This is the schematic diagram of valve turning in the embodiment of the present invention. O is the world coordinate system. is the point expressed in the world coordinate system, which is the position of the left arm's high-friction coefficient rubber-coated wheel. P is the point expressed in the world coordinate system, which is at the high-friction coefficient rubber-coated wheel of the right arm. O is the point expressed in the world coordinate system, which is the center position of the valve. The center position of the valve can be identified and located through the steps in S10. Move the left and right arms to both sides of the valve respectively, and keep the positions of the ends of the left and right arms in the xyz directions the same except for the clamping direction. The positions and postures in other directions are kept consistent. Also, keep the rubber-coated wheel with high friction coefficient and the valve center the same in the xyz directions except for the clamping direction, and the other positions are consistent.

[0082] S203. Clamp based on adaptive variable admittance control. Specifically, under admittance control, use the high-friction rubber-coated wheel to approach the middle point of the valve. When the friction wheel touches the outer periphery of the valve, quickly perform force tracking. And this force also greatly shows the influence on turning the valve. Therefore, a relatively large clamping force is required.

[0083] S204. Implement turning the valve. Specifically, the power source for turning the valve depends on the rotation of the end joints of the left and right robotic arms. The end joints of the left and right robotic arms rotate in the same direction to drive the valve to rotate accordingly.

[0084] The present invention consists of a dual-arm robot composed of a left arm and a right arm. The left arm is designated as L, and the right arm is designated as R. Here, i represents the left and right arms, that is . The specific control method for clamping in S203 adopts Figure 7 the schematic diagram of adaptive variable admittance control shown as follows. The specific control steps are as follows:

[0085] S301 F is the force data read by the six-axis force sensor. This force data needs to be obtained through gravity compensation and coordinate transformation (transformed into the world coordinate system through a coordinate transformation algorithm).

[0086] S302. Subtract this force data from the desired force and substitute it into the adaptive variable admittance controller of the present invention. It is required that the force in the clamping direction is a constant force, and the dual arms need to maintain a constant force clamping. For example, Figure 7 the adaptive variable admittance controller in

[0087]

[0088] is as follows: M is the mass coefficient matrix, D is the damping coefficient matrix, K is the stiffness coefficient matrix. Among them, , and are constant matrices, can be adjusted according to the adaptive control law to track the force. Its specific update rate is as follows:

[0089]

[0090] In the above formula, , which is used to prevent the denominator in the formula from being 0, is the update rate.

[0091] By adding the desired trajectory to the calculated by the adaptive admittance controller, the actual motion trajectories of the left and right arms are calculated and this value is input into the dual-arm robot system.

[0092] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as a limitation of the present invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the present invention defined by the appended claims.

Claims

1. A method for a humanoid two-armed robot to tighten a valve, characterized in that, The method includes the following steps: S101. Obtain a valve image dataset based on a binocular vision camera; S102. Annotate the valve image dataset to generate an annotated dataset, where the annotated dataset includes the bounding box of the valve and the position of the center point of the valve; S103. Perform enhancement processing on the annotated dataset to expand the dataset; S104. Identify and locate the valve based on a preset binocular vision algorithm and a YOLO model, and identify the position, border, and center point of the valve; The steps of identifying and locating the valve by the binocular vision algorithm and the YOLO model are as follows: (1) Use the trained YOLOv8 model to detect different types of valves and obtain the prediction boxes of valve targets: , where and respectively represent the coordinates of the upper left corner and the lower right corner of the prediction box; (2) Use image alignment to extract the point cloud information in the depth image that aligns with the prediction box: , where each point cloud data is represented as a three-dimensional coordinate with the camera coordinate system as the origin ; (3) Centralize the point cloud data and calculate the centroid of the corresponding point cloud within the prediction box and the centralized point cloud data , and the calculation formulas are as follows: , ; (4) Calculate the covariance matrix of the centered point cloud data , and the calculation formula is as follows: , where is the transpose of the centered point cloud data; (5) Solve the eigenvalues of the covariance matrix and eigenvectors , and the calculation formula is as follows: ; (6) Point cloud normal vector Calculate. Assume that the eigenvector corresponding to the largest eigenvalue is Then this eigenvector is the normal vector of the point cloud data ; (7) Point cloud normal vector and the center of the prediction box are transformed from the camera coordinate system to the dual-arm robot world coordinate system. The transformation formula is: , , where is the world coordinate system, is the global camera coordinate system, represents the rotation transformation matrix from the coordinate system to the coordinate system, and represents the offset vector from the (8) Determine the pre-grasp position of the valve and set the rough positioning result as the target position of the robotic arm , which is determined by the center of the prediction box moving 40 cm along the normal vector of the point cloud in this area The direction is obtained, and the calculation formula is as follows: , After obtaining the target position, use the inverse kinematics of the robotic arm to calculate the joint angles corresponding to the valve grasping points: ; S201. Identify and locate the valve through steps S101 to S104 to obtain the border position and the center point position of the valve; S202. Move the dual-arm robot to both sides of the valve, and keep the positions and poses of the ends of the two arms consistent in all directions except the clamping direction; S203. Perform clamping control based on an adaptive variable admittance controller; S204. Drive the rotation of the valve through the rotation of the end joints of the dual-arm robot to achieve valve tightening.

2. The method for tightening the valve of a humanoid two-armed robot according to claim 1, characterized in that, The dual-arm robot includes a left arm and a right arm. Actuators are provided at the ends of the left arm and the right arm. The actuator includes a flange, a rubber-coated wheel, a six-axis force sensor, and an operating part. After the six-axis force sensor, the rubber-coated wheel, and the flange center are aligned, they are connected in sequence. After connection, the six-axis force sensor is accommodated in the rubber-coated wheel, the flange is fixed outside the rubber-coated wheel, the six-axis force sensor is connected to the end of the left arm or the right arm, and the flange is connected to the operating part.

3. The method for tightening the valve of the humanoid two-armed robot according to claim 2, characterized in that The operating part is a two-finger gripper.

4. The method for tightening the valve of a humanoid two-armed robot according to claim 2, wherein The clamping control by the adaptive variable admittance controller includes: Read the force data obtained by the six-axis force sensor ; Calculate the expected force and the actual force to obtain the error . Then, input the error into the adaptive variable admittance controller to control the two arms to maintain a constant force in the clamping direction; The adaptive variable admittance controller dynamically adjusts control parameters through an adaptive control algorithm. The control parameters include adjusting the damping coefficient matrix b, the stiffness coefficient matrix k, and the mass coefficient matrix m.

5. The method for tightening the valve of the humanoid dual-arm robot according to claim 4, wherein, The control formula for the control parameters of the adaptive variable admittance controller is: where, m is the mass coefficient matrix, b is the damping coefficient matrix, and k is the stiffness coefficient matrix, is the position error, representing the difference between the actual position and the desired position at time t, is the velocity error, representing the difference between the actual velocity and the desired velocity at time t, is the acceleration error, representing the difference between the actual acceleration and the desired acceleration at time t, is the position error at time t + 1, is the velocity error at time t + 1, is the position acceleration error at time t + 1, and T is the time step; ∆b is an adaptive damping adjustment parameter, and its update rate is: , where , is the update rate, represents the adaptive parameter at the current time t, represents the adaptive parameter at time t+1; The desired trajectory is added to the output of the adaptive admittance controller to obtain the actual motion trajectories of the left and right arms .

6. A system for implementing the method for tightening a valve of the humanoid double-arm robot according to claim 1, characterized in that, Including: A binocular vision camera for obtaining a valve image dataset; A binocular vision algorithm and a YOLO model for obtaining an image dataset and performing valve identification and positioning; A dual-arm robot, including a left arm and a right arm. Actuators are provided at the ends of the left arm and the right arm, and are used to clamp the valve and perform a tightening operation under adaptive variable admittance control. The actuator has a six-axis force sensor for real-time detection of the valve clamping force and transmitting force data to the adaptive variable admittance controller; An adaptive variable admittance controller for adjusting the clamping force of the two arms according to the feedback of the force sensor, ensuring a constant valve clamping force and controlling the rotation of the dual-arm manipulator to achieve valve tightening.

Images