Humanoid double-arm robot valve tightening method and system

Abstract

The invention belongs to the technical field of humanoid robot control, and particularly relates to a humanoid double-arm robot valve tightening method and system, and the method comprises the steps: obtaining a valve image data set based on a binocular vision camera; labeling the valve image data set to generate a labeled data set; performing enhancement processing on the annotation data; recognizing and positioning the valve based on a preset binocular vision algorithm and YOLO, and recognizing the position, the frame and the center point of the valve; the two-arm robot is moved to the two sides of the valve, and the positions and poses of the tail ends of the two arms in other directions except the clamping direction are kept consistent; clamping control is carried out based on a self-adaptive variable admittance controller; the valve is driven to rotate through rotation of the joint at the tail end of the double-arm robot, so that the valve is tightened. According to the invention, rapid and accurate valve identification is realized, and the reliability and flexibility of the system in practical application are greatly enhanced.

Description

Technical Field

[0001] The invention belongs to the technical field of humanoid robot control, and in particular relates to a valve tightening method and system of a humanoid dual-arm robot. Background Art

[0002] Screw-type valve devices are crucial in many industrial scenarios, such as petrochemical plants, nuclear power plants, and hydropower stations. The valves in these places usually involve high pressure and high temperature environments, requiring high reliability and high precision operations. At present, most of the devices for switching valves use dedicated mechanical structures. Although they can complete specific valve twisting actions, the limitations of these devices are particularly prominent in emergency situations. For example, when an emergency occurs and the valve needs to be closed quickly, the operator must first replace the end effector, which is not only time-consuming but also may increase safety risks. In addition, the existing electric grippers and pneumatic grippers are insufficient in clamping force and cannot effectively cope with the operation requirements of large torque valves. In high torque environments, these grippers often cannot provide sufficient clamping force, resulting in unstable operation or failure to complete the task. This makes it imperative to develop a more powerful clamping system to ensure that the valve can be operated quickly and safely at critical moments.

[0003] In terms of valve positioning, YOLO (You Only Look Once) is currently commonly used for recognition, combined with point cloud library (PCL) for target feature extraction. Although this is an advanced technical means, the complexity of the process may lead to insufficient real-time performance and low recognition efficiency. In rapidly changing working conditions, this cumbersome process affects the system response speed, which in turn affects the safety and effectiveness of the entire operation. Therefore, simplifying the recognition process and improving the recognition accuracy will be an important direction for improving the application of this technology. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art and provide a humanoid dual-arm robot valve tightening method and system to achieve fast and accurate valve identification, greatly enhancing the reliability and flexibility of the system in practical applications.

[0005] Technical solution: The humanoid dual-arm robot valve tightening method of the present invention comprises the following steps: S101, obtaining a valve image dataset based on a binocular vision camera; S102, annotating the valve image data set to generate an annotated data set, wherein the annotated data set includes a boundary box of the valve and a center point position of the valve; S103, performing enhancement processing on the annotated data set to expand the data set; S104, identifying and locating the valve based on a preset binocular vision algorithm and a YOLO model, and identifying the position, border, and center point of the valve; S201, identifying and locating the valve through steps S101 to S104, and obtaining the frame position and center point position of the valve; S202, moving the dual-arm robot to both sides of the valve, keeping the positions and postures of the ends of the dual arms consistent in other directions except the clamping direction; S203, performing clamping control based on an adaptive variable admittance controller; S204. The valve is rotated by rotating the end joint of the dual-arm robot to tighten the valve.

[0006] To further improve the above technical solution, the dual-arm robot includes a left arm and a right arm, and actuators are provided at the ends of the left arm and the right arm. The actuators include a flange, a rubber-coated wheel, a six-dimensional force sensor, and a working part. The six-dimensional force sensor, the rubber-coated wheel, and the flange are connected in sequence after the centers are aligned. After the connection, the six-dimensional force sensor is accommodated in the rubber-coated wheel, and the flange is fixed outside the rubber-coated wheel. The six-dimensional force sensor is connected to the end of the left arm or the right arm, and the flange is connected to the working part.

[0007] Furthermore, the working part is a two-finger gripper.

[0008] Furthermore, the steps of using the binocular vision algorithm and the YOLO model to identify and locate the valve are as follows: (1) Use the trained YOLOv8 model to detect different types of valves and obtain the prediction box of the valve target: ,in, and Respectively represent the coordinates of the upper left corner and lower right corner of the prediction box; (2) Use image alignment to extract point cloud information in the depth image that is aligned with the prediction box: , where each point cloud data Expressed as three-dimensional coordinates with the camera coordinate system as the origin ; (3) Centralize the point cloud data and calculate the corresponding point cloud centroid within the prediction box and centralized point cloud data , the calculation formulas are as follows: , ; (4) Calculate the covariance matrix of the centralized point cloud data , the calculation formula is as follows: ,in, It is the transpose of the centralized point cloud data; (5) Solve the eigenvalues ​​of the covariance matrix and the eigenvector , the calculation formula is as follows: ; (6) Point cloud normal vector Calculate, assuming that the eigenvector corresponding to the largest eigenvalue is , then the feature vector is the normal vector of the point cloud data ; (7) Point cloud normal vector and the prediction box center The conversion formula from the camera coordinate system to the dual-arm robot world coordinate system is: , ,in, is the world coordinate system, is the global camera coordinate system, express Coordinate system to The rotation transformation matrix of the coordinate system, express Coordinate system to The offset vector of the coordinate system; (8) Determine the valve pre-grasping position and set the rough positioning result as the target position of the robot arm , which is located at the center of the prediction box The normal vector of the point cloud along the area The calculation formula is as follows: , (9) After obtaining the target position, the inverse kinematics of the robot arm is used to calculate the joint angle corresponding to the valve grasping point: .

[0009] Furthermore, the adaptive variable admittance controller performs clamping control including: reading force data obtained by the six-dimensional force sensor , and perform gravity compensation and coordinate transformation; calculate the expected force With actual force The error between , and bring it into the adaptive variable admittance controller to control the double arms to maintain a constant force in the clamping direction; the adaptive variable admittance controller dynamically adjusts the control parameters through an adaptive control algorithm, including adjusting the damping coefficient matrix b, the stiffness coefficient matrix k and the mass coefficient matrix m.

[0010] Furthermore, the control formula of the control parameter of the adaptive variable admittance controller is: Among them, m is the mass coefficient matrix, b is the damping coefficient matrix, k is the stiffness coefficient matrix, is the position error, which represents the difference between the actual position and the expected position at time t. is the speed error, which represents the difference between the actual speed and the expected speed at time t. is the acceleration error, which represents the difference between the actual acceleration and the expected acceleration at time t. is the position error at time t+1, is the speed error at time t+1, is the position acceleration error at time t+1, T is the time step; ∆b ​​is the adaptive damping adjustment parameter, and its update rate is: , where , is the update rate, represents the adaptive parameter at the current time t, represents the adaptive parameters at time t+1; The expected trajectory The output of the adaptive admittance controller Add them together to get the actual movement trajectory of the left and right arms .

[0011] The system for implementing the above-mentioned humanoid dual-arm robot valve tightening method comprises: A binocular vision camera is used to obtain valve image datasets; Binocular vision algorithm and YOLO model are used to obtain image data sets and identify and locate valves; A dual-arm robot, comprising a left arm and a right arm, wherein the ends of the left arm and the right arm are both provided with actuators for clamping the valve and performing tightening operations under adaptive variable admittance control, wherein the actuator has a six-dimensional force sensor for real-time detection of the valve clamping force and transmission of force data to the adaptive variable admittance controller; The adaptive variable admittance controller is used to adjust the clamping force of the double arms according to the feedback of the force sensor, ensure the constant clamping force of the valve and control the rotation of the double-arm robot to achieve the tightening of the valve.

[0012] Beneficial effects: Compared with the prior art, the advantages of the present invention are: the present invention combines the YOLO target detection algorithm and the binocular vision algorithm in the valve recognition process to realize automatic recognition and positioning of the valve; the binocular vision algorithm improves the valve recognition accuracy through stereo vision and depth calculation technology, and can identify and locate valves in complex industrial environments; binocular vision can realize RGB recognition of images, and deep vision obtains depth information. Based on the fusion of the two information, the spatial posture information and edge detection information of the valve can be identified, thereby improving the accuracy and rate of target recognition. . Based on the adaptive variable admittance controller of the present invention, the clamping force and motion trajectory of the double arms are adjusted in real time to meet the clamping and tightening requirements of different valve types.

[0013] The valve twisting device of the present invention has a simple structure and relatively low cost. Compared with the traditional use of a special valve twisting device, it is obviously a waste of time to replace the end effector in some emergency places. This structure can be equipped with a five-finger / two-finger tool at the end to quickly perform other operations; there is no angle limit, the action is simple and efficient, and there is no need to change hands after the angle limit of steering the wheel, which is more advantageous than manual valve twisting; visual control is simple and easy to implement, and clamping can be achieved without other complex algorithms; high friction coefficient rubber-coated wheels can protect the six-dimensional force sensor from being damaged in high-risk environments. This solution uses high friction coefficient rubber-coated wheels to provide a larger torque. If two fingers are used to clamp and rotate, there is often a risk of slipping, and the torque that can be provided is relatively small. It is difficult to twist a relatively large valve. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram of an end effector connection device according to an embodiment of the present invention; Figure 2 It is a schematic diagram of the terminal structure of the device for implementing the present invention; Figure 3 This is a flow chart of valve recognition based on YOLO according to an embodiment of the present invention; Figure 4 This is a rendering of the valve recognition effect using YOLO in an embodiment of the present invention; Figure 5 This is a flow chart of double-arm valve turning based on YOLO recognition and positioning in an embodiment of the present invention; Figure 6 This is a diagram showing the effect of turning a valve according to an embodiment of the present invention; Figure 7 This is the adaptive variable admittance control according to an embodiment of the present invention.

[0015] In the figure: 1: end effector flange; 2: high friction coefficient rubber-coated wheel; 3: six-dimensional force sensor; 4: two-finger gripper. DETAILED DESCRIPTION

[0016] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0017] Example 1: Figure 1 , Figure 2The end effector connection device shown includes an end effector flange 1, a high friction coefficient rubber-coated wheel 2, a six-dimensional force sensor 3, and a two-finger clamp 4. The six-dimensional force sensor 3 is connected to the end of the robot arm, and the high friction coefficient rubber-coated wheel 2 wraps the six-dimensional force sensor 3. The end effector flange 1, the high friction coefficient rubber-coated wheel 2, and the six-dimensional force sensor 3 align their holes and can be connected by hexagon socket bolts. The two-finger clamp 4 is connected to the end effector flange 1, and the two-finger clamp 4 at the end can be changed according to actual work requirements.

[0018] like Figure 3 As shown in the figure, the valve recognition flow chart based on YOLO, the specific process is as follows: S101, obtaining a valve image dataset based on a binocular vision camera, and specifically selecting a source of the dataset, such as valves in industrial environments such as hydropower stations and petrochemical plants; S102, labeling the collected valve data set to generate a labeled data set, and manually labeling using the LabelImg tool to mark the valve boundary box and the position of the center point; S103, enhancing the labeled data, formulating a specific enhancement strategy, and processing the image by various methods such as rotation, flipping, scaling, and brightness adjustment to expand the data set; S104, identifying and locating the valve based on the preset binocular vision algorithm and the YOLO model, and identifying and locating the valve in combination with the preset binocular vision algorithm, basic parameters and the target YOLO model.

[0019] By combining the target YOLO model with the binocular vision algorithm, the automatic recognition and positioning of the valve can be achieved. The specific steps are as follows: 1. Use the trained YOLOv8 model to detect different types of valves and obtain the prediction box of the valve target: in, and Respectively represent the coordinates of the upper left corner and lower right corner of the prediction box; 2. Get the RGB image and depth image taken by the binocular camera, and use image alignment to extract the point cloud data aligned with the prediction box in the depth image: Among them, each point cloud data Expressed as three-dimensional coordinates with the camera coordinate system as the origin ; 3. Centralize the point cloud data and calculate the corresponding point cloud centroid within the prediction box and centralized point cloud data , the calculation formulas are as follows: 4. Calculate the covariance matrix of the centralized point cloud data , the calculation formula is as follows in, It is the transpose of the centralized point cloud data; 5. Solve the eigenvalues ​​of the covariance matrix and the eigenvector , the calculation formula is as follows: 6. Point cloud normal vector Calculate, assuming that the eigenvector corresponding to the largest eigenvalue is , then the feature vector is the normal vector of the point cloud data ; 7. Point cloud normal vector and the prediction box center The conversion formula from the camera coordinate system to the dual-arm robot world coordinate system is: in, is the world coordinate system, is the global camera coordinate system, express Coordinate system to The rotation transformation matrix of the coordinate system, express Coordinate system to The offset vector of the coordinate system; 8. Determine the valve pre-grasping position and set the rough positioning result as the target position of the robot arm , the location has the center of the prediction box The normal vector of the point cloud along the area The calculation formula is as follows: 9. After obtaining the target position, use the inverse kinematics of the robot arm to calculate the joint angle corresponding to the valve grabbing point: like Figure 4 As shown in FIG. 1 , this is the effect diagram of YOLO identifying the valve. Based on YOLO, the valve is identified and located to obtain its center position and the position of the valve frame.

[0020] like Figure 5As shown in the figure, the double-arm valve twisting flow chart based on YOLO recognition and positioning, the specific clamping process is as follows: S201, valve identification and positioning, according to the above valve identification process, the valve can be identified and positioned, so as to read the positions of the upper left corner and the upper right corner of the valve frame, and read the position of the valve center; S202, move the left and right arms to both sides of the valve, Figure 6 Schematic diagram of valve turning according to an embodiment of the present invention, O is the world coordinate system, is the point expressed in the world coordinate system, which is the rubber wheel with high friction coefficient on the left arm. is a point expressed in the world coordinate system, which is the rubber-coated wheel with high friction coefficient on the right arm; O is a point expressed in the world coordinate system, which is the center position of the valve, wherein the center position of the valve can be obtained by identifying and locating the valve in the step S10; the left and right arms are moved to the two sides of the valve respectively, and the positions of the ends of the left and right arms in the xyz direction are kept different only in the clamping direction, while the positions and postures in other directions are kept consistent, and the rubber-coated wheel with high friction coefficient and the center of the valve are kept consistent in the xyz direction except for the clamping direction; S203, clamping based on adaptive variable admittance control, specifically, under admittance control, use a high friction rubber-coated wheel to approach the middle point of the valve, and quickly perform force tracking when the friction wheel contacts the periphery of the valve. This force also greatly affects the twisting of the valve, so a relatively large clamping force is required; S204, the valve is turned. The specific power source for turning the valve is the rotation of the end joints of the left and right mechanical arms. The end joints of the left and right mechanical arms rotate in one direction, thereby driving the valve to rotate.

[0021] The present invention comprises a double-arm robot composed of a left arm and a right arm, wherein the left arm is coded as L and the right arm is coded as R, wherein i represents the left and right arms, that is, The specific control method of S203 clamping is Figure 7 The adaptive variable admittance control principle diagram shown in the figure, the specific control steps are as follows: S301, The force data is read by the six-dimensional force sensor. The force data needs to be obtained through gravity compensation and coordinate conversion (converted to the world coordinate system through the coordinate conversion algorithm); S302, by using the force data and expectation Subtraction is introduced into the adaptive variable admittance controller of the present invention, in which the force in the clamping direction is required to be constant force, and double-arm clamping is required to maintain constant force clamping, such as Figure 7 The adaptive variable admittance controller in is as follows: in, is the quality coefficient matrix, is the damping coefficient matrix, is the stiffness coefficient matrix, where , and is a constant matrix, Force tracking is done based on adaptive control rate adjustments, with the following update rates: In the above formula, , used to prevent the denominator from being 0, is the update rate.

[0022] By putting the expected trajectory The adaptive admittance controller calculates Add together to calculate the actual movement trajectory of the left and right arms , and input this value into the dual-arm robot system.

[0023] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A valve tightening method using a humanoid dual-arm robot, characterized in that: The method comprises the following steps: S101, obtaining a valve image dataset based on a binocular vision camera; S102, annotating the valve image data set to generate an annotated data set, wherein the annotated data set includes a boundary box of the valve and a center point position of the valve; S103, performing enhancement processing on the annotated data set to expand the data set; S104, identifying and locating the valve based on a preset binocular vision algorithm and a YOLO model, and identifying the position, border, and center point of the valve; S201, identifying and locating the valve through steps S101 to S104, and obtaining the frame position and center point position of the valve; S202, moving the dual-arm robot to both sides of the valve, keeping the positions and postures of the ends of the dual arms consistent in other directions except the clamping direction; S203, performing clamping control based on an adaptive variable admittance controller; S204. The valve is rotated by rotating the end joint of the dual-arm robot to tighten the valve.

2. The humanoid dual-arm robot valve tightening method according to claim 1, characterized in that: The dual-arm robot includes a left arm and a right arm, and actuators are provided at the ends of the left arm and the right arm. The actuators include a flange, a rubber-coated wheel, a six-dimensional force sensor, and an operating part. The six-dimensional force sensor, the rubber-coated wheel, and the flange are connected in sequence after their centers are aligned. After the connection, the six-dimensional force sensor is accommodated in the rubber-coated wheel, and the flange is fixed outside the rubber-coated wheel. The six-dimensional force sensor is connected to the end of the left arm or the right arm, and the flange is connected to the operating part.

3. The humanoid dual-arm robot valve tightening method according to claim 2, characterized in that: The working part is a two-finger clamp.

4. The humanoid dual-arm robot valve tightening method according to claim 1, characterized in that: The steps of using the binocular vision algorithm and the YOLO model to identify and locate the valve are as follows: (1) Use the trained YOLOv8 model to detect different types of valves and obtain the prediction box of the valve target: ,in, and Respectively represent the coordinates of the upper left corner and lower right corner of the prediction box; (2) Use image alignment to extract point cloud information in the depth image that is aligned with the prediction box: , where each point cloud data Expressed as three-dimensional coordinates with the camera coordinate system as the origin ; (3) Centralize the point cloud data and calculate the corresponding point cloud centroid within the prediction box and centralized point cloud data , the calculation formulas are as follows: , ; (4) Calculate the covariance matrix of the centralized point cloud data , the calculation formula is as follows: ,in, It is the transpose of the centralized point cloud data; (5) Solve the eigenvalues ​​of the covariance matrix and the eigenvector , the calculation formula is as follows: ; (6) Point cloud normal vector Calculate, assuming that the eigenvector corresponding to the largest eigenvalue is , then the feature vector is the normal vector of the point cloud data ; (7) Point cloud normal vector and the prediction box center The conversion formula from the camera coordinate system to the dual-arm robot world coordinate system is: , ,in, is the world coordinate system, is the global camera coordinate system, express Coordinate system to The rotation transformation matrix of the coordinate system, express Coordinate system to The offset vector of the coordinate system; (8) Determine the valve pre-grasping position and set the rough positioning result as the target position of the robot arm , which is located at the center of the prediction box The normal vector of the point cloud along this area The calculation formula is as follows: , (9) After obtaining the target position, the inverse kinematics of the robot arm is used to calculate the joint angle corresponding to the valve grasping point: .

5. The valve tightening method of the humanoid dual-arm robot according to claim 2, characterized in that: The adaptive variable admittance controller performs clamping control including: Read the force data obtained by the six-axis force sensor ; Calculating Expected Power With actual force The error between , the error It is brought into the adaptive variable admittance controller to control the double arms to maintain a constant force in the clamping direction; The adaptive variable admittance controller dynamically adjusts control parameters through an adaptive control algorithm, and the control parameters include adjusting the damping coefficient matrix b, the stiffness coefficient matrix k and the mass coefficient matrix m.

6. The valve tightening method of the humanoid dual-arm robot according to claim 5, characterized in that: The control formula of the control parameter of the adaptive variable admittance controller is: Among them, m is the mass coefficient matrix, b is the damping coefficient matrix, k is the stiffness coefficient matrix, is the position error, which represents the difference between the actual position and the expected position at time t. is the speed error, which represents the difference between the actual speed and the expected speed at time t. is the acceleration error, which represents the difference between the actual acceleration and the expected acceleration at time t. is the position error at time t+1, is the speed error at time t+1, is the position acceleration error at time t+1, T is the time step; ∆b ​​is the adaptive damping adjustment parameter, and its update rate is: , where , is the update rate, represents the adaptive parameter at the current time t, represents the adaptive parameters at time t+1; The expected trajectory The output of the adaptive admittance controller Add them together to get the actual movement trajectory of the left and right arms .

7. A system for implementing the humanoid dual-arm robot valve tightening method according to claim 1, characterized in that: include: A binocular vision camera is used to obtain valve image datasets; Binocular vision algorithm and YOLO model are used to obtain image data sets and identify and locate valves; A dual-arm robot, comprising a left arm and a right arm, wherein the ends of the left arm and the right arm are both provided with actuators for clamping the valve and performing tightening operations under adaptive variable admittance control, wherein the actuator has a six-dimensional force sensor for real-time detection of the valve clamping force and transmission of force data to the adaptive variable admittance controller; The adaptive variable admittance controller is used to adjust the clamping force of the double arms according to the feedback of the force sensor, ensure the constant clamping force of the valve and control the rotation of the double-arm robot to achieve the tightening of the valve.

Images