A grasping control method for UR5 robot in V-REP simulation environment

Abstract

This invention discloses a grasping control method for a UR5 robot in a V-REP simulation environment, comprising the following steps: S1, collecting various data from the end-effector camera module of the robotic arm in the simulation environment; S2, compressing the images and transmitting them to an Android device for display, creating joystick and slider controls; S3, collecting operation commands on the Android device using the joystick and slider controls, encoding the collected operation command data and transmitting it to a PC, where the PC receives the control commands and decodes them into six-degree-of-freedom motion information; S4, establishing a DH model of the robotic arm, and inversely solving the velocities of each joint based on the motion commands; S5, constraining the joint velocities obtained in step S4 by configuring the range of angles and angular accelerations, thereby obtaining a more realistic simulation effect; S6, encoding the position information into a format required by the simulation environment and sending it to the simulation environment for control.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and robotic arm control, specifically to a grasping control method for a UR5 robot in a V-REP simulation environment. Background Technology

[0002] The widespread application of simulation technology in robotics has led to lower cost and lighter weight in robot algorithm development. However, in the development of redundant robotic arms, to achieve rapid and low-cost development goals, it is often necessary to simulate and control the robotic arm in a simulation environment first. The multi-joint and complex structure of the robotic arm makes remote control in the simulation environment very unintuitive. A flexible robotic arm simulation control platform (A Flexible Robotic Arm Control Simulation Platform and Its Simulation Method, Yang Yang, Nanjing University of Posts and Telecommunications) was constructed. This platform includes a mathematical model of the flexible robotic arm and a simulation interface. This solution has a UI interface, but requires parameter configuration for robotic arm control, a cumbersome process. Chen Yongzhen et al. from Shenyang University of Chemical Technology proposed a three-dimensional simulation teaching system for robotic arms with teaching functions, but it still requires configuring the angles of each joint. For multi-joint robotic arms, it is difficult to obtain the joint angles of the target posture without calculation. A robotic arm simulation control method, device, and storage medium (Jiang Shiyun) simulates the movement of a robotic arm by acquiring waypoints of a real robotic arm at a 1:1 scale. However, this method suffers from drawbacks such as high cost and system complexity. All the aforementioned patents rely on algorithms to autonomously control the robotic arm's movement, requiring manual parameter configuration, which is complex. Furthermore, for systems requiring physical assistance, the cost is high. Additionally, for robotic arm structures with the eye on the hand, first-person perspective operation cannot be tested. Summary of the Invention

[0003] To address these issues and the high cost of experimenting with robotic arms in real-world environments, this invention provides a UR5 robot grasping control method within a V-REP simulation environment. The method replicates the UR5 robotic arm and RealSense RGB-D camera at a 1:1 scale within the V-REP simulation environment, performing kinematic modeling and Jacobian calculations on the UR5 robotic arm. An Android device is used as the controller to communicate with a PC, ultimately enabling grasping and other operations on the robotic arm within the simulation environment. This invention uses an Android phone as the operating terminal, interacting via network to perceive the simulation environment and implement control operations. By describing the rotational and translational motions in a Cartesian coordinate system configured at the robotic arm's end effector, the method allows for intuitive manipulation of the robotic arm within the simulation environment. This provides a highly convenient human-computer interaction method for debugging robotic arms in simulation environments.

[0004] The present invention is achieved by at least one of the following technical solutions.

[0005] A grasping control method for a UR5 robot in a V-REP simulation environment includes the following steps:

[0006] S1. Collect various data from the end-effector camera module of the robotic arm in the simulation environment;

[0007] S2. Compress the image and transmit it to the Android device for display; create joystick and slider controls on the Android device;

[0008] S3. On the Android side, data generated by the joystick and slider is collected through the joystick and slider control, encoded and transmitted to the PC. The PC receives the control commands and decodes them into six degrees of freedom motion information.

[0009] S4. Establish the DH model of the robotic arm and inversely solve the velocity of each joint according to the motion command.

[0010] S5. By configuring the range of angle and angular acceleration, the joint velocity obtained in step S4 is constrained, thereby obtaining a more realistic simulation effect.

[0011] S6. Combine the velocity of each joint obtained in step S5 with the information of the joint angle and time interval at the previous moment to calculate a composite list of angle information for each joint. By calling the interface function of vrep, the list is traversed and sent to the simulation environment for control.

[0012] Further, step S1 includes the following steps:

[0013] S101. Create a UR5 robotic arm model in the V-REP simulation environment, and configure each joint of UR5 in V-REP according to the kinematic model of UR5.

[0014] S102. Create an STL appearance model file for the RealSense-D435i camera. Attach the vision sensor and depth sensor from the V-REP simulation environment to the STL appearance model file and configure them according to the parameters of the RealSense-D435i camera to achieve simulation of the RGB-D camera RealSense-D435i.

[0015] S103. In V-REP, mount the realsense-D435i to the UR5 robotic arm to realize the Eye-in-hand model;

[0016] S104. Obtain various types of data from the simulation environment through the V-REP interface, including depth map data, color map data, and position and velocity data of each joint in UR5.

[0017] Furthermore, step S2, which involves compressing the image and transmitting it to the Android device for display, includes the following steps:

[0018] S201. The PC performs JPEG compression on the color image acquired in step S1;

[0019] S202. Transmit the compressed data to the Android device via TCP communication.

[0020] After receiving the data, the S203 and Android devices decode and display it.

[0021] Furthermore, creating a joystick control involves the following steps:

[0022] S301. Draw a circular feasible area on the Android device screen to define the range of motion of the joystick;

[0023] S302. Draw a small disk in the center of the circular feasible area as a joystick position indicator;

[0024] S303. In the Android client, a Java class for a joystick control is created, inheriting from the View class in the Android SDK. The onTouchEvent function is used to listen for touch coordinates applied to the joystick control. When a touch operation is triggered, the onTouchEvent function receives the MotionEvent parameter. The MotionEvent parameter obtains the touch operation applied to the joystick control, including pressing, moving, lifting, and moving out of the area. At the same time, the getX() and getY() functions are called through the MotionEvent parameter to obtain the real-time position of the touch operation, which is used to update the position of the small disk in real time, so that the position of the small disk follows the movement of the finger. Simultaneously, the position of the center of the small disk on the screen is calculated in real time, and the horizontal and vertical pixel differences relative to the position of the center of the feasible area on the screen are calculated. These differences are then divided by the radius of the feasible area to obtain the horizontal and vertical proportions as the output control signal. It can be seen that the output range of the control signal is [-1, 1], and the output is a normalized output. After receiving the signal, the PC client scales according to its own speed limit.

[0025] Furthermore, creating a slider control involves the following steps:

[0026] S401. Draw a one-dimensional feasible area in the horizontal direction on the Android device screen to define the range of motion of the slider. Here, one-dimensional means that the slider can only move in one direction within the feasible area.

[0027] S402, Draw points in a one-dimensional feasible region round The slider serves as an indicator of the slider's position.

[0028] S403. Create a joystick control class that inherits from the View class in the Android SDK. Replace the system's own onTouchEvent function by overriding the onTouchEvent function to listen for touch operation response logic. When a touch operation is triggered, the Android system calls the onTouchEvent function and passes in a MotionEvent object. The MotionEvent parameter obtains the touch operation performed on the control, including pressing, moving, lifting, and moving out of the area. At the same time, the getX() and getY() functions are called through the MotionEvent object to obtain the touch coordinate values ​​performed on the control, so that the position of the small disc follows the finger movement in the horizontal direction. When the position of the circular slider exceeds the boundary of the feasible area, it stops at the edge of the feasible area and outputs the position value of the small disc in the feasible area, which is in the range of [-1,1]. When the touch operation is released, the circular slider returns to the center position and the output is 0.

[0029] Further, step S4 involves establishing a DH model of the robotic arm and inversely solving the position operations of each joint based on motion commands, including the following steps:

[0030] S401 and Android send data to the PC in byte form. The PC receives the byte-form control instructions and decodes them into six-degree-of-freedom motion information, namely [vx,vy,vz,rx,ry,rz]. It divides this into two groups: vx,vy, and vz are the translational velocities of the robotic arm's end effector along the x, y, and z axes in the robotic arm's end effector coordinate system; rx, ry, and rz are the rotational velocities of the robotic arm's end effector around the x, y, and z axes in the robotic arm's end effector coordinate system.

[0031] S402. Construct the DH model of the robotic arm, perform inverse kinematics solution, obtain the relationship between the velocity of each joint and the motion velocity of the end joint in the base coordinate system of the robotic arm, and obtain the Jacobian matrix.

[0032] S403. Convert the velocity in step S401 into the motion velocity in the robot arm's base coordinate system, and obtain the angular velocity of each joint by multiplying it with the Jacobian matrix obtained in step S402.

[0033] Furthermore, the velocity in the world coordinate system is transformed into the motion velocity in the robot arm's base coordinate system, specifically by extracting the transformation matrix of the camera coordinate system in the base coordinate system. Given the rotation matrix R, construct the transformation matrix T as follows. vc2b :

[0034]

[0035] Furthermore, the angular velocities of each joint are:

[0036]

[0037] in Let V represent the rotational speed of each joint of the robotic arm, V represent the velocity vector at the end effector of the robotic arm, and J represent the velocity vector at the end effector of the robotic arm. vc2b This represents the transformation matrix that converts velocity into motion velocity in the robot arm's base coordinate system.

[0038] Furthermore, step S5 constrains the joint velocity obtained in step S8 by configuring the range of angle and angular acceleration, including the following steps:

[0039] S501, Configure the angle range of the joint [q] - ,q + ], where q - q represents the lower limit of the joint angle. + Indicates the upper limit of the joint angle;

[0040] S502, Configure the range of joint movement speed in Indicates the lower limit of joint velocity. Indicates the upper limit of joint velocity;

[0041] S503, Configure the angular acceleration range of joint motion. in This indicates the lower limit of joint angular acceleration. Indicates the upper limit of joint angular acceleration;

[0042] S504. Obtain the current angle values ​​of each joint of the robotic arm;

[0043] S505. Based on the angle range, motion speed range, angular acceleration range and current angle data configured in steps S501 to S504 above, calculate the upper and lower limits of the joint velocity at the current position, and limit the angular velocity output in step S8.

[0044] Further, in step S504, the current angle value of the DH model of the robotic arm is obtained through the VREP interface; wherein, the current angle of the DH model refers to the rotation angle of each coordinate system fixed at the joint in the coordinate system of the DH model.

[0045] Compared with existing technologies, the beneficial effects of the present invention are as follows:

[0046] 1. Simple and intuitive operation. Direct control of the robotic arm's end effector movement, rather than configuring each joint. This invention addresses the challenges of numerous control variables and high difficulty in operating redundant robotic arms in simulation environments. It establishes a method to control redundant robotic arms in simulation environments by controlling the six degrees of freedom of translation and rotation at the end effector. This method offers advantages such as intuitive control and effectively reducing the control variables of redundant robotic arms with more than six degrees of freedom.

[0047] 2. Multi-view operation. Control can be achieved by observing the VREP simulation interface or by observing first-view images received by the mobile phone. This invention aims to use the most common Android mobile phone as the operation entry point, employing image transmission, robotic arm control, and other technologies. The robotic arm's movement can be controlled via a joystick and slider. This means the movement of the robotic arm can be observed and controlled from the VREP interface, or first-view control can be achieved through images transmitted from the robotic arm's end-effector camera received on the Android device.

[0048] 3. Low cost. No additional controller needs to be built; simply install an app on your mobile phone to use it. Attached Figure Description

[0049] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0050] Figure 1 This is a system architecture flowchart of a UR5 robot grasping control method in a V-REP simulation environment according to an embodiment of the present invention;

[0051] Figure 2 This is a structural diagram of the UR5 robotic arm in the VREP simulation environment according to an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of the coordinate system configured for the DH modeling of the robotic arm in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the Android joystick control and slider control according to an embodiment of the present invention. Detailed Implementation

[0054] This section will describe in detail specific embodiments of the present invention. Preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the drawings is to supplement the textual description with graphics, so that people can intuitively and vividly understand each technical feature and overall technical solution of the present invention, but they should not be construed as limiting the scope of protection of the present invention.

[0055] Example 1

[0056] A grasping control method for a UR5 robot in a V-REP simulation environment includes the following steps:

[0057] S1. Collect various data from the end-effector camera module of the robotic arm in the simulation environment;

[0058] S2. Compress the image and transmit it to the Android device for display; create a joystick control and a slider control on the Android device; creating the joystick control includes the following steps:

[0059] S301. Draw a circular feasible area on the screen of an Android device (phone or Android tablet) to define the range of motion of the joystick;

[0060] S302. Draw a small disk in the center of the circular feasible area as a joystick position indicator;

[0061] S303. In the Android client, create a Java class for a joystick control that inherits from the View class in the Android SDK. Override the onTouchEvent function of the View class in the Android SDK to listen for touch coordinates applied to the joystick control. When a touch operation is triggered, the Android system calls the onTouchEvent function and automatically passes the MotionEvent parameter. The MotionEvent parameter is automatically constructed and passed by the system, allowing programmers to obtain the touch operations applied to the joystick control, including pressing, moving, releasing, and moving out of the area. Additionally, the getX() and getY() sub-functions of the MotionEvent parameter can be called.

[0062] (The `onTouchEvent` function is an SDK function in Android programming. `getX()` and `getY()` are functions automatically passed to this function by the Android system. They are system-provided interfaces used to obtain the operation performed on this space.) The system obtains the real-time position of the touch screen operation and uses it to update the position of the small disk in real time, allowing the disk to follow the finger's movement. Simultaneously, it calculates the horizontal and vertical pixel differences between the center of the small disk and the center of the feasible area on the screen, and divides these differences by the radius of the feasible area to obtain the horizontal and vertical percentages as the output control signal. The output range of the control signal is [-1, 1], and the output is normalized. After receiving the signal, the PC scales it according to its own speed limit. For example, if the received control signal is 0.5 and the maximum speed of the robotic arm is 2 m / s, then the PC multiplies the received 0.5 control signal by the maximum speed of 2 m / s, resulting in a value of 0.5 * 2 = 1 m / s sent to the simulation environment.

[0063] Creating a slider control involves the following steps:

[0064] S401. Draw a one-dimensional feasible area in the horizontal direction on the screen of an Android device (phone or Android tablet) to define the range of motion of the slider. Here, one dimension means that the slider can only move in one direction within the feasible area, such as only horizontally or only vertically.

[0065] S402, Draw points in a one-dimensional feasible region round The slider serves as an indicator of the slider's position.

[0066] S403. In the Android application being developed, create a joystick control class that inherits from the View class in the Android SDK. As above, listen for touch operations by overriding the onTouchEvent function. When a touch operation is triggered, the onTouchEvent function will pass in the MotionEvent parameter. The MotionEvent parameter obtains the touch operation performed on the control, including pressing, moving, lifting, and moving out of the area. At the same time, the getX() and getY() functions are called through the MotionEvent parameter to obtain the touch coordinate values ​​performed on the control, so that the position of the small disc follows the finger movement in the horizontal direction. When the position of the circular slider exceeds the boundary of the feasible area, it stops at the edge of the feasible area and outputs the position value of the small disc in the feasible area, which is in the range of [-1, 1]. For example, when the slider can only move left and right, the leftmost position is -1 and the rightmost position is 1. When the touch operation is released, the circular slider returns to the center position and the output is 0.

[0067] S3. On the Android side, data generated by the joystick and slider is collected through the joystick and slider control, encoded and transmitted to the PC. The PC receives the control commands and decodes them into six degrees of freedom motion information.

[0068] S4. Establish the DH model of the robotic arm, and inversely solve the velocity of each joint according to the motion commands, including the following steps:

[0069] S401 and Android send data to the PC in byte form. The PC receives the byte-form control instructions and decodes them into six-degree-of-freedom motion information, namely [vx,vy,vz,rx,ry,rz]. It divides this into two groups: vx,vy, and vz are the translational velocities of the robotic arm's end effector along the x, y, and z axes in the robotic arm's end effector coordinate system; rx, ry, and rz are the rotational velocities of the robotic arm's end effector around the x, y, and z axes in the robotic arm's end effector coordinate system.

[0070] S402. Construct the DH model of the robotic arm, perform inverse kinematics solution, obtain the relationship between the velocity of each joint and the motion velocity of the end joint in the base coordinate system of the robotic arm, and obtain the Jacobian matrix.

[0071] S403. Convert the velocity in step S401 into the motion velocity in the robot arm's base coordinate system, and obtain the angular velocity of each joint by multiplying it with the Jacobian matrix obtained in step S402.

[0072] S5. By configuring the range of angle and angular acceleration, the joint velocity obtained in step S4 is constrained, thereby obtaining a more realistic simulation effect.

[0073] S6. Combine the velocity of each joint obtained in step S5 with the information of the joint angle and time interval at the previous moment to calculate a composite list of angle information for each joint. By calling the interface function of vrep, the list is traversed and sent to the simulation environment for control.

[0074] Example 2

[0075] like Figure 1 , Figure 2 As shown, the present invention provides a UR5 robot grasping control method in a V-REP simulation environment, comprising the following steps:

[0076] S1. In the simulation environment, acquire RGB images from the end-effector camera module of the robotic arm in the simulation environment;

[0077] S2. Compress the image and transmit it to the Android device for display;

[0078] S3. Create a joystick control on the Android device.

[0079] S4. Create a slider control on the Android side.

[0080] S5. On the Android device, operation commands are collected through joystick and slider controls;

[0081] S6. Encode the collected data and transmit it to the PC.

[0082] S7: The PC receives control commands and decodes them into six-degree-of-freedom motion information.

[0083] S8. By establishing the DH model of the robotic arm, the velocity of each joint is solved inversely according to the motion command.

[0084] S9. By configuring the range of angle and angular acceleration, the joint velocity obtained in S8 is constrained, thereby obtaining a more realistic simulation effect.

[0085] S10. Encode the location information into a format required by the simulation environment and send it to the simulation environment for control.

[0086] In the V-REP simulation environment, the simulation environment setup operation in step S1 includes the following steps:

[0087] S101. Create the UR5 robotic arm model in V-REP. Configure the joints of UR5 in V-REP based on the kinematic model of UR5.

[0088] S102. Create an STL appearance model file for the RealSense-D435i camera, attach the vision sensor and depth sensor from the V-REP simulation environment to it, and configure them according to the parameters of the RealSense-D435i camera. This achieves the simulation of the RGB-D camera RealSense-D435i.

[0089] S103. In V-REP, attach the realsense-D435i to the UR5 robotic arm to realize the Eye-in-hand model.

[0090] S104. Read the image data stream from the simulation environment through the V-REP data interface and resize it to the form (w,h,c), where w is the row of the image, h is the column of the image, and c is the number of channels of the image.

[0091] In step S2, the image compression and transmission display operation includes the following steps:

[0092] S201. After obtaining the camera image from the V-REP simulation environment on the PC, compress it using the jpeg-2000 algorithm.

[0093] S202. Send the compressed data to the Android device via TCP protocol.

[0094] After receiving the data, the S203 and Android devices decompress it using the jpeg-2000 algorithm and construct a bitmap image for display.

[0095] In step S8, the transformation operation from motion control command to joint movement includes the following steps:

[0096] S801, control commands transmitted from Android, including the translation speed v along the coordinate axes in the camera coordinate system. x ,v y ,v z and the rotational speed r around the coordinate axis x ,r y ,r z , and capture signal g.

[0097] S802. First, establish the representation of the camera coordinate system in the base coordinate system using the DH model. T represents the transformation matrix, c in the lower left corner represents the current coordinate system, and 1 in the upper left corner represents the reference coordinate system. The transformation matrix of the camera coordinate system in the base coordinate system can be obtained through forward kinematics operations. This coordinate system is a homogeneous transformation matrix, including a 3x3 rotation matrix R in the upper left corner, a 3x1 translation vector t in the upper right corner, a 1x3 vector 0 in the lower left corner, and a 1x1 number 1 in the lower right corner.

[0098] S803, transform the translation and rotation velocities in the camera coordinate system to the base coordinate system. Specifically, extract... Given the rotation matrix R, construct the transformation matrix T as follows. vc2b :

[0099]

[0100] S804, The velocity vector V = [v x ,v y ,v z ,r x ,r y ,r z ] T Right multiply by T vc2b The translational and rotational speeds of the camera in the robot arm's base coordinate system are obtained.

[0101] S805, by adjusting the camera transformation matrix Calculate the Jacobian matrix J, and then multiply the translational and rotational velocities in the robot arm's base coordinate system by J to obtain the velocities of each joint. Finally, by limiting the joint accelerations, a more realistic robot arm motion state is obtained. Therefore, the entire transformation process is as follows: in V represents the rotational speed of each joint of the robotic arm, and V represents the velocity vector at the end of the robotic arm.

[0102] In step S9, the joint velocity obtained in S8 is constrained by configuring the range of angle and angular acceleration. Direct constraint can be used, and the specific steps include:

[0103] S901, Configure the angle range of the joint [q] - ,q + ], where q - q represents the lower limit of the joint angle. + Indicates the upper limit of the joint angle;

[0104] S902, Configuration of joint movement speed range in Indicates the lower limit of joint velocity. Indicates the upper limit of joint velocity;

[0105] S903, Configure the angular acceleration range of joint motion. in This indicates the lower limit of joint angular acceleration. This indicates the upper limit of joint angular acceleration.

[0106] S904, Regarding the current joint rotation speed obtained in step S8 Joint angular velocity at the previous moment And with the control period T, the angular acceleration of each joint can be calculated as follows:

[0107] S905, Limiting Acceleration Calculate the current angular velocity using the constrained acceleration.

[0108] S906, Regarding the current joint rotation speed obtained in step S8 Current joint angle q t And with the control period T, the angle of each joint at the next moment can be calculated.

[0109] S907, Limiting the angle q t ′ +1 =min(max(q) t+1 ,q - ),q + ), where q t ′ +1 This is the constrained angle value. The current velocity is deduced from the constrained angle value.

[0110] S908, Set for and The one with the smaller absolute value, for Amplitude limiting is applied to obtain the final output angular velocity value.

[0111] Example 3

[0112] A grasping control method for a UR5 robot in a V-REP simulation environment includes the following steps:

[0113] S1. In the simulation environment, acquire RGB images from the end-effector camera module of the robotic arm in the simulation environment;

[0114] S2. Compress the image and transmit it to the Android device for display;

[0115] S3. Create a joystick control on the Android device.

[0116] S4. Create a slider control on the Android side.

[0117] S5. On the Android device, operation commands are collected through joystick and slider controls;

[0118] S6. Encode the collected data and transmit it to the PC.

[0119] S7: The PC receives control commands and decodes them into six-degree-of-freedom motion information.

[0120] S8. By establishing the DH model of the robotic arm, the velocity of each joint is solved inversely according to the motion command.

[0121] S9. By configuring the range of angle and angular acceleration, the joint velocity obtained in S8 is constrained, thereby obtaining a more realistic simulation effect.

[0122] S10. Encode the location information into a format required by the simulation environment and send it to the simulation environment for control.

[0123] Preferably, the simulation environment in step S1 is V-REP. The RGB-D camera appearance uses the RealSense D435i STL model, which is imported into V-REP. An RG2 gripper is attached to the UR5_connection node at the end of the UR5 robotic arm model. A camera holder is attached to the root node of the RG2 model. The RealSense STL appearance model is attached under the holder node. Two vision sensors are attached under the appearance model node. To accelerate computation, one sensor acquires only RGB image information, and the other only acquires depth information. In the simulation environment, each module is moved to a suitable position and orientation through translation and rotation. Then, the data from the vision sensors is obtained through the V-REP API interface. The obtained data is one-dimensional data. This one-dimensional raw data is grouped into three pixels to form a pixel value, and then arranged in a row-first, column-second order to obtain the final image data.

[0124] Preferred, such as Figure 3 As shown in Table 1, the DH model parameters used in step S8 are as follows:

[0125] Table 1 DH Model Parameters

[0126]

[0127] In the table above, the coordinate system is defined as follows: Figure 3

[0128] As shown, where α i This represents the angle between the z-axis of coordinate system i+1 and the z-axis of coordinate system i, with the positive direction of rotation being counterclockwise rotation around the x-axis of coordinate system i+1; a i d represents the translation distance of coordinate system i+1 relative to coordinate system i along the x-axis of coordinate system i, in meters. i θ represents the distance that coordinate system i+1 has translated relative to coordinate system i along the z-axis of coordinate system i, in meters. i Let coordinate system i rotate around the z-axis, with counterclockwise as the positive direction.

[0129] Control commands transmitted from Android include the translation speed v along the coordinate axes in the camera coordinate system. x ,v y ,v z and the rotational speed r around the coordinate axis x ,r u ,r z , and capture signal g.

[0130] First, the camera coordinate system is represented in the base coordinate system using the DH model. T represents the transformation matrix, c in the lower left corner represents the current coordinate system, and 1 in the upper left corner represents the reference coordinate system. The transformation matrix of the camera coordinate system in the base coordinate system can be obtained through forward kinematics operations. This coordinate system is a homogeneous transformation matrix, including a 3x3 rotation matrix R in the upper left corner, a 3x1 translation vector t in the upper right corner, a 1x3 vector 0 in the lower left corner, and a 1x1 number 1 in the lower right corner.

[0131] Then transform the translational and rotational velocities from the camera coordinate system to the base coordinate system. The specific steps are as follows: extract... Given the rotation matrix R, construct the following transformation matrix:

[0132]

[0133] The velocity vector V = [v x ,v y ,v z ,r x ,r y ,r z ] T Right multiply by T vc2b The translational and rotational speeds of the camera in the robot arm's base coordinate system are obtained.

[0134] By adjusting the camera transformation matrix Calculate the Jacobian matrix J, and then multiply the translational and rotational velocities in the robot arm's base coordinate system by J to obtain the velocities of each joint. Finally, by limiting the joint accelerations, a more realistic robot arm motion state is obtained. Therefore, the entire transformation process is as follows:

[0135] In step S9, the joint velocity obtained in S8 is constrained by configuring the range of angle and angular acceleration. Direct constraint can be used, and the specific steps include:

[0136] S901, Configure the angle range of the joint [q] - ,q + ], where q - q represents the lower limit of the joint angle. + Indicates the upper limit of the joint angle;

[0137] S902, Configuration of joint movement speed range in Indicates the lower limit of joint velocity. Indicates the upper limit of joint velocity;

[0138] S903. By configuring the upper and lower limits of the angle and angular velocity as described above, the final upper and lower limits of the angular velocity are obtained. in Represents angular velocity, and the lower limit of angular velocity is... The upper limit of angular velocity is Where q represents the angle information at the current moment.

[0139] Example 4

[0140] S1. In the simulation environment, acquire RGB images from the end-effector camera module of the robotic arm in the simulation environment;

[0141] S2. Compress the image and transmit it to the Android device for display;

[0142] S3. Create a joystick control on the Android device.

[0143] S4. Create a slider control on the Android side.

[0144] S5. On the Android device, operation commands are collected through joystick and slider controls;

[0145] S6. Encode the collected data and transmit it to the PC.

[0146] S7: The PC receives control commands and decodes them into six-degree-of-freedom motion information.

[0147] S8. By establishing the DH model of the robotic arm, the velocity of each joint is solved inversely according to the motion command.

[0148] S9. By configuring the range of angle and angular acceleration, the joint velocity obtained in S8 is constrained, thereby obtaining a more realistic simulation effect.

[0149] S10. Encode the location information into a format required by the simulation environment and send it to the simulation environment for control.

[0150] Preferably, the simulation environment used in step S1 is V-REP, and the model can be any robotic arm with a structure similar to UR5, such as... Figure 3

[0151] As shown.

[0152] Preferably, the remote control and slider control in steps S3 and S4 can be replaced by an external game controller, mainly to collect motion control commands and achieve motion control by listening to the data from the game controller.

[0153] Preferably, in step S8, any robotic arm with a similar structure can adopt the following approach: Figure 3

[0154] The coordinate system of the DH model is shown.

[0155] Control commands transmitted from Android include the translation speed v along the coordinate axes in the camera coordinate system. x ,v y ,v z and the rotational speed r around the coordinate axis x ,r y ,r z , and capture signal g.

[0156] First, the camera coordinate system is represented in the base coordinate system using the DH model. T represents the transformation matrix, c in the lower left corner represents the current coordinate system, and 1 in the upper left corner represents the reference coordinate system. The transformation matrix of the camera coordinate system in the base coordinate system can be obtained through forward kinematics operations. This coordinate system is a homogeneous transformation matrix, including a 3x3 rotation matrix R in the upper left corner, a 3x1 translation vector t in the upper right corner, a 1x3 vector 0 in the lower left corner, and a 1x1 number 1 in the lower right corner.

[0157] Then transform the translational and rotational velocities from the camera coordinate system to the base coordinate system. The specific steps are as follows: extract... Given the rotation matrix R, construct the following transformation matrix:

[0158]

[0159] The velocity vector V = [v x ,v y …v z ,r x ,r y ,r z ] T Right multiply by T vc2b The translational and rotational speeds of the camera in the robot arm's base coordinate system are obtained.

[0160] By adjusting the camera transformation matrix Calculate the Jacobian matrix J, and then multiply the translational and rotational velocities in the robot arm's base coordinate system by J to obtain the velocities of each joint. Finally, by limiting the joint accelerations, a more realistic robot arm motion state is obtained. Therefore, the entire transformation process is as follows:

[0161] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A UR5 robot arm grasping control method in a V-REP simulation environment, characterized in that, The method comprises the following steps: S1, collecting various data of the camera module at the end of the UR5 robot arm in the V-REP simulation environment, including depth map data, color map data, position and speed data of each joint of the UR5 robot arm; S2, compressing the color map data and transmitting it to the Android device for display; creating a joystick control and a slider control in the Android device; S3, collecting the data generated by the joystick and the slider in the Android device through the joystick control and the slider control, encoding and transmitting it to the PC end, and the PC end receives the control instruction and decodes it into a six-degree-of-freedom motion instruction; S4, establishing a DH model of the robot arm, and inversely solving the angular velocity of each joint of the UR5 robot arm according to the six-degree-of-freedom motion instruction; S5, by configuring the range of angle and angular acceleration, the angular velocity of the joint obtained in step S4 is constrained, so that a more realistic simulation effect is obtained; S6, combining the joint angle and time interval information of the previous moment, the angular information list of each joint is calculated, and the interface function of the V-REP simulation environment is called to traverse and send to the V-REP simulation environment for control.

2. The UR5 robot arm grasping control method in a V-REP simulation environment according to claim 1, characterized in that, Step S1 comprises the following steps: S101, creating a kinematic model of the UR5 robot arm in the V-REP simulation environment, and configuring each joint of the UR5 robot arm in the V-REP simulation environment according to the kinematic model of the UR5 robot arm; S102, creating an STL appearance model file of the realsense-D435i camera, mounting the visual sensor and depth sensor in the V-REP simulation environment under the STL appearance model file, and configuring according to the parameters of the realsense-D435i camera to achieve the simulation of the realsense-D435i camera; S103, mounting the realsense-D435i camera to the UR5 robot arm in the V-REP simulation environment to realize the Eye-in-hand model; S104, obtaining various data in the V-REP simulation environment through the interface of the V-REP simulation environment, including depth map data, color map data, position and speed data of each joint of the UR5 robot arm.

3. The UR5 robot arm grasping control method in a V-REP simulation environment according to claim 2, characterized in that, The operation of compressing the color map data and transmitting it to the Android device for display in step S2 comprises the following steps: S201, the PC end JPEG compresses the color map data collected in step S1; S202, the compressed data is transmitted to the Android device in the form of TCP communication; S203, after the Android device receives the data, it decodes and displays it.

4. The UR5 robot arm grasping control method in a V-REP simulation environment according to claim 1, characterized in that, Creating a joystick control comprises the following steps: S301, drawing a circular feasible region in the screen of the Android device, which is used to clearly define the motion range of the joystick; S302, drawing a small disc in the center of the circular feasible region as a joystick position indicator; S303, in the Android device, a joystick control java class is created and inherits the View class in the Android sdk, an onTouchEvent function is used to listen to the real-time position of the touch operation acting on the joystick control, when the touch operation is triggered, the onTouchEvent function is passed in a MotionEvent object, the MotionEvent object obtains the touch operation acting on the joystick control, including pressing, moving, lifting and moving out of the area, at the same time, the getX() and getY() functions are called through the MotionEvent object to obtain the real-time position of the touch operation, and are used to update the position of the small disc in real time, so that the position of the small disc follows the finger movement, at the same time, the center of the small disc in the screen is calculated in real time, the pixel difference value in the horizontal and vertical directions of the center of the circular feasible region in the screen, and the horizontal and vertical direction is divided by the radius of the circular feasible region to obtain the proportion as the output control signal, it is known that the output range of the output control signal is [-1, 1], and the output is normalized output; after receiving the output control signal, the PC end scales according to its own speed limit.

5. The UR5 robot arm grasping control method in a V-REP simulation environment according to claim 1, characterized in that, The creation of the slider control includes the following steps: S401, a one-dimensional feasible region in the horizontal direction is drawn in the screen of the Android device, which is used to clearly define the movement range of the slider, and the one-dimensional feasible region herein means that the slider can only move in one direction in the one-dimensional feasible region; S402, a circular slider is drawn at the midpoint of the one-dimensional feasible region as a position indicator of the slider; S403, a joystick control java class is created and inherits the View class in the Android sdk, the system's own onTouchEvent function is replaced by rewriting the onTouchEvent function to listen to the response logic of the touch operation, when the touch operation is triggered, the Android device calls the rewritten onTouchEvent function and passes in a MotionEvent object, the MotionEvent object obtains the touch operation acting on the joystick control java class, including pressing, moving, lifting and moving out of the area, at the same time, the getX() and getY() functions are called through the MotionEvent object to obtain the real-time position value of the touch operation acting on the joystick control java class, so that the position of the circular slider follows the finger movement in the horizontal direction, when the position of the circular slider exceeds the boundary of the one-dimensional feasible region, it stays at the edge of the one-dimensional feasible region, at the same time, the position value of the circular slider in the one-dimensional feasible region is output, ranging from -1 to 1, when the touch operation is released, the circular slider returns to the center position, and the output is 0.

6. The UR5 robot arm grasping control method in a V-REP simulation environment according to claim 1, characterized in that, Step S4 of establishing the DH model of the mechanical arm, inversely solving the angular velocity of each joint of the UR5 mechanical arm according to the six-degree-of-freedom motion instruction, including the following steps: S401, the Android device sends data to the PC in the form of bytes, and the PC receives the six-degree-of-freedom motion instructions in the form of bytes and decodes them into six-degree-of-freedom motion information, i.e., [vx, vy, vz, rx, ry, rz], which is divided into two groups, where vx, vy, and vz are the translational motion velocities of the UR5 robot arm along the x, y, and z axes in the robot arm end coordinate system; rx, ry, and rz are the rotational velocities of the UR5 robot arm around the x, y, and z axes in the robot arm end coordinate system; S402, a DH model of the robot arm is constructed, inverse kinematics is solved, the relationship between the angular velocity of each joint of the UR5 robot arm and the motion velocity of the UR5 robot arm end in the robot arm base coordinate system is obtained, and a Jacobian matrix is acquired; S403, the velocity in step S401 is converted into motion velocity in the robot arm base coordinate system, and the Jacobian matrix obtained in step S402 is multiplied to obtain the angular velocity of each joint of the UR5 robot arm.

7. The UR5 robot arm grasping control method in a V-REP simulation environment according to claim 6, characterized in that, The motion speed of the UR5 robot arm end in the V-REP simulation environment is converted to the motion speed in the robot arm base coordinate system, specifically based on the rotation matrix R in the transformation matrix of the camera coordinate system in the robot arm base coordinate system , which is constructed as follows : 。 8. The UR5 robot arm grasping control method in a V-REP simulation environment according to claim 6, characterized in that, Step S5 restricts the angular velocity of the joint obtained in step S4 by configuring the range of the angle and the angular acceleration, including the following steps: S501, configure a range of angles of a joint , wherein represents a lower limit of a joint angle, represents an upper limit of a joint angle; S502, configure an angular velocity range of the joint , wherein represents a lower limit of the angular velocity of the joint, represents an upper limit of the angular velocity of the joint; S503, configuring an angular acceleration range of articulation , wherein denotes a lower limit of the joint angular acceleration, denotes an upper limit of the joint angular acceleration; S504, the current angular value of each joint of the UR5 robot arm is acquired; S505, according to the configured angle range, joint angular velocity range, angular acceleration range, and current angular value of each joint in steps S501 to S504, the upper and lower limits of the current position joint angular velocity are calculated, and the joint angular velocity in step S4 is limited in amplitude.

9. The UR5 robot arm grasping control method in a V-REP simulation environment according to claim 8, characterized in that, In step S504, the current angular value of each joint of the robot arm DH model is acquired through the interface of the V-REP simulation environment; wherein the current angular value of each joint of the robot arm DH model refers to the rotation angle of each coordinate system fixed at the joint in the coordinate system established when the robot arm DH model is established.

Images