Robot flexible grabbing method and system based on touch sensor

Abstract

The invention relates to the technical field of pose control, in particular to a robot flexible grabbing method and system based on a touch sensor, and the method comprises the steps: carrying out the image collection of an environment where a to-be-operated object is located based on a camera integrated in a robot, so as to obtain an image of the environment where the to-be-operated object is located; and acquiring a real-time pose of an end effector of the robot based on inertial measurement equipment integrated in the robot, and performing pose estimation on a to-be-operated object in the environment image based on a pre-trained pose estimation model to obtain the pose of the to-be-operated object. By combining the environment image acquired by the camera and the real-time pose information provided by the inertial measurement equipment, the robot can estimate the pose of the object to be operated in real time, and the position and the pose of the object can be accurately speculated by utilizing the pre-trained pose estimation model, so that the accuracy of the operation of the robot is improved. Therefore, the operation of the end effector of the robot can be accurately butted with the object, and errors in the operation process are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of posture control, and in particular to a robot flexible grasping method and system based on a tactile sensor. Background Art

[0002] A tactile sensor is a sensor that can sense physical information such as touch, pressure, vibration, deformation, etc. on the surface of an object. Its working principle is usually based on mechanical deformation, pressure changes or electrical signal changes. It can convert the information of the touched object into electrical signals, thereby providing real-time feedback for robots or other devices.

[0003] Traditional methods usually rely on single sensor information (for example, only vision or only tactile feedback), and cannot combine multiple feedbacks of vision, inertial measurement and tactile sensors like the present invention. The use of a single sensor limits the accuracy of object posture estimation, which can easily lead to large errors in robot operation, especially in dynamic or complex environments. Traditional methods often lack precise end-effector kinematic control and usually rely on relatively simple path planning. They find it difficult to maintain high operating accuracy when facing delicate operations (such as precision grasping or micro-manipulations). There is no real-time adjustment or optimization of object posture errors, resulting in poor coordination of grasping accuracy, operation path and action, affecting the stability and reliability of the operation. Traditional methods usually do not have the ability to respond to changes in object state in real time and dynamically. When the shape or size of the object changes or the environment is uncertain, it is difficult for the robot operation to adapt to these changes. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a robot flexible grasping method and system based on tactile sensors.

[0005] The technical solution adopted to solve the above technical problems is: a robot flexible grasping method based on tactile sensors, comprising:

[0006] Based on the camera integrated in the robot, the environment where the object to be operated is located is imaged to obtain an image of the environment where the object to be operated is located;

[0007] Acquiring a real-time pose of an end effector of the robot based on an inertial measurement device integrated in the robot, and estimating the pose of an object to be operated in the environment image based on a pre-trained pose estimation model to obtain the pose of the object to be operated;

[0008] Constructing a kinematic function of the end effector of the robot, taking the error between the real-time posture of the end effector and the posture of the object to be operated as the minimum, and calculating a first operating parameter of the end effector according to the kinematic function;

[0009] Controlling the end effector to contact the object to be operated based on a first operating parameter of the end effector, and performing tactile perception of the object to be operated based on a tactile sensor integrated in the end effector to obtain a contact image;

[0010] Acquire a depth image corresponding to the contact image, and calculate an offset coefficient between the end effector and the object to be operated based on the depth image;

[0011] A second operating parameter of the end effector is calculated according to the kinematic function based on an offset coefficient between the end effector and the object to be operated, and the object to be operated is flexibly grasped based on the second operating parameter.

[0012] Preferably, the pose estimation model adopts an improved YOLOv3 network, and the backbone network of the YOLOv3 network is improved based on a hybrid attention model.

[0013] Preferably, the hybrid attention model includes an attention module and an aggregation model, wherein the attention module includes a coordinate information unit, a channel information unit and a spatial attention unit, wherein the coordinate information unit is expressed as follows:

[0014]

[0015] in, Represents the feature map output by the coordinate information unit, Conv represents the convolution operation, Concat represents the channel cascade operation, X i1 (i,j) represents the first input feature map, x coord and coord Represents the coordinate information of the X and Y axis channels in the coordinate information unit;

[0016] The expression of the channel information unit is as follows:

[0017]

[0018] in, represents the feature map output by the channel information unit, σ represents the Sigmoid activation function, ω 1 and b 1 Represents the weight vector and bias vector of the channel information unit, H and W represent the height and width of the feature map;

[0019] The expression of the spatial attention unit is as follows:

[0020]

[0021] in, represents the feature map output by the spatial attention unit, GN represents group normalization, ω 2 and b 2 represents the weight vector and bias vector of the spatial attention unit, X i2 (i,j) represents the second input feature map.

[0022] Preferably, constructing the kinematic function of the end effector of the robot includes:

[0023] A constraint function of the end effector of the robot on the reference plane around the instantaneous rotation center is established, wherein the expression of the constraint function is as follows:

[0024]

[0025] Where E represents the constraint function, β represents the pitch angle of the end effector, L represents the right shoulder joint of the end effector, and v R and v L represents the rotational linear velocity of the left and right shoulder joints of the end effector, X r and Y r represents the positive and reverse movement directions of the end effector, v A represents the rotational velocity of the reference plane, v O represents the linear velocity of the rotation center, and c represents the cosine function, η represents the action angle, R 0 and L 0 Represents the left and right shoulder joints of the end effector.

[0026] Preferably, constructing the kinematic function of the end effector of the robot further includes:

[0027] The characteristic vector of the end effector of the robot is obtained based on the constraint function, wherein the expression of the characteristic vector is as follows:

[0028]

[0029] Among them, O r represents the feature vector of the end effector;

[0030] A kinematic function of the end effector of the robot is constructed based on the characteristic vector of the end effector, wherein the expression of the kinematic function is as follows:

[0031]

[0032] Among them, O(t) represents the kinematic function, T(x, y, z) represents the process of translating from the origin of the world coordinate system to the current position of the robot, l(Z new,φ) represents the rotation angle around the Z axis in the new coordinate system after transformation, R(Y new ,-β) represents the rotation angle around the Y axis in the new coordinate system after transformation, Indicates the equivalent roll angle around the X-axis in the transformed new coordinate system.

[0033] Preferably, obtaining the depth image corresponding to the contact image includes:

[0034] Binarizing the contact image to obtain a background image corresponding to the contact image, and subtracting the contact image from the background image to obtain a difference image;

[0035] The difference image is subjected to contour extraction to obtain the center position and radius of the contact area, and the surface gradient of the tactile skin deformation corresponding to each pixel position in the contact area is calculated based on the actual radius of the calibration sphere.

[0036] Preferably, obtaining the depth image corresponding to the contact image further includes:

[0037] Establishing a mapping relationship between the surface gradient and the pixel intensity at the point, and establishing a mapping table with the pixel value as an index;

[0038] Traversing the pixel intensity of each pixel in the difference image, substituting the pixel intensity into the index address of the mapping table to obtain the corresponding surface gradient;

[0039] Depth information of the surface of the object to be operated is calculated from the surface gradient based on solving the Poisson equation, and a depth image corresponding to the contact image is acquired based on the depth information.

[0040] Preferably, calculating the offset coefficient between the end effector and the object to be operated based on the depth image includes:

[0041] Performing area screening on the depth image based on a preset depth threshold to obtain a valid contact area, and acquiring a distribution array of the valid contact area in a camera coordinate system;

[0042] The eigenvalues ​​and eigenvectors of the covariance matrix of the distribution array are calculated, the eigenvalues ​​are sorted, and the eigenvector corresponding to the largest eigenvalue is taken, and the center point of the distribution array is calculated.

[0043] Preferably, calculating the offset coefficient between the end effector and the object to be operated based on the depth image further includes:

[0044] The offset angle between the eigenvector corresponding to the largest eigenvalue and the Y axis of the camera coordinate system is calculated, and the offset distance between the center point of the effective contact area and the center point of the camera coordinate system is calculated, wherein the offset coefficient includes the offset angle and the offset distance.

[0045] The technical solution adopted to solve the above technical problems is: a robot flexible grasping method system based on tactile sensor, which is applicable to the robot flexible grasping method based on tactile sensor, including:

[0046] An image acquisition unit, wherein the image acquisition unit is used to acquire an image of the environment in which the object to be operated is located based on a camera integrated in the robot, so as to obtain an image of the environment in which the object to be operated is located;

[0047] A posture estimation unit, the posture estimation unit is used to obtain the real-time posture of the end effector of the robot based on an inertial measurement device integrated in the robot, and to estimate the posture of the object to be operated in the environment image based on a pre-trained posture estimation model to obtain the posture of the object to be operated;

[0048] A first posture control unit, the first posture control unit is used to construct a kinematic function of the end effector of the robot, taking the error between the real-time posture of the end effector and the posture of the object to be operated as the minimum, and calculating a first operating parameter of the end effector according to the kinematic function;

[0049] a tactile sensing unit, the tactile sensing unit being used to control the end effector to contact the object to be operated based on a first operating parameter of the end effector, and to perform tactile sensing on the object to be operated based on a tactile sensor integrated in the end effector to obtain a contact image;

[0050] an offset calculation unit, the offset calculation unit being used to obtain a depth image corresponding to the contact image, and to calculate an offset coefficient between the end effector and the object to be operated based on the depth image;

[0051] A second posture control unit is used to calculate a second operating parameter of the end effector according to the kinematic function based on an offset coefficient between the end effector and the object to be operated.

[0052] The beneficial effects of the present invention are as follows: (1) By combining the environmental image acquired by the camera and the real-time posture information provided by the inertial measurement device, the robot can estimate the posture of the object to be operated in real time. By using the pre-trained posture estimation model, the position and posture of the object can be accurately inferred, thereby ensuring that the operation of the robot end effector can accurately dock with the object, reducing the error during the operation process, and the tactile sensor in the end effector can provide real-time contact feedback. Based on the contact image and the depth image, the offset coefficient between the actuator and the object can be calculated. By analyzing the tactile perception information, the robot can adaptively adjust its action during the operation to achieve more flexible and precise operation and avoid damage caused by excessive or insufficient contact; (2) By constructing the kinematic function of the robot's end effector, the present invention can minimize the error based on the error between the posture of the object and the real-time posture of the actuator, thereby optimizing the path and action of the robot operation, which helps to improve the accuracy and response speed of the robot, especially when performing precision operations such as grasping and moving; (3) By calculating the offset coefficient and combining the kinematic function, the robot can adjust the operation parameters in real time, thereby dynamically responding to the posture changes of the object. This adaptive adjustment enables the robot to maintain high flexibility and stability when facing objects of different shapes and sizes and operating in uncertain environments. The combined use of vision, inertial measurement and tactile feedback can improve the safety of robot operations. The robot can detect any potential collisions or errors in real time based on feedback information, thereby adjusting its movements to avoid damaging objects or equipment. This is especially important when handling fragile items or operating in sensitive environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic flow chart of the steps of an overall method in an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of the system architecture of an overall system in an embodiment of the present invention.

[0055] Figure numerals: 1. Image acquisition unit; 2. Posture estimation unit; 3. First posture control unit; 4. Tactile perception unit; 5. Offset calculation unit; 6. Second posture control unit. DETAILED DESCRIPTION

[0056] Embodiment 1, as Figure 1 As shown, the present invention proposes a robot flexible grasping method based on a tactile sensor, comprising:

[0057] S1. Capturing an image of the environment where the object to be operated is located based on a camera integrated in the robot to obtain an image of the environment where the object to be operated is located;

[0058] S2, obtaining the real-time pose of the end effector of the robot based on the inertial measurement device integrated in the robot, and estimating the pose of the object to be operated in the environment image based on the pre-trained pose estimation model to obtain the pose of the object to be operated;

[0059] S3, constructing a kinematic function of the end effector of the robot, taking the minimum error between the real-time posture of the end effector and the posture of the object to be operated as the goal, and calculating the first operating parameter of the end effector according to the kinematic function;

[0060] S4, controlling the end effector to contact the object to be operated based on the first operation parameter of the end effector, and performing tactile perception on the object to be operated based on the tactile sensor integrated in the end effector to obtain a contact image;

[0061] S5, obtaining a depth image corresponding to the contact image, and calculating an offset coefficient between the end effector and the object to be operated based on the depth image;

[0062] S6. Calculate a second operating parameter of the end effector according to a kinematic function based on an offset coefficient between the end effector and the object to be operated, and perform flexibly grasping the object to be operated based on the second operating parameter.

[0063] In the present invention, the camera is an image acquisition device installed inside the robot, which is used to obtain images of the robot's surrounding environment in real time, including the object to be operated and its position. These images can provide key information about the environment, object shape and position; the inertial measurement unit (IMU) is used to measure the acceleration, angular velocity and other data of the robot or its components (such as the end effector). Common inertial measurement devices include gyroscopes, accelerometers, etc., which can help estimate the real-time position and posture (pose) of the robot; the end effector is the operating part of the robot, usually a tool such as grasping, cutting, welding, etc., which performs tasks of interacting with objects; the kinematic function describes the geometric relationship between the end effector of the robot and its joints or other components. Through this function, it can be calculated how the pose of the end effector changes with the change of the angle or position of each joint of the robot. The kinematic function helps to achieve the conversion from the current pose to the target pose; the first operating parameter refers to the preliminary action or adjustment that the end effector needs to perform before contacting the object, usually refers to parameters such as position, angle, speed, etc., which are used to ensure that the robot operates the object correctly; the second operating parameter refers to the geometric relationship between the end effector of the robot and its joints or other components. After touching the object, the adjustment parameters are calculated based on the feedback from the tactile sensor, image data and other information. These parameters help the robot optimize its movements to ensure that the task is completed more accurately. A tactile sensor is a sensor used to sense physical contact information such as contact force and pressure. It is usually installed on the end effector of the robot. It can help the robot sense the contact with the object and thus control the operation more accurately. The contact image is data collected by the tactile sensor, usually a pressure distribution map or a force sensing image, which reflects the contact between the end effector and the object. The contact image helps the robot determine the current contact state with the object and make corresponding action adjustments. A depth image is an image that represents the distance from each pixel in the image to the camera. It is usually obtained by a depth camera (such as a stereo camera or a structured light camera). The depth image helps the robot understand the three-dimensional structure of the object and its relative position to other objects in the environment.

[0064] Embodiment 2: A robot flexible grasping method based on tactile sensors is proposed in the present invention. Compared with embodiment 1, this embodiment also includes: the pose estimation model adopts an improved YOLOv3 network, and the backbone network of the YOLOv3 network is improved based on a hybrid attention model.

[0065] In this embodiment, YOLOv3: is the third version in the YOLO series. Compared with previous versions, YOLOv3 is optimized in accuracy and speed, supports multi-scale detection, and has better performance. It detects objects through a specific convolutional neural network (CNN) architecture, including prediction of bounding boxes, classification of object categories, etc.; Backbone Network: refers to the part of the deep learning model used for feature extraction. In YOLOv3, the role of the backbone network is to extract useful features from the input image. These features will be used for subsequent target detection tasks (such as bounding box prediction, category classification, etc.). Common backbone networks include ResNet, VGG, Darknet, etc.

[0066] In an optional embodiment, the hybrid attention model includes an attention module and an aggregation model, wherein the attention module includes a coordinate information unit, a channel information unit, and a spatial attention unit, wherein the coordinate information unit is expressed as follows:

[0067]

[0068] in, Represents the feature map output by the coordinate information unit, Conv represents the convolution operation, Concat represents the channel cascade operation, X i1 (i,j) represents the first input feature map, x coord and coord Represents the coordinate information of the X and Y axis channels in the coordinate information unit;

[0069] The expression of the channel information unit is as follows:

[0070]

[0071] in, represents the feature map output by the channel information unit, σ represents the Sigmoid activation function, ω 1 and b 1 Represents the weight vector and bias vector of the channel information unit, H and W represent the height and width of the feature map;

[0072] The expression of the spatial attention unit is as follows:

[0073]

[0074] in, represents the feature map output by the spatial attention unit, GN represents group normalization, ω 2 and b 2 represents the weight vector and bias vector of the spatial attention unit, X i2 (i,j) represents the second input feature map.

[0075] In an optional embodiment, constructing a kinematic function of an end effector of a robot includes:

[0076] A1. Establish a constraint function for the robot's end effector on the reference plane around the instantaneous rotation center, where the constraint function is expressed as follows:

[0077]

[0078] Where E represents the constraint function, β represents the pitch angle of the end effector, L represents the right shoulder joint of the end effector, and v R and v L represents the rotational linear velocity of the left and right shoulder joints of the end effector, X r and Y r represents the positive and reverse movement directions of the end effector, v A represents the rotational velocity of the reference plane, v O represents the linear velocity of the rotation center, and c represents the cosine function, η represents the action angle, R 0 and L 0 Represents the left and right shoulder joints of the end effector.

[0079] It should be noted that the reference plane is a reference plane used to describe the position of an object in space. Usually, it is an imaginary two-dimensional plane that serves as a reference for the movement of an object or robot in three-dimensional space. For example, the ground is often used as a reference plane. In the kinematic model, the reference plane is used to describe the relative motion of the end effector, usually in a two-dimensional coordinate system; the action angle usually refers to the angle between two objects, and their motion or interaction is affected by this angle. In robot kinematics, the action angle may refer to certain joint angles of the robotic arm or the orientation angle of the object, which affects the motion and posture of the end effector; the cosine function is a trigonometric function, usually used to describe the proportional relationship between an angle and the adjacent sides and hypotenuse in a right triangle. In robot kinematics, the cosine function can be used to represent the relationship between the rotation angle and the position change, especially when dealing with rotational transformations in space.

[0080] In an optional embodiment, constructing the kinematic function of the end effector of the robot further includes:

[0081] A2. Obtain the feature vector of the robot's end effector based on the constraint function, where the expression of the feature vector is as follows:

[0082]

[0083] Among them, O rrepresents the feature vector of the end effector;

[0084] A3. Construct the kinematic function of the end effector of the robot based on the feature vector of the end effector, where the expression of the kinematic function is as follows:

[0085]

[0086] Among them, O(t) represents the kinematic function, T(x, y, z) represents the process of translating from the origin of the world coordinate system to the current position of the robot, l(Z new ,φ) represents the rotation angle around the Z axis in the new coordinate system after transformation, R(Y new ,-β) represents the rotation angle around the Y axis in the new coordinate system after transformation, Indicates the equivalent roll angle around the X-axis in the transformed new coordinate system.

[0087] In an optional embodiment, obtaining a depth image corresponding to the contact image includes:

[0088] B1. Binarizing the contact image to obtain a background image corresponding to the contact image, and subtracting the contact image from the background image to obtain a difference image;

[0089] B2. Perform contour extraction on the difference image to obtain the center position and radius of the contact area, and calculate the surface gradient of the tactile skin deformation corresponding to each pixel position in the contact area based on the actual radius of the calibration sphere.

[0090] It should be noted that image binarization is the conversion of the pixel values ​​of an image into two possible values ​​(usually 0 and 255), which is usually used to simplify image processing, turn the image into a black and white image, and highlight the target object or a specific area. For example, by selecting a suitable threshold, pixels greater than the threshold are set to white (255) and pixels less than the threshold are set to black (0). Binarization helps to separate the area of ​​interest from the background for subsequent processing; the background image is the image obtained after the image is binarized, usually referring to the non-target area or background area in the original image. By comparing or subtracting with the original image, the background image will help to remove interference and non-target parts from the image, making the target area more prominent; image subtraction refers to subtracting the background image from the original image to obtain a difference image. This operation can help remove background information in the image and highlight the target area. For example, in a contact image, the background may be static, while the contact area represents the change or part of interest. By subtracting, information containing only the change in the contact area is obtained; the difference image is an image obtained by subtracting the background image from the original image. It shows the difference or change between the original image and the background image. In a contact image, the difference image usually highlights the contact area because the image content of the contact part is different from the background. The difference image usually helps to identify the change part (i.e., the contact area) in the image; contour extraction is a technique in image processing used to identify and extract the boundaries of objects in an image. In a difference image, the edge of the contact area can be determined by contour extraction. Common methods include edge detection (such as the Canny algorithm) or using gradient information in the image to detect edges. The extracted contour can help determine the specific position and shape of the contact area; the center position of the contact area refers to the coordinate representing the center of the contact area in the extracted contour. Generally speaking, the center position of the contact area can be obtained by calculating the geometric center (i.e., the center of mass) of the contour. This center position may be an important reference point for subsequent analysis (such as tactile skin deformation calculation); the calibration sphere is a sphere of known size, usually used to measure and calibrate the system. In this case, the actual radius of the calibration sphere refers to the physical size of the sphere. This known value can help convert the pixel values ​​in the image into actual physical units (such as millimeters or centimeters). In this way, the size in the image can be associated with the physical size in the real world; the surface gradient refers to the rate of change on the surface of an object. In the contact image, the surface gradient describes the rate of change of the skin deformation in the contact area. This gradient can help infer the direction and intensity of the force on the skin. The calculation of the surface gradient is usually related to the edge or color change of the image, and is usually achieved by solving the derivative or gradient of the image; based on the actual radius of the calibration sphere, the pixel size of the contact area in the image can be converted into actual physical units.The known radius of the calibration sphere allows for scaling to correspond the pixel spacing in the image to the actual physical spacing, thus providing a true physical measurement of the surface gradient of skin deformation.

[0091] In an optional embodiment, obtaining a depth image corresponding to the contact image further includes:

[0092] B3, establish a mapping relationship between the surface gradient and the pixel intensity of the point, and establish a mapping table with the pixel value as an index;

[0093] B4, traversing the pixel intensity of each pixel in the difference image, substituting the pixel intensity into the index address of the mapping table to obtain the corresponding surface gradient;

[0094] B5. Calculate the depth information of the surface of the object to be operated from the surface gradient based on solving the Poisson equation, and obtain a depth image corresponding to the contact image based on the depth information.

[0095] It should be noted that a mapping table is a data structure used to establish a relationship between two variables. The mapping table is used to use pixel intensity (grayscale value) as an index to find the corresponding surface gradient value through this index. The mapping table is actually a lookup table, in which each pixel intensity has a corresponding surface gradient value. This mapping relationship helps to infer the actual physical gradient based on the brightness or change of the image; Poisson's equation is a common type of partial differential equation, which is usually used to describe potential field problems in physics. For the calculation of surface deformation, Poisson's equation can help infer depth information from known surface gradient information. The surface gradient usually reflects the rate of change of the local surface, and Poisson's equation is used to recover the depth or height information of the object from this gradient field; Depth information is information that describes the position or shape of the surface of an object in three-dimensional space. In a contact image, depth information reflects the deformation of the surface of an object after being acted upon by an external force. Depth information is very important for analyzing tactile feedback, pressure distribution on the surface of an object, etc. By calculating the depth image, the deformation of the surface of the object during the contact process can be obtained; a depth image refers to an image in which the value of each pixel represents the depth or distance of a point on the surface of the object.

[0096] In an optional embodiment, calculating the offset coefficient between the end effector and the object to be operated based on the depth image includes:

[0097] C1. Performing area screening on the depth image based on a preset depth threshold to obtain a valid contact area, and obtaining a distribution array of the valid contact area in the camera coordinate system;

[0098] C2. Calculate the eigenvalues ​​and eigenvectors of the covariance matrix of the distribution array, sort the eigenvalues, take the eigenvector corresponding to the largest eigenvalue, and calculate the center point of the distribution array.

[0099] In an optional embodiment, calculating the offset coefficient between the end effector and the object to be operated based on the depth image also includes:

[0100] C1. Calculate the offset angle between the eigenvector corresponding to the largest eigenvalue and the Y axis of the camera coordinate system, and calculate the offset distance between the center point of the effective contact area and the center point of the camera coordinate system, where the offset coefficient includes the offset angle and the offset distance.

[0101] Embodiment three, as Figure 2 As shown, a robot flexible grasping method system based on a tactile sensor proposed in the present invention is applicable to the robot flexible grasping method based on a tactile sensor, comprising:

[0102] An image acquisition unit 1 is used to acquire an image of the environment where the object to be operated is located based on a camera integrated in the robot to obtain an image of the environment where the object to be operated is located;

[0103] A posture estimation unit 2, which is used to obtain the real-time posture of the end effector of the robot based on an inertial measurement device integrated in the robot, and to estimate the posture of the object to be operated in the environment image based on a pre-trained posture estimation model to obtain the posture of the object to be operated;

[0104] The first posture control unit 3 is used to construct a kinematic function of the end effector of the robot, taking the error between the real-time posture of the end effector and the posture of the object to be operated as the minimum, and calculating the first operating parameter of the end effector according to the kinematic function;

[0105] A tactile sensing unit 4, which is used to control the end effector to contact the object to be operated based on the first operating parameter of the end effector, and to perform tactile sensing on the object to be operated based on the tactile sensor integrated in the end effector to obtain a contact image;

[0106] An offset calculation unit 5, the offset calculation unit 5 is used to obtain a depth image corresponding to the contact image, and calculate an offset coefficient between the end effector and the object to be operated based on the depth image;

[0107] The second posture control unit 6 is used to calculate the second operation parameter of the end effector according to the kinematic function based on the offset coefficient between the end effector and the object to be operated.

[0108] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A robot flexible grasping method based on tactile sensor, characterized in that: include: Based on the camera integrated in the robot, the environment where the object to be operated is located is imaged to obtain an image of the environment where the object to be operated is located; Acquiring a real-time pose of an end effector of the robot based on an inertial measurement device integrated in the robot, and estimating the pose of an object to be operated in the environment image based on a pre-trained pose estimation model to obtain the pose of the object to be operated; Constructing a kinematic function of the end effector of the robot, taking the error between the real-time posture of the end effector and the posture of the object to be operated as the minimum, and calculating a first operating parameter of the end effector according to the kinematic function; Controlling the end effector to contact the object to be operated based on a first operating parameter of the end effector, and performing tactile perception of the object to be operated based on a tactile sensor integrated in the end effector to obtain a contact image; Acquire a depth image corresponding to the contact image, and calculate an offset coefficient between the end effector and the object to be operated based on the depth image; A second operating parameter of the end effector is calculated according to the kinematic function based on an offset coefficient between the end effector and the object to be operated, and the object to be operated is flexibly grasped based on the second operating parameter.

2. A robot flexible grasping method based on tactile sensor according to claim 1, characterized in that: The pose estimation model adopts an improved YOLOv3 network, and improves the backbone network of the YOLOv3 network based on a hybrid attention model.

3. The robot flexible grasping method based on tactile sensor according to claim 2 is characterized in that: The hybrid attention model includes an attention module and an aggregation model, wherein the attention module includes a coordinate information unit, a channel information unit and a spatial attention unit, wherein the expression of the coordinate information unit is as follows: in, Represents the feature map output by the coordinate information unit, Conv represents the convolution operation, Concat represents the channel cascade operation, X i1 (i,j) represents the first input feature map, x coord and coord Represents the coordinate information of the X and Y axis channels in the coordinate information unit; The expression of the channel information unit is as follows: in, represents the feature map output by the channel information unit, σ represents the Sigmoid activation function, ω1 and b1 represent the weight vector and bias vector of the channel information unit, and H and W represent the height and width of the feature map; The expression of the spatial attention unit is as follows: in, represents the feature map output by the spatial attention unit, GN represents group normalization, ω2 and b2 represent the weight vector and bias vector of the spatial attention unit, X i2 (i,j) represents the second input feature map.

4. The robot flexible grasping method based on tactile sensor according to claim 1 is characterized in that: Constructing the kinematic function of the end effector of the robot, including: A constraint function of the end effector of the robot on the reference plane around the instantaneous rotation center is established, wherein the expression of the constraint function is as follows: Where E represents the constraint function, β represents the pitch angle of the end effector, L represents the right shoulder joint of the end effector, and v R and v L represents the rotational linear velocity of the left and right shoulder joints of the end effector, X r and Y r represents the positive and reverse movement directions of the end effector, v A represents the rotational velocity of the reference plane, v O represents the linear velocity of the rotation center, and c represents the cosine function, η represents the action angle, and R0 and L0 represent the left and right shoulder joints of the end effector.

5. The robot flexible grasping method based on tactile sensor according to claim 4 is characterized in that: Constructing the kinematic function of the end effector of the robot also includes: The characteristic vector of the end effector of the robot is obtained based on the constraint function, wherein the expression of the characteristic vector is as follows: Among them, O r represents the feature vector of the end effector; A kinematic function of the end effector of the robot is constructed based on the characteristic vector of the end effector, wherein the expression of the kinematic function is as follows: Among them, O(t) represents the kinematic function, T(x, y, z) represents the process of translating from the origin of the world coordinate system to the current position of the robot, l(Z new ,φ) represents the rotation angle around the Z axis in the new coordinate system after transformation, R(Y new ,-β) represents the rotation angle around the Y axis in the new coordinate system after transformation, Indicates the equivalent roll angle around the X-axis in the transformed new coordinate system.

6. The robot flexible grasping method based on tactile sensor according to claim 5 is characterized in that: Acquiring a depth image corresponding to the contact image, comprising: Binarizing the contact image to obtain a background image corresponding to the contact image, and subtracting the contact image from the background image to obtain a difference image; The difference image is subjected to contour extraction to obtain the center position and radius of the contact area, and the surface gradient of the tactile skin deformation corresponding to each pixel position in the contact area is calculated based on the actual radius of the calibration sphere.

7. The robot flexible grasping method based on tactile sensor according to claim 6 is characterized in that: Acquiring a depth image corresponding to the contact image also includes: Establishing a mapping relationship between the surface gradient and the pixel intensity at the point, and establishing a mapping table with the pixel value as an index; Traversing the pixel intensity of each pixel in the difference image, substituting the pixel intensity into the index address of the mapping table to obtain the corresponding surface gradient; Depth information of the surface of the object to be operated is calculated from the surface gradient based on solving the Poisson equation, and a depth image corresponding to the contact image is acquired based on the depth information.

8. The robot flexible grasping method based on tactile sensor according to claim 7 is characterized in that: Calculating the offset coefficient between the end effector and the object to be operated based on the depth image includes: Performing area screening on the depth image based on a preset depth threshold to obtain a valid contact area, and acquiring a distribution array of the valid contact area in a camera coordinate system; The eigenvalues ​​and eigenvectors of the covariance matrix of the distribution array are calculated, the eigenvalues ​​are sorted, and the eigenvector corresponding to the largest eigenvalue is taken, and the center point of the distribution array is calculated.

9. The robot flexible grasping method based on tactile sensor according to claim 8 is characterized in that: Calculating the offset coefficient between the end effector and the object to be operated based on the depth image also includes: The offset angle between the eigenvector corresponding to the largest eigenvalue and the Y axis of the camera coordinate system is calculated, and the offset distance between the center point of the effective contact area and the center point of the camera coordinate system is calculated, wherein the offset coefficient includes the offset angle and the offset distance.

10. A robot flexible grasping method system based on a tactile sensor, which is applicable to a robot flexible grasping method based on a tactile sensor according to any one of claims 1 to 9, characterized in that: include: An image acquisition unit (1), the image acquisition unit (1) being used to acquire an image of the environment in which the object to be operated is located based on a camera integrated in the robot, so as to obtain an image of the environment in which the object to be operated is located; A posture estimation unit (2), the posture estimation unit (2) being used to obtain a real-time posture of an end effector of the robot based on an inertial measurement device integrated in the robot, and to perform posture estimation on an object to be operated in the environment image based on a pre-trained posture estimation model, so as to obtain the posture of the object to be operated; A first posture control unit (3), the first posture control unit (3) is used to construct a kinematic function of the end effector of the robot, taking the error between the real-time posture of the end effector and the posture of the object to be operated as the minimum, and calculating a first operating parameter of the end effector according to the kinematic function; A tactile sensing unit (4), the tactile sensing unit (4) being used to control the end effector to contact the object to be operated based on a first operating parameter of the end effector, and to perform tactile sensing on the object to be operated based on a tactile sensor integrated in the end effector, so as to obtain a contact image; An offset calculation unit (5), the offset calculation unit (5) being used to obtain a depth image corresponding to the contact image, and to calculate an offset coefficient between the end effector and the object to be operated based on the depth image; A second posture control unit (6), the second posture control unit (6) is used to calculate a second operating parameter of the end effector according to the kinematic function based on an offset coefficient between the end effector and the object to be operated, and to perform flexibly grasping the object to be operated based on the second operating parameter.

Images