A robot flexible grasping method and system based on tactile sensor

Abstract

The present invention relates to the field of posture control technology, specifically a tactile sensor-based flexible grasping method and system for a robot, comprising: acquiring images of the environment in which an object to be operated is located based on a camera integrated in the robot to obtain an image of the environment in which the object to be operated is located; acquiring the real-time posture of the robot's end effector based on an inertial measurement device integrated in the robot, and estimating the posture of the object to be operated in the environment image based on a pre-trained posture estimation model to obtain the posture of the object to be operated. By combining the environment image acquired by the camera and the real-time posture information provided by the inertial measurement device, the present invention enables the robot to estimate the posture of the object to be operated in real time. By using the pre-trained posture estimation model, the position and posture of the object can be accurately inferred, thereby ensuring that the operation of the robot's end effector can accurately dock with the object, reducing errors during the operation process.

Description

Technical Field

[0001] The present invention relates to the technical field of posture control, and in particular to a robot flexible grasping method and system based on a tactile sensor. Background Art

[0002] A tactile sensor is a sensor that can sense physical information such as touch, pressure, vibration, and deformation on the surface of an object. Its working principle is usually based on mechanical deformation, pressure changes, or electrical signal changes. It can convert information about the touched object into electrical signals, thereby providing real-time feedback for robots or other devices.

[0003] Traditional methods usually rely on single sensor information (for example, only vision or only tactile feedback) and cannot combine multiple feedbacks of vision, inertial measurement and tactile sensors like the present invention. The use of a single sensor limits the accuracy of object posture estimation, which can easily lead to large errors in robot operation, especially in dynamic or complex environments. Traditional methods often lack precise end-effector kinematic control and usually rely on relatively simple path planning. It is difficult to maintain high operational accuracy when facing fine operations (such as precision grasping or micro-operations). There is no real-time adjustment or optimization of object posture errors, resulting in poor coordination of grasping accuracy, operation path and action, affecting the stability and reliability of the operation. Traditional methods usually do not have the ability to respond to changes in object state in real time. When the shape or size of the object changes or the environment is uncertain, it is difficult for the robot operation to adapt to these changes. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a robot flexible grasping method and system based on tactile sensors.

[0005] The technical solution adopted to solve the above technical problems is: a robot flexible grasping method based on tactile sensors, comprising:

[0006] The camera integrated in the robot collects images of the environment where the object to be operated is located, so as to obtain an image of the environment where the object to be operated is located;

[0007] Acquiring a real-time pose of an end effector of the robot based on an inertial measurement device integrated in the robot, and estimating the pose of an object to be operated in the environment image based on a pre-trained pose estimation model to obtain the pose of the object to be operated;

[0008] Constructing a kinematic function of the end effector of the robot, minimizing the error between the real-time posture of the end effector and the posture of the object to be operated, and calculating a first operating parameter of the end effector according to the kinematic function;

[0009] controlling the end effector to contact the object to be operated based on a first operating parameter of the end effector, and performing tactile perception of the object to be operated based on a tactile sensor integrated in the end effector to obtain a contact image;

[0010] Acquiring a depth image corresponding to the contact image, and calculating an offset coefficient between the end effector and the object to be operated based on the depth image;

[0011] A second operating parameter of the end effector is calculated according to the kinematic function based on an offset coefficient between the end effector and the object to be operated, and the object to be operated is flexibly grasped based on the second operating parameter.

[0012] Preferably, the pose estimation model adopts an improved YOLOv3 network, and the backbone network of the YOLOv3 network is improved based on a hybrid attention model.

[0013] Preferably, the hybrid attention model includes an attention module and an aggregation model, wherein the attention module includes a coordinate information unit, a channel information unit and a spatial attention unit, wherein the coordinate information unit is expressed as follows:

[0014]

[0015] in, Represents the feature map output by the coordinate information unit, Conv represents the convolution operation, Concat represents the channel cascade operation, X i1 (i, j) represents the first input feature map, x coord and y coord Represents the coordinate information of the X and Y axis channels in the coordinate information unit;

[0016] The expression of the channel information unit is as follows:

[0017]

[0018] in, Represents the feature map output by the channel information unit, σ represents the Sigmoi d activation function, ω1 and b1 represent the weight vector and bias vector of the channel information unit, H and W represent the height and width of the feature map;

[0019] The expression of the spatial attention unit is as follows:

[0020]

[0021] in, represents the feature map output by the spatial attention unit, GN represents group normalization, ω2 and b2 represent the weight vector and bias vector of the spatial attention unit, X i2 (i,j) represents the second input feature map.

[0022] Preferably, constructing the kinematic function of the end effector of the robot includes:

[0023] A constraint function for the robot's end effector around the instantaneous rotation center on the reference plane is established, wherein the constraint function is expressed as follows:

[0024]

[0025] Where E represents the constraint function, β represents the pitch angle of the end effector, L represents the right shoulder joint of the end effector, and v R and v L represents the rotational velocity of the left and right shoulder joints of the end effector, X r and Y r Indicates the forward and reverse movement directions of the end effector, v A represents the rotational velocity of the reference plane, v O represents the linear velocity of the rotation center, and c represents the cosine function, η represents the action angle, and R0 and L0 represent the left and right shoulder joints of the end effector.

[0026] Preferably, constructing the kinematic function of the end effector of the robot further includes:

[0027] A feature vector of the end effector of the robot is obtained based on the constraint function, wherein the expression of the feature vector is as follows:

[0028]

[0029] Among them, O r represents the feature vector of the end effector;

[0030] A kinematic function of the end effector of the robot is constructed based on the characteristic vector of the end effector, wherein the expression of the kinematic function is as follows:

[0031]

[0032] Among them, O(t) represents the kinematic function, T(x,y,z) represents the process of translating from the origin of the world coordinate system to the current position of the robot, l(Z new ,φ) represents the rotation angle around the Z axis in the new coordinate system after transformation, R(Y new,-β) represents the rotation angle around the Y axis in the new coordinate system after transformation, Indicates the equivalent roll angle around the X-axis in the transformed new coordinate system.

[0033] Preferably, obtaining the depth image corresponding to the contact image includes:

[0034] performing image binarization on the contact image to obtain a background image corresponding to the contact image, and subtracting the contact image from the background image to obtain a difference image;

[0035] The difference image is subjected to contour extraction to obtain the center position and radius of the contact area, and the surface gradient of the tactile skin deformation corresponding to each pixel position in the contact area is calculated based on the actual radius of the calibration sphere.

[0036] Preferably, obtaining the depth image corresponding to the contact image further includes:

[0037] Establishing a mapping relationship between the surface gradient and the pixel intensity at the point, and establishing a mapping table with the pixel value as an index;

[0038] Traversing the pixel intensity of each pixel in the difference image, substituting the pixel intensity into the index address of the mapping table to obtain the corresponding surface gradient;

[0039] Depth information of the surface of the object to be operated is calculated from the surface gradient based on solving the Poisson equation, and a depth image corresponding to the contact image is acquired based on the depth information.

[0040] Preferably, calculating the offset coefficient between the end effector and the object to be operated based on the depth image includes:

[0041] Performing regional screening on the depth image based on a preset depth threshold to obtain a valid contact area, and acquiring a distribution array of the valid contact area in a camera coordinate system;

[0042] The eigenvalues ​​and eigenvectors of the covariance matrix of the distribution array are calculated, the eigenvalues ​​are sorted, and the eigenvector corresponding to the largest eigenvalue is obtained to calculate the center point of the distribution array.

[0043] Preferably, calculating the offset coefficient between the end effector and the object to be operated based on the depth image further includes:

[0044] Calculate the offset angle between the eigenvector corresponding to the largest eigenvalue and the Y axis of the camera coordinate system, and calculate the offset distance between the center point of the effective contact area and the center point of the camera coordinate system, wherein the offset coefficient includes the offset angle and the offset distance.

[0045] The technical solution adopted to solve the above technical problems is: a robot flexible grasping method system based on a tactile sensor, which is applicable to the above-mentioned robot flexible grasping method based on a tactile sensor, comprising:

[0046] An image acquisition unit, configured to acquire an image of the environment in which the object to be operated is located based on a camera integrated in the robot, so as to obtain an image of the environment in which the object to be operated is located;

[0047] a pose estimation unit, configured to obtain a real-time pose of an end effector of the robot based on an inertial measurement device integrated in the robot, and to perform pose estimation on an object to be operated in the environment image based on a pre-trained pose estimation model to obtain a pose of the object to be operated;

[0048] a first posture control unit, configured to construct a kinematic function for an end effector of the robot, minimize an error between a real-time posture of the end effector and a posture of the object to be operated, and calculate a first operating parameter of the end effector according to the kinematic function;

[0049] a tactile sensing unit, configured to control the end effector to contact the object to be operated based on a first operating parameter of the end effector, and to perform tactile sensing of the object to be operated based on a tactile sensor integrated in the end effector to obtain a contact image;

[0050] an offset calculation unit, configured to obtain a depth image corresponding to the contact image and calculate an offset coefficient between the end effector and the object to be operated based on the depth image;

[0051] A second posture control unit is configured to calculate a second operating parameter of the end effector according to the kinematic function based on an offset coefficient between the end effector and the object to be operated.

[0052] The beneficial effects of the present invention are as follows: (1) By combining the environmental image acquired by the camera and the real-time posture information provided by the inertial measurement device, the robot can estimate the posture of the object to be operated in real time. By using the pre-trained posture estimation model, the position and posture of the object can be accurately inferred, thereby ensuring that the operation of the robot's end effector can accurately dock with the object, reducing the error during the operation process, and the tactile sensor in the end effector can provide real-time contact feedback. Based on the contact image and depth image, the offset coefficient between the actuator and the object can be calculated. By analyzing the tactile perception information, the robot can adaptively adjust its movement during the operation to achieve more flexible and precise operation and avoid damage caused by excessive or insufficient contact; (2) By constructing the kinematic function of the robot's end effector, the present invention can minimize the error between the object's posture and the actuator's real-time posture, thereby optimizing the robot's operation path and movement, which helps to improve the robot's accuracy and response speed, especially when performing precision operations such as grasping and moving; (3) By calculating the offset coefficient and combining it with the kinematic function, the robot can adjust the operation parameters in real time, thereby dynamically responding to the change in the object's posture. This adaptive adjustment enables the robot to maintain high flexibility and stability when facing objects of different shapes and sizes and operating in uncertain environments. By combining vision, inertial measurement, and tactile feedback, the safety of robot operations can be improved. The robot can detect any potential collisions or errors in real time based on feedback information, thereby adjusting its movements to avoid damaging objects or equipment. This is especially important when handling fragile items or operating in sensitive environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic flow chart of the steps of the overall method in one embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the system architecture of the overall system in an embodiment of the present invention.

[0055] Figure numerals: 1. Image acquisition unit; 2. Posture estimation unit; 3. First posture control unit; 4. Tactile perception unit; 5. Offset calculation unit; 6. Second posture control unit. DETAILED DESCRIPTION

[0056] Example 1, as Figure 1 As shown, the present invention proposes a robot flexible grasping method based on a tactile sensor, comprising:

[0057] S1. Capturing an image of the environment in which the object to be operated is located based on a camera integrated in the robot to obtain an image of the environment in which the object to be operated is located;

[0058] S2. Obtaining the real-time pose of the robot's end effector based on an inertial measurement device integrated in the robot, and estimating the pose of the object to be operated in the environment image based on a pre-trained pose estimation model to obtain the pose of the object to be operated;

[0059] S3, constructing a kinematic function of the end effector of the robot, taking the error between the real-time posture of the end effector and the posture of the object to be operated as the minimum, and calculating a first operating parameter of the end effector according to the kinematic function;

[0060] S4, controlling the end effector to contact the object to be operated based on the first operating parameter of the end effector, and performing tactile perception of the object to be operated based on the tactile sensor integrated in the end effector to obtain a contact image;

[0061] S5. Acquire a depth image corresponding to the contact image, and calculate an offset coefficient between the end effector and the object to be operated based on the depth image;

[0062] S6. Calculate a second operating parameter of the end effector according to the kinematic function based on the offset coefficient between the end effector and the object to be operated, and perform flexibly grasping the object to be operated based on the second operating parameter.

[0063] In the present invention, the camera is an image acquisition device installed inside the robot, which is used to obtain real-time images of the robot's surrounding environment, including the object to be operated and its position. These images can provide key information about the environment, object shape and position; the inertial measurement unit (IMU) is used to measure the acceleration, angular velocity and other data of the robot or its components (such as the end effector). Common inertial measurement devices include gyroscopes, accelerometers, etc., which can help estimate the real-time position and posture (pose) of the robot; the end effector is the operating part of the robot, usually a tool such as grasping, cutting, welding, etc., which performs tasks that interact with objects; the kinematic function describes the geometric relationship between the end effector of the robot and its joints or other components. Through this function, how the pose of the end effector changes with the angle or position of each joint of the robot can be calculated. The kinematic function helps to achieve the conversion from the current pose to the target pose; the first operating parameter refers to the preliminary action or adjustment that the end effector needs to perform before contacting the object, usually referring to parameters such as position, angle, and speed, used to ensure that the robot correctly operates the object; the second operating parameter refers to the initial action or adjustment that the end effector needs to perform before contacting the object. After touching an object, adjustment parameters are calculated based on tactile sensor feedback, image data and other information. These parameters help the robot optimize its movements to ensure more accurate task completion. A tactile sensor is a sensor used to sense physical contact information such as contact force and pressure. It is usually installed on the end effector of the robot. It can help the robot sense the contact situation with the object, thereby controlling the operation more accurately. The contact image is data collected by the tactile sensor, usually a pressure distribution map or a force sensing image, which reflects the contact situation between the end effector and the object. The contact image helps the robot judge the current contact status with the object and make corresponding action adjustments. The depth image is an image that represents the distance from each pixel in the image to the camera. It is usually obtained by a depth camera (such as a stereo camera or a structured light camera). The depth image helps the robot understand the three-dimensional structure of the object and its relative position to other objects in the environment.

[0064] Example 2: A robot flexible grasping method based on a tactile sensor proposed in the present invention. Compared with Example 1, this embodiment also includes: the pose estimation model adopts an improved YOLOv3 network, and the backbone network of the YOLOv3 network is improved based on the hybrid attention model.

[0065] In this embodiment, YOLOv3 is the third version in the YOLO series. Compared with previous versions, YOLOv3 is optimized in terms of accuracy and speed, supports multi-scale detection, and has better performance. It detects objects through a specific convolutional neural network (CNN) architecture, including bounding box prediction and object category classification. Backbone Network refers to the part of the deep learning model used for feature extraction. In YOLOv3, the backbone network extracts useful features from the input image. These features will be used for subsequent target detection tasks (such as bounding box prediction and category classification). Common backbone networks include ResNet, VGG, Darknet, etc.

[0066] In an optional embodiment, the hybrid attention model includes an attention module and an aggregation model, wherein the attention module includes a coordinate information unit, a channel information unit, and a spatial attention unit, wherein the coordinate information unit is expressed as follows:

[0067]

[0068] in, Represents the feature map output by the coordinate information unit, Conv represents the convolution operation, Concat represents the channel cascade operation, X i1 (i, j) represents the first input feature map, x coord and y coord Represents the coordinate information of the X and Y axis channels in the coordinate information unit;

[0069] The expression of the channel information unit is as follows:

[0070]

[0071] in, Represents the feature map output by the channel information unit, σ represents the Sigmoi d activation function, ω1 and b1 represent the weight vector and bias vector of the channel information unit, H and W represent the height and width of the feature map;

[0072] The expression of the spatial attention unit is as follows:

[0073]

[0074] in, represents the feature map output by the spatial attention unit, GN represents group normalization, ω2 and b2 represent the weight vector and bias vector of the spatial attention unit, X i2 (i,j) represents the second input feature map.

[0075] In an optional embodiment, constructing a kinematic function of the robot's end effector includes:

[0076] A1. Establish a constraint function for the robot's end effector around its instantaneous rotation center on the reference plane. The constraint function is expressed as follows:

[0077]

[0078] Where E represents the constraint function, β represents the pitch angle of the end effector, L represents the right shoulder joint of the end effector, and v R and v L represents the rotational velocity of the left and right shoulder joints of the end effector, X r and Y r Indicates the forward and reverse movement directions of the end effector, v A represents the rotational velocity of the reference plane, v O represents the linear velocity of the rotation center, and c represents the cosine function, η represents the action angle, and R0 and L0 represent the left and right shoulder joints of the end effector.

[0079] It should be noted that the reference plane is a reference plane used to describe the position of an object in space. Usually, it is an imaginary two-dimensional plane that serves as a reference for the movement of an object or robot in three-dimensional space. For example, the ground is often used as a reference plane. In kinematic models, the reference plane is used to describe the relative motion of the end effector, usually in a two-dimensional coordinate system. The action angle usually refers to the angle between two objects, and their motion or interaction is affected by this angle. In robot kinematics, the action angle may refer to certain joint angles of the robotic arm or the orientation angle of the object, which affects the motion and posture of the end effector. The cosine function is a trigonometric function that is usually used to describe the proportional relationship between an angle and the adjacent sides and hypotenuse of a right triangle. In robot kinematics, the cosine function can be used to represent the relationship between the rotation angle and the position change, especially when dealing with rotational transformations in space.

[0080] In an optional embodiment, constructing the kinematic function of the robot's end effector further includes:

[0081] A2. Obtain the eigenvector of the robot's end effector based on the constraint function, where the expression of the eigenvector is as follows:

[0082]

[0083] Among them, O r represents the feature vector of the end effector;

[0084] A3. Construct the kinematic function of the robot's end effector based on the eigenvector of the end effector. The expression of the kinematic function is as follows:

[0085]

[0086] Among them, O(t) represents the kinematic function, T(x,y,z) represents the process of translating from the origin of the world coordinate system to the current position of the robot, l(Z new ,φ) represents the rotation angle around the Z axis in the new coordinate system after transformation, R(Y new ,-β) represents the rotation angle around the Y axis in the new coordinate system after transformation, Indicates the equivalent roll angle around the X-axis in the transformed new coordinate system.

[0087] In an optional embodiment, obtaining a depth image corresponding to the contact image includes:

[0088] B1. Binarize the contact image to obtain a background image corresponding to the contact image, and subtract the contact image from the background image to obtain a difference image;

[0089] B2. Perform contour extraction on the difference image to obtain the center position and radius of the contact area, and calculate the surface gradient of the tactile skin deformation corresponding to each pixel position in the contact area based on the actual radius of the calibration sphere.

[0090] It should be noted that image binarization is the conversion of the pixel values ​​of an image into two possible values ​​(usually 0 and 255). This is usually used to simplify image processing, turning the image into a black and white image, and highlighting the target object or a specific area. For example, by selecting a suitable threshold, pixels greater than the threshold are set to white (255) and pixels less than the threshold are set to black (0). Binarization helps to separate the area of ​​interest from the background, facilitating subsequent processing; the background image is the image obtained after image binarization, usually referring to the non-target area or background area in the original image. By comparing or subtracting it from the original image, the background image will help to remove interference and non-target parts from the image, making the target area more prominent; image subtraction refers to subtracting the background image from the original image to obtain a difference image. This operation can help remove background information from an image and highlight the target area. For example, in a contact image, the background may be static, while the contact area represents the changing or interesting part. By subtracting them, information containing only the changes in the contact area is obtained. A difference image is an image obtained by subtracting the background image from the original image. It shows the difference or changes between the original image and the background image. In a contact image, the difference image usually highlights the contact area because the image content of the contact part is different from the background. The difference image usually helps to identify the changing part (i.e., the contact area) in the image. Contour extraction is a technique in image processing used to identify and extract the boundaries of objects in an image. In a difference image, the edges of the contact area can be determined through contour extraction. Common methods include edge detection (such as the Canny algorithm) or using gradient information in the image to detect edges. The extracted contour can help determine the specific location and shape of the contact area; the center position of the contact area refers to the coordinate representing the center of the contact area in the extracted contour. Generally speaking, the center position of the contact area can be obtained by calculating the geometric center (i.e., center of mass) of the contour. This center position may be an important reference point for subsequent analysis (such as tactile skin deformation calculation); the calibration sphere is a sphere of known size, commonly used for measurement and calibration systems. In this case, the actual radius of the calibration sphere refers to the physical size of the sphere. This known value can help convert pixel values ​​in the image into actual physical units (such as millimeters or centimeters). In this way, the dimensions in the image can be associated with real-world physical dimensions; the surface gradient refers to the rate of change on the surface of an object. In a contact image, the surface gradient describes the rate of change of skin deformation in the contact area. This gradient can help infer the direction and intensity of the force on the skin. The calculation of the surface gradient is usually related to the edge or color change of the image and is usually achieved by solving the derivative or gradient of the image; based on the actual radius of the calibration sphere, the pixel size of the contact area in the image can be converted into actual physical units.The known radius of the calibration sphere allows for scaling, aligning the pixel spacing in the image with the actual physical spacing, thus providing a true physical measurement of the surface gradient of skin deformation.

[0091] In an optional embodiment, obtaining a depth image corresponding to the contact image further includes:

[0092] B3. Establish a mapping relationship between the surface gradient and the pixel intensity at the point, and create a mapping table with the pixel value as the index;

[0093] B4. traverse the pixel intensity of each pixel in the difference image and substitute the pixel intensity into the index address of the mapping table to obtain the corresponding surface gradient;

[0094] B5. Calculate the depth information of the surface of the object to be operated from the surface gradient based on solving the Poisson equation, and obtain a depth image corresponding to the contact image based on the depth information.

[0095] It should be noted that a mapping table is a data structure used to establish a relationship between two variables. A mapping table uses pixel intensity (grayscale value) as an index to find the corresponding surface gradient value. A mapping table is actually a lookup table, in which each pixel intensity has a corresponding surface gradient value. This mapping relationship helps to infer the actual physical gradient based on the brightness or change of the image. The Poisson equation is a common type of partial differential equation, often used to describe potential field problems in physics. For the calculation of surface deformation, the Poisson equation can help infer depth information from known surface gradient information. The surface gradient generally reflects the rate of change of the local surface, and the Poisson equation is used to recover the depth or height information of the object from this gradient field. Depth information describes the position or shape of the object surface in three-dimensional space. In a contact image, depth information reflects the deformation of the object surface after being subjected to an external force. Depth information is very important for analyzing tactile feedback and surface pressure distribution. By calculating the depth image, the deformation of the object surface during the contact process can be obtained. A depth image is an image in which the value of each pixel represents the depth or distance of a point on the object surface.

[0096] In an optional embodiment, calculating the offset coefficient between the end effector and the object to be operated based on the depth image includes:

[0097] C1. Perform regional screening on the depth image based on a preset depth threshold to obtain a valid contact area and obtain a distribution array of the valid contact area in the camera coordinate system;

[0098] C2. Calculate the eigenvalues ​​and eigenvectors of the covariance matrix of the distribution array, sort the eigenvalues, and take the eigenvector corresponding to the largest eigenvalue. Calculate the center point of the distribution array.

[0099] In an optional embodiment, calculating the offset coefficient between the end effector and the object to be operated based on the depth image further includes:

[0100] C1. Calculate the offset angle between the eigenvector corresponding to the largest eigenvalue and the Y-axis of the camera coordinate system, and calculate the offset distance between the center point of the effective contact area and the center point of the camera coordinate system. The offset coefficient includes the offset angle and the offset distance.

[0101] Example 3, as Figure 2 As shown, the present invention proposes a tactile sensor-based robot flexible grasping method system, which is applicable to the tactile sensor-based robot flexible grasping method, including:

[0102] An image acquisition unit 1 is used to acquire an image of the environment in which the object to be operated is located based on a camera integrated in the robot, so as to obtain an image of the environment in which the object to be operated is located;

[0103] The pose estimation unit 2 is used to obtain the real-time pose of the robot's end effector based on the inertial measurement device integrated in the robot, and to estimate the pose of the object to be operated in the environment image based on a pre-trained pose estimation model to obtain the pose of the object to be operated;

[0104] The first posture control unit 3 is used to construct a kinematic function of the end effector of the robot, with the goal of minimizing the error between the real-time posture of the end effector and the posture of the object to be operated, and calculate the first operating parameter of the end effector according to the kinematic function;

[0105] a tactile sensing unit 4, which is used to control the end effector to contact the object to be operated based on the first operating parameter of the end effector, and to perform tactile sensing on the object to be operated based on a tactile sensor integrated in the end effector to obtain a contact image;

[0106] An offset calculation unit 5 is used to obtain a depth image corresponding to the contact image and calculate an offset coefficient between the end effector and the object to be operated based on the depth image;

[0107] The second posture control unit 6 is configured to calculate a second operating parameter of the end effector according to a kinematic function based on an offset coefficient between the end effector and the object to be operated.

[0108] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A robot flexible grasping method based on tactile sensor, characterized in that: include: The camera integrated in the robot collects images of the environment where the object to be operated is located, so as to obtain an image of the environment where the object to be operated is located; Acquiring a real-time pose of an end effector of the robot based on an inertial measurement device integrated in the robot, and estimating the pose of an object to be operated in the environment image based on a pre-trained pose estimation model to obtain the pose of the object to be operated; Constructing a kinematic function of the end effector of the robot, minimizing the error between the real-time posture of the end effector and the posture of the object to be operated, and calculating a first operating parameter of the end effector according to the kinematic function; controlling the end effector to contact the object to be operated based on a first operating parameter of the end effector, and performing tactile perception of the object to be operated based on a tactile sensor integrated in the end effector to obtain a contact image; Acquiring a depth image corresponding to the contact image, and calculating an offset coefficient between the end effector and the object to be operated based on the depth image; calculating a second operating parameter of the end effector according to the kinematic function based on an offset coefficient between the end effector and the object to be operated, and performing flexible grasping on the object to be operated based on the second operating parameter; The pose estimation model adopts an improved YOLOv3 network, and improves the backbone network of the YOLOv3 network based on a hybrid attention model; The hybrid attention model includes an attention module and an aggregation model, wherein the attention module includes a coordinate information unit, a channel information unit, and a spatial attention unit, wherein the expression of the coordinate information unit is as follows: ； in, Represents the feature map output by the coordinate information unit, represents the convolution operation, Indicates channel cascade operation, represents the first input feature map, and Represents the coordinate information of the X and Y axis channels in the coordinate information unit; The expression of the channel information unit is as follows: ； in, Represents the feature map output by the channel information unit, represents the Sigmoid activation function, and Represents the weight vector and bias vector of the channel information unit, and Indicates the height and width of the feature map; The expression of the spatial attention unit is as follows: ； in, The feature map representing the output of the spatial attention unit, represents group normalization, and represents the weight vector and bias vector of the spatial attention unit, represents the second input feature map; Constructing the kinematic function of the end effector of the robot, including: A constraint function for the robot's end effector around the instantaneous rotation center on the reference plane is established, wherein the constraint function is expressed as follows: ； in, represents the constraint function, represents the pitch angle of the end effector, represents the right shoulder joint of the end effector, and represents the rotational linear velocity of the left and right shoulder joints of the end effector, and Indicates the forward and reverse movement directions of the end effector, represents the rotational velocity of the reference plane, represents the linear velocity of the rotation center, and , represents the cosine function, represents the action angle, and Represents the left and right shoulder joints of the end effector.

2. The robot flexible grasping method based on tactile sensor according to claim 1 is characterized in that: Constructing the kinematic function of the end effector of the robot also includes: A feature vector of the end effector of the robot is obtained based on the constraint function, wherein the expression of the feature vector is as follows: ； in, represents the feature vector of the end effector; A kinematic function of the end effector of the robot is constructed based on the characteristic vector of the end effector, wherein the expression of the kinematic function is as follows: ； in, represents the kinematic function, Represents the process of translating from the origin of the world coordinate system to the current position of the robot. Represents the rotation angle around the Z axis in the new coordinate system after transformation, Represents the rotation angle around the Y axis in the new coordinate system after transformation, Indicates the equivalent roll angle around the X-axis in the transformed new coordinate system.

3. The robot flexible grasping method based on tactile sensor according to claim 2 is characterized in that: Acquiring a depth image corresponding to the contact image, comprising: performing image binarization on the contact image to obtain a background image corresponding to the contact image, and subtracting the contact image from the background image to obtain a difference image; The difference image is subjected to contour extraction to obtain the center position and radius of the contact area, and the surface gradient of the tactile skin deformation corresponding to each pixel position in the contact area is calculated based on the actual radius of the calibration sphere.

4. The robot flexible grasping method based on tactile sensor according to claim 3 is characterized in that: Acquiring a depth image corresponding to the contact image further includes: Establishing a mapping relationship between the surface gradient and the pixel intensity at the point, and establishing a mapping table with the pixel value as an index; Traversing the pixel intensity of each pixel in the difference image, substituting the pixel intensity into the index address of the mapping table to obtain the corresponding surface gradient; Depth information of the surface of the object to be operated is calculated from the surface gradient based on solving the Poisson equation, and a depth image corresponding to the contact image is acquired based on the depth information.

5. The robot flexible grasping method based on tactile sensor according to claim 4 is characterized in that: Calculating an offset coefficient between the end effector and the object to be operated based on the depth image includes: Performing regional screening on the depth image based on a preset depth threshold to obtain a valid contact area, and acquiring a distribution array of the valid contact area in a camera coordinate system; The eigenvalues ​​and eigenvectors of the covariance matrix of the distribution array are calculated, the eigenvalues ​​are sorted, and the eigenvector corresponding to the largest eigenvalue is obtained to calculate the center point of the distribution array.

6. The robot flexible grasping method based on tactile sensor according to claim 5, characterized in that: Calculating an offset coefficient between the end effector and the object to be operated based on the depth image further includes: Calculate the offset angle between the eigenvector corresponding to the largest eigenvalue and the Y axis of the camera coordinate system, and calculate the offset distance between the center point of the effective contact area and the center point of the camera coordinate system, wherein the offset coefficient includes the offset angle and the offset distance.

7. A tactile sensor-based robot flexible grasping system, which is applicable to the tactile sensor-based robot flexible grasping method according to any one of claims 1 to 6, characterized in that: include: An image acquisition unit (1), the image acquisition unit (1) being used to acquire an image of the environment in which the object to be operated is located based on a camera integrated in the robot, so as to obtain an image of the environment in which the object to be operated is located; A posture estimation unit (2), the posture estimation unit (2) is used to obtain the real-time posture of the end effector of the robot based on an inertial measurement device integrated in the robot, and to estimate the posture of the object to be operated in the environment image based on a pre-trained posture estimation model to obtain the posture of the object to be operated; A first posture control unit (3), the first posture control unit (3) is used to construct a kinematic function of the end effector of the robot, taking the error between the real-time posture of the end effector and the posture of the object to be operated as the minimum, and calculating a first operating parameter of the end effector according to the kinematic function; A tactile perception unit (4), the tactile perception unit (4) being used to control the end effector to contact the object to be operated based on a first operating parameter of the end effector, and to perform tactile perception of the object to be operated based on a tactile sensor integrated in the end effector to obtain a contact image; An offset calculation unit (5), the offset calculation unit (5) is used to obtain a depth image corresponding to the contact image, and calculate an offset coefficient between the end effector and the object to be operated based on the depth image; A second posture control unit (6) is used to calculate a second operating parameter of the end effector according to the kinematic function based on an offset coefficient between the end effector and the object to be operated, and to perform flexible grasping of the object to be operated based on the second operating parameter.

Images