Tactile fingertip sensor and its operating system and data processing method

Abstract

The present invention relates to the field of tactile sensor technology, and in particular to a tactile fingertip sensor and its operating system and data processing method. The tactile fingertip sensor includes a tactile skin, a marker array sticker and a monocular camera. The marker array sticker is pasted on the inner side of the tactile skin. The tactile skin and the bottom of the tactile fingertip sensor form a hollow cavity. The monocular camera is installed at the bottom of the tactile fingertip sensor and is located in the hollow cavity. The field of view of the monocular camera covers the inner side of the tactile skin. The tactile fingertip sensor of the present invention, when a mechanical finger wearing the sensor touches an object, the sensor captures an internal marker array image through a monocular camera, inputs it to a host computer for image processing and data analysis, and finally converts it into a variety of tactile information, including contact surface deformation, the magnitude and direction of pressure at each marker, contact torque, etc. Such a design can not only sense contacts in different directions and provide distributed tactile information, but also ensure that the sensor structure is compact and lightweight.

Description

Technical Field

[0001] The present invention relates to the technical field of tactile sensors, and in particular to a tactile fingertip sensor and an operating system and a data processing method thereof. Background Art

[0002] Tactile perception is an important part of robots' acquisition of external information. Giving robots a sense of touch is of great significance for the study of robot's operating skills. Many different tactile sensing and perception systems have been designed at home and abroad for robot perception and manipulation tasks, such as object feature recognition, slip detection, grasping stability assessment, and dexterous operation of manipulators. According to the principle and shape of the sensor, tactile sensors used on fingertips can be roughly divided into three categories: tactile arrays, bionic fingertips, and visual tactile sensors. Due to limited spatial resolution, tactile arrays and bionic fingertips are difficult to achieve high-precision positioning of small objects; while visual tactile sensors can obtain high-resolution tactile information.

[0003] Visual tactile sensors are important tools for flexible grasping and dexterous manipulation. They can provide information such as the interaction force between mechanical fingers and objects, as well as contact surface characteristics and deformation. Among them, tactile sensors that apply optical principles, such as sensors based on "marker detection", can obtain rich and specific tactile information. Moreover, this type of sensor has a transparent elastic shell with known marks embedded inside, which can convert "physical contact" into "marker displacement" for detection. However, existing visual tactile sensors still have the following shortcomings:

[0004] First, binocular cameras are usually used to obtain depth information of marking points based on the principle of parallax. However, the duplex structure of binocular cameras will cause the sensor to be bulky and overloaded, making it inconvenient to install the sensor on the mechanical finger, reducing operational flexibility. Second, such sensors are generally designed as planar structures, such as the GelSight sensor. The sensor with a planar structure has a limited sensing area and can only sense contact in a single direction. Moreover, this structure may affect the grasping operation of the mechanical finger. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a tactile fingertip sensor and its operating system and data processing method, which solves the problem that the duplex structure of the binocular camera causes the sensor to be bulky and overloaded, making it inconvenient to install the sensor on the mechanical finger and reducing the operational flexibility, and also solves the problem that the sensor with a planar structure has a limited sensing area and can only sense contact in a single direction.

[0006] The present invention solves the above technical problems by the following technical means:

[0007] One aspect of the present invention is to provide a tactile fingertip sensor, comprising a tactile skin, a marker array sticker and a monocular camera, wherein the marker array sticker is pasted on the inner side of the tactile skin, the tactile skin and the bottom of the tactile fingertip sensor form a hollow cavity, the monocular camera is installed at the bottom of the tactile fingertip sensor and is located in the hollow cavity, and the field of view of the monocular camera covers the inner side of the tactile skin.

[0008] Furthermore, the tactile skin is formed into a shell-like structure having a hemisphere and a cylinder, the hemisphere and the cylinder are formed as one piece, and the monocular camera is installed on a side of the cylinder away from the hemisphere.

[0009] Furthermore, a plurality of ArUco marks are printed on the mark array sticker, and the plurality of ArUco marks are distributed in an array, each ArUco mark has a side length of 2 mm, and a distance of 1 mm between two adjacent ArUco marks.

[0010] Another aspect of the present invention is to provide an operating system including the above-mentioned tactile fingertip sensor, which also includes a host computer. The tactile fingertip sensor is worn at the end of a mechanical finger used to grasp an object, and the tactile fingertip sensor and the mechanical finger are both connected to the host computer via a cable.

[0011] Another aspect of the present invention is to provide a data processing method based on the above operating system, comprising the following steps:

[0012] The monocular camera captures the tactile skin deformation image and transmits the deformation image to the host computer;

[0013] The ArUco code marker detection program runs inside the host computer. After a sufficient number of markers are detected, the 3D position and posture information of each marker point is saved.

[0014] The host computer runs data processing algorithms including marker displacement measurement, torque estimation and 3D modeling.

[0015] Further, the process of the ArUco code mark detection program is as follows:

[0016] Create a marker, select the predefined dictionary of the ArUco module, and generate a specific ArUco marker given the ArUco code tag ID and pixel size;

[0017] Image capture: Use a monocular camera to capture color images containing ArUco markers;

[0018] Adaptive threshold segmentation, converting color images into grayscale images, and performing threshold processing to extract contours and filter out non-square polygons;

[0019] Corner point detection: detect the positions of the four corner points of the ArUco markers in the image, store them as point vectors in the order of upper left, upper right, lower right, and lower left, and store multiple markers in the image together in a point vector container;

[0020] Pose detection, build the PnP mapping equation based on the monocular camera's internal parameters, camera distortion parameters and the actual physical side length of the ArUco marker, and call the solvePnP function to obtain the spatial coordinates and pose state of the ArUco marker in the camera coordinate system.

[0021] Further, the marker displacement is measured as follows:

[0022] Assume that marker i passes through the ArUco code marker detection program at time t, and the coordinates of the marker center point in the camera coordinate system are [x i (t),y i (t),z i (t)] T , the pixel coordinate is [u i (t), v i (t)] T According to the camera hand-eye calibration, the conversion relationship between the camera coordinate system and the pixel coordinate system is as follows:

[0023]

[0024] Where f is the focal length of the camera, (u0, v0) is the offset of the origin of the image coordinate system relative to the upper left corner of the image, and z i (t) is the depth information obtained by detecting the ArUco code;

[0025] Calculate the change p of the marker position from the initial state i (t)-p i (0), or the change p of the subsequent frame i (t)-p i (t-Δt), the movement of the center point of the marker can be obtained.

[0026] Further, the torque is estimated as follows:

[0027] Estimate the contact force n. For each ArUco marker, calculate the initial state (x0, y0, z0) and the current state (x t ,y t ,z t ) coordinate offset (d x ,d y ,d z ) is used to estimate the contact force n as follows:

[0028] (d x ,d y,d z )=(x t -x0,y t -y0,z t -z0)

[0029] n=[c x d x ,c y d y ,c z d z ]

[0030] Calculate the torque τ around the center of the sensor sphere as follows:

[0031]

[0032] τ=c τ n×r

[0033] Among them, c x , c y , c z , c τ is the conversion coefficient, which represents the direction and weight parameter of each vector, and r refers to the distance from the force point to the rotation axis.

[0034] Further, the three-dimensional modeling is as follows:

[0035] Collect the spatial coordinate values ​​of each marker obtained in the ArUco code marker detection program step;

[0036] Split the spatial coordinate values ​​of each marker and store them in the x, y, and z arrays in turn to obtain three-dimensional values;

[0037] Draw a discrete point cloud based on the three-dimensional values;

[0038] Complete the second-order fitting of the z value to x and y to achieve three-dimensional reconstruction of the surface where the mark is located.

[0039] When the mechanical fingertip sensor of the present invention touches an object, the sensor uses a monocular camera to capture an image of the internal marker array, which is input to a host computer for image processing and data analysis, and finally converted into a variety of tactile information, including the deformation of the contact surface, the magnitude and direction of the pressure at each marker, contact torque, etc. This design can not only sense contacts in different directions and provide distributed tactile information, but also ensure that the sensor structure is compact and lightweight.

[0040] The present invention adopting the above technical solution has the following advantages:

[0041] (1) The optical method based on marker detection can effectively obtain the three-dimensional position and posture information of each point on the sensor surface, fully reflecting the deformation situation.

[0042] (2) In terms of structural design, the tactile fingertip sensor of the present invention adopts a strategy of using a monocular camera to detect a specific marker array instead of a binocular camera to obtain depth information, making the sensor structure more compact and lightweight (the total mass of the sensor is 3 grams, which is significantly lighter than that of an ordinary binocular camera), reducing the manufacturing cost, and facilitating sensor packaging and deployment on a mechanical finger.

[0043] (3) The touch fingertip sensor of the present invention simulates the way human fingers obtain touch and designs the sensor into the shape of a fingertip. Compared with a flat structure, this design is not only convenient for wearing mechanical fingers, but also expands the sensing area, so that contacts from different directions can be sensed. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic structural diagram of a tactile fingertip sensor of the present invention;

[0045] Figure 2 is the tactile skin model of the present invention;

[0046] Figure 3 is the internal image of the tactile skin of the tactile fingertip sensor of the present invention;

[0047] Figure 4 It is a schematic diagram of the structure of the operating system of the present invention;

[0048] Figure 5 is a schematic diagram of a camera coordinate system-pixel coordinate system in an embodiment of the present invention;

[0049] Figure 6 is a schematic diagram of torque calculation in an embodiment of the present invention;

[0050] Figure 7 is a schematic diagram of three-dimensional reconstruction steps in an embodiment of the present invention;

[0051] Figure 8 is a data processing flow chart of the operating system of the present invention;

[0052] Among them, there are tactile skin 1, marker array sticker 2, monocular camera 3, mechanical finger 4, tactile fingertip sensor 5, object 6, and host computer 7. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Example 1

[0055] This embodiment describes a tactile fingertip sensor, which includes a tactile skin 1, a marker array sticker 2, and a monocular camera 3. The tactile skin 1 is used to convert physical contact into visual information that can be captured by the camera. The markers in the marker array sticker 2 contain position information and posture information, which can be decoded and obtained by a computer. The monocular camera 3 is used to capture the internal image of the tactile skin 1. The marker array sticker 2 is attached to the inner side of the tactile skin 1. The tactile skin 1 and the bottom of the tactile fingertip sensor form a hollow cavity. The monocular camera 3 is installed at the bottom of the tactile fingertip sensor and is located in the hollow cavity. The field of view of the monocular camera 3 covers the inner side of the tactile skin 1.

[0056] The touch skin 1 is the outermost layer of the touch fingertip sensor. In order to fully reflect the deformation, it needs to be made of elastic material. In this embodiment, epoxy resin material is selected to prepare the touch skin 1. The touch skin 1 is formed into a shell structure with a hemisphere and a cylinder. The hemisphere and the cylinder are formed in one piece. In this embodiment, the inner diameter of the hemisphere is 8mm, the outer shell thickness is 2mm, and the cylinder height is 15mm. The model of the touch skin 1 is as follows: Figure 2 shown.

[0057] In this embodiment, the marker array sticker 2 is made of PVC material, and a number of ArUco markers are printed on the marker array sticker 2. The dictionary type is DICT_4*4_50. The number of ArUco markers are distributed in an array. The side length of each ArUco marker is 2 mm, and the interval between two adjacent ArUco markers is 1 mm. The internal image of the tactile skin 1 after the marker array sticker 2 is attached is as follows: Figure 3 shown.

[0058] In this embodiment, the monocular camera 3 is installed on the side of the cylinder away from the hemisphere. The monocular camera 3 has a lens object distance of 6 cm, a focal length of 1 cm, a lens diameter of 14 mm, and a field of view angle of 70°.

[0059] Example 2

[0060] This embodiment describes an operating system including the tactile fingertip sensor of embodiment 1. Figure 4 As shown, the operating system also includes a host computer 7, and the tactile fingertip sensor 5 is worn at the end of the mechanical finger 4 used to grasp the object 6. The tactile fingertip sensor 5 and the mechanical finger 4 are connected to the host computer 7 through a cable. When the mechanical finger 4 grasps the object 6, the elastic surface of the tactile skin 1 of the tactile fingertip sensor 5 will be deformed, causing the marker array to be displaced, and the monocular camera 3 transmits the captured image to the host computer 7 for image processing, estimates the force and deformation of the grasped object, and then controls the force applied by the mechanical finger 4. For example, if the applied force is judged to be too large, the control is reduced; if the applied force is judged to be too small, the control is increased.

[0061] Example 3

[0062] This embodiment describes the workflow of the operating system of Embodiment 2, that is, the data processing flow and method. The workflow is as follows: Figure 8 As shown, the details are as follows:

[0063] S1. The monocular camera 3 captures the deformation image of the tactile skin 1 and transmits the deformation image to the host computer 7.

[0064] When the mechanical finger 4 grasps an object, the tactile fingertip sensor 5 touches the object 6, and the elastic tactile skin 1 is deformed due to compression. The monocular camera 3 is used to capture the deformation image of the tactile skin 1, and the deformation image is transmitted to the host computer 7.

[0065] S2. The host computer 7 runs the ArUco code mark detection program internally, and after determining that a sufficient number of marks are detected, saves the three-dimensional position and posture information of each mark point.

[0066] The ArUco code consists of an equal-width black border and a specific binary matrix. The black border facilitates rapid detection of images and provides location information; the internal binary code is used to verify the label and enhance recognition robustness. By recognizing the ArUco marker, the precise three-dimensional position, posture direction and internal tag ID of the ArUco code can be obtained. The Aruco code can be recognized using only a monocular camera 3 to obtain the projection relationship between the two-dimensional plane and the three-dimensional space. Therefore, it can replace the binocular camera or infrared camera to obtain the depth information in the Z-axis direction. The ArUco marker array is arranged inside the tactile sensor, and the monocular camera 3 is used to capture images to obtain posture information at different points, thereby quantifying and analyzing deformation and realizing tactile perception.

[0067] The specific process of the ArUco marker detection procedure is as follows:

[0068] S21. Create a marker. First, select the predefined dictionary of the ArUco module. The dictionary determines the size, encoding content, etc. of the created ArUco marker. Then, the ArUco code tag ID and pixel size are given to generate a specific ArUco marker.

[0069] S22. Image capture: Use the monocular camera 3 to capture a color image containing the ArUco mark.

[0070] S23. Adaptive threshold segmentation, after converting the color image into a grayscale image, threshold processing is performed to extract the shape contour, filter out non-square polygons, and use additional filtering to reduce the interference of random noise and improve computational efficiency.

[0071] S24. Corner point detection, detect the positions of the four corner points of the ArUco mark in the image, and store them as point vectors in the order of upper left, upper right, lower right, and lower left. Store multiple marks in the image together in a point vector container. According to the positions of the four corner points, the ArUco code border can be drawn and the center coordinates of the mark can be obtained.

[0072] S25. Posture detection, construct the PnP (perspective-n-points) mapping equation based on the camera internal parameters, camera distortion parameters and the actual physical side length of the ArUco code, and call the solvePnP function in the OpenCV library to solve the spatial coordinates and posture state of the ArUco marker in the camera coordinate system, represented by the rotation and translation vector. For the convenience of observation and visualization, the X, Y, and Z axes corresponding to the marker posture are drawn with red, green, and blue line segments respectively. After obtaining the rotation vector, it can be converted into a rotation matrix, and then the Euler angle is obtained to reflect the rotation angle of each axis relative to the camera coordinate system.

[0073] S3. The host computer 7 runs data processing algorithms including marker displacement measurement, moment estimation and three-dimensional modeling.

[0074] After the ArUco code mark detection program is run inside the host computer 7, the corresponding mark displacement measurement, moment estimation and three-dimensional modeling data processing algorithms are executed, as follows:

[0075] S31. Marker displacement measurement. When contact occurs, the elastic tactile skin 1 is deformed, causing the embedded marker to move. Assuming that marker i passes through the ArUco code marker detection program at time t, the coordinates of the marker center point in the camera coordinate system are [x i (t),y i (t),z i (t)] T , the pixel coordinate is [u i (t),v i (t)] T , the schematic diagram of the geometric relationship between the camera coordinate system and the pixel coordinate system is shown in Figure 5. According to the camera hand-eye calibration, the conversion relationship between the camera coordinate system and the pixel coordinate system is as follows:

[0076]

[0077] Where f is the focal length of the camera, (u0, v0) is the offset of the origin of the image coordinate system relative to the upper left corner of the image, and z i (t) is the depth information obtained by detecting the ArUco code;

[0078] Calculate the change p of the marker position from the initial state i (t)-p i(0), or the change p of the subsequent frame i (t)-p i (t-Δt), the movement of the center point of the marker can be obtained.

[0079] S32. Torque estimation: Applying different forces on the tactile fingertip sensor 5 will cause different degrees of deformation. After multiple experiments, the nonlinear relationship between the marker coordinate offset and the applied force can be obtained. For each ArUco marker, calculate the initial state (x0, y0, z0) and the current state (x t ,y t ,z t ) coordinate offset (d x ,d y ,d z ) is used to estimate the contact force n as follows:

[0080] (d x ,d y ,d z )=(x t -x0,y t -y0,z t -z0)

[0081] n=[c x d x ,c y d y ,c z d z ]

[0082] Calculate the torque τ around the center of the sensor. The torque calculation diagram is as follows: Figure 6 As shown, the details are as follows:

[0083]

[0084] τ=c τ n×r

[0085] Among them, c x , c y , c z , c τ is the conversion coefficient, which represents the direction and weight parameter of each vector. r refers to the distance from the force point to the rotation axis. The magnitude of the torque is equal to the product of the tangential component of the contact force n and the distance from the force to the center of the ball.

[0086] S33. Three-dimensional modeling, discrete marker points can be equivalent to point cloud data. According to the position information of each point, approximate three-dimensional reconstruction can be achieved to visualize the deformation of the elastic shell. Figure 7 Schematic diagram of the three-dimensional modeling steps.

[0087] The steps of 3D modeling are as follows:

[0088] S331. Collect the spatial coordinate values ​​of each marker obtained in the ArUco code marker detection program step, use the monocular camera 3 to capture the internal image of the lower hemisphere shell, perform ArUco code marker detection, obtain the position and posture information of each marker, and represent its posture with a three-color coordinate axis.

[0089] S332. Split the spatial coordinate values ​​of each marker and store them into the x, y, and z arrays in sequence to obtain three-dimensional values.

[0090] S333. Draw a discrete point cloud based on the three-dimensional numerical values.

[0091] S334. Call the plot_surface function in the Matplotlib drawing library to complete the second-order fitting of the z value to x, y, and draw an approximate three-dimensional surface of the surface where the mark is located based on the discrete point cloud, so as to fully reflect the deformation of the mark.

[0092] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should be included in the scope of the claims of the present invention. The techniques, shapes, and structural parts not described in detail in the present invention are all known technologies.

Claims

1. A data processing method based on an operating system, characterized in that: The operating system includes a tactile fingertip sensor and a host computer, wherein the tactile fingertip sensor is worn at the end of a mechanical finger used to grasp an object, and the tactile fingertip sensor and the mechanical finger are both connected to the host computer via a cable; The tactile fingertip sensor comprises a tactile skin, a marker array sticker and a monocular camera, wherein the marker array sticker is attached to the inner side of the tactile skin, the tactile skin and the bottom of the tactile fingertip sensor form a hollow cavity, the monocular camera is installed at the bottom of the tactile fingertip sensor and is located in the hollow cavity, and the field of view of the monocular camera covers the inner side of the tactile skin; The tactile skin is formed into a shell-like structure having a hemisphere and a cylinder, the hemisphere and the cylinder are formed in one piece, and the monocular camera is mounted on a side of the cylinder away from the hemisphere; The marker array sticker is printed with a plurality of ArUco markers, the plurality of ArUco markers are distributed in an array, each ArUco marker has a side length of 2 mm, and a distance of 1 mm between two adjacent ArUco markers; The method comprises the following steps: The monocular camera captures the tactile skin deformation image and transmits the deformation image to the host computer; The ArUco code marker detection program runs inside the host computer. After a sufficient number of markers are detected, the 3D position and posture information of each marker point is saved. The host computer runs data processing algorithms including marker displacement measurement, moment estimation, and 3D modeling; The process of the ArUco code mark detection program is as follows: Create a marker, select the predefined dictionary of the ArUco module, give the ArUco code tag ID and pixel size, and generate the ArUco marker; Image capture: Use a monocular camera to capture color images containing ArUco markers; Adaptive threshold segmentation, converting color images into grayscale images, and performing threshold processing to extract contours and filter out non-square polygons; Corner point detection: detect the positions of the four corner points of the ArUco markers in the image, store them as point vectors in the order of upper left, upper right, lower right, and lower left, and store multiple markers in the image together in a point vector container; Pose detection: construct the PnP mapping equation based on the internal parameters of the monocular camera, the camera distortion parameters and the actual physical side length of the ArUco marker, and call the solvePnP function to obtain the spatial coordinates and pose state of the ArUco marker in the camera coordinate system; The moment is estimated as follows: Estimate the contact force n. For each ArUco marker, calculate the initial state (x0, y0, z0) and the current state (x t ,y t ,z t ) coordinate offset (d x ,d y ,d z ) is used to estimate the contact force n as follows: (d x ,d y ,d z )=(x t -x0,y t -y0,z t -z0) n=[c x d x ,c y d y ,c z d z ] Calculate the torque τ around the center of the sensor sphere as follows: τ=c τ n×r Among them, c x , c y , c z , c τ is the conversion coefficient, which represents the direction and weight parameter of each vector, r refers to the distance from the force point to the rotation axis, and h represents the z-axis coordinate difference from the center of the hemisphere of the tactile skin of the tactile fingertip sensor to the horizontal plane of the coordinate axis.

2. The data processing method according to claim 1, characterized in that: The marker displacement is measured as follows: Assume that marker i passes through the ArUco code marker detection program at time t, and the coordinates of the marker center point in the camera coordinate system are [x i (t),y i (t),z i (t)] T , the pixel coordinate is [u i (t),v i (t)] T According to the camera hand-eye calibration, the conversion relationship between the camera coordinate system and the pixel coordinate system is as follows: Where f is the focal length of the camera, (u0, v0) is the offset of the origin of the image coordinate system relative to the upper left corner of the image, and z i (t) is the depth information obtained by detecting the ArUco code; Calculate the change p of the marker position from the initial state i (t)-p i (0), or the change p of the subsequent frame i (t)-p i (t-Δt), the movement of the center point of the marker can be obtained.

3. The data processing method according to claim 1, characterized in that: The three-dimensional modeling is as follows: Collect the spatial coordinate values ​​of each marker obtained in the ArUco code marker detection program step; Split the spatial coordinate values ​​of each marker and store them in the x, y, and z arrays in turn to obtain three-dimensional values; Draw a discrete point cloud based on the three-dimensional values; Complete the second-order fitting of the z value to x and y to achieve three-dimensional reconstruction of the surface where the mark is located.

Images