Flexible visual-haptic finger, hand, and sensing method based on fin effect structure

Abstract

The flexible visual tactile finger based on the fin effect structure, the hand claw and the sensing method provided by the disclosure, the visual tactile finger comprises a finger body, a finger body base and an image acquisition unit, the finger body is a fin effect structure made of a flexible transparent material, a flexible semi-transparent sensing layer is arranged on a surface of the finger body opposite to an object, an array of mark points is arranged on two side surfaces of the finger body respectively and connected with the sensing layer, a flexible light shielding layer is arranged on the surface of the sensing layer and the remaining side surfaces of the finger body, and a camera in the image acquisition unit is located directly below the side of the finger body provided with the sensing layer. The hand claw comprises a plurality of the visual tactile fingers and a driving mechanism for driving each finger. The sensing method comprises sequentially performing overall deformation reconstruction, local deformation reconstruction and complete deformation reconstruction of a contact surface of the finger. The disclosure overcomes the problem of poor tactile feedback capability of the existing flexible hand claw, has low illumination requirements, simple structure, convenient manufacturing, is suitable for multiple application scenarios and has high application value.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of flexible manipulator and sensor, and particularly proposes a flexible visual-tactile manipulator based on fin ray effect structure and a sensing method thereof. BACKGROUND

[0002] Manipulator is one of the core technologies of intelligent robot industry, and is widely used to replace human labor to perform sorting, digging and other repetitive or dangerous work, which significantly improves production efficiency and safety. However, the rigid structure of the traditional manipulator is easy to cause damage to the workpiece and the environment during operation, which poses a safety hazard to production. Flexible gripper is a kind of manipulator made of flexible material or having flexible structure, which can effectively buffer the impact caused by accidental touch and has higher application value in environmental adaptability. Tactile sensor is a sensor that can feedback the position, force and surface morphology of contact and other tactile information, which can allow the manipulator to adjust the working state in real time, and has become an indispensable part of realizing feedback control of manipulator.

[0003] The gripper of the finger based on fin ray effect (fish fin hand) is a kind of bionic structure flexible gripper with high flexibility, which has been widely applied to the field of object grasping and manipulation. However, this kind of gripper allows significant deformation during work, and has high requirements for structural design when integrated with common contact type tactile sensor, which limits the popularization and application of fish fin hand. Among various types of tactile sensors, visual-based tactile sensor has high resolution, simple principle and structure, and the sensing elements are independent of contact. Moreover, thanks to the development of deep learning technology, the sensing ability has been greatly improved, and it has broad application prospects in the field of medical image processing and other fields, and has great potential in combination with flexible gripper. The combination of fish fin hand and visual-tactile sensor can enable the gripper to adjust the working state in real time according to the tactile feedback, and realize more stable grasping. However, most of the existing fish fin hands lack tactile perception function, or can only perceive contact in specific areas, so it is urgent to design a fish fin hand with more comprehensive tactile perception ability. SUMMARY

[0004] The present disclosure aims to at least partially solve one of the technical problems in the related art.

[0005] To this end, the present disclosure proposes a flexible visual-tactile finger based on fin ray effect structure, a gripper and a sensing method. The flexible gripper proposed by the present application has omnidirectional tactile sensing ability, and simplifies the structure design and manufacturing method. It has low illumination requirements, is easy to manufacture, is suitable for multiple scenarios, and has high application value.

[0006] In order to achieve the above purpose, the present disclosure adopts the following technical solutions:

[0007] The first aspect of the present disclosure provides a flexible visual-tactile finger based on a fin effect structure, comprising:

[0008] a finger body, which is a fin effect structure made of a flexible transparent material, and is provided with a flexible semi-transparent sensing layer on the surface of the side of the finger body facing an object, an array of marker points on each of the two side surfaces of the finger body adjacent to the sensing layer, and a flexible light shielding layer on the surface of the sensing layer and the remaining side surfaces of the finger body;

[0009] a finger base connected to the root of the finger body, wherein the finger base has a transparent bottom plate;

[0010] an image acquisition unit, which comprises a light-tight shell fixed to the bottom of the finger base, and a light source and a camera arranged in the shell, wherein the camera is located directly below the side of the finger body provided with the sensing layer, and the light source is arranged around the lens of the camera.

[0011] In some embodiments, the wall thickness of the finger body on the side provided with the sensing layer gradually increases as the position approaches the finger root.

[0012] In some embodiments, the sensing layer is made of a soft material with a hardness of 5A.

[0013] In some embodiments, the marker points in the array of marker points follow the rule that the radius and spacing of the marker points increase in proportion to the distance from the camera, so as to ensure that the spacing and size of each marker point in the image are close to consistent.

[0014] In some embodiments, the color of the light source is single white, and a isolation ring is further installed between the camera and the light source to prevent the appearance of light spots in the image captured by the camera.

[0015] The second aspect of the present disclosure provides a flexible hand claw, comprising:

[0016] a plurality of fingers, wherein each finger is a visual-tactile finger according to any one of the embodiments of the first aspect of the present disclosure;

[0017] a driving mechanism connected to the shell of each finger, for providing driving force to each visual-tactile finger to realize the gripping and releasing of the object.

[0018] A sensing method based on the visual-tactile finger according to any one of the embodiments of the first aspect of the present disclosure, comprising:

[0019] Step 1) overall deformation reconstruction of the contact surface:

[0020] The light-shielding layer deformed by contact with the object is defined as a contact surface, a real-time image of the object pressing the visual-tactile finger surface is acquired by using the camera, the three-dimensional space coordinates of the overall deformation of the contact surface without local deformation are obtained by combining the non-distortion pinhole camera model, the geometric constraint that the contact surface is of equal width at each point when the object presses the visual-tactile finger, and the tracking of the marker point features in the real-time image, the reconstruction of the overall deformation of the contact surface is realized, and the contact surface without local deformation is defined as an unlocally deformed surface;

[0021] Step 2) Reconstruction of local deformation of the contact surface:

[0022] Step 2-1) Record the images of the contact surface deformed in as many different ways as possible as a reference video by using the camera, so as to track the positions of the marker points under various deformations;

[0023] Step 2-2) Press a calibration ball with a known radius to the contact surface, obtain a calibration image by using the camera, and select a frame from the reference video that is closest to the position of the marker point in the calibration image as a calibration reference image, calculate the normalized brightness difference between the calibration image and the calibration reference image, obtain the center position of the calibration ball in the calibration image according to the shadow area generated by pressing the contact surface by the calibration ball, obtain the local deformation three-dimensional space coordinates corresponding to the deformation of the shadow area by using the center position and the non-distortion pinhole camera model, calculate the pressing depth corresponding to each point in the shadow area by using the coordinates and the three-dimensional space coordinates of the overall deformation of the contact surface corresponding to the shadow area obtained in step 1), and fit the mapping between the normalized brightness difference and the pressing depth by using the pressing depth corresponding to each point in the shadow area and the normalized brightness difference, so as to complete the calibration;

[0024] Step 2-3) Select a frame from the reference video that is closest to the position of the marker point in the real-time image as a real-time reference image, calculate the normalized brightness difference between the real-time reference image and the real-time image, obtain the real-time pressing depth of each point of the contact surface by using the mapping, and realize the reconstruction of the local deformation of the contact surface;

[0025] Step 3) Reconstruction of complete deformation of the contact surface:

[0026] Subtract the real-time pressing depth of each point of the contact surface from the three-dimensional space coordinates of the overall deformation of the contact surface to obtain the complete deformation of each point of the contact surface, and realize the reconstruction of the complete deformation of the contact surface.

[0027] In some embodiments, step 1) specifically comprises the following steps:

[0028] 1-1) Define a non-distortion pinhole camera model, and the expression is as follows:

[0029] zp′=Kp

[0030] in This is the camera intrinsic parameter matrix; p′ and p are the pixel coordinates of a pixel in a frame of an image and its corresponding 3D coordinates in the camera coordinate system, respectively, and are defined as follows:

[0031]

[0032] The image's u-axis is parallel to the camera's x-axis, the image's v-axis is parallel to the camera's y-axis, the origin of the camera's coordinate system is set at the camera's optical center, and the z-axis is defined to point downwards.

[0033] 1-2) Assemble the visual-tactile finger according to the alignment of the finger width direction with the v-axis and y-axis, and use a camera to acquire real-time images of the visual-tactile finger when an object presses on it; define the interface between the finger body and the sensing layer as the observation surface, and assume that the following geometric constraints are satisfied when the finger undergoes overall deformation: In the camera coordinate system, the distance between points p1 and p2 at the same height on the left and right edges of the observation surface is always equal to the width E of the finger, therefore:

[0034] ||p1-p2||2=E

[0035] z1=z2

[0036] Let p′1 and p′2 be the two pixels in the real-time image corresponding to points p1 and p2, respectively. Pixels p′1 and p′2 satisfy the following:

[0037] u1 = u2

[0038] Where z1 and z2 are the z-axis coordinates of points p1 and p2, respectively, and u1 and u2 are the u-axis coordinates of pixels p′1 and p′2, respectively.

[0039] 1-3) Trace the left and right edges of the observation surface in the real-time image, obtain the three-dimensional spatial coordinates of all pixels on the left and right edges of the observation surface using the expression described in step 1-2), and obtain the three-dimensional spatial coordinates p of any pixel on the observation surface using linear interpolation. VS ;

[0040] 1-4) For the three-dimensional spatial coordinate scale p VS By performing polynomial fitting and differentiation, the tangent inclination angle θ at each point is obtained. The three-dimensional spatial coordinates P of each pixel on the overall deformable contact surface without local deformation are then obtained according to the following formula. LUS :

[0041]

[0042]

[0043] where t is the initial thickness of the sensing layer, t0 is the equivalent thickness of the sensing layer along the optical axis of the camera at each point of the left and right edges of the observation plane. v where t is the initial thickness of the sensing layer, t0 is the equivalent thickness of the sensing layer along the optical axis of the camera at each point of the left and right edges of the observation plane.

[0044] In some embodiments, step 2-2) specifically comprises the following steps:

[0045] 2-2-1) Press a calibration ball with a known diameter R onto the contact surface to cause the finger to deform globally and locally, and obtain a calibration image;

[0046] 2-2-2) Compare the positions of the marker points in each frame of the reference video with the positions of the marker points in the calibration image, and find the frame closest to the calibration image as the calibration reference image, the comparison method being to calculate the sum of distances of the marker points in the calibration image and each frame of the reference video Take the frame of the reference video with the smallest sum of distances as the calibration reference image, The formula is expressed as:

[0047]

[0048] where m' and m are the pixel coordinates of the marker points in the calibration image and the i-th frame of the reference video, respectively; C and respectively, are the pixel coordinates of the marker points in the calibration image and the i-th frame of the reference video, respectively;

[0049] 2-2-3) Calculate the normalized brightness difference according to the following formula:

[0050]

[0051] where I and I are the brightness of the calibration image and the calibration reference image, respectively; C and I refC where I and I are the brightness of the calibration image and the calibration reference image, respectively;

[0052] 2-2-4) Circle the circular contour of the shadow area generated by the calibration ball pressed on the contact surface in the calibration image, and mark the pixel coordinates c r ' and the pixel coordinates p r ' of a point on the circle, the three-dimensional distance between the two points is approximately the radius R of the calibration ball, so:

[0053] ||p r -c r ||2=R

[0054] z r =z cr

[0055] where c r is the pixel coordinates of the center of the circle corresponding to c rThe three-dimensional spatial coordinates of the point corresponding to ′ in the camera coordinate system, i.e., the position of the sphere's center, p r To be with p r The three-dimensional spatial coordinates of the point corresponding to ′ in the camera coordinate system, z r and z cr p r and c r z-axis coordinate;

[0056] By combining the above formula and the expression of the distortion-free pinhole camera model, the approximate location of the sphere's center can be obtained;

[0057] 2-2-5) Update the approximate position of the ball's center according to the following formula:

[0058]

[0059] Among them, L c =||c r ||2 is the distance between the center of the sphere and the optical center of the camera, and c is the exact position of the center of the sphere;

[0060] 2-2-6) Using the accurate position c of the sphere's center and the distortion-free pinhole camera model, calculate the three-dimensional spatial coordinates p of each pixel in the shadow area after deformation. D Referring to the method described in step 1), the three-dimensional spatial coordinates p of the point located on the undeformed surface corresponding to the shaded area are obtained. LUSD The pressing depth d at each point on the contact surface is obtained according to the following formula. D :

[0061] d D =z D -z LUS

[0062] Among them, z D and z LUS They are p D and p LUSD z-axis coordinate;

[0063] 2-2-7) with For independent variable pairs and d D Polynomial fitting was performed to obtain the normalized mapping M() between the brightness difference and the pressing depth;

[0064] Steps 2-3) specifically include the following steps:

[0065] The time-normalized luminance difference is obtained frame by frame from the real-time image by performing steps 2-2-2) to 2-2-3). The real-time pressure depth d is then obtained from the mapping M() to complete the local reconstruction:

[0066]

[0067] In some embodiments, the sensing method further comprises judging the force direction by tracking the marker points in real time and summing the displacement of the marker points from their initial positions, specifically comprising the following steps:

[0068] The displacement of the marker points from their initial positions is summed according to the following formula:

[0069] o' =∑(m' -m' * )

[0070] wherein, is the overall displacement of the marker points in the real-time image, m' is the pixel coordinate of the marker points in the real-time image, m' * is the initial pixel coordinate of the marker points in the image;

[0071] According to the set threshold value ε, it is judged whether the finger and the object are in contact, when ||o' ||2>ε, it is determined that the finger and the object are in contact once, and the force direction of the finger in the camera coordinate system is defined according to the following formula:

[0072]

[0073] wherein, o is a unit vector representing the force direction, o' u and o' v are the u-axis and v-axis coordinates of the marker points in the real-time image, respectively;

[0074] The finger side with the normal vector closest to o among all the sides of the finger is determined as the force receiving surface, and the method for judging the closeness is the vector inner product.

[0075] The features and advantages of the present disclosure are:

[0076] The present disclosure combines the Fin Ray fish fin hand with the visual tactile sensor, proposes a flexible finger with visual tactile perception ability and a hand claw with the finger, uses the visual tactile sensor to obtain higher resolution, simplifies the lighting configuration and manufacturing method, and overcomes the problem of lack of tactile feedback ability of the flexible finger and hand claw, and can realize feedback control of the hand claw.

[0077] The present disclosure also proposes a sensing method suitable for the proposed flexible finger, so that the finger can reconstruct the overall and local deformation of the contact surface and perceive the contact force, and realize the visual tactile function.

[0078] The present disclosure can be applied as an end effector of a mechanical arm to complete the work of flexible grasping, tactile sensing and feedback control, so that the mechanical hand claw can be better controlled and interacted with the environment. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1is a cross-sectional view of a flexible visual-haptic finger based on fin effect structure according to an embodiment of the present disclosure, wherein the finger body is not cut.

[0080] Figure 2 is a partial cross-sectional view of a flexible visual-haptic hand based on fin effect structure according to an embodiment of the present disclosure, wherein the flexible visual-haptic finger is cut along the screw axis, and the rest is half cut.

[0081] Figure 3 is a local deformation schematic diagram of a flexible visual-haptic finger based on fin effect structure according to an embodiment of the present disclosure.

[0082] Figure 4 is a whole flow chart of a flexible visual-haptic finger sensing method based on fin effect structure according to an embodiment of the present disclosure.

[0083] Figure 5 is a calibration schematic diagram of a visual-haptic finger based on fin effect structure according to an embodiment of the present disclosure.

[0084] In the figure:

[0085] 1, finger body, 11, perception layer, 12, light shielding layer, 13, marker point array, 2, finger body base, 21, transparent bottom plate, 3, shell, 4, isolation ring, 5, light source, 6, camera;

[0086] 10, flexible visual-haptic finger, 20, driving mechanism, 21, motor box, 22, motor, 23, lead screw, 24, sliding block, 25, connecting table, 26, connecting rod, 27, flange coupling.

[0087] 1a, observation surface, 1b, contact surface, 1c, non-local deformation surface, 1d, indentation depth, 1e, imaging thickness, 1f, equivalent thickness, 1g, perception layer thickness.

[0088] 1A, calibration ball, 1B, shadow area boundary point, 1C, ball center, 1D, ball center distance. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0090] On the contrary, the present application covers any alternative, modification, equivalent method and scheme defined by the claims on the essence and scope of the present application. Further, in order to make the public better understand the present application, some specific details are described in detail in the following detailed description of the present application. The present application can also be completely understood without the description of these details by those skilled in the art.

[0091] The structures, proportions, sizes, etc. shown in the drawings of the specification are merely used to cooperate with the content disclosed in the specification, to be understood and read by those skilled in the art, and are not used to limit the conditions that can be implemented by the present application, so they do not have technical significance. Any modification of the structure, change of the proportional relationship, or adjustment of the size, without affecting the effects that can be produced by the present application and the purposes that can be achieved, should still fall within the scope of the technical content disclosed by the present application. At the same time, the terms such as "front", "back", "left", "right", "middle" and "one" used in the specification are only for the convenience of clear description, and are not used to limit the scope of the present application, and the change or adjustment of the relative relationship, without substantially changing the technical content, is also considered as the scope of the present application that can be implemented.

[0092] The first aspect embodiment of the present disclosure proposes a flexible visual tactile finger based on fin effect structure, the structure is as shown in Figure 1 , comprising:

[0093] The finger body 1 is a fin effect structure made of flexible transparent material, a flexible semi-transparent perception layer 11 is arranged on the surface of the finger body 1 opposite to the object, an array of marker points 13 is arranged on the two side surfaces of the finger body 1 respectively, and a flexible light shielding layer 12 is arranged on the surface of the perception layer 11 and the remaining side surfaces of the finger body 1.

[0094] The finger body base 2 is fixedly connected with the root of the finger body 1, and the finger body base 2 has a transparent bottom plate 21.

[0095] The image acquisition unit comprises a shell 3 fixed to the bottom of the finger body base 2 and a light source 5 and a camera 6 arranged in the shell 3, the camera 6 is located directly below the side of the finger body 1 provided with the perception layer 11, and the light source 5 is arranged around the lens of the camera 6.

[0096] In the following, for the convenience of description, the side of the finger body 1 provided with the perception layer 11 is defined as the front side of the flexible visual tactile finger, the two sides of the finger body provided with the array of marker points 13 are defined as the left side and the right side of the flexible visual tactile finger, and the side opposite to the front side is defined as the rear side of the flexible visual tactile finger, and the side orientation marks are shown in Figure 1 .

[0097] In some embodiments, the finger body 1 is made of transparent elastic silicone, and is shaped like a fish fin, and can be integrally formed by multiple times of casting in a mold. In a specific embodiment, the finger body has a width (the distance between the left and right sides) of 20 mm, a height of 92 mm, and contains 6 fin strips between the front and back walls, with a fin strip spacing of 10 mm and a fin strip thickness of 3 mm. The connection between the fin strips and the front and back walls is thinned to 2 mm. The front wall of the finger body 1 gradually increases in thickness from the position close to the root of the finger to provide a larger observation field of view for the camera 6. In this embodiment, the front wall of the finger body 1 gradually increases in thickness from 2.5 mm to 10 mm, ensuring that the camera 6 can directly observe the entire sensing layer 11.

[0098] Further, the sensing layer 11 on the finger body 1 is made of translucent elastic silicone, and has a thickness of 1 mm to 2 mm. The thinner the sensing layer 11, the more sensitive the sensor; the thicker the sensing layer 11, the greater the sensing depth. The sensing layer 11 is formed on the front surface of the finger body 1 by casting, and a black opaque elastic silicone layer is cast on the surface of the sensing layer 11 facing away from the front surface of the finger body 1 to form a light shielding layer 12, which generally has a thickness of less than 0.3 mm. When an object presses the finger body 1, the sensing layer 11 thins at the corresponding position, and the reflected light becomes darker. Different pressing depths can be shown in different brightness in the image captured by the camera 6. Using this principle, the local deformation of the contact surface can be reconstructed. The remaining side surfaces of the finger body 1 also need to be coated with a light shielding layer to block ambient light. Since these surfaces are wrapped by the side walls of the mold during casting, black opaque silicone needs to be applied after demolding and left to solidify.

[0099] Further, the sensing layer 11 of this embodiment uses a soft material with a hardness of 5A, which is close to the hardness of human skin, and has a higher deformation fit when contacting an object, which is beneficial to reconstructing the surface morphology of the contacted object.

[0100] Further, two rows of dot-shaped markers are laser-engraved on the left and right sides of the finger body 1 to form a marker array 13. The engraved markers follow the rule that the radius and spacing increase in proportion to the distance from the camera 6, thereby ensuring that the spacing and size of each marker in the image are close to consistent. In this embodiment, there are 14 markers in each row, with a spacing that gradually increases from 4 mm to 10 mm from the root to the tip of the finger, and a size that gradually increases from a radius of 0.3 mm to a radius of 0.5 mm.

[0101] In some embodiments, the root of the finger body 1 is fixed to the finger base 2, which is composed of four side walls and a transparent bottom plate 21. The root of the finger body 1 is bonded to the side walls of the finger base 2, and is pressed tightly by the transparent bottom plate 21 of the finger base 2. The transparent bottom plate 21 can be made of acrylic plate 3.

[0102] In some embodiments, the image acquisition unit is located below the finger body 1, including an opaque shell 4 and a light source 5 and a camera 6 located therein. Among them, the camera 6 is fixed on the bottom of the shell 4 by a screw, and its optical axis is parallel to the height direction of the finger body, and it is opposite to the contact surface of the finger contacting the object. In a specific embodiment of the present application, the acquisition resolution of the camera 6 is 1920x1080, FPS (frames per second) = 30, and FOV (field of view) = 85°. The light source 6 uses monochromatic white light illumination, and specifically can select LED light plate. The light plate is installed in the slot of the shell 4, and the camera 6 passes through the round hole on the light plate to limit the light plate. A black isolation ring 4 is installed between the camera 6 and the light source 5 to prevent light spots from appearing in the image captured by the camera 6.

[0103] It can be understood that, Figure 1 The flexible visual tactile finger based on the fin strip effect structure shown in the figure places the camera below the perception layer 11 and shoots obliquely instead of being installed on the back side of the finger, avoiding interference with the flexible deformation of the finger; The finger makes changes to the original FinRay structure, and the front wall thickness gradually increases as it approaches the root, which ensures that the camera 6 can directly observe the entire perception layer 11 and reconstruct as much contact surface deformation as possible; The perception layer 11 and the light shielding layer 12 cast on the front side of the finger body 1 enable the finger to have the ability of local tactile perception; The marker point array 13 of the finger body 1 is formed on the left and right sides, avoiding interference with the processing of the image and the reconstruction of the deformation when hitting the front surface.

[0104] The second aspect embodiment of the present disclosure proposes a flexible hand claw assembled by the above visual tactile finger, and the partial cross-sectional view and the left view are as shown in Figure 2 The cross-sectional view, the flexible visual tactile finger 10 is cut along the screw axis, and the rest is half cut. The flexible hand claw includes a plurality of visual tactile fingers 10 and a driving mechanism 20; wherein,

[0105] The visual tactile finger 10 adopts the flexible visual tactile finger provided by the first aspect embodiment of the present disclosure;

[0106] The driving mechanism 20 is used to provide driving force for each visual-tactile finger 10 to realize the gripping and releasing of the object. The driving mechanism 20 comprises a motor box 21, a motor 22, a lead screw 23, a sliding block 24, a connecting table 25 and a plurality of connecting rods 26. The motor 22 is installed in a reserved slot at the bottom of the motor box 21. One end of the lead screw 23 is fixed with a flange coupling 27 through a set screw, and the flange coupling 27 is fixedly sleeved on the rudder disc of the motor 22, so that the lead screw 23 is driven to rotate by the motor 22. The other end of the lead screw 23 protrudes from the top cover of the motor box 21 and is sleeved with the sliding block 24, which is threadedly connected with the lead screw 23 and is driven to move axially along the lead screw 23 by the rotation of the lead screw 23. The connecting table 25 is fixedly sleeved on the sliding block 24 through a screw, and the connecting table 25 is connected with the shell 4 in each visual-tactile finger 10 through a pin shaft. Meanwhile, the shell 4 in each visual-tactile finger 10 is also connected with one end of a corresponding connecting rod 26 through a pin shaft, and the other end of each connecting rod 26 is connected with the top cover of the motor box 21 through a pin shaft.

[0107] The third aspect embodiment of the present disclosure provides a sensing method based on the above visual-tactile finger. Specifically, the contact surface of the finger when contacting the object is reconstructed. Since the finger has local deformation and overall deformation when contacting the object, the complete reconstruction of the contact surface can be completed through local deformation reconstruction and overall deformation reconstruction. Before describing the specific steps of the method, the structure of the visual-tactile finger will be described first. Figure 3 The two deformations and the following surfaces are defined.

[0108] The interface between the object 1 and the sensing layer 11 is defined as the observation surface 1a. Since the thickness of the light shielding layer 12 is much smaller than the thickness of the sensing layer 11 and the object 1, the light shielding layer 12 is approximated as a plane, and the light shielding layer 12 that is in contact with the object and deformed is defined as the contact surface 1b. A virtual contact surface, the non-local deformation surface 1c, is defined by translating each point of the observation surface 1a along the normal by the initial thickness of the sensing layer 11. The non-local deformation surface 1c is considered to be the surface formed when the object presses the contact surface but only causes global deformation, at which time the normal thickness of the sensing layer 11 does not change. In combination with the above definitions, the global deformation is defined as the deformation that causes the contact surface to change from the initial shape to the shape of the non-local deformation surface 1c, and the local deformation is defined as the deformation that causes the contact surface to change from the shape of the non-local deformation surface 1c to the shape of the actual contact surface 1b. The depth of the contact surface 1b into the non-local deformation surface 1c along the optical axis of the camera 6 is defined as the pressing depth 1d, and the thickness of the sensing layer 11 after being pressed in the same direction is referred to as the imaging thickness 1e, which determines the brightness of the pressed position in the field of view of the camera. The distance between the non-local deformation surface 1c and the observation surface 1a along the optical axis of the camera is defined as the equivalent thickness 1f of the sensing layer 11, which is equal to the sum of the pressing depth 1d and the imaging thickness 1e. The normal thickness of the sensing layer 11 is defined as the sensing layer thickness 1g, which is determined during casting and does not change with deformation.

[0109] Referring to Figure 4 The sensing method of the embodiment includes the following steps:

[0110] Step 1) Global deformation reconstruction of the contact surface

[0111] The camera 6 is used to obtain a real-time image of the object pressing the surface of the visual tactile finger 10. The non-local deformation three-dimensional spatial coordinates of the global deformation of the contact surface 1c are obtained by combining the non-distortion pinhole camera model, the geometric constraint that the contact surface 1b is of uniform width when the object presses the visual tactile finger 10, and the tracking of the marker point features in the real-time image, thereby realizing the reconstruction of the global deformation of the contact surface 1b.

[0112] Step 2) Local deformation reconstruction of the contact surface

[0113] Step 2-1) The camera 6 is used to record images of the contact surface 1b in as many different deformations as possible as a reference video to track the positions of the marker points under various deformations.

[0114] Step 2-2) Press the calibration ball with known radius onto the contact surface 1b, obtain the calibration image by using the camera 6, and select a frame from the reference video that is closest to the position of the mark point in the calibration image as the calibration reference image, and calculate the normalized brightness difference between the calibration image and the calibration reference image; at this time, it is considered that the part of the ball surface pressed onto the contact surface 1b coincides with the two surfaces of the contact surface, and the position of the ball center in the calibration image can be obtained according to the shadow area generated by the calibration ball pressing the contact surface 1b, and the local deformed three-dimensional space coordinates corresponding to the shadow area after deformation are obtained by using the ball center position and the non-distortion pinhole camera model, the pressing depth corresponding to each point in the shadow area is calculated by using the three-dimensional space coordinates of the overall deformation of the contact surface 1b obtained in step 1) and the three-dimensional space coordinates of the overall deformation of the contact surface 1b, and the mapping of the normalized brightness difference and the pressing depth is fitted by using the pressing depth corresponding to each point in the shadow area and the normalized brightness difference, and the calibration is completed.

[0115] Step 2-3) Select a frame from the reference video recorded in step 2-1) that is closest to the position of the mark point in the real-time image obtained in step 1) as the real-time reference image, calculate the normalized brightness difference between the real-time reference image and the real-time image, obtain the real-time pressing depth of each point of the contact surface 1b by using the mapping obtained in step 2-2), and realize the reconstruction of the local deformation of the contact surface 1b.

[0116] Step 3) Complete deformation reconstruction of the contact surface

[0117] The complete deformation of each point of the contact surface 1b is obtained by using the three-dimensional space coordinates of the overall deformation of the contact surface 1b obtained in step 1) and the real-time pressing depth of each point of the contact surface 1b obtained in step 2), and the complete deformation reconstruction of the contact surface 1b is realized.

[0118] In some embodiments, step 1) mainly uses the non-distortion pinhole camera model, image edge feature tracking, geometric constraint assumption and linear interpolation to reconstruct the overall deformation of the contact surface 1b, and the specific steps are as follows:

[0119] 1-1) Define the non-distortion pinhole camera model, and its expression is as follows:

[0120] zp′=Kp

[0121] wherein, is the intrinsic matrix of the camera 6, which is obtained by using the existing Zhang Zhengyou method for calibration; p' and p are respectively the pixel coordinates of a pixel point in an image and the corresponding three-dimensional coordinates in the camera coordinate system, which are defined as:

[0122] p′=(u,v,1) T

[0123] p=(x,y,z) T

[0124] The image's u-axis is parallel to the camera's x-axis, the image's v-axis is parallel to the camera's y-axis, the origin of the camera's coordinate system is set at the camera's optical center, and the z-axis is defined to point downwards. The superscript T represents transpose.

[0125] 1-2) Assemble the visual-tactile finger, wherein the width direction of the finger is aligned with the v-axis and y-axis; after assembly, use camera 6 to acquire the original image of the contact surface 1b when the object presses the visual-tactile finger of this embodiment in real time. The original image is then subjected to grayscale filtering to obtain a real-time image; assume that the following geometric constraints are satisfied when the finger undergoes overall deformation: in the camera coordinate system, the distance between points p1 and p2 at the same height on the left and right edges of the finger observation surface 1a is always equal to the width E of the finger, therefore:

[0126] ||p1-p2||2=W

[0127] z1=z2

[0128] Let p′1 and p′2 be the two pixels corresponding to points p1 and p2 in the real-time image. Since the width direction of the finger is aligned with the v-axis and y-axis, pixels p′1 and p′2 satisfy the following:

[0129] u1 = u2

[0130] Where z1 and z2 are the z-axis coordinates of points p1 and p2, respectively, and u1 and u2 are the u-axis coordinates of pixels p′1 and p′2, respectively.

[0131] 1-3) Trace the left and right edges of the observation surface 1a in the real-time image, and use the expression described in step 1-2) to obtain the three-dimensional spatial coordinates of all pixels on the left and right edges of the observation surface 1a. Use linear interpolation to obtain the three-dimensional spatial coordinate p of any pixel on the observation surface 1a. VS ;

[0132] 1-4) Perform polynomial fitting and differentiation on the three-dimensional spatial coordinates corresponding to each pixel point on the left and right edges of the observation surface 1a to obtain the tangent inclination angle θ at each point, combined with the initial thickness t of the sensing layer 11. v (Right now Figure 3 The equivalent thickness t0 (i.e., 1g) at each of the left and right edges can be obtained. Figure 3 In 1f), the three-dimensional spatial coordinates p of any pixel on the observation surface 1a are then... VS Adding the equivalent thickness t0, we obtain the three-dimensional spatial coordinates p of each pixel on the overall deformable contact surface without local deformation, i.e., the undeformed surface 1c. LUS That is, its expression is:

[0133]

[0134] p LUS = p VS + (0, 0, t0) T

[0135] In one embodiment of the application, the camera captures the contact surface images with resolution of 1920x1080 at 30 FPS, and uses a checkerboard to calibrate the intrinsic matrix K of the camera, and the finger width W is 20 mm.

[0136] In some embodiments, step 2) is mainly to reconstruct the local deformation of the contact surface 1b using the distortionless pinhole camera model, image marker point feature tracking and local normalized intensity difference, and the specific steps are as follows:

[0137] 2-1) Use the camera 6 to record the images of the contact surface 1b when it is deformed as much as possible to obtain a reference video, use a deep learning model to track the positions of the marker points under various deformations and record them.

[0138] 2-2) Calibrate the normalized intensity difference-pressing depth mapping M(), as shown in Figure 5 , the specific steps are as follows:

[0139] 2-2-1) Press a calibration ball 1A with a known diameter R on the contact surface 1b to make the finger deform globally and locally, obtain a calibration image, track the positions of the marker points and record them.

[0140] 2-2-2) Compare the positions of the marker points in the calibration image with the positions of the marker points in each frame of the reference video recorded in step 2-1), find the frame closest to the calibration image as the calibration reference image, and the comparison method is to calculate the sum of distances of the marker points in the calibration image and each frame of the reference video Take the frame of the reference video with the smallest sum of distances as the calibration reference image, The formula is expressed as:

[0141]

[0142] Wherein, represents the sum of distances of the marker points in the calibration image and the i-th frame of the reference video, m' C and are the pixel coordinates of the marker points in the calibration image and the i-th frame of the reference video, respectively.

[0143] 2-2-3) Subtract the calibration reference image from the calibration image to obtain the intensity difference, and calculate the normalized intensity difference to avoid the influence of different distances between the contact position and the light source, and the normalization process is expressed as:

[0144]

[0145] Among them, I C and I refC These represent the brightness of the calibration image and the calibration reference image, respectively.

[0146] 2-2-4) After the calibration ball 1A is pressed onto the contact surface 1b, the thickness of the contact layer in the pressed area decreases from the equivalent thickness 1f to the imaging thickness 1e, and the reflected light gradually weakens. Therefore, a convex circular contour shadow area is generated in the calibration image. During calibration, the circular contour of the shadow area in the calibration image is circled, i.e. Figure 5 The boundary points of the shaded area shown in Figure 1B are marked, along with the pixel coordinates c corresponding to the center of the circular outline. r The pixel coordinates p of a point on the circle. r If the three-dimensional distance between two points is approximately equal to the radius R of the calibration sphere 1A, then:

[0147] ||p r -c r ||2=R

[0148] Among them, c r To be with c r The three-dimensional spatial coordinates of the point corresponding to ′ in the camera coordinate system, p r To be with p r The three-dimensional spatial coordinates of the point corresponding to ′ in the camera coordinate system;

[0149] At the same time, c r and p r Since they are at the same height in the z-direction, we have:

[0150] z r =z cr

[0151] Among them, z r z cr p r and c r z-axis coordinate;

[0152] Combining the above formulas and the expression for the distortion-free pinhole camera model, the approximate position c of the sphere center 1C in the camera coordinate system can be obtained. r .

[0153] 2-2-5) Iterate once to estimate the center position of the ball more accurately. Since the shooting distance is relatively far and the lateral width of the finger is small, it can be assumed that the camera is directly facing the contact surface 1b of the calibration ball 1A, therefore:

[0154]

[0155] Where L c =||c r||2 is the distance from the center of the sphere to the center of the sphere, i.e., the distance between the center of the sphere, 1C, and the optical center of the camera, where c is the exact position of the center of the sphere.

[0156] 2-2-6) Using the accurate position c of the sphere's center and the distortion-free pinhole camera model, calculate the three-dimensional spatial coordinates p of each pixel in the shadow area after deformation. D Referring to the method described in step 1), the original three-dimensional spatial coordinates p of the point located on the undeformed surface 1c corresponding to the shadow area are obtained. LUSD , will p D With p LUSD Subtracting the z-axis coordinates yields the pressing depth d at each point on the contact surface when the calibration ball 1A presses against the contact surface 1b. D (Right now Figure 3 The formula for calculating 1d in the formula is:

[0157] d D =z D -z LUSD

[0158] Among them, z D and z LUSD They are p D and P LUSD The z-axis coordinate.

[0159] 2-2-7) with For independent variable pairs Polynomial fitting of dD yields the normalized mapping M() between the brightness difference and the pressing depth.

[0160] 2-3) Real-time reconstruction of local deformation of the finger contact surface. During reconstruction, steps 2-2-2) and 2-2-3) are repeated frame by frame for the real-time acquired images to obtain the real-time normalized brightness difference. The real-time pressure depth d is then obtained from the mapping M() to complete the local reconstruction:

[0161]

[0162] 3) Reconstruct the complete deformation of the contact surface

[0163] Steps 1) and 2) reconstruct the local and overall deformations of the contact surface. Further, this embodiment uses the following formula to reconstruct the complete deformation of the contact surface:

[0164] p CS =p LUS -(0, 0, d) T

[0165] Where, p CS It represents the three-dimensional coordinates of any point on the contact surface.

[0166] In one embodiment of the present application, the thickness of the front side sensing layer of the probe is 2 mm. The calibration uses a small steel ball with a diameter of 6 mm to press on the contact surface, while the pressed area sensing layer thins, the reflected light decreases, the brightness darkens, and a convex, circular profile shadow area appears in the image. By calibrating the ball center position and corresponding brightness difference, each pixel in the shadow area can provide a set of pressing depth-brightness difference data, and finally 15546 sets of data are collected and fitted into a quadratic polynomial. Subsequent deformation reconstruction uses the existing deep learning model C o -Tracker tracks the position of the marker point, with a processing speed of about 0.1-0.2 seconds per frame.

[0167] In some embodiments, the present disclosure proposes a sensing method further comprising identifying the force direction of the tactile finger, by tracking the marker point in real time and summing the displacement of the marker point position from its initial position to determine the force direction, specifically:

[0168] The displacement of the marker point position from its initial position is summed according to the following formula:

[0169] o' = ∑(m' - m' * )

[0170] Wherein, is the overall displacement of the marker point in the real-time image, m' is the pixel coordinate of the marker point in the real-time image, m' * is the initial pixel coordinate of the marker point in the image;

[0171] According to the set threshold value ε, it is judged whether the finger and the object have contact, when ||o' ||2≤ε, it is judged that the finger does not contact the object, when ||o' ||2>ε, it is determined that the finger contacts the object once, and then the contact force direction of the finger in the camera coordinate system is defined according to the following formula:

[0172]

[0173] Wherein, o is a unit vector representing the contact force direction, o' u and o' u are the u-axis and v-axis coordinates of the marker point in the real-time image, respectively;

[0174] Among all the sides of the finger (four sides in this embodiment), the side of the finger whose normal vector is closest to o is determined as the force receiving surface, and the method for judging the closeness is the vector inner product.

[0175] It should be noted that in the working process of the present finger, only one side where the sensing layer 11 is located is used as the grasping working surface, and the remaining sides can detect force in order to prevent possible false touch and ensure timely response.

[0176] In one embodiment of the present application, when the force sensor pressure head gradually pushes the finger body from different directions, different positions 0-60 mm away from the fingertip in the z direction, the sensing method based on the above principle can always detect the force deformation of the finger before the force reaches 2N, and return the correct force surface. The experiment was conducted 72 times, and each side of the finger body was tested 18 times.

[0177] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the present specification, the exemplary description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0178] Although the embodiments of the present disclosure have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirit of the present disclosure, and the scope of the present disclosure is defined by the claims and their equivalents.

Claims

1. A flexible visuo-haptic finger based on a fin effect structure, characterized in that, The device comprises: a finger body made of a flexible transparent material in a fin effect structure, a flexible semi-transparent sensing layer being arranged on the surface of the finger body opposite to the object, an array of marker points being arranged on the two side surfaces of the finger body respectively adjacent to the sensing layer, and a flexible light shielding layer being arranged on the surface of the sensing layer and the remaining side surfaces of the finger body; the wall thickness of the finger body on the side with the sensing layer gradually increases as the position approaches the finger root; the marker points in the array of marker points follow the rule that the radius and spacing of the marker points increase in proportion to the distance from the camera, so as to ensure that the spacing and size of the marker points in the image are close to consistent; a finger body base connected to the root of the finger body, the finger body base having a transparent bottom plate; an image acquisition unit comprising a light-tight shell fixed to the bottom of the finger body base, and a light source and a camera arranged in the shell, the camera being located directly below the side of the finger body with the sensing layer, and the light source being arranged around the lens of the camera.

2. The visio-haptic finger according to claim 1, characterized in that, The sensing layer is made of a soft material with a hardness of 5A.

3. The visio-haptic finger of claim 1, wherein, The color of the light source is single white, and a isolation ring is further arranged between the camera and the light source to prevent the appearance of light spots in the image captured by the camera.

4. A flexible gripper, characterized by The device comprises: a plurality of fingers, the fingers being the tactile vision finger according to any one of claims 1-3; a driving mechanism connected to the shell of each finger, for providing driving force for each tactile vision finger to realize the gripping and releasing of the object.

5. A sensing method based on the visual-haptic finger according to any one of claims 1 to 3, characterized in that, The device comprises: Step 1) overall deformation reconstruction of the contact surface: the light shielding layer deformed by contact with the object is defined as the contact surface, the real-time image of the object pressing the surface of the tactile vision finger is acquired by the camera, the three-dimensional spatial coordinates of the overall deformation of the contact surface without local deformation are obtained by combining the non-distortion pinhole camera model, the geometric constraint that the contact surface is of equal width at each position when the object presses the tactile vision finger, and the tracking of the marker point features in the real-time image, the overall deformation reconstruction of the contact surface is realized, and the contact surface without local deformation is defined as the non-local deformation surface; Step 2) local deformation reconstruction of the contact surface: Step 2-1) record the images of the contact surface deformed as much as possible in different ways as reference videos by the camera, so as to track the positions of the marker points under various deformations; Step 2-2) press a calibration ball with a known radius onto the contact surface, obtain a calibration image by using the camera, and select a frame from the reference video that is closest to the position of the marker point in the calibration image as a calibration reference image, calculate the normalized brightness difference between the calibration image and the calibration reference image; obtain the position of the ball center in the calibration image from the shadow area generated by pressing the contact surface with the calibration ball, obtain the corresponding local deformed three-dimensional space coordinates of the shadow area after deformation by using the ball center position and the non-distorted pinhole camera model, calculate the corresponding pressing depth of each point in the shadow area by using the coordinates and the three-dimensional space coordinates of the overall deformation of the contact surface corresponding to the shadow area obtained in step 1), and fit the mapping between the normalized brightness difference and the pressing depth by using the corresponding pressing depth of each point in the shadow area, thereby completing the calibration; Step 2-3) select a frame from the reference video that is closest to the position of the marker point in the real-time image as a real-time reference image, calculate the normalized brightness difference between the real-time reference image and the real-time image, obtain the real-time pressing depth of each point on the contact surface by using the mapping, and realize the reconstruction of the local deformation of the contact surface; Step 3) complete deformation reconstruction of the contact surface: Subtract the real-time pressing depth of each point on the contact surface from the three-dimensional space coordinates of the overall deformation of the contact surface to obtain the complete deformation of each point on the contact surface, thereby realizing the reconstruction of the complete deformation of the contact surface.

6. The sensing method of claim 5, wherein, Step 1) specifically includes the following steps: 1-1) define a non-distorted pinhole camera model, which has the following expression: zp' = Kp wherein is the camera intrinsic matrix; p' and p are the pixel coordinate of a pixel on a frame and its corresponding three-dimensional coordinate in the camera coordinate system, respectively, which are defined as: wherein the u-axis of the image is parallel to the x-axis of the camera coordinate system, the v-axis of the image is parallel to the y-axis of the camera coordinate system, the origin of the camera coordinate system is set at the camera optical center, and the positive direction of the z-axis is downward; 1-2) assemble the visual-tactile finger according to the finger width direction aligned with the v-axis and the y-axis, and obtain a real-time image of the object pressing the visual-tactile finger by using the camera; define the interface between the finger and the sensing layer as an observation surface, and set that when the finger is deformed as a whole, the following geometric constraints are met: in the camera coordinate system, the distance between points p1 and p2 on the left and right edges of the observation surface at the same height is always equal to the width W of the finger, so there is: ||p1-p2||2 = W z1 = z2 wherein p'1 and p'2 are two pixel points in the real-time image corresponding to points p1 and p2, respectively, and pixel points p'1 and p'2 satisfy: u1 = u2 wherein z1 and z2 are the z-axis coordinates of points p1 and p2, respectively, and u1 and u2 are the u-axis coordinates of pixel points p'1 and p'2, respectively; 1-3) Tracking the left and right edges of the observed surface in the real-time image, obtaining the three-dimensional space coordinates corresponding to all the pixels on the left and right edges of the observed surface by using the expression in step 1-2), and obtaining the three-dimensional space coordinates p of any pixel point on the observed surface by using linear interpolation VS ; 1-4) to the three-dimensional space coordinates p VS Polynomial fitting and derivation are performed to obtain the tangent angle θ at each location, and the three-dimensional space coordinates p of each pixel point on the overall deformed contact surface without local deformation are obtained according to the following formula LUS : Among them, t v t0 is the initial thickness of the sensing layer, and t0 is the equivalent thickness t0 at each point along the optical axis of the camera on the left and right sides of the observation surface.

7. The sensing method of claim 5, wherein, Step 2-2) specifically includes the following steps: 2-2-1) press a calibration ball with a known diameter R onto the contact surface to cause the finger to deform as a whole and locally, and obtain a calibration image; 2-2-2) Comparing the positions of the marker points in each frame of the reference video with the positions of the marker points in the calibration image, finding the frame closest to the calibration image as the calibration reference image, and the comparison method is to calculate the sum of the distances of the marker points in the calibration image and each frame of the reference video Taking the frame of the reference video with the smallest sum of distances as the calibration reference image, The formula is expressed as: wherein m' C and are the pixel coordinates of the marker points in the i-th image of the calibration image and of the reference video, respectively. 2-2-3) The normalized luminance difference is calculated according to the following formula where I C and I refC are the luminance of the calibration image and the calibration reference image, respectively. 2-2-4) Circle out the circular contour of the shadow area caused by the pressing of the calibration ball on the contact surface in the calibration image, and mark the pixel coordinates c corresponding to the center of the circle r ' and the pixel coordinates p of a point on the circle r ', and the three-dimensional distance between the two points is approximately the radius R of the calibration ball, then we have: ||p r -c r ||2=R z r = z cr where c r is the three-dimensional spatial coordinate of the point corresponding to c r in the camera coordinate system, i.e. the sphere center position, p r is the three-dimensional spatial coordinate of the point corresponding to p r in the camera coordinate system, z r and z cr are the z-axis coordinates of p r and c r respectively; The rough position of the ball center can be obtained by combining the above formula with the expression of the non-distorted pinhole camera model; 2-2-5) update the rough position of the ball center according to the following formula: where L c = ||c r is the distance of the sphere center from the camera optical center, and c is the exact position of the sphere center. 2-2-6) Calculate the three-dimensional space coordinates p of each pixel point in the shadow area after deformation using the accurate position c of the sphere center and the non-distortion pinhole camera model D , and the three-dimensional space coordinates p of the points in the shadow area corresponding to the non-local deformation surface are obtained according to the method in step 1) LUSD , and the pressing depth d of each point on the contact surface is obtained according to the following formula D : d D = z D - z LUS where z D and z LUS are the z-axis coordinates of p D and p LUSD respectively. 2-2-7) with For independent variable pairs and d D Polynomial fitting was performed to obtain the normalized brightness difference and the mapping M( ) between the pressing depth; Step 2-3) specifically includes the following steps: The real-time image is processed frame by frame according to the operations of steps 2-2-2) to 2-2-3) to obtain the time-normalized brightness difference The real-time pressing depth d is obtained from the mapping M() again to complete the local reconstruction:

8. The sensing method of claim 6, wherein, The sensing method further comprises judging the force direction received by the finger by tracking the marker point in real time and summing the displacement of the marker point position from its initial position, and specifically comprises the following steps: The displacement of the marker point position from its initial position is summed according to the following formula: wherein, is the overall shift of the marker point in the live image, m' is the pixel coordinate of the marker point in the live image, m' * is the initial pixel coordinate of the marker point in the image; According to a set threshold value ε, whether the finger and the object are in contact is judged, when ||o'||2>ε, it is determined that the finger and the object are in contact once, and the contact force direction of the finger in the camera coordinate system is defined according to the following formula: where o is a unit vector representing the force direction, o u and o' v are the u-axis and v-axis coordinates of the marker point in the real-time image, respectively; The finger side surface closest to o in the normal vector of all side surfaces of the finger is determined as the force receiving surface, and the method for judging the closeness is the vector inner product.

Images