Fin ray effect structure-based flexible visual tactile finger, paw and sensing method

By integrating flexible visual tactile sensing technology on the fin-bar effect structure hand paws, omnidirectional tactile sensing and feedback control are achieved, solving the problems of complex hand paw design and lack of tactile perception, and enhancing the application value of mechanical hand paws.

CN118977269AActive Publication Date: 2024-11-19TSINGHUA UNIVERSITY

Patent Information

Application Number
CN202411099226.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-19
Estimated Expiration
2044-08-12

AI Technical Summary

Technical Problem

The existing fin-effect structure of the handclaw lacks omnidirectional tactile sensing capabilities and is complex in design, which limits its promotion and application in mechanical handclaws.

Method used

Using flexible visual tactile fingers based on fin strip effect structure, fin strip effect structure fingers made of flexible transparent materials combine translucent sensing layer, marking dot array and flexible light-shading layer, combined with distortion-free pinhole camera model and deep learning technology , realize the reconstruction of the overall and local deformation of the contact surface, and simplify structural design and production methods.

Benefits of technology

The omnidirectional tactile sensing capability of the hand paw is realized, the resolution is improved, the lighting configuration and production methods are simplified, the problem of the hand paw lacking tactile feedback capability is overcome, and the interaction between the mechanical hand paw and the environment can be better controlled.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118977269A_ABST
    Figure CN118977269A_ABST
Patent Text Reader

Abstract

According to the flexible visual tactile finger based on the fin ray effect structure, the paw and the sensing method, the visual tactile finger comprises a finger body, a finger body base and an image acquisition unit, the finger body is of the fin ray effect structure made of a flexible transparent material, a flexible semitransparent sensing layer is arranged on the surface of the side, right facing an object, of the finger body, and the flexible semitransparent sensing layer is arranged on the surface of the side, right facing the object, of the finger body. Mark point arrays are arranged on the two side faces, connected with the sensing layer, of the finger body respectively, and flexible shading layers are arranged on the surface of the sensing layer and the other side surfaces of the finger body. A camera in the image acquisition unit is located under the side, provided with the sensing layer, of the finger body. The gripper comprises a plurality of visual touch fingers and a driving mechanism used for driving the visual touch fingers. The sensing method comprises the step of sequentially carrying out overall deformation reconstruction, local deformation reconstruction and complete deformation reconstruction on a finger contact surface. The flexible gripper overcomes the problem that an existing flexible gripper is poor in tactile feedback capacity, and the flexible gripper is low in illumination requirement, simple in structure, convenient to manufacture, multiple in applicable scene and high in application value.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of flexible manipulators and sensors, and particularly proposes a flexible visual-tactile manipulator based on a fin effect structure and a sensing method thereof. Background Art

[0002] Manipulators are one of the core technologies of the intelligent robot industry. They are widely used to replace human labor in performing repetitive or dangerous tasks such as sorting and excavation, significantly improving production efficiency and safety. However, the rigid structure of traditional manipulators can easily cause damage to workpieces and the environment during manipulation, posing a safety hazard to production. Flexible grippers are a type of manipulator grippers made of flexible materials or with flexible structures. They can effectively buffer the impact caused by accidental touches, have more advantages than rigid manipulators in environmental adaptability, and have extremely high application value. Tactile sensors are sensors that can provide feedback on tactile information such as contact position, force, and surface morphology. They allow manipulators to adjust their working status in real time and have become an indispensable part of realizing manipulator feedback control.

[0003] The gripper based on the fin ray effect (Fin Ray Effect) structure of the finger (Fin Hand) is a kind of bionic structure flexible gripper with extremely strong flexibility, which has been widely used in the field of object grasping and manipulation. However, this kind of gripper allows significant deformation during operation, and has high requirements for structural design when integrated with common contact tactile sensors, which limits the promotion and application of the Fin Hand. Among various types of tactile sensors, the vision-based tactile sensor has high resolution, simple principle structure, and the sensing element does not rely on contact. Thanks to the development of deep learning technology, the sensing ability has been greatly improved. It has broad application prospects in fields such as medical image processing, and has great potential in combination with flexible grippers. Combining the Fin Hand with the visual tactile sensor can enable the gripper to adjust the working state in real time according to tactile feedback to achieve more stable grasping. However, most of the existing Fin Hand designs lack tactile perception function, or can only perceive contact in a specific area. Therefore, it is urgent to design a Fin Hand with more comprehensive tactile perception ability. Summary of the invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the present invention discloses a flexible visual-tactile finger, gripper and sensing method with a fin effect structure. The flexible gripper proposed in the present invention has omnidirectional tactile sensing capability and simplifies the structural design and manufacturing method. It has low lighting requirements, is easy to manufacture, is applicable to many scenarios, and has high application value.

[0006] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0007] The first aspect of the present disclosure provides a flexible visual-tactile finger based on a fin effect structure, comprising:

[0008] The finger body is a fin effect structure made of a flexible transparent material, a flexible translucent sensing layer is provided on the surface of the finger body facing the object, and a marking point array is provided on the two side surfaces of the finger body connected to the sensing layer, and a flexible light shielding layer is provided on the surface of the sensing layer and the other side surfaces of the finger body;

[0009] A finger body base, connected to the root of the finger body, and the finger body base has a transparent bottom plate;

[0010] The image acquisition unit includes a light-proof shell fixed to the bottom of the finger base, a light source and a camera arranged in the shell, the camera is located directly below one side of the finger with the sensing layer, and the light source is arranged around the lens of the camera.

[0011] In some embodiments, the wall thickness of the finger body on the side where the sensing layer is provided gradually increases as the position approaches the finger base.

[0012] In some embodiments, the sensing layer is made of a soft material with a hardness of 5A.

[0013] In some embodiments, the marker points in the marker point array follow a rule that the radius and spacing of the marker points increase in proportion to the distance from the camera, so as to ensure that the spacing and size between the marker points in the image are close to the same.

[0014] In some embodiments, the color of the light source is single white, and an isolation ring is installed between the camera and the light source to prevent light spots from appearing in the image captured by the camera.

[0015] A second aspect of the present disclosure provides a flexible gripper, comprising:

[0016] A plurality of fingers, wherein the fingers are the visual-tactile fingers according to any embodiment of the first aspect of the present disclosure;

[0017] The driving mechanism is connected to the shell of each finger and is used to provide driving force for each visual-tactile finger to achieve grasping and releasing of the object.

[0018] A visual-tactile finger sensing method according to any embodiment of the first aspect of the present disclosure comprises:

[0019] Step 1) Reconstruction of the overall deformation of the contact surface:

[0020] The shading layer that is deformed by contact with the object is defined as a contact surface, a real-time image of the object pressing the surface of the visual-tactile finger is obtained by using the camera, the real-time image is combined with a distortion-free pinhole camera model, a geometric constraint that the width of each part of the contact surface is equal when the object presses the visual-tactile finger, and feature tracking of the marker points in the real-time image to obtain the three-dimensional spatial coordinates of the overall deformation of the contact surface without local deformation, so as to achieve reconstruction of the overall deformation of the contact surface, and the contact surface without local deformation is defined as a non-locally deformed surface;

[0021] Step 2) Reconstruction of local deformation of contact surface:

[0022] Step 2-1) using the camera to record images of the contact surface when it undergoes as many different deformations as possible as reference videos to track the positions of the marker points under various deformations;

[0023] Step 2-2) Press a calibration ball with a known radius onto the contact surface, use the camera to obtain a calibration image, and select a frame closest to the position of the marked point in the calibration image from the reference video as the calibration reference image, and calculate the normalized brightness difference between the calibration image and the calibration reference image; obtain the center position of the calibration ball in the calibration image according to the shadow area generated by the calibration ball pressing the contact surface in the calibration image, and use the center position and the distortion-free pinhole camera model to obtain the local deformation three-dimensional space coordinates corresponding to the deformation of the shadow area, and use the coordinates and the three-dimensional space coordinates of the overall deformation of the contact surface corresponding to the shadow area obtained in step 1) to calculate the pressing depth corresponding to each point in the shadow area, and use the pressing depth corresponding to each point in the shadow area and the normalized brightness difference to fit the mapping of the normalized brightness difference and the pressing depth to complete the calibration;

[0024] Step 2-3) selecting a frame closest to the position of the marked point in the real-time image from the reference video as the real-time reference image, calculating the normalized brightness difference between the real-time reference image and the real-time image, and using the mapping to obtain the real-time pressing depth of each point on the contact surface to achieve reconstruction of the local deformation of the contact surface;

[0025] Step 3) Complete deformation reconstruction of the contact surface:

[0026] The complete deformation of each point on the contact surface is obtained by subtracting the real-time pressing depth of each point on the contact surface from the three-dimensional spatial coordinates of the overall deformation of the contact surface, thereby realizing reconstruction of the complete deformation of the contact surface.

[0027] In some embodiments, step 1) specifically includes the following steps:

[0028] 1-1) Define the distortion-free pinhole camera model, which is expressed as follows:

[0029] zp′=Kp

[0030] in is the camera intrinsic parameter matrix; p′ and p are the pixel coordinates of a pixel on a frame image and its corresponding three-dimensional coordinates in the camera coordinate system, respectively, and the two are defined as:

[0031]

[0032] Among them, the u-axis of the image is parallel to the x-axis of the camera coordinate system, the v-axis of the image is parallel to the y-axis of the camera coordinate system, the origin of the camera coordinate system is set at the optical center of the camera, and the positive direction of the z-axis is set downward;

[0033] 1-2) Assemble the visual-tactile finger in a manner that the finger width direction is aligned with the v-axis and the y-axis, and use a camera to obtain a real-time image when an object presses the visual-tactile finger; define the interface between the finger body and the sensing layer as an observation surface, and assume that the finger satisfies the following geometric constraints when it is deformed as a whole: in the camera coordinate system, the distance between the point p1 and the point p2 at the same height on the left and right edges of the observation surface is always equal to the width E of the finger, so:

[0034] ||p1-p2||2=E

[0035] z1=z2

[0036] Assume that the two pixel points corresponding to the point p1 and the point p2 in the real-time image are p′1 and p′2 respectively, and the pixel points p′1 and p′2 satisfy:

[0037] u1=u2

[0038] Among them, z1 and z2 are the z-axis coordinates of point p1 and point p2 respectively, and u1 and u2 are the u-axis coordinates of pixel point p′1 and pixel point p′2 respectively;

[0039] 1-3) Track the left and right edges of the observation surface in the real-time image, use the expression in step 1-2) to obtain the three-dimensional spatial coordinates corresponding to all pixels on the left and right edges of the observation surface, and use linear interpolation to obtain the three-dimensional spatial coordinates p of any pixel point on the observation surface VS ;

[0040] 1-4) Mark the three-dimensional space coordinates VS Perform polynomial fitting and derivation to obtain the tangent inclination angle θ at each location, and obtain the three-dimensional spatial coordinates P of each pixel point on the overall deformation contact surface without local deformation according to the following formula: LUS :

[0041]

[0042]

[0043] Among them, t v is the initial thickness of the sensing layer, and t0 is the equivalent thickness t0 of the left and right edges of the observation surface along the camera optical axis.

[0044] In some embodiments, step 2-2) specifically includes the following steps:

[0045] 2-2-1) Press a calibration ball with a known diameter R onto the contact surface to cause overall and local deformation of the finger, and obtain a calibration image;

[0046] 2-2-2) Compare the position of the marking point in each frame of the reference video with the position of the marking point in the calibration image, and find the closest frame as the calibration reference image. The comparison method is to calculate the sum of the distances between the calibration image and the marking points in each frame of the reference video. Take the frame of reference video with the smallest sum of the distances as the calibration reference image. The formula is:

[0047]

[0048] Among them, m′ C and are the pixel coordinates of the marked points in the ith frame of the calibration image and the reference video respectively;

[0049] 2-2-3) Calculate the normalized brightness difference according to the following formula:

[0050]

[0051] Among them, I C and I refC are the brightness of the calibration image and the calibration reference image respectively;

[0052] 2-2-4) Circle the circular outline of the shadow area produced by the calibration ball pressing on the contact surface in the calibration image, and mark the pixel coordinate c corresponding to the center of the circle. r ′ and the pixel coordinates p of a point on the circle r ′, the three-dimensional distance between the two points is approximately the radius R of the calibration sphere, then:

[0053] ||p r -c r ||2=R

[0054] z r =z cr

[0055] Among them, c r For c r′ is the three-dimensional space coordinate of the point corresponding to it in the camera coordinate system, that is, the position of the center of the sphere, p r For r ′ is the three-dimensional space coordinate of the point corresponding to it in the camera coordinate system, z r and z cr They are p r and c r The z-axis coordinate of

[0056] Combining the above formula with the expression of the distortion-free pinhole camera model, the rough position of the center of the sphere can be obtained;

[0057] 2-2-5) Update the rough position of the center of the ball according to the following formula:

[0058]

[0059] Among them, L c =||c r ||2 is the distance between the center of the sphere and the optical center of the camera, and c is the exact position of the center of the sphere;

[0060] 2-2-6) Using the accurate position c of the sphere center and the distortion-free pinhole camera model, calculate the three-dimensional spatial coordinates p corresponding to each pixel point in the shadow area after deformation D , refer to the method described in step 1) to obtain the three-dimensional spatial coordinates p of the point on the non-locally deformed surface corresponding to the shadow area LUSD , and the pressing depth d of each point on the contact surface is obtained according to the following formula D :

[0061] d D =z D -z LUS

[0062] Among them, z D and z LUS They are p D and p LUSD The z-axis coordinate of

[0063] 2-2-7) For the independent variable and d D Perform polynomial fitting to obtain the mapping M() between normalized brightness difference and pressing depth;

[0064] Step 2-3) specifically includes the following steps:

[0065] The normalized brightness difference is obtained by performing the operations of steps 2-2-2) to 2-2-3) on the real-time image frame by frame. Then the real-time compression depth d is obtained by mapping M() to complete the local reconstruction:

[0066]

[0067] In some embodiments, the sensing method further includes determining the direction of the force applied to the finger by tracking the marker point in real time and summing the offset between the marker point position and its initial position, specifically including the following steps:

[0068] The offset between the marker position and its initial position is summed as follows:

[0069] o′=∑(m′-m′ * )

[0070] in, is the overall offset of the marker point in the real-time image, m′ is the pixel coordinate of the marker point in the real-time image, and m′ * is the initial pixel coordinate of the marker point in the image;

[0071] The threshold ε is set to determine whether the finger is in contact with the object. When ||o′||2>ε, it is determined that the finger is in contact with the object once, and the contact force direction of the finger in the camera coordinate system is defined according to the following formula:

[0072]

[0073] Where o is the unit vector indicating the direction of the force, o′ u and o′ v They are the u-axis and v-axis coordinates of the marked points in the real-time image respectively;

[0074] The side of the finger whose normal vector is closest to o among all the sides of the finger is determined as the force-bearing surface, and the method for judging the degree of proximity is the vector inner product.

[0075] The characteristics and beneficial effects of the present disclosure are:

[0076] The present invention combines the Fin Ray hand with a visual-tactile sensor to propose a flexible finger with visual-tactile perception capabilities and a gripper having the finger. The visual-tactile sensor is used to obtain a higher resolution, and the lighting configuration and manufacturing method are simplified, thereby overcoming the problem that the flexible fingers and grippers lack tactile feedback capabilities, and feedback control of the gripper can be achieved.

[0077] The present disclosure also proposes a sensing method suitable for the proposed flexible finger, so that the finger can reconstruct the overall and local deformation of the contact surface and sense the contact force, thereby realizing visual and tactile functions.

[0078] The present disclosure can be applied as an end effector of a robotic arm to perform tasks such as flexible grasping, tactile sensing, and feedback control, so that the robotic gripper can be better controlled and interact with the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1It is a cross-sectional view of a flexible visual-tactile finger based on a fin effect structure according to an embodiment of the present disclosure, wherein the finger body is not cut away.

[0080] Figure 2 It is a partial cross-sectional view of a flexible visual-tactile gripper based on a fin effect structure according to a specific embodiment of the present disclosure, wherein the flexible visual-tactile finger is cut along the axis of the screw and the rest of the part is half-cut.

[0081] Figure 3 It is a schematic diagram of local deformation of a flexible visual-tactile finger based on a fin effect structure according to an embodiment of the present disclosure.

[0082] Figure 4 It is an overall flow chart of a flexible visual-tactile finger sensing method based on a fin effect structure according to a specific embodiment of the present disclosure.

[0083] Figure 5 It is a schematic diagram of calibration of a visual-tactile finger based on a fin effect structure according to an embodiment of the present disclosure.

[0084] In the figure:

[0085] 1. Finger, 11. Sensing layer, 12. Shading layer, 13. Marking point array, 2. Finger base, 21. Transparent bottom plate, 3. Shell, 4. Isolation ring, 5. Light source, 6. Camera;

[0086] 10. Flexible visual-tactile finger, 20. Driving mechanism, 21. Motor box, 22. Motor, 23. Lead screw, 24. Slider, 25. Connecting table, 26. Connecting rod, 27. Flange coupling.

[0087] 1a, observation surface, 1b, contact surface, 1c, non-local deformation surface, 1d, indentation depth, 1e, imaging thickness, 1f, equivalent thickness, 1g, sensing layer thickness.

[0088] 1A, calibration sphere, 1B, boundary point of shadow area, 1C, center of sphere, 1D, distance between centers of sphere. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0090] On the contrary, the present application covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present application as defined by the claims. Further, in order to make the public have a better understanding of the present application, some specific details are described in detail in the detailed description of the present application below. Those skilled in the art can fully understand the present application without the description of these details.

[0091] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions that can be implemented in this application, so they have no substantial technical significance. Any modification of the structure, change in the proportional relationship, or adjustment of the size, without affecting the effects that can be produced by this application and the purposes that can be achieved, should still fall within the scope of the technical contents disclosed in this application. At the same time, the terms such as "front", "back", "left", "right", "middle" and "one" cited in this specification are only for the convenience of narration, and are not used to limit the scope of the implementation of this application. The change or adjustment of their relative relationship, without substantial change in the technical content, should also be regarded as the scope of the implementation of this application.

[0092] The first embodiment of the present disclosure proposes a flexible visual-tactile finger based on a fin effect structure, the structure of which is as follows: Figure 1 As shown, including:

[0093] The finger body 1 is a fin effect structure made of a flexible transparent material. A flexible translucent sensing layer 11 is provided on the surface of the finger body 1 facing the object. Marking point arrays 13 are provided on the two side surfaces of the finger body 1 connected to the sensing layer 11. A flexible light shielding layer 12 is provided on the surface of the sensing layer 11 and the other side surfaces of the finger body 1.

[0094] The finger body base 2 is fixedly connected to the root of the finger body 1, and the finger body base 2 has a transparent bottom plate 21;

[0095] The image acquisition unit includes a shell 3 fixed to the bottom of the finger base 2 and a light source 5 and a camera 6 arranged in the shell 3. The camera 6 is located directly below one side of the finger 1 where the sensing layer 11 is provided, and the light source 5 is arranged around the lens of the camera 6.

[0096] For the convenience of description, the side of the finger 1 provided with the sensing layer 11 is defined as the front side of the flexible visual-tactile finger, the two sides of the finger 1 provided with the marking point array 13 are defined as the left and right sides of the flexible visual-tactile finger, and the side opposite to the front side is defined as the back side of the flexible visual-tactile finger. Figure 1 .

[0097] In some embodiments, the finger body 1 is made of transparent elastic silicone and is fin-shaped as a whole, which can be formed by multiple integrated mold castings. In a specific embodiment, the width of the finger body (the distance between the left and right sides) is 20 mm, the height is 92 mm, and there are 6 fins between the front and rear side walls. The fin spacing is 10 mm and the fin thickness is 3 mm. The connection between the fins and the front and rear side walls becomes thinner to 2 mm. The wall thickness of the front side of the finger body 1 gradually increases as it approaches the root of the finger, providing a larger observation field for the camera 6. In this embodiment, the wall thickness of the front side of the finger body 1 gradually increases from 2.5 mm to 10 mm, ensuring that the camera 6 can directly observe the entire perception layer 11.

[0098] Furthermore, the sensing layer 11 on the finger body 1 is made of translucent elastic silicone, and its thickness is set at 1mm to 2mm. The thinner the sensing layer 11, the more sensitive the sensor; the thicker the sensing layer 11, the greater the sensing depth. The sensing layer 11 is formed on the front surface of the finger body 1 by casting, and a black opaque elastic silicone layer is cast on the surface of the sensing layer 11 facing away from the front side of the finger body 1 to form a light-shielding layer 12, and its thickness is generally within 0.3mm. When an object presses the finger body 1, the sensing layer 11 becomes thinner at the corresponding position, and the reflected light becomes darker. Different pressing depths can show different brightness in the image taken by the camera 6. Using this principle, the local deformation of the contact surface can be reconstructed. The remaining side surfaces of the finger body 1 also need to be coated with a light-shielding layer to block ambient light. Since these surfaces are wrapped by the side walls of the mold during casting, they need to be coated with black opaque silicone after demolding and left to solidify.

[0099] Furthermore, the sensing layer 11 of the present embodiment uses a soft material with a hardness of 5A, which is close to the hardness of human skin. The deformation produced when contacting an object has a higher degree of fit, which is conducive to reconstructing the surface morphology of the contacted object.

[0100] Furthermore, two columns of dot marks are engraved on the left and right sides of the finger body 1 by laser to form a marking point array 13. The marking points follow the rule that the radius and spacing increase in proportion to the distance from the camera 6, thereby ensuring that the spacing and size between the marking points in the image are close to the same. In this embodiment, there are 14 marking points in each column, and the spacing gradually increases from 4mm to 10mm from the base to the fingertip, and the size gradually increases from a radius of 0.3mm to 0.5mm.

[0101] In some embodiments, the root of the finger body 1 is fixed to the finger base 2, which is composed of four side walls and a transparent bottom plate 21. The root of the finger body 1 is bonded to the side wall of the finger base 2 and is pressed by the transparent bottom plate 21 of the finger base 2. The transparent bottom plate 21 can be made of an acrylic plate 3.

[0102] In some embodiments, the image acquisition unit is located below the finger body 1, and includes an opaque housing 4 and a light source 5 and a camera 6 located therein. The camera 6 is fixed to the bottom of the housing 4 by screws, and its optical axis is parallel to the height direction of the finger body, and is facing the contact surface of the finger contacting the object. In a specific embodiment of the present application, the acquisition resolution of the camera 6 is 1920×1080, FPS (frames per second) = 30, FOV (field of view) = 85°. The light source 6 uses monochromatic white light for illumination, and specifically an LED light board can be used. The light board is installed in the slot of the housing 4, and the camera 6 passes through the circular hole on the light board to limit the light board. A black isolation ring 4 is installed between the camera 6 and the light source 5 to prevent light spots from appearing in the image taken by the camera 6.

[0103] Understandably, Figure 1 The flexible visual-tactile finger based on the fin-ray effect structure shown places the camera below the sensing layer 11 for oblique shooting instead of installing it on the back side of the finger, avoiding interference with the flexible deformation of the finger; the finger has modified the original FinRay structure, and its front wall thickness gradually increases as it approaches the root, which ensures that the camera 6 can directly observe the entire sensing layer 11 and reconstruct as much contact surface deformation as possible; the sensing layer 11 and the shading layer 12 cast on the front side of the finger body 1 enable the finger to have the ability of local tactile perception; the marking point array 13 of the finger body 1 is formed on the left and right sides, avoiding interference with the deformation of image reconstruction when hitting the front.

[0104] The second embodiment of the present disclosure provides a flexible gripper assembled from the above-mentioned visual-tactile fingers, the partial cross-sectional view and left side view of the gripper are as follows: Figure 2 As shown in (a) and (b), in the cross-sectional view, the flexible visual-tactile finger 10 is cut along the screw axis, and the rest of the part is half-cut. The flexible gripper includes a plurality of visual-tactile fingers 10 and a driving mechanism 20; wherein,

[0105] The visual-tactile finger 10 adopts the flexible visual-tactile finger provided by the embodiment of the first aspect of the present disclosure;

[0106] The driving mechanism 20 is used to provide driving force for each visual-tactile finger 10 to achieve the grasping and releasing of the object. The driving mechanism 20 includes a motor box 21, a motor 22, a lead screw 23, a slider 24, a connecting platform 25 and a plurality of connecting rods 26. The motor 22 is installed in a slot reserved at the bottom of the motor box 21. One end of the lead screw 23 is fixed to a flange coupling 27 by a set screw, and the flange coupling 27 is fixedly sleeved on the steering plate of the motor 22, so that the motor 22 drives the lead screw 23 to rotate. The other end of the lead screw 23 protrudes from the motor box 21. A top cover is provided, and a slider 24 is sleeved on a lead screw 23 protruding from the motor box 21. The slider 24 is threadedly matched with the lead screw 23. The rotation of the lead screw 23 drives the slider 24 to move along the axial direction of the lead screw 23. A connecting platform 25 is fixedly sleeved on the slider 24 by screws. The connecting platform 25 is connected to the outer shell 4 in each visual-tactile finger 10 through a pin shaft. At the same time, the outer shell 4 in each visual-tactile finger 10 is also connected to one end of a corresponding connecting rod 26 through a pin shaft, and the other end of each connecting rod 26 is connected to the top cover of the motor box 21 through a pin shaft.

[0107] The third aspect of the present disclosure proposes a sensing method based on the visual-tactile finger, specifically, to reconstruct the contact surface of the finger when it contacts the object. Since the finger has both local deformation and overall deformation when it contacts the object, the contact surface can be completely reconstructed by local deformation reconstruction and overall deformation reconstruction. Figure 3 Define these two deformations and the following surfaces:

[0108] The interface between the finger 1 and the sensing layer 11 is defined as the observation surface 1a. Since the thickness of the shading layer 12 is much smaller than the thickness of the sensing layer 11 and the finger 1, the shading layer 12 is approximated as a plane, and the shading layer 12 that is deformed by contact with the object is defined as the contact surface 1b. A virtual contact surface - the non-local deformation surface 1c is defined. This surface is obtained by translating each point of the observation surface 1a along the normal direction by a distance equal to the initial thickness of the sensing layer 11. The non-local deformation surface 1c is considered to be a surface formed by an object pressing the contact surface but only deforming it as a whole. At this time, the normal thickness of the sensing layer does not change. Combined with the above definitions, the overall deformation is defined as the deformation that changes the contact surface from its initial shape to the shape of the non-local deformation surface 1c, and the local deformation is defined as the deformation that changes the contact surface from the shape of the non-local deformation surface 1c to the shape of the real contact surface 1b. The depth of the contact surface 1b entering the non-local deformation surface 1c along the optical axis direction of the camera 6 is defined as the pressing depth 1d. The thickness of the sensing layer 11 after being pressed in the same direction is called the imaging thickness 1e, which determines the brightness of the pressing position in the camera field of view. The distance between the non-local deformation surface 1c and the observation surface 1a along the optical axis direction of the camera is defined as the equivalent thickness 1f of the sensing layer 11, which is equal to the sum of the pressing depth 1d and the imaging thickness 1e. The normal thickness of the sensing layer 11 is defined as the sensing layer thickness 1g, which is determined during casting and does not change with deformation.

[0109] See also Figure 4 , the sensing method of this embodiment includes the following steps:

[0110] Step 1) Reconstruction of the overall deformation of the contact surface

[0111] A real-time image of the object pressing the surface of the visual-tactile finger 10 is obtained by using the camera 6. The real-time image is combined with a distortion-free pinhole camera model, a geometric constraint that the contact surface 1b is of equal width at all locations when the object presses the visual-tactile finger 10, and feature tracking of the marker points in the real-time image to obtain the three-dimensional spatial coordinates of the overall deformation of the contact surface 1c without local deformation, thereby reconstructing the overall deformation of the contact surface 1b.

[0112] Step 2) Reconstruction of local deformation of contact surface

[0113] Step 2-1) Using the camera 6 to record images of the contact surface 1b when it undergoes as many different deformations as possible as reference videos, so as to track the positions of the marking points under various deformations;

[0114] Step 2-2) Press a calibration ball with a known radius onto the contact surface 1b, obtain a calibration image using the camera 6, and select a frame closest to the position of the marking point in the calibration image from the reference video as the calibration reference image, and calculate the normalized brightness difference between the calibration image and the calibration reference image; at this time, it is considered that the partial spherical surface pressed onto the contact surface 1b and the contact surface coincide with each other, and the center position of the calibration ball in the calibration image can be obtained according to the shadow area generated by the calibration ball pressing the contact surface 1b in the calibration image, and the local deformation three-dimensional space coordinates corresponding to the deformation of the shadow area are obtained using the center position and the distortion-free pinhole camera model, and the pressing depth corresponding to each point in the shadow area is calculated using the coordinates and the three-dimensional space coordinates of the overall deformation of the contact surface 1b corresponding to the shadow area obtained according to step 1), and the mapping of the normalized brightness difference and the pressing depth is obtained by fitting the pressing depth corresponding to each point in the shadow area with the normalized brightness difference, and the calibration is completed;

[0115] Step 2-3) Filter out a frame closest to the position of the marked point in the real-time image obtained in step 1) from the reference video recorded in step 2-1) as the real-time reference image, calculate the normalized brightness difference between the real-time reference image and the real-time image, and use the mapping obtained in step 2-2) to obtain the real-time pressing depth of each point on the contact surface 1b, so as to achieve reconstruction of the local deformation of the contact surface 1b;

[0116] Step 3) Complete deformation reconstruction of the contact surface

[0117] The complete deformation of each point on the contact surface 1b is obtained by using the three-dimensional spatial coordinates of the overall deformation of the contact surface 1b obtained in step 1) and the real-time pressing depth of each point on the contact surface 1b obtained in step 2), thereby reconstructing the complete deformation of the contact surface 1b.

[0118] In some embodiments, step 1) mainly uses a distortion-free pinhole camera model, image edge feature tracking, geometric constraint assumptions, and linear interpolation to reconstruct the overall deformation of the contact surface 1b. The specific steps are as follows:

[0119] 1-1) Define the distortion-free pinhole camera model, which is expressed as follows:

[0120] zp′=Kp

[0121] in, is the intrinsic parameter matrix of camera 6, which is calibrated by the existing Zhang Zhengyou method; p′ and p are the pixel coordinates of a pixel point on a frame image and its corresponding three-dimensional coordinates in the camera coordinate system, respectively, and the two are defined as:

[0122] p′=(u,v,1) T

[0123] p = (x, y, z) T

[0124] Among them, the u-axis of the image is parallel to the x-axis of the camera coordinate system, the v-axis of the image is parallel to the y-axis of the camera coordinate system, the origin of the camera coordinate system is set at the optical center of the camera, and the positive direction of the z-axis is set downward, and the superscript T represents transposition;

[0125] 1-2) Assemble the visual-tactile finger, wherein the width direction of the finger is aligned with the v-axis and the y-axis; after the assembly is completed, use the camera 6 to obtain in real time the original image of the contact surface 1b when the object presses the visual-tactile finger of this embodiment, and perform grayscale filtering on the original image to obtain a real-time image; assume that the finger satisfies the following geometric constraints when the overall deformation occurs: in the camera coordinate system, the distance between the point p1 and the point p2 at the same height on the left and right edges of the finger observation surface 1a is always equal to the width E of the finger, so:

[0126] ||p1-p2||2=W

[0127] z1=z2

[0128] Assume that the two pixels corresponding to point p1 and point p2 in the real-time image are p′1 and p′2 respectively. Since the width direction of the finger is aligned with the v-axis and the y-axis, the pixel point p′1 and the pixel point p′2 satisfy:

[0129] u1=u2

[0130] Among them, z1 and z2 are the z-axis coordinates of point p1 and point p2 respectively, and u1 and u2 are the u-axis coordinates of pixel point p′1 and pixel point p′2 respectively;

[0131] 1-3) Track the left and right edges of the observation surface 1a in the real-time image, use the expression in step 1-2) to obtain the three-dimensional spatial coordinates corresponding to all pixel points on the left and right edges of the observation surface 1a, and use linear interpolation to obtain the three-dimensional spatial coordinates p of any pixel point on the observation surface 1a VS ;

[0132] 1-4) Perform polynomial fitting and derivation on the three-dimensional spatial coordinates corresponding to each pixel point on the left and right edge lines of the observation surface 1a to obtain the tangent inclination angle θ at each location, combined with the initial thickness t of the sensing layer 11 v (Right now Figure 3 1g in the figure) can obtain the equivalent thickness t0 (i.e. Figure 3 1f), and then the three-dimensional space coordinates p of any pixel point on the observation surface 1a VS Adding the equivalent thickness t0, we can obtain the three-dimensional spatial coordinates p of each pixel point on the overall deformed contact surface without local deformation, that is, the non-locally deformed surface 1c. LUS , that is, its expression is:

[0133]

[0134] p LUS =p VS +(0, 0, t0) T

[0135] In a specific embodiment of the present application, the camera collects contact surface images with a resolution of 1920×1080 at 30 FPS, uses a checkerboard to calibrate the camera intrinsic parameter matrix K, and the finger width W is 20 mm.

[0136] In some embodiments, step 2) mainly uses a distortion-free pinhole camera model, image marker feature tracking, and local normalized brightness difference to reconstruct the local deformation of the contact surface 1b, and the specific steps are as follows:

[0137] 2-1) Use camera 6 to record images of the contact surface 1b when it undergoes as many different deformations as possible as reference videos, and use a deep learning model to track the positions of the marker points under various deformations and record them.

[0138] 2-2) Calibrate and obtain the mapping M() of normalized brightness difference-pressing depth, such as Figure 5 As shown, the specific steps are as follows:

[0139] 2-2-1) Press a calibration ball 1A with a known diameter R onto the contact surface 1b to cause overall and local deformation of the finger, obtain a calibration image, and track and record the position of its marking points.

[0140] 2-2-2) Compare the position of the marker points in each frame of the reference video recorded in step 2-1) with the position of the marker points in the calibration image, and find the closest frame as the calibration reference image. The comparison method is to calculate the sum of the distances between the calibration image and the marker points in each frame of the reference video. Take the frame of reference video with the smallest sum of the distances as the calibration reference image. The formula is:

[0141]

[0142] in, Represents the sum of the distances between the calibration image and the marked points in the i-th frame of the reference video, m′ C and are the pixel coordinates of the marked points in the ith frame of the calibration image and the reference video, respectively.

[0143] 2-2-3) Subtract the calibration image from the calibration reference image to obtain the brightness difference, and calculate the normalized brightness difference In order to avoid the influence of different distances between the contact position and the light source, the normalization process is expressed as:

[0144]

[0145] Among them, I C and I refC are the brightness of the calibration image and the calibration reference image respectively.

[0146] 2-2-4) After the calibration ball 1A is pressed on the contact surface 1b, the thickness of the contact layer in the pressed area decreases from the equivalent thickness 1f to the imaging thickness 1e, and the reflected light gradually weakens, thereby generating a convex circular contour shadow area in the calibration image. During calibration, the circular contour of the shadow area in the calibration image is circled, that is, Figure 5 The boundary point of the shadow area shown in 1B, and the pixel coordinate c corresponding to the center of the circle outline are marked at the same time. r ′ and the pixel coordinates p of a point on the circle r ′, the three-dimensional distance between the two points is approximately the radius R of the calibration sphere 1A, then:

[0147] ||p r -c r ||2=R

[0148] Among them, c r For c r ′ is the three-dimensional space coordinate of the point corresponding to it in the camera coordinate system, p r For r ′The three-dimensional space coordinates of the corresponding point in the camera coordinate system;

[0149] At the same time, c r and p r The heights in the z direction are equal, so:

[0150] z r =z cr

[0151] Among them, z r 、z cr They are p r and c r The z-axis coordinate of

[0152] Combining the above formula with the expression of the distortion-free pinhole camera model, the rough position c of the sphere center 1C in the camera coordinate system can be obtained: r .

[0153] 2-2-5) Iterate once to estimate the center position of the ball more accurately. Since the shooting distance is far and the horizontal width of the finger is small, it can be considered that the camera is facing the contact surface 1b of the calibration ball 1A, so:

[0154]

[0155] Where L c =||c r||2 is the distance between the center of the sphere 1D, that is, the distance between the center of the sphere 1C and the optical center of the camera, and c is the exact position of the center of the sphere.

[0156] 2-2-6) Using the accurate position c of the sphere center and the distortion-free pinhole camera model, calculate the three-dimensional spatial coordinates p corresponding to each pixel point in the shadow area after deformation D , refer to the method described in step 1) to obtain the original three-dimensional space coordinates p of the point located on the non-local deformation surface 1c corresponding to the shadow area LUSD , p D With p LUSD Subtract the z-axis coordinates of the calibration ball 1A from the contact surface 1b to obtain the pressing depth d of each point on the contact surface when the calibration ball 1A presses the contact surface 1b. D (Right now Figure 3 1d), the calculation formula is:

[0157] d D =z D -z LUSD

[0158] Among them, z D and z LUSD They are p D and P LUSD The z-axis coordinate of .

[0159] 2-2-7) For the independent variable A polynomial fitting is performed on dD to obtain the mapping M() of normalized brightness difference and pressing depth.

[0160] 2-3) Real-time reconstruction of the local deformation of the finger contact surface. During reconstruction, steps 2-2-2) and 2-2-3) are repeated frame by frame for the real-time acquired image to obtain the real-time normalized brightness difference Then the real-time compression depth d is obtained by mapping M() to complete the local reconstruction:

[0161]

[0162] 3) Reconstruct the complete deformation of the contact surface

[0163] Step 1) and step 2) reconstruct the local and overall deformation of the contact surface. Further, this embodiment uses the following formula to reconstruct the complete deformation of the contact surface:

[0164] p CS =p LUS -(0, 0, d) T

[0165] Among them, p CS are the three-dimensional coordinates of any point on the contact surface.

[0166] In a specific embodiment of the present application, the thickness of the sensing layer on the front side of the finger is 2 mm. The calibration uses a small steel ball with a diameter of 6 mm to press on the contact surface. At the same time, the sensing layer in the pressed area becomes thinner, the reflected light becomes less, the brightness becomes darker, and a shadow area with a convex shape and a circular outline appears in the image. The center position of the ball is found through calibration and corresponds to the brightness difference. Then, each pixel in the shadow area can provide a set of pressing depth-brightness difference data. Finally, 15,546 sets of data are collected and fitted into a quadratic polynomial. The subsequent deformation reconstruction uses the existing deep learning model C o -Tracker tracks the position of the marker, with a processing speed of about 0.1 to 0.2 seconds per frame.

[0167] In some embodiments, the sensing method proposed in the present disclosure further includes identifying the force direction of the visual-tactile finger, by real-time tracking of the marker point and summing the offset between the marker point position and its initial position to determine the force direction, specifically:

[0168] The offset between the marker position and its initial position is summed as follows:

[0169] o′=∑(m′-m′ * )

[0170] in, is the overall offset of the marker point in the real-time image, m′ is the pixel coordinate of the marker point in the real-time image, and m′ * is the initial pixel coordinate of the marker point in the image;

[0171] According to the set threshold ε, it is determined whether the finger is in contact with the object. When ||o′||2≤ε, it is determined that the finger is not in contact with the object. When ||o′||2>ε, it is determined that the finger is in contact with the object once. Then, the contact force direction of the finger in the camera coordinate system is defined according to the following formula:

[0172]

[0173] Where o is the unit vector indicating the direction of the contact force, o′ u and o′ u They are the u-axis and v-axis coordinates of the marked points in the real-time image respectively;

[0174] Among all the sides of the finger (four sides in this embodiment), the side of the finger whose normal vector is closest to o is determined as the force-bearing surface, and the method for determining the degree of proximity is the vector inner product.

[0175] It should be noted that, during operation, the finger only uses the side where the sensing layer 11 is located as the gripping working surface, and the other sides can detect force in order to prevent possible false touches, thereby ensuring a timely response.

[0176] In a specific embodiment of the present application, when the force sensor pressure head gradually pushes the finger from different directions and different positions in the z direction 0 to 60 mm away from the fingertip, the sensing method based on the above principle can always detect the deformation of the finger before the force reaches 2N and return to the correct force surface. The experiment was carried out 72 times in total, and each side of the finger was tested 18 times.

[0177] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0178] Although embodiments of the present disclosure have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and alterations may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A flexible visual-tactile finger based on a fin effect structure, characterized in that: include: The finger body is a fin effect structure made of a flexible transparent material, a flexible translucent sensing layer is provided on the surface of the finger body facing the object, and a marking point array is provided on the two side surfaces of the finger body connected to the sensing layer, and a flexible light shielding layer is provided on the surface of the sensing layer and the other side surfaces of the finger body; A finger body base, connected to the root of the finger body, and the finger body base has a transparent bottom plate; The image acquisition unit includes a light-proof shell fixed to the bottom of the finger base, a light source and a camera arranged in the shell, the camera is located directly below one side of the finger with the sensing layer, and the light source is arranged around the lens of the camera.

2. The visual-tactile finger according to claim 1, characterized in that: The wall thickness of the finger body on the side where the sensing layer is provided gradually increases as the position approaches the finger root.

3. The visual-tactile finger according to claim 1, characterized in that: The sensing layer is made of a soft material with a hardness of 5A.

4. The visual-tactile finger according to claim 1, characterized in that: The marking points in the marking point array follow the rule that the radius and spacing of the marking points increase in proportion to the distance from the camera, so as to ensure that the spacing and size between the marking points in the image tend to be consistent.

5. The visual-tactile finger according to claim 1, characterized in that: The color of the light source is single white, and an isolation ring is installed between the camera and the light source to prevent light spots from appearing in the image taken by the camera.

6. A flexible gripper, characterized in that: include: A plurality of fingers, wherein the fingers are visual-tactile fingers according to any one of claims 1 to 5; The driving mechanism is connected to the shell of each finger and is used to provide driving force for each visual-tactile finger to achieve grasping and releasing of the object.

7. A sensing method based on the visual-tactile finger according to any one of claims 1 to 5, characterized in that: include: Step 1) Reconstruction of the overall deformation of the contact surface: The shading layer that is deformed by contact with the object is defined as a contact surface, a real-time image of the object pressing the surface of the visual-tactile finger is obtained by using the camera, the real-time image is combined with a distortion-free pinhole camera model, a geometric constraint that the width of each part of the contact surface is equal when the object presses the visual-tactile finger, and feature tracking of the marker points in the real-time image to obtain the three-dimensional spatial coordinates of the overall deformation of the contact surface without local deformation, so as to achieve reconstruction of the overall deformation of the contact surface, and the contact surface without local deformation is defined as a non-locally deformed surface; Step 2) Reconstruction of local deformation of contact surface: Step 2-1) using the camera to record images of the contact surface when it undergoes as many different deformations as possible as reference videos to track the positions of the marker points under various deformations; Step 2-2) Press a calibration ball with a known radius onto the contact surface, use the camera to obtain a calibration image, and select a frame closest to the position of the marked point in the calibration image from the reference video as the calibration reference image, and calculate the normalized brightness difference between the calibration image and the calibration reference image; obtain the center position of the calibration ball in the calibration image according to the shadow area generated by the calibration ball pressing the contact surface in the calibration image, and use the center position and the distortion-free pinhole camera model to obtain the local deformation three-dimensional space coordinates corresponding to the deformation of the shadow area, and use the coordinates and the three-dimensional space coordinates of the overall deformation of the contact surface corresponding to the shadow area obtained in step 1) to calculate the pressing depth corresponding to each point in the shadow area, and use the pressing depth corresponding to each point in the shadow area and the normalized brightness difference to fit the mapping of the normalized brightness difference and the pressing depth to complete the calibration; Step 2-3) selecting a frame closest to the position of the marked point in the real-time image from the reference video as the real-time reference image, calculating the normalized brightness difference between the real-time reference image and the real-time image, and using the mapping to obtain the real-time pressing depth of each point on the contact surface to achieve reconstruction of the local deformation of the contact surface; Step 3) Complete deformation reconstruction of the contact surface: The complete deformation of each point on the contact surface is obtained by subtracting the real-time pressing depth of each point on the contact surface from the three-dimensional spatial coordinates of the overall deformation of the contact surface, thereby realizing reconstruction of the complete deformation of the contact surface.

8. The sensing method according to claim 7, characterized in that: Step 1) specifically includes the following steps: 1-1) Define the distortion-free pinhole camera model, which is expressed as follows: zp′=Kp in is the camera intrinsic parameter matrix; p' and p are the pixel coordinates of a pixel on a frame image and its corresponding three-dimensional coordinates in the camera coordinate system, respectively, and the two are defined as: Among them, the u-axis of the image is parallel to the x-axis of the camera coordinate system, the v-axis of the image is parallel to the y-axis of the camera coordinate system, the origin of the camera coordinate system is set at the optical center of the camera, and the positive direction of the z-axis is set downward; 1-2) Assemble the visual-tactile finger in a manner that the finger width direction is aligned with the v-axis and the y-axis, and use a camera to obtain a real-time image when an object presses the visual-tactile finger; define the interface between the finger body and the sensing layer as an observation surface, and assume that the finger satisfies the following geometric constraints when it is deformed as a whole: in the camera coordinate system, the distance between the point p1 and the point p2 at the same height on the left and right edges of the observation surface is always equal to the width W of the finger, so: ||p1-p2||2=W z1=z2 Assume that the two pixel points corresponding to the point p1 and the point p2 in the real-time image are p'1 and p'2 respectively, and the pixel points p'1 and p'2 satisfy: u1=u2 Among them, z1 and z2 are the z-axis coordinates of point p1 and point p2 respectively, and u1 and u2 are the u-axis coordinates of pixel point p'1 and pixel point p'2 respectively; 1-3) Track the left and right edges of the observation surface in the real-time image, use the expression in step 1-2) to obtain the three-dimensional spatial coordinates corresponding to all pixels on the left and right edges of the observation surface, and use linear interpolation to obtain the three-dimensional spatial coordinates p of any pixel point on the observation surface VS ; 1-4) Mark the three-dimensional space coordinates VS Perform polynomial fitting and derivation to obtain the tangent inclination angle θ at each location, and obtain the three-dimensional spatial coordinates p of each pixel point on the overall deformation contact surface without local deformation according to the following formula: LUS : Among them, t v is the initial thickness of the sensing layer, and t0 is the equivalent thickness t0 of the left and right edges of the observation surface along the camera optical axis.

9. The sensing method according to claim 7, characterized in that: Step 2-2) specifically includes the following steps: 2-2-1) Press a calibration ball with a known diameter R onto the contact surface to cause overall and local deformation of the finger, and obtain a calibration image; 2-2-2) Compare the position of the marking point in each frame of the reference video with the position of the marking point in the calibration image, and find the closest frame as the calibration reference image. The comparison method is to calculate the sum of the distances between the calibration image and the marking points in each frame of the reference video. Take the frame of reference video with the smallest sum of the distances as the calibration reference image. The formula is: Among them, m' c and are the pixel coordinates of the marked points in the ith frame of the calibration image and the reference video respectively; 2-2-3) Calculate the normalized brightness difference according to the following formula: Among them, I C and I refC are the brightness of the calibration image and the calibration reference image respectively; 2-2-4) Circle the circular outline of the shadow area produced by the calibration ball pressing on the contact surface in the calibration image, and mark the pixel coordinate c corresponding to the center of the circle. r 'And the pixel coordinates p of a point on the circle r ', the three-dimensional distance between the two points is approximately the radius R of the calibration sphere, then: ||p r -c r ||2=R With r =from cr Among them, c r For c r 'The three-dimensional space coordinates of the corresponding point in the camera coordinate system, that is, the center position of the sphere, p r For r 'The three-dimensional space coordinates of the corresponding point in the camera coordinate system, z r and z cr They are p r and c r The z-axis coordinate of Combining the above formula with the expression of the distortion-free pinhole camera model, the rough position of the center of the sphere can be obtained; 2-2-5) Update the rough position of the center of the ball according to the following formula: Among them, L c =||c r ||2 is the distance between the center of the sphere and the optical center of the camera, and c is the exact position of the center of the sphere; 2-2-6) Using the accurate position c of the sphere center and the distortion-free pinhole camera model, calculate the three-dimensional spatial coordinates p corresponding to each pixel point in the shadow area after deformation D , refer to the method described in step 1) to obtain the three-dimensional spatial coordinates p of the point on the non-locally deformed surface corresponding to the shadow area LUSD , and the pressing depth d of each point on the contact surface is obtained according to the following formula D : d D =z D -z LUS Among them, z D and z LUS They are p D and p LUSD The z-axis coordinate of 2-2-7) For the independent variable and d D Perform polynomial fitting to obtain the mapping M() between normalized brightness difference and pressing depth; Step 2-3) specifically includes the following steps: The normalized brightness difference is obtained by performing the operations of steps 2-2-2) to 2-2-3) on the real-time image frame by frame. Then the real-time compression depth d is obtained by mapping M() to complete the local reconstruction:

10. The sensing method according to claim 8, characterized in that: The sensing method further includes determining the direction of the force applied to the finger by real-time tracking of the marker point and summing the offset between the marker point position and its initial position, specifically including the following steps: The offset between the marker position and its initial position is summed as follows: o'=∑(m'-m' * ) in, is the overall offset of the marker point in the real-time image, m' is the pixel coordinate of the marker point in the real-time image, and m' * is the initial pixel coordinate of the marker point in the image; The threshold ε is set to determine whether the finger is in contact with the object. When ||o'||2>ε, it is determined that the finger is in contact with the object once, and the contact force direction of the finger in the camera coordinate system is defined according to the following formula: Where o is the unit vector indicating the direction of the force, o' u and o' v They are the u-axis and v-axis coordinates of the marked points in the real-time image respectively; The side of the finger whose normal vector is closest to o among all the sides of the finger is determined as the force-bearing surface, and the method for judging the degree of proximity is the vector inner product.

Citation Information

Patent Citations

  • Structured light technology-based projector calibration method

    CN108230399A

  • Manipulator grabbing method based on tactile-visual fusion perception and manipulator

    CN115625713A

  • Grabbing device control method based on machine vision, grabbing device and storage medium

    CN117381799A

  • Visual tactile finger for mechanical jaw

    CN117773985A

  • Robotic manipulator with visual guidance & tactile sensing

    US20230073681A1

Cited By

  • Flexible visual tactile finger and gripper based on multi-mirror optical system

    CN121340344A