Tactile Sensor

Abstract

A tactile sensor includes a cap having a top surface and a lower surface. The lower surface includes pins, each having a marking. A portion of the lower surface is attachable to a device. A camera is located in a field of view of the marking, capturing an image of the marking in motion by elastic deformation of the top surface of the cap. A processor receives the captured image and determines a set of relative positions of the marking in the captured image by identifying image position coordinates measured in the image among the captured images. A stored machine vision algorithm is used to determine a net force tensor acting on the top surface by comparing the set of relative positions of the marking with a stored set of previously learned relative positions of the marking in motion. The device is controlled via a controller in response to the net force tensor determined in the processor.

Description

Technical Field

[0001] The present disclosure relates to tactile sensing, and more particularly, to a tactile sensor having a plurality of micromechanical pins attached to the underside of an elastomeric cap and, once an external force is applied to a top surface, converts the pattern of force into a measurable quantity. Background Art

[0002] Robots are used for assembly, but correct assembly depends on the robot always holding the part in exactly the same position and orientation. Incorrect gripping results in assembly failure, damaged parts, and possible damage to the gripper or the robot itself. Even if a part is gripped correctly, if the received part is incorrect (or the part is contaminated or deformed), the robot will try to force the assembly, resulting in poor assembly, damaged parts, or damage to the robot or robot gripper. For this reason, tactile sensors are used in robotic applications, for example, to assist in correct assembly and various tasks of modern devices.

[0003] In robotics, tactile sensors provide valuable data about the state of contact between a robotic hand and an object when grasping the object. The action of grasping an object by a robotic hand or gripper may include different physical effects for ensuring a stable grip between the gripper and the object to be grasped. For example, some effects may include: (a) an impact effect (i.e., like a forceps that physically grasps by impact directly acting on the object); (b) an invasion effect (i.e., a row of needles that physically penetrate the surface of the object); (c) a contraction effect (i.e., an attractive force applied to the surface of the object, such as a vacuum effect); (d) or an abutment effect (i.e., direct contact is required for adhesion to occur, such as glue).

[0004] Tactile sensors can identify the presence and pressure of an object, as well as the position of an object in a gripper for robotic applications that include assemblies. However, most robotic sensors are based on designs similar to pressure sensors and only measure compression forces, without considering shear motion or other types of forces or pressures. Among these, shear sensors are useful for detecting the motion of a gripped object. Summary of the invention

[0005] The present disclosure relates to tactile sensing, and more particularly to a tactile sensor having a plurality of micromechanical pins attached to the underside of an elastomeric cap and which convert the magnitude of the force into a measurable quantity once an external force is applied to an outer impact surface of the cap.

[0006] Some embodiments of the elastomeric cap include mechanical flexure amplification pins, i.e., micromechanical pins (hereinafter referred to as "pins") that flex under pressure, shear, and torque. Each tip of the pin carries at least one mark or reference mark, such as white or colored paint or retroreflective paint, where these marks can be easily tracked using machine vision algorithms against a dark background. The flexural motion of the elastomeric cap under pressure, bias pressure, shear, or torque can be recorded, resulting in a higher sensitivity level than the sensitivity level of a conventional tactile sensor. In fact, the pin can be modified to include one or more lever arms extending away from the pin, i.e., similar to a tree-like figure or shape; wherein at least one traceable mark can be included at the tip of each lever arm. Based on experiments, adding one or more lever arms to the pin results in a higher sensitivity amplification level than the sensitivity amplification level of the pin without one or more lever arms. Therefore, one or more lever arms amplify the sensitivity amplification level of each pin, resulting in a higher sensitivity of the elastomeric cap.

[0007] Unlike conventional tactile sensors that can only record or measure compressive forces without considering shear movement or other types of forces or pressures, the present disclosure is sensitive to compressive forces, shear, torque, extrusion, and expansion forces on the elastomeric cap. Embodiments of the present disclosure are configured to record and measure patterns of forces acting on the outer impact surface and causing flexural motion of the outer impact surface. For example, the pattern of forces recorded or measured using one or more cameras or webcams includes flexural motion of the outer impact surface under pressure, bias pressure, shear force, or torque force. By way of non-limiting example, one or more cameras observe a marker or reference marker at an angle of tilt (0 to ~20) degrees or at some other angle. At least one embodiment of the present disclosure includes the use of a machine vision algorithm that matches the marker motion with a pre-stored or pre-learned set of marker flexures using interpolation (if necessary) to determine the pattern or net force tensor of the force on the outer impact surface of the elastomeric cap.

[0008] In more detail, and in a simple preferred embodiment, there may be tactile sensors, i.e., elastomeric tactile sensors embedded in the fingers of a robotic gripper; preferably, the robotic gripper includes controllable force capabilities, rather than single-bit "close with full force" and "open with full force" controls. For example, as the gripper closes, a pattern of forces is generated against the outer impact surface, and a camera records an image of the elastomeric cap or the underside of the cap, which includes markings located on markings or micromechanical pins. Since the micromechanical pins are very light in mass, very long compared to their diameter, and freely moving, they provide a 3-30 times extension of the movement caused by the strain of the elastomeric cap, which also makes even small angular deformations highly visible and therefore highly measurable. As will be described in further detail, the elastomeric cap, specifically the pins and lever arms with markings, can be customized according to different types of intended measurable sensitivity, which can depend on the type of pattern of forces applied to the outer impact surface of the cap.

[0009] For example, aspects that are customized to obtain different measurable sensitivities may include: customization of different structural configurations of the elastomeric cap, including a combination of pattern and structural configuration (attributes) of the outer impact surface or pin configuration on the lower surface. At least one sample of customization may include: a combination of attributes, i.e., ridges, cones, smoothness, rough textures, in conjunction with the pin structural design on the underside of the cap (i.e., each pin position on the lower surface, pin diameter, amount of elasticity of each pin, pin length, number of rods per pin, pattern of pin structural design); customized part-specific shapes that fit the specific items to be assembled in a specific robotic task, especially those patterned on the outer impact surface. The reason for customization may be to determine a higher level of measurable sensitivity for a specific force pattern or a single force according to a user-specified application. For example, if the gripper is configured to undertake a series of tasks or movements involving shear forces, the user is able to structurally configure a combination of attributes and structural design of the pins together, which will result in recording a higher level of measurable sensitivity specific to shear forces as well as a pattern of forces, if desired.

[0010] At least one realization gained from the experiments is that the discovered aspects of the customized measurable sensitivity to a specific force pattern or a single force include a micromechanical rod attached or positioned at the interface between the flat portion and the radius section of the underside of the cap. It was found that when the cap was depressed 0.1 mm, the radius section would be forced to change curvature and bend more tightly. More precisely, the radius section would deform from a 10 mm radius to a 9.9 mm radius. Assuming that the cap material is relatively incompressible and bends rather than bulges, this translates into a change in the arc length of the cap from exactly 90 degrees and 15.7079 mm long without deformation to a constant length of 15.7079 mm on a 9.9 mm radius, such that 15.7079 = 9.9 mm * new angle, and the new angle is 1.5866 radians or 90.905 degrees, which is a deformation of 0.905 degrees in radius. Since the flat top of the outer impact surface of the cap is supported by the object being pressed, the flat top of the outer impact surface will remain mostly flat, and most of the accommodation will be in the sidewalls. Therefore, the micromechanical pin will move upward 0.1 mm (with the cap) and bend outward about 0.9 degrees.

[0011] Assuming our micromachined pin is 20mm long, consistent with our experimental results, the lateral displacement due to the tilt in the micromachined pin is sin(0.9 degrees)*20mm=.31mm. The pin also moves vertically, with a total movement of just under 0.4mm. This shows a gain of about 4 times, which is significant, and the sensitivity increase can be measured via the markings on the pin specific to the pattern of force applied to the outer impact surface of the cap.

[0012] Of course, this is an oversimplification due to the two-dimensional nature of the example. Two effects need to be addressed: first, the edge of the elastomeric cap at the base of the radius is fixed in position and angle relative to the camera, and second, the base edge of the cap has a much larger stress-bearing cross-section than the flat junction of the cap top. To a first approximation, the cross-section at the fixed root of the elastomeric cap is about twice that at the flat top, and the microrod experiences double the strain (and deformation, and tilt). Second, the edge of the cap being fixed pushes deformation into the central region of the cap, again doubling the strain (and deformation, and tilt) experienced at the base of the microrod.

[0013] In summary, for 0.1 mm deformation on the outer impact surface of the elastomeric cap, there was tip motion of the microrod tip that experienced approximately 1.6 mm of movement, which was predicted based on hand calculations and finite element modeling, and later confirmed by actual experiments using 3D printed elastomeric caps with microrods and fiducial markers. Most of this motion amplification was due to the tip and tilt experienced at the base of the microrod, and there was no effect on the desired aspects of tipping, tilting, and rotation in response to external surface forces, such as reinforcing or supporting the gel, which is at least one important aspect of the present disclosure.

[0014] The observed 1.6mm motion of the fiducial mark is greater than the diameter of the pin itself, and is a much larger motion than the direct observation of a surface moving at 0.1mm. This is the reason and purpose of the free moving micro lever in the present disclosure; this is due to the very large gain of 16 times in the observed displacement due to the strain deformation of the elastomeric cap. This observed displacement is very noticeable and can be measured using a simple camera such as a webcam.

[0015] Due to this microrod deformation gain, the images captured by the camera have larger motion in the fiducial markers and better signal-to-noise ratio compared to traditional tactile sensor methods, which directly image the top surface of the contact surface through grazing illumination or through a hemispherical gel proxy.

[0016] The use of independently moving microrods also allows different parts of the cap to experience and report very different local strains and forces. For example, one part of the cap could be in shear from left to right, and the other side of the cap could be in shear from right to left, and because the microrods are free to move rather than being trapped in a continuous gel, the custom configuration of the sensor is able to correctly detect this as a surface torque, rather than being insensitive to a specific set of forces, as the hemispherical gel is insensitive to these specific sets of forces.

[0017] Several experiments were conducted to generate force signal outputs using the motion of the fiducial markers amplified by these microrods. Some aspects considered included: structurally arranging a fixed camera at different locations within the cavity of the gripper. For example, the fixed camera captured an image of a fiducial marker attached to the pin. The image was sent to a computer, where, according to at least one experiment, an inexpensive webcam was connected via USB to a computer running Debian Linux software, but obviously other experiments included different types of cameras, camera locations, camera mounting arrangements, and other types of software.

[0018] On a computer, image processing software can be used to extract the XY position of each fiducial marker. Obviously, there are many different but similar image processing software methods, but the preferred method involves using the open source package "OpenCV". At least one realization gained is that not every marker needs to be in the field of view of one camera or multiple cameras, all of which will depend on the specific application.

[0019] Because the number of such XY pairs is equal to the number of fiducial markers and is small (at least for some of these particular experiments, only seven fiducial marker-bearing pins were used), the next step is to pair-match the set of observed fiducial marker XY positions against a pre-stored library of fiducial marker XY positions generated earlier by providing known and controlled forces on the elastomeric cap and recording the XY positions of the fiducial markers found by the image processing chain. For example, for a total of 7+4*4+8+2*(3+3*3)=55 different sets of XY positions, each set containing multiple individual XY positions, one for each fiducial marker, we can apply 0, 1, 2, 5, 10, 15, and 20 Newtons of compressive force at each of six locations on the cap (which we define as "zones"), 1, 2, 5, and 10 Newtons of shear force in each of the four principal in-plane directions for a typical cap, -10, -5, -2, -1, 1, 2, 5, and 10 Newton meters of torque, 2, 5, and 10 Newtons of compression in the axial direction, and 2, 5, and 10 Newtons of lateral compression and expansion at three different axial locations (zone 1 to zone 2, zone 3 to zone 4, and zone 5 to zone 6). For experiments using an elastomeric cap with seven pins, each carrying a fiducial marker, this results in seven XY positions associated with each of the 55 known and controlled force settings.

[0020] Each set of XY positions associated with a particular known and controlled force setting is labeled; the labels may be human understandable text strings such as NO_CONTACT, 10_NT_COMPRESSION, 2_NT_SHEAR_EAST, 2_NT_CENTRAL_PINCH, etc. Each set of XY positions may also be associated with a force vector, say, Z1 (compression in zone 1), Z2 (compression in zone 2), through Z6 (compression in zone 6) X-shear (negative numbers indicate shear in the opposite direction), Y (shear), A (total torque on the X axis and P (squeeze) 0 (axial squeeze) and forces in P1 through P3 (zone 1 / zone 2 squeeze / expand, zone 3 / zone 4 squeeze / expand, and zone 5 / zone 6 squeeze / expand).

[0021] For this particular set of experiments, fewer than 50 pre-stored XY sets are stored in the memory of the pre-stored library database, providing excellent resolution and usability. Of course, more than 50 pre-stored XY sets can be stored in the pre-stored library database, but for these sets of experiments, 50 pre-stored XY sets appear to be sufficient to achieve some goals according to some aspects of the present disclosure.

[0022] The next step involves using the Pythagorean metric (square root of the sum of the squares of the X and Y differences); determining the best match of the observed fiducial XY position to the pre-stored XY sets in the library. At least one surprising result is that this calculation is very fast; each observed XY position is compared to each pre-stored XY position in the pre-stored library; a "running calculation" is kept for each of the approximately 50 pre-stored sets, based on the assumption that each fiducial marker is represented only once in each pre-stored set, so each pre-stored set running calculation is incremented by the Pythagorean distance between the observed XY fiducial position and the closest pre-stored XY position of any fiducial in any given pre-stored set. In some applications, during experimentation, we found that there was no need to keep track of which observed XY fiducial position corresponds to which XY pin; just "closest XY, in some applications where we don't care about copying or reuse", which leads to excellent results with extremely simple coding.

[0023] More strictly speaking, the Pythagorean metric in the preferred embodiment working for a cap with k reference pins can be expressed as follows:

[0024] For any observed XY set OBS, it contains k XY pairs OBS_X[0...k-1], OBS_Y[0...k-1]

[0025] And, any stored set of XY positions STO, containing STO_X[0...j-1], STO_Y[0...j-1], uses the function PDIST(arg1,arg2,arg3,arg4) defined as arg1 minus arg2 squared plus the square root of arg3 minus arg4 squared (i.e., SQRT((arg1–arg2)^2+(arg3–arg4)^2); this is the Pythagorean distance formula.

[0026] Then, the preferred distance value between any observed (OBS)XY set and any stored (STO) set is:

[0027] ·Preferred_Distance=SUM_over_all k(MIN_over_all_j(PDIST(OBS_X[k],STO_X[k],OBS_Y[k],STO_Y[k])))

[0028] As mentioned above, the shocking result from this experiment is that this calculation is incredibly fast; with 10 fiducial markers and 100 pre-stored library sets, there are a total of 1000 XY pairwise comparisons, which totals to 2000 square operations, 1000 addition operations, and 1000 square root operations or about 4000 floating point operations. This means and recognizes that this is a very tractable method with excellent results, and in terms of process configuration, even the cheapest microprocessor performs at well above the camera frame rate. Of course, more than 10 markers can be tested, this particular amount was used for the experiment.

[0029] The next step involves running a sum of calculations using Pythagorean distances to sort the list of pre-stored XY position sets into an ordered quality of match list in order from lowest Pythagorean distance (best match) to lowest (worst match).

[0030] In some applications, a single best match may be the only expectation and output of the sensor processing. For example, "very good clamping" vs. "no object found" vs. "too tight clamping" vs. "clipped on edge" may be the labels for some of the stored XY position sets, and so these will be output by the sensor processor. Note that there is nothing prohibited from having multiple stored XY position sets with the same label; there may be five to ten "clipped on edge" possibilities, and depending on the application, it may not be necessary or even useful for the downstream system to know exactly which one is the best match - what matters is that they are all edge clamped.

[0031] In other applications, a sensor result may be output, upon selection, that is some particular weighted, inversely weighted, or linear combination of the forces that were in effect when a particular stored set of XY positions was recorded.

[0032] It is also reasonable to limit the number of stored XY sets to be evaluated in weighting, inverse weighting, or linear interpolation; typically, the best matching (lowest Pythagorean distance) two or three stored XY position sets give the best accuracy for one-dimensional problems, 4 to 6 for two-dimensional problems, 6 to 8 for three-dimensional problems, etc. As will be appreciated from implementation, the disclosed method is highly customized to the user's specific application needs / requirements (i.e., regarding the types of different types of force patterns or measurable sensitivity of a single force), all while being executed at well above the camera frame rate using very inexpensive components (including the use of an inexpensive microprocessor).

[0033] As described above, the pin transforms something that is difficult to measure (such as a pattern of forces or a net force tensor) into something that is easy to measure. That is, the measurement is made easier based on the image motion of the tip or end of a free micromechanical pin, i.e., the image motion of a marker at the outermost end of the pin is tracked by a camera, such as a cheap webcam, that views the pin tip at an angle. The pin tip can be continuously viewed by the camera before the pin tip moves (i.e., before a force is applied to the outer impact surface of the elastomeric cap), during the pin tip movement (i.e., while a force is being applied to the outer impact surface of the elastomeric cap), and after the pin tip movement stops (i.e., after the externally applied force has stopped).

[0034] Some other aspects learned while experimenting with different combinations of properties of the outer impact surface and combinations of pin configurations resulted in useful realizations, such as:

[0035] A) intentionally causing the pins of the present disclosure to become substantially non-parallel even under symmetrical forces, which was later discovered to be at least a key aspect of detecting asymmetric forces, shear, and torque on the pin tip;

[0036] B) Some configurations of the elastomeric cap of the present disclosure are constructed with fewer pins, by way of non-limiting example, the number of pins for a thumb-sized sensor is in the range of 5 to 10 pins to obtain a substantially measurable force tensor. Larger numbers of pins, such as over 100 pins or 200 pins, do not provide measurable sensitivity levels that meet some of the goals of the present disclosure, so for some embodiments of the present disclosure, a much smaller number of pins is targeted.

[0037] C) Asymmetric Pins: Some embodiments of the present disclosure intentionally place or arrange fewer pins asymmetrically on the underside of the cap. In this way, between the rounded corners and the flat surface of the cap, some pins are placed at lower angles of the asymmetric cap or on the edge of the cap. Some experimental results show greater sensitivity of the cap compared to pins located in the middle of the cap (i.e., not on the edge). In addition, the pin lengths can be staggered, and for highly optimized sensors, for example, when using finite element modeling (FEM), the FEM may indicate that some pins have asymmetric (highly elliptical or even flying buttress-like) bases. In this way, even for symmetric forces, this facilitates non-parallel motion; and

[0038] D) Oblique Camera Angle: Instead of looking along the Z axis (which requires 75-100mm vertical height), look at an oblique angle (typically around 10 degrees, and staggered pin lengths, allowing a zero degree viewing angle, parallel to the plane of the gripper tip). It is therefore appreciated that this provides the opportunity to use a sensor that is only about 10mm to 20mm thick without losing sensitivity to force, shear, and torque, and allows the use of the elastomeric cap sensor of the present disclosure for both compact spaces and compact space assemblies. By way of non-limiting example, an additional benefit / advantage is that the sensor can be used in a "teaching glove" worn by a human that records the appropriate force tensors to be applied during a robotic assembly operation. Note that other camera angles can be utilized, including 0-10 degrees, 10-15 degrees, 10-20 degrees, and 15-30 degrees.

[0039] In summary, by way of non-limiting example, some of the recognized structural configurations of the discovered caps may include: (1) dimensional thickness, such as uniform thickness or non-uniform thickness of the outer impact surface; (2) dimensional shape, such as uniform cap shape (such as an elliptical cap shape, a semi-circular cap shape, a hemispherical cap shape, etc.), non-uniform shape; (3) dimensional height, width, length, and depth of the cap; (4) the type of material of the cap, and the location of the different types of materials that make up the cap; (5) how the cap is constructed / manufactured, including the type of three-dimensional (3D) printing, molding manufacturing process, etc. In addition, the number of pins to be mounted / attached / molded to the underside of the elastomeric cap can depend on many factors (as learned from experimentation), for example, it is recognized that the number of pins can depend on the specific application based on the user's intent and goals in terms of achieving a predetermined level of sensitivity, durability, etc.

[0040] Another imaging processing method that was tested and will be used with some embodiments of the present disclosure can be a finite element modeling (FEM) method. For example, to process the data generated from the captured images of the camera including the movement of the pin, the FEM method can be used to process the data into a metamaterial design. A metamaterial is an object whose internal structure produces a final object whose physical property (electrical, magnetic, mechanical, thermal) properties are significantly different from the bulk properties of its "real" constituent materials, that is, negative ε, μ, almost arbitrary light speed c, Poisson's ratio ν, or thermal conductivity κ. Because the cap has an elastomeric outer surface that behaves like a moderately hard rubber (Poisson's ratio close to 1.0 and elastic modulus on the order of 0.01 GPai, typical values ​​for rubber), while the internal micro-rods move freely within the observation area, simulating a material with a Poisson's ratio close to 0.0 and an elastic modulus also close to 0), analyzing the cap as a unit must either simulate the cap as a multi-component metamaterial, or must include fine details of the cap, cap edge attachments, cap texture and surface feature elements, and micro-rods. This second method is preferred.

[0041] Using this second approach, the FEM model can predict the elastomer deformation under different loads. In addition, the elastomer pin lifts the surface contour and converts the surface tilt into an XY motion that is easy to image, i.e., using a low-cost webcam image of the pin tip. Moreover, the structural design of the cap with few fiducial markers provides simple image thresholding with no inter-frame tracking / ambiguity. In fact, the motion of the pin tip, which is converted into a force map, can be tracked using OpenCV. This minimizes the computational cost and maintains a high frame rate.

[0042] During the experiments, the inverse FEM proved to be computationally expensive. It was realized that the training of the model should use real data. One aspect further learned from the experiments was that for the case of labels with grasp position, angular pose, shear and torque, using the Euclidean distance in 14-dimensional space proved to be beneficial. For example, the computation time was very fast, i.e., the amount of computational cost proved to be very low, resulting in the ability to process in real time with excellent visualization on a 1GHz cpu (e.g., Raspberry Pi Zero).

[0043] In many cases, it may not be necessary to calculate the actual force tensor or force pattern, such that a simple comparison of the pin tip position to a preset library of correct and incorrect images may be sufficient to determine whether the machine grasp is appropriate or inappropriate, or to interpolate a library of known deformations to produce a good estimate of the actual force tensor on the sensor in real time.

[0044] Other embodiments of the present disclosure also address today's industrial needs by providing benefits such as low cost, low mass, and low profile simple compact tactile sensors while delivering highly sensitive sensing of external forces applied to the elastomeric tactile sensors. Some of these benefits allow the compact tactile sensors of the present disclosure to be used in technologies where existing conventional tactile sensor use has proven to be too costly for various technical industries. For example, simply placing delicate foods such as fruit, sushi, or baked goods requires a degree of touch control that makes such labor expensive for human employees, but a robot equipped with an elastomeric sensor will be a much cheaper place.

[0045] As described above, most conventional robotic grippers have many problems, including little or no tactile sensing, which poses problems for conventional robots used in flexible assembly lines. At least one of the many reasons is that these conventional dedicated grippers designed for high-precision part grasping and grasp verification cannot be used in flexible assembly lines because they are not operationally capable of incorporating a certain flexibility that allows the gripper to react to changes (whether predicted or unpredictable). Similarly, conventional robots have a low level of ability to know whether the actual assembly operation was successful. In order to overcome these problems of conventional robots, some embodiments of the present disclosure are configured to provide, by way of non-limiting example, a mechanical robotic universal gripper finger tip with recordable, measurable sensitivity of a usable, force mode, a single force, or a total force set in terms of pressure, translation, rotation, shear, compression, and extension for flexible assembly lines.

[0046] According to an embodiment of the present disclosure, a tactile sensor for a robot includes an elastically deformable element, which includes an outer impact surface and a lower surface having pins, ridges, or both. Each lower surface pin or ridge includes a mark, and the elastically deformable element is attached to a gripper of the robot. A camera is located in a cavity of the gripper to capture an image of the mark. The memory has stored data, which includes image data of a set of pre-learned mark positions with corresponding pre-learned force patterns, each set of pre-learned mark positions corresponding to the pre-learned force pattern. An image processor is operably connected to the camera and the memory. The image processor is configured to detect the mark position in the captured image. Based on a distance function, the mark position detected in the captured image is compared with the set of pre-learned mark positions to determine the quality of the matching value of each set of pre-learned mark positions. The best quality matching pre-learned force pattern is determined using a user-selected best matching function applied to the determined quality matching value to calculate the pre-learned mark positions of the best matching set and the corresponding best quality matching pre-learned force pattern. A pattern of forces acting on the elastically deformable element is identified based on the determined best matching pre-learned force pattern. The identified pattern of forces is output to initiate a gripper action of the gripper based on the identified pattern of forces acting on the deformable element.

[0047] According to another embodiment of the present disclosure, a tactile sensor for a robot has a deformable element, which includes an outer impact surface and a lower surface with pins, ridges, or both. Each lower surface pin or ridge includes a mark, and the deformable element is attached to a gripper of the robot. A camera is located in a cavity of the gripper to capture an image of the mark. The memory has stored data, and the stored data includes image data of a set of pre-learned mark positions with corresponding pre-learned force patterns, each set of pre-learned mark positions corresponding to the pre-learned force pattern. The tactile sensor includes an image processor operably connected to the camera and the memory. The image processor is configured to: detect the position of the mark in the captured image. Based on a distance function, the detected mark position in the captured image is compared with the set of pre-learned mark positions to determine the quality of the matching value of each set of pre-learned mark positions. The best matching function selected by the user is used to determine the pre-learned force pattern of the best quality match. Based on the determined best matching pre-learned force pattern, the pattern of force acting on the deformable element is identified. The identified pattern of forces is output to initiate a clamping action of the clamper based on the identified pattern of forces acting on the deformable element.

[0048] Another embodiment of the present disclosure, a method for tactile sensing, having: an elastically deformable element, the elastically deformable element having an outer impact surface and a lower surface having pins, ridges, or both, the lower surface pins or ridges including a marker. The elastically deformable element is attached to a gripper of a robot. A camera is located in a cavity of the gripper to capture an image of the marker. The method includes: capturing an image from a camera. Accessing a memory having stored data, the stored data including image data of a set of pre-learned marker positions having a pre-learned force pattern, each set of pre-learned marker positions corresponding to a pre-learned force pattern. Detecting the position of the marker in the captured image. Evaluating the captured image to detect each marker position in the captured image, and comparing the marker position detected in the captured image with the set of pre-learned marker positions of the image data based on a distance function to obtain a quality of matching value for each set of pre-learned marker positions. Applying a user-selected best matching function to the quality matching value to calculate a best matching set of pre-learned marker positions and a corresponding best quality matching pre-learned force pattern. Based on the determined best matching pre-learned force pattern, a force pattern acting on the elastically deformable element is identified. The identified force pattern is output to initiate a gripper action of the gripper based on the identified force pattern acting on the elastically deformable element.

[0049] Another embodiment of the present disclosure includes a tactile sensor for a robot, the tactile sensor having a deformable element, the deformable element including an outer impact surface and a lower surface having pins, ridges, or both. Each lower surface pin or ridge includes a marker. The deformable element is attached to a gripper of the robot. A camera is located in a cavity of the gripper to capture an image of the marker. The memory has stored data, the stored data including image data of a set of pre-learned marker positions with corresponding pre-learned force patterns, each set of pre-learned marker positions corresponding to the pre-learned force pattern. The tactile sensor includes an image processor operably connected to the camera and the memory. The image processor is configured to: detect the position of the marker in the captured image. Based on a distance function, the detected marker position in the captured image is compared with the set of pre-learned marker positions to determine the quality of the match value of each set of pre-learned marker positions. Using a user-selected best match function, the best quality matching pre-learned force pattern is determined. Based on the determined best matching pre-learned force pattern, the pattern of force acting on the deformable element is identified. Output the identified force pattern to initiate the clamping action of the clamp based on the identified force pattern acting on the deformable element, wherein the identified force pattern includes one or a combination of the following: vertical pressure Z, center pressure, bias pressure, lateral force X, shear force Y, torque in pitch, yaw or roll direction, that is, compressive lateral force X, compressive shear force Y and tensile force.

[0050] The embodiments of the present disclosure will be further explained with reference to the accompanying drawings. The drawings shown are not necessarily drawn to scale, with emphasis instead generally being placed upon illustrating the principles of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] [ Figure 1A ]

[0052] Figure 1A is a schematic diagram illustrating an embodiment of an elastomeric cap configured with a clamp according to an embodiment of the present disclosure.

[0053] [ Figure 1B ]

[0054] Figure 1B is a schematic diagram illustrating some components of a sensor control computer connected to an elastomeric cap, the cap being attached to a gripper, and the gripper being part of a robotic assembly, according to some embodiments of the present disclosure.

[0055] [ Figure 1C ]

[0056] Figure 1C is a schematic diagram illustrating an embodiment of an elastomeric cap according to some embodiments of the present disclosure.

[0057] [ Figure 1D ]

[0058] Figure 1D is a schematic diagram illustrating an embodiment of an elastomeric cap configured with a clamp according to an embodiment of the present disclosure.

[0059] [ Figure 2A ]

[0060] Figure 2A is a flowchart illustrating an implementation of some steps for implementing a method according to some implementations of the present disclosure.

[0061] [ Figure 2B ]

[0062] Figure 2B is a flowchart illustrating an implementation of some steps for implementing a method according to some implementations of the present disclosure.

[0063] [ Figure 2C ]

[0064] Figure 2C is an example of some embodiments of the present disclosure. Figure 2A A block diagram of some of the steps of performing a pair-wise matching of a pre-stored library of XY positions stored in memory with a set of observed XY positions of fiducial markers in step 225 .

[0065] [ Figure 2D ]

[0066] Figure 2D is a block diagram illustrating some steps for generating a pre-stored library of XY positions according to some embodiments of the present disclosure.

[0067] [ Figure 2E ]

[0068] Figure 2E is a block diagram illustrating some steps for using image measurement functionality according to some embodiments of the present disclosure.

[0069] [ Figure 3A ]

[0070] Figure 3A is a flow chart illustrating some steps of finite element modeling (FEM) according to some embodiments of the present disclosure.

[0071] [ Figure 3B ]

[0072] Figure 3B is an example of using according to some embodiments of the present disclosure Figure 3A Schematic diagram of aspects of the FEM.

[0073] [ Figure 3C ]

[0074] Figure 3C is an example of using according to some embodiments of the present disclosure Figure 3A Schematic diagram of aspects of the FEM.

[0075] [ Figure 4A ]

[0076] Figure 4A is a schematic diagram illustrating an embodiment of a robotic assembly including a gripper and two elastomeric sensors according to some embodiments of the present disclosure.

[0077] [ Figure 4B ]

[0078] Figure 4B is a schematic diagram illustrating an embodiment of an elastomeric cap having pins on an inner surface and ridges attached to locations on an outer impact surface of the elastomeric cap according to some embodiments of the present disclosure.

[0079] [ Figure 4C ]

[0080] Figure 4C is a schematic diagram illustrating an embodiment of an elastomeric cap having pins on an inner surface and a cone attached to a location on an outer impact surface of the elastomeric cap according to some embodiments of the present disclosure.

[0081] [ Figure 4D ]

[0082] Figure 4D is a schematic diagram illustrating an embodiment of an elastomeric cap having a pin with micro-rods or micro-arms attached along and extending from the pin, such that each micro-rod or micro-arm may include a marking, according to some embodiments of the present disclosure.

[0083] [ Figure 5A ]

[0084] Figure 5A is a schematic diagram of an embodiment of an elastomeric cap arranged with a straight gripper cavity configuration illustrating the structure of a gripper according to some embodiments of the present disclosure.

[0085] [ Figure 5B ]

[0086] Figure 5B is a schematic diagram illustrating an embodiment of an elastomeric cap arranged with a bent or curved holder cavity configuration regarding the structure of a holder according to some embodiments of the present disclosure.

[0087] [ Figure 5C ]

[0088] Figure 5C is a photograph of an embodiment of an elastomeric cap attached to a straight gripper cavity configuration and with another elastomeric cap facing upward to show the underside of the pins illustrating the structure of the gripper according to some embodiments of the present disclosure.

[0089] [ Figure 5D ]

[0090] Figure 5D is a photograph of an embodiment of an elastomeric cap attached to a bent or curved gripper cavity configuration and a "teaching glove" worn by a human illustrating the structure of the gripper in accordance with some embodiments of the present disclosure.

[0091] [ Figure 6 ]

[0092] Figure 6 A is a photograph illustrating an embodiment of an elastomeric cap with an illuminated view of a pin with no pressure or force applied to an outer impact surface (not shown) of the elastomeric cap in accordance with some embodiments of the present disclosure.

[0093] Figure 6 B is the same view after image processing according to some embodiments of the present disclosure, where the set of detected XY features are marked as grey circles.

[0094] Figure 6 C is a photograph illustrating an embodiment of an elastomeric cap with an illuminated view of a pin with a force applied to an outer impact surface (not shown) of the elastomeric cap in accordance with some embodiments of the present disclosure.

[0095] Figure 6 D is the same view after image processing according to some embodiments of the present disclosure, where the set of detected XY features are marked as grey circles.

[0096] Figure 6 E is a photograph illustrating an embodiment of an elastomeric cap with an illuminated view of a pin with a torque force applied to an outer impact surface (not shown) of the elastomeric cap in accordance with some embodiments of the present disclosure.

[0097] Figure 6 F is the same view after image processing according to some embodiments of the present disclosure, where the set of detected XY features are marked as grey circles.

[0098] Figure 6 G is a photograph illustrating an embodiment of an elastomeric cap with an illuminated view of a pin with proximal pressure applied to an outer impact surface (not shown) of the elastomeric cap in accordance with some embodiments of the present disclosure.

[0099] Figure 6 H is the same view after image processing according to some embodiments of the present disclosure, where the set of detected XY features are marked as grey circles.

[0100] Figure 6 I is a photograph illustrating an embodiment of an elastomeric cap with an illuminated view of a pin with a lateral shear force applied to an outer impact surface (not shown) of the elastomeric cap according to some embodiments of the present disclosure.

[0101] Figure 6 J is the same view after image processing according to some embodiments of the present disclosure, where the set of detected XY features are marked as grey circles.

[0102] Figure 6 K is a photograph of an embodiment of an elastomeric cap with an illuminated view of a pin with an off center faraxial shear force applied to an outer impact surface (not shown) of the elastomeric cap according to some embodiments of the present disclosure.

[0103] Figure 6 L is the same view after image processing according to some embodiments of the present disclosure, where the set of detected XY features are marked as grey circles.

[0104] [ Fig. 7A ]

[0105] Fig. 7A is a schematic diagram illustrating some components of an alternative robotic system according to some embodiments of the present disclosure, the alternative robotic system including a robot having a gripper with an elastically deformable cap and various subsystems.

[0106] [ Figure 7B ]

[0107] Figure 7B is a schematic diagram illustrating an alternative processor according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0108] Although the above-identified drawings illustrate the presently disclosed embodiments, other embodiments may be envisioned as noted in the discussion. The present disclosure presents exemplary embodiments by way of representation and not limitation. Those skilled in the art may devise many other modifications and embodiments that fall within the scope and spirit of the principles of the presently disclosed embodiments.

[0109] The present disclosure relates to tactile sensing, and more particularly to a tactile sensor or elastomeric tactile sensor having a plurality of micromechanical pins attached to the underside of an elastomeric cap and converting the pattern of force into a measurable quantity once an external force is applied to a top surface.

[0110] Figure 1A is a schematic diagram illustrating an embodiment of an elastomeric cap according to an embodiment of the present disclosure. The tactile sensor includes an elastically deformable cap 18 having a top surface 16 and a bottom surface 17. The bottom surface 17 includes a plurality of pins 13, each pin 13 including a reference tip marker 15. Thus, when a force is applied to the top surface 16, the elastically deformable cap 18 is deformed, causing the pins 13 to move, and the reference tip markers 15 to undergo a flexural motion. The elastomeric tactile sensor may include a camera 10 that enters the catheter 11 and into the cavity 12A of the holder 8. The elastomeric cap 18 is connected to the cavity opening 12B of the holder, wherein one or all of the reference tip markers 15 (i.e., attached to the pins 13) are in the field of view of the camera 10.

[0111] The camera 10 can be positioned in the field of view of the fiducial tip marker 15 at an oblique angle to capture images of the fiducial tip marker 15 before, during, and after movement. The marker 15 moves through a pattern of forces or a net total force tensor when in motion when pressure, force, torque, compression, stretching, etc. is applied to the outer impact surface 16 corresponding to the movement of the pin 13 that moves the marker 15.

[0112] Figure 1B is an example of some embodiments of the present disclosure including connecting to Figure 1A Schematic diagram of some components of the sensor control computer 113 of the elastomeric cap 16, the cap being attached to Figure 1A The gripper 8 is a gripper 8 of the embodiment of the present invention, and the gripper is part of the robot assembly. The sensor control computer 113 includes a hardware processor 120 connected to a storage 130 via a bus 106. The hardware processor can implement or execute stored instructions stored in the memory 122 or the storage 130, and specifically, the hardware processor 120 can access a pre-stored XY set 131. The storage 130 can store a pre-stored XY set 131 including labels and force values. The processor 120 is connected to the memory 122 via the bus 106. Optionally, the pre-stored XY set with labels and force values ​​can be stored in the storage 103, the memory 122, or both, all depending on the specific application predetermined by the user.

[0113] The control module 140 connected via bus 106 is connected to the robot control computer 142 so that the robot control computer 143 can communicate back to the control module 140 via 143. For example, the control module 140 can be configured by the robot control computer to check the gripper force status once and report back to confirm or refute the gripper, or to continuously monitor the gripper status at a frame rate and interrupt robot control when the gripper indicates a good or invalid grip.

[0114] Still refer to Figure 1B , the robot control computer 142 is connected to the robot 144, wherein the robot 144 can communicate back to the robot control computer 142 via 145. For example, the robot control computer 142 can command the robot 144 to move, open a gripper, or close a gripper.

[0115] In addition, the robot 144 is connected to the deformation of the outer impact surface 148, wherein the physical environment 146 is also connected to the deformation of the outer impact surface 148. In addition, the deformation of the outer impact surface 148 is an external pressure or force that constitutes a pattern of forces or a net force tensor that moves the pin. Wherein at least one camera 133 captures images of the markings and pin positions to obtain video image data. For example, the deformation of the outer impact surface 148 can come from the force applied to the Figure 1A The external pressure, force, etc. of the outer impact surface 16. The external or external pressure, force, etc. can be a force mode, and the embodiments of the present disclosure can be directed to the external impact surface 16. Figure 1A The amount of measurable sensitivity is determined for each pressure or force of the force pattern that impacts the outer surface 16 .

[0116] Still refer to Figure 1BOptionally, the sensor control computer 113 may include a network interface controller 150 adapted to connect the system 100 to a network 151 via the bus 106, the network 151 connecting the control system 100 to a wireless system that can wirelessly receive the camera image data 133 of the captured images of the marker and pin positions, as well as other data. Also optionally, the sensor control system via the bus 106 may include a transmitter interface 160 connected to a transmitter 165 to wirelessly transmit data to a remote robot control computer (not shown), a remote robot (not shown), or even a remote control module (not shown). In addition, the sensor control system 113 via the bus 106 may be connected to a control interface 170, which is connected to one or more power sources 128. The control interface 170 may provide data including action commands to the power source or other device. It is contemplated that the power source 128 may be one or a combination of a battery, an alternative power source, a 120 volt power source, an alternative energy source (i.e., sun, water, wind, etc.). Additionally, the sensor control computer may include a receiver interface 180 connected to a receiver 185 via bus 106 to receive video image data or other data.

[0117] Figure 1C is an example of some embodiments of the present disclosure. Figure 1A Schematic diagram of an embodiment of an elastomeric cap. The elastomeric cap 18 includes an outer impact surface 16, a lower surface 17, pins 13A-13G, and indicia 15A-15G located on the pins 13A-13G. It is contemplated that the edge 19 or radius of the elastomeric cap 18 can be configured to mount to a clamp mounting surface (see Figure 1D ). Of course, depending on the specific application predetermined by the user, the edge or radius can be configured into a different structural configuration that can be attached to another type of mounting structure.

[0118] With respect to some elastomeric cap 18 configurations according to some embodiments, the elastomeric cap 18 includes a plurality of micromechanical pins 13A-13G positioned on the underside 17 of the elastomeric cap. Each micromechanical pin (hereinafter referred to as a "pin") 13A-13G can extend from the underside 17 of the cap and extend outward. Positioned on the outermost end (away from the underside) on the pins 13A-13G, a mark or fiducial tip mark (white or colored paint or retroreflective paint) 15A-15G has a motion damped only by air. Depending on the specific application, the marks 15A-15G can be positioned anywhere on the pins.

[0119] Still refer to Figure 1C, the pins 13A-13G can be made of an elastomeric material, such as silicone rubber, polyurethane, thermoplastic elastomer, natural rubber, polyisoprene, polyamide (nylon), polyethylene, polypropylene polyvinyl chloride, plasticized photopolymerized acrylic, or a mixture thereof. The outer impact surface 17 of the elastomeric cap 18 can be made of one deformable material or multiple deformable materials. By way of non-limiting example, the elastomeric cap 18 can be made of silicone rubber, polyurethane, thermoplastic elastomer, natural rubber, polyisoprene, polyvinyl chloride, plasticized photopolymerized acrylic, etc., so that it is conceivable that the pins 13A-13G can be made of the same material as the elastomeric cap 18 or a different material, or a combination of materials. For example, the pins 13A-13G can be made of nylon, while the body and impact surface 16 of the cap 18 can be made of polyurethane and manufactured according to double shot injection molding.

[0120] As described above, the pins 13A-13G convert something that is difficult to measure (such as a pattern of forces or a net force tensor) into something that is easy to measure. That is, measurement is made easier based on the image motion 15A-15G of the tip or end of the free micromechanical pin, i.e., the image motion of the marker 15A-15G at the outermost end of the pin is tracked by a camera, such as a cheap webcam, that views the pin tip at an angle. The pin tip 15A-15G can be continuously viewed by the camera before the pin tip moves (i.e., before force is applied to the outer impact surface of the elastomeric cap), during the pin tip movement (i.e., while force is being applied to the outer impact surface 16 of the elastomeric cap 18), and after the pin tip movement has stopped (i.e., after the externally applied force has stopped).

[0121] Figure 1D is a schematic diagram illustrating an embodiment of an elastomeric cap configured with a gripper according to an embodiment of the present disclosure. The gripper assembly 43 may include two fingers or grippers 38A, 38B, each gripper including an elastomeric cap 28A, 28B attached to the gripper 38A, 38B. The gripper assembly 43 may have an extended mounting structure 42A, 42B for attaching the fingers or grippers 38A, 38B to the gripper assembly 43. The attachment may be by a mechanical device including bolts, screws, etc., or by a magnetic device, or by some other attachment arrangement. In addition, the gripper 38B shows that the camera 30 can be inserted into the catheter 31 to be located in the gripper 38B so as to observe the pin 26 of the elastomeric cap 28B to capture an image of the position of the pin 26 and the mark 27.

[0122] It is envisioned that there may be multiple cameras, each with overlapping views of the marker, where the camera includes an illumination source. The illumination source is an artificial source of visible illumination, or observable properties and effects of light that may or may not be part of the camera and mounted separately. In addition, the marker may include different marker types, including different sizes, shapes, or colors, depending on the application predetermined by the user. For example, the marker may be white, such as white "liquid paper" paint (as used in some of our experiments), or some type of retroreflector (sometimes called a retroreflector or cataphote), which may be a device or surface that reflects radiation (usually light) back to its source (or camera) with minimal scattering. In a retroreflector, the wavefront of the radiation is reflected directly back to the source or camera. It works over a wide range of incident angles, unlike a plane reflector, which will only do so if the reflector is perfectly perpendicular to the wavefront and the angle of incidence is zero. By being directed, the reflection of a retroreflector is brighter than that of a diffuse reflector. Microbead coatings and coatings are commonly used as flexible or applied retroreflector materials in safety vests, joggers’ apparel, backpacks, and high-visibility highway markings.

[0123] Still refer to Figure 1D Some aspects of different types of markers are that different types of illumination can be used to illuminate or measure marker motion. For example, different types of illumination can each be associated with a corresponding marker that allows the marker to be measured by capturing an image from a camera. In addition, the illumination can include the visible light spectrum visible to the human eye and camera. Visible light is generally defined as wavelengths in the 400-700 nanometer (nm) range between infrared (with longer wavelengths) and ultraviolet (with shorter wavelengths), or 4.00×10 -7 to 7.00×10 -7 m. This wavelength implies a frequency range of approximately 430–750 terahertz (THz). Depending on the specific application, the illumination may be all types of electromagnetic radiation (EMR), and other components may be used to utilize the type of EMR depending on the specific application if the user has an unlimited fee account.

[0124] Figure 2A and Figure 2B is a flowchart illustrating an implementation of some steps for implementing a method according to some implementations of the present disclosure.

[0125] Figure 2AStep 215 includes capturing images of the moving fiducial tip marker from the starting point to the final endpoint using a camera, video camera, or webcam. For example, a fixed camera captures an image of a miniature bar pin carrying the fiducial marker. As in at least one experiment, the image is sent to a computer or processor, where an inexpensive webcam can be connected to a computer running software (such as Debian Linux software) via USB. However, other experiments can obviously include different types of cameras, camera positions, camera mounting layouts, and other types of software.

[0126] Figure 2A Step 220 includes identifying measured image coordinates of locations in the image of the captured image. On a computer, image processing software can be used to extract the XY position of each fiducial marker located on the lower surface of the cap. Obviously, there are many different but similar methods, but a preferred method includes using the open source package "OpenCV". It is recognized that depending on the specific application, not every marker needs to be in the field of view of the camera. Initially, during this experimental color, the webcam image has been reduced to a grayscale image, but this is not required and is optional.

[0127] Reference Figure 2B Optionally, step 221 may include a method step for thresholding the image. At this point, only black (background) and white (fiducial markers) are retained as black or white pixels. As a non-limiting example, a preferred threshold level is a pixel brightness value of 245 on a scale of 0 to 255. This method appears to render the fiducials as irregular white spots in black areas, as well as small white spots caused by noise in the sensor or fabric lint contamination.

[0128] In addition, refer to Figure 2B Optionally, step 222 may include a method step for denoising the white spots by replacing each pixel value with the lowest pixel value found in a 3x3 region centered around each pixel via a first method called the OpenCV "erosion" operator. This erosion method is performed 3 times, which removes noisy pixels and lint contamination from the image; this is because these noise features are typically less than 6 pixels across.

[0129] In addition, refer to Figure 2BOptionally, step 223 may include a method step for re-expanding the remaining white spots by inverse erosion known as OpenCV "dilation". It replaces each pixel value with the highest pixel value found in a 3x3 region centered on that pixel. This results in expanding the remaining white spots into a relatively stable, relatively circular mask centered on the area where the fiducial marker is located. We then use this mask image with the grayscale image to essentially eliminate all noise and contamination from the grayscale image, because the areas / regions / places after the erosion / dilation process where no stable areas have been restored are black (zero) and when logically ANDed with the original image, produce a black background that makes the marker detection fast and very low noise.

[0130] In addition, refer to Figure 2B Optionally, step 224 may include a method step using a standard OpenCV blob detector "Simple Blob Detector" to find the centers of these blobs, which are the XY locations of the fiducial markers on the end of the micro-rod pins.

[0131] There are many ways to implement blob detection. For ease of understanding, consider an example that applies to Figure 2B A simple circular spot detector of step 224 that finds bright circular spots on a dark background.

[0132] The simple blob detector works as follows: For each pixel in the input image, sum the values ​​of all pixels inside a circular radius of R1 pixels; call this sum SUM1. Then, sum the values ​​of all pixels within a radius of R2 pixels but outside the R1 radius; that is, the sum of all pixel values ​​within the ring from R1 to R2. Call this sum SUM2. The output value of the original pixel is SUM1–SUM2. We repeat this summing process for each pixel in the input image.

[0133] Now, suppose we set R2 to the square root of 2 (about 1.414) times R1, and we see that the areas of the R1 circle and the R2 ring are equal. Since the area of ​​the R1 circle and the area of ​​the R2-R1 ring are equal, on average, for a uniform (but noisy) image, the sum SUM1 of the R1 circle and the sum SUM2 of the R2-R1 ring will be very close to being equal, and the value of SUM1 minus SUM2 will be very close to zero.

[0134] For convenience, we refer to this center-minus-ring-sum operation on an image as Circle Minus Ring Filtering, or CMRF; it accepts an incoming image, performs a circle-minus-ring operation centered at each pixel, and returns the result. Unless otherwise specified, the CMRF function for radius R1 will default to R2 = SQRT(2)*R1. Note that the raw return result for a CMRF pixel may be more than 8 bits (values ​​of 255); for the purposes of this explanation, we will assume that the returned image allows multi-byte values. Other more advanced methods may scale or renormalize, but a discussion of those methods is beyond the scope of this description and is available in the public literature.

[0135] We then test our CMRF on an input image with a single white pixel and observe the output pixels on and around our original white pixel. If R1=1, the R1 circle contains 5 pixels, R2 is 1.414, and there are 4 pixels in the R1-R2 ring, our output blob detection image SUM1 minus SUM2 is black (zero) except in the area within one pixel of the white pixel we tested, which has a value of white (in a black and white image with one byte per pixel, it would be 255).

[0136] Now notice. The behavior of this detector as we increase R1 and R2, keeping R2 at the value of R1*SQRT(2). As we increase the radius of the R1 circle and the R2-R1 ring, larger and larger white circles will appear centered on a single white pixel. However, the CMRF value SUM1 minus SUM2 does not increase as R1 increases; it remains at 255.

[0137] We repeat this experiment with a white circle of radius 5 on a black background, and increase the value of R1 (and R2) from 1 to 2, to 3, etc. We then see that for values ​​of R1 from 1 to 5, the maximum CMRF value obtained for SUM1 minus SUM2 increases from 1 to ~20,000 (depending on your circle drawing algorithm), and then for R1 of 6 or greater, no further increase in the CMRF SUM1 minus SUM2 is seen.

[0138] We also notice that the pixel with the maximum value of SUM1 minus SUM2 is the center of our blob. Moving away by even a one pixel step, we move some white (value 255) pixels that are outside the SUM1 (positive contribution) region of the CMRF but inside the SUM2 (negative contribution) region. Thus, we find that the pixel with the maximum CMRF value will find the center of the blob.

[0139] From here we can now see how to detect individual blobs. We repeatedly perform CMRF, varying R1 from slightly below a pre-selected minimum blob radius, R1min, to slightly above a pre-selected maximum blob radius, R1max. The pixel with the largest CMRF value is the center of the blob; the value of R1 when this value is found is the radius of the blob.

[0140] In the case of multiple spots, the same procedure is sufficient with only one change. Instead of selecting a single pixel with the maximum CMRF value over the entire image, we look for the maximum CMRF value excluding pixels within a certain radius R and above a convenient threshold T percentage. Therefore, it is possible to find more than one spot, or no spots at all.

[0141] In our experiments, Rmin was ∼5 pixels, Rmax was ∼20, the blob exclusion radius was Rmax, and the T percent threshold was 90% (ie, the blob CMRF value must be at least 90% of the maximum possible value for a full white round blob).

[0142] We emphasize that this is a very simplified method given only for the sake of completeness, and that better (but more complex) methods that may be used in step 224 are published in the open literature and available in the OpenCV open source software.

[0143] Reference Figure 2A , step 225 includes determining a set of relative positions of the fiducial markers in the captured image. For example, because the number of such XY pairs is equal to the number of fiducial markers and is small (for these experiments, only seven fiducial pins were used), the next step 225 may be to pair-wise match the set of observed fiducial marker XY positions with a pre-stored library of XY positions.

[0144] The pre-stored library of XY positions was previously generated through multiple experiments. For this particular set of experiments, less than 50 pre-stored XY sets are stored in the memory of the pre-stored library database, resulting in excellent resolution and availability. Of course, more than 50 pre-stored XY sets can be stored in the pre-stored library database, but for these sets of experiments, 50 pre-stored XY sets appear to be sufficient to achieve some goals according to some aspects of the present disclosure.

[0145] Figure 2A Step 230 includes steps for calculating a force pattern or determining a net force tensor acting on the top surface of the elastically deformable cap. As a general overview, the force pattern can be determined by matching a set of relative positions of the fiducial tip markers to a stored set of previously learned relative positions of the fiducial tip markers placed in motion, and then identifying the force pattern associated with the stored set of previously learned relative positions of the fiducial tip markers.

[0146] therefore, Figure 2A Step 230 of begins with the following method steps: Using the Pythagorean metric (square root of the sum of the squares of the X and Y differences); determine the best match of the observed fiducial XY position to the pre-stored XY sets in the library. This calculation is very fast; each observed XY position is compared to each pre-stored XY position in the pre-stored library; a "running calculation" is maintained for each of the approximately 50 pre-stored sets, based on the assumption that each fiducial marker is represented only once in each pre-stored set, so the running calculation for each pre-stored set increases by the Pythagorean distance between the observed XY fiducial position and the nearest pre-stored XY position of any fiducial in any given pre-stored set. While conducting experiments for this step, it was initially assumed that a single marker was required or necessary to be tracked over time (i.e., the leftmost marker in the front row corresponds to the fourth XY coordinate in the fourth pre-stored library set, etc.). However, later experiments showed that this assumption was a completely unnecessary calculation and could be omitted without loss of accuracy.

[0147] Still refer to Figure 2A Step 230, the astounding result from this experiment is that this calculation is exceptionally fast; with 10 fiducial markers and 100 pre-stored library sets, there are a total of 1000 XY pairwise comparisons, i.e., a total of 2000 square operations, 1000 addition operations, and 1000 square root operations or about 4000 floating point operations. This means and recognizes that this is a very tractable method with excellent results, and in terms of process configuration, even the cheapest microprocessor can execute at a much higher rate than the camera frame rate.

[0148] Figure 2A Step 235 of may include the method step of running the calculated sum using the Pythagorean distance to sort the list of pre-stored XY position sets into a ranking quality of the matching list.

[0149] In some applications, a single best match is the only expectation and output of the sensor processing. For example, "well clamped" vs. "object not found" vs. "too tight clamped" vs. "clamped on edge" might be labels for some of the stored XY position sets, and thus would be output by the sensor processor. Note that there is nothing prohibiting having multiple stored XY position sets with the same label; there could be five to ten "clamped on edge" possibilities, and depending on the application, it may not be necessary or even useful for the downstream system to know exactly which one it is.

[0150] Still refer to Figure 2AIn step 230, in other applications, a sensor result may be outputted based on the selection, which is some particular weighted, inversely weighted, or linear combination of the forces that were in effect when a particular stored set of XY positions was recorded. For example, if downward forces of 1, 2, 5, and 10 Newtons were recorded and the corresponding reference XY position sets were stored, and then the sensor was subjected to an unknown force, an image may be captured, where the image is processed as described above to obtain the Pythagorean distance, and then the following may be performed:

[0151] For the XY set stored by 1 Newton, the Pythagorean distance = 350

[0152] For the XY set stored by 2 Newton, the Pythagorean distance = 100

[0153] For the XY set stored by 5 Newton, the Pythagorean distance = 50

[0154] For the XY set stored by 10 Newton, the Pythagorean distance = 400

[0155] …which can then be calculated as follows:

[0156] The sum of the reciprocals of the weights = 1 / 350 + 1 / 100 + 1 / 50 + 1 / 400 = 0.0353

[0157] Weight of 1 Newton sample = 1*[1 / 350] / 0.0353 = 0.080

[0158] Weight of 2 Newton sample = 2*[1 / 100] / 0.0353 = 0.566

[0159] Weight of 5 Newton sample = 5*[1 / 50] / 0.0353 = 2.832

[0160] Weight of 10 Newton sample = 10*[1 / 400] / 0.0353 = 0.708

[0161] Total weighted sum: 4.186

[0162] ...shows that the approximate force on the sensing surface of the elastomeric cap is 4.186 Newtons.

[0163] The same process can be extended to multiple forces operating alone or in combination, such as 1, 2, 5, 10 Newtons of downward force, -10, -5, -2, -1, 0, 1, 2, 5, and 10 Newtons of shear in X, -10, -2, 2, and 10 Newtons of shear in Y, and -10, -2, +2, and +10 Newtons of torque in Z.

[0164] Still refer to Figure 2AStep 230. It is also reasonable to limit the number of stored XY sets to be evaluated in weighting, inverse weighting, or linear interpolation; typically, the best match (lowest Pythagorean distance) two or three stored XY position sets give the best accuracy for one-dimensional problems, 4 to 6 for two-dimensional problems, 6 to 8 for three-dimensional problems, etc. As you can see, the disclosed method can be highly customized to a user's specific application needs / requirements while using very inexpensive components and executing at much higher than camera frame rates using inexpensive microprocessors.

[0165] Still refer to Figure 2A At step 230, one aspect could indicate that the problem could be inverse finite element modeling, however, later experiments showed that for a human thumb-sized robotic gripper cap, between 300 and 1000 grid points (each with three degrees of freedom) were required for sufficient fidelity in a linear finite element model (and more if we allow for nonlinearities). Meaning, this suggests that the computational phase involves solving a system of up to 3000x3000 linear equations at the camera frame rate, which is certainly computationally expensive, if not completely impossible with current conventional tactile sensor technology.

[0166] Figure 2C is an example of some embodiments of the present disclosure. Figure 2A The pre-stored library of XY positions stored in memory in step 225 is matched pair-wise with the set of observed XY positions of the fiducial markers ( Figure 2A The block diagram of some steps of step 225 of FIG. 1 is a block diagram of some steps of step 225 of FIG. 1 . Based on the following steps, the pairwise matching distance (PBPMD) of the observed set of marker XY positions and the pre-stored library of XY positions stored in the memory is calculated:

[0167] Figure 2C Step 251 includes setting the currently stored set of XY positions to the above-mentioned stored set of XY positions.

[0168] Figure 2C Step 253 includes executing the following routine PBPMD to find the pairwise matching distances of the observed XY position set to the currently stored XY position set to obtain the pairwise matching distances of that observed XY position set to the observed XY position set.

[0169] Figure 2C Step 255 includes calculating the PBPMD of the currently stored set of XY positions and the observed set of XY positions:

[0170] Figure 2C Step 257 includes:

[0171] [--Set CURRENT_STORED_XY_SET = Current stored XY position set

[0172] --Set variable THIS_SET_TOTAL_PYTHAG_DIST=0

[0173] --For each observed XY position in the set of observed XY positions: ]

[0174] [ --set THIS_LOC_MIN_DIST=10000000.

[0175] -- Set THIS_OBS_X and THIS_OBS_Y = the X and Y values ​​of the observed XY position

[0176] --For each stored XY position in CURRENT_STORED_XY_SET: ]

[0177] [--Set THIS_STORED_X and THIS_STORED_Y = X and Y values ​​of the stored XY location--Set THIS_DIST = SQRT((THIS_OBS_X-THIS_STORED_X)^2+(THIS_OBS_Y-THIS_STORED_Y)^2)

[0178] --if THIS_DIST <THIS_LOC_MIN_DIST:]

[0179] [ADD THIS_LOC_MIN_DIST TO THIS_SET_TOTAL_PYTHAG_DIST]

[0180] [Returns THIS_SET_TOTAL_PYTHAG_DIST as the pairwise matching distance.]

[0181] Figure 2D is a block diagram illustrating some steps for generating a pre-stored library of XY positions according to some embodiments of the present disclosure. Generating a pre-stored library of XY positions 260 may include step 261, which includes applying a known force (such as pressure, shear, torque, compression or expansion force) to the outer surface of the cap.

[0182] Figure 2D Step 263 includes defining a label for the imposed force, such as "GRIP OK" or "Y shear = 30 Newtons".

[0183] Figure 2DStep 265 includes capturing an image of the markings on the interior of the cap.

[0184] Figure 2D Step 267 includes processing the image according to the same operations in the same order as steps 221 (thresholding), 222 (erosion), 223 (dilation), and 224 (blob detection) to obtain a set of XY positions.

[0185] Figure 2D Step 269 includes storing the set of XY positions with the defined labels as one of a pre-stored library of sets of XY positions stored in the memory.

[0186] Figure 2E is a block diagram illustrating some steps for using the image measurement function 270 according to some embodiments of the present disclosure. Step 271 includes identifying a starting position of a marker from one of a plurality of captured images.

[0187] Figure 2E Step 273 includes identifying a final position of the marker from another captured image of the plurality of captured images.

[0188] Figure 2E Step 275 comprises associating the start position of the identified marker with the measured image coordinates in the one captured image.

[0189] Figure 2E Step 277 comprises associating the final position of the identified marker with the measured image coordinates in the further captured image.

[0190] Figure 2E Step 279 includes comparing the measured image coordinates of the fiducial markers from the starting position measurement coordinates to the final position measurement coordinates to identify a set of relative positions of the fiducial markers in motion.

[0191] Figure 2E Step 280 includes using an image processor to convert the relative position of each marker measured in image coordinates between the one captured image and the other captured image, summing the relative positions as Euclidean distances to form a matching distance between the one captured image and the other captured image.

[0192] Figure 3Ais a flow chart illustrating some steps of using finite element modeling (FEM) 310 according to some embodiments of the present disclosure. FEM is an excellent place to show or illustrate that the sensitivity measurement of elastic deformation of the top surface of the elastomeric cap using multiple pins on the lower surface of the elastomeric cap is amplified and larger when compared to the amount of measurable sensitivity of flexural motion of the "only" outer impact surface of the elastomeric cap. For example, step 315 includes creating a CAD model of the elastomeric cap, including the inner surface, outer surface, edge, micro-rods and markers.

[0193] Figure 3A Step 320 includes converting the CAD model into a mesh suitable for use in a FEM software package.

[0194] Figure 3A Step 325 includes defining a fixed portion or portions of the mesh that generally correspond to the edges of the elastomeric cap.

[0195] Figure 3A Step 330 includes defining a test force or forces applied to the cap, which test forces typically correspond to pressure, shear, torque, compression, or expansion forces applied to the cap.

[0196] Figure 3A Step 335 includes defining the Young's modulus and optionally the yield strength of each material or materials making up the cap.

[0197] Figure 3A Step 340 includes executing a FEM software package on the mesh, the fixture or fixtures, and the test force or forces, which typically involves solving three force balance equations (one each for X, Y, and Z forces) simultaneously for each mesh point.

[0198] Figure 3A Step 345 includes interpreting the results of the FEM solution, which is typically the X displacement, Y displacement, and Z displacement that each grid point will experience when subjected to the test force or forces.

[0199] Figure 3B and Figure 3C is an example of using according to some embodiments of the present disclosure Figure 3A Schematic diagram of aspects of finite element modeling (FEM). For example, Figure 3B A CAD model of an elastomeric cap is illustrated, and the mesh of thin lines shows the mesh that will be used for finite element modeling. Within each short line in the mesh, the material behavior is assumed to be linear.

[0200] Figure 3CThe results of the FEM solution of the CAD model in 3B are illustrated, and the material is set to an elastomer with a Shore durometer stiffness of 80A (approximately equal to the rubber used in the soles of hiking boots), the fixed part of the grid is the edge of the cap, and the test force is vertically upward below the foreground micro-rod pin at the far right. Note that although the tip of the foreground micro-rod at the far right has a large movement, the adjacent left foreground micro-rod tip has even greater movement. This is due to the stress of the test force causing the tip and tilt in the cap to deform. The base of the left micro-rod moves less vertically than the base of the right micro-rod, but the overall movement is much greater, and the movement is mainly in the -X direction. This makes it possible to disambiguate the different forces.

[0201] Figure 4A 4 is a schematic diagram of an embodiment of a robotic assembly including grippers 408A, 408B and two elastomeric sensors 418A, 418B according to some embodiments of the present disclosure. For this particular application, at least one goal may be to pick up an object 495 from a container 496 based on the object pose 297. The system includes a 6-axis robotic arm 404 with grippers 408A, 408B. A 3D sensor 409 may be arranged on the arm 404 to acquire data of a scene including the object 495 and the container 496. The grippers 408A, 408B may then pick up the object 495 from the container 496 based on their poses. It should be noted that the container may contain different objects.

[0202] As in Figure 2A As indicated in step 235 of , a single best match may be desired as the output of the sensor processing. For example, "good clamp" vs. "no object found" vs. "too tight clamp" vs. "clamped on edge" may be labels for some stored XY position sets, and thus they will be output by the sensor processor.

[0203] An example 3D sensor uses structured light generated by a projector. Other sensors such as stereo cameras and time-of-flight ranging sensors are also feasible. The sensor acquires 3D scene data, such as a point cloud. The 3D sensor is calibrated relative to the robot arm 404. Therefore, the pose of the object 495 can be estimated in the coordinate system of the 3D sensor, which can be transformed into the coordinate system of the robot arm 404, allowing the object to be grasped and picked up by controlling the robot arm 404 according to the pose via the robot control computer 403. The scene data can be processed by a control module, which can implement the determination method to be performed in the sensor control computer 401. The sensor control computer 401 may include a memory and an input / output interface known in the art.

[0204] However, a differentiating factor that is novel over conventional robotic assemblies is the inclusion of elastomeric sensors in the elastomeric caps 418A, 418B. For example, today's manufacturing facilities include flexible assembly lines, however, most conventional robotic grippers have many issues, including little or no tactile sensing, which presents problems for these conventional robots used for flexible assembly lines. At least one reason is that these conventional dedicated grippers designed for high precision part grasping and grasp verification cannot be used for flexible assembly lines. This is because these conventional dedicated gripper fingers are not operationally capable of incorporating a level of flexibility that allows the gripper to react to changing (whether predicted or unpredictable) circumstances. Similarly, conventional robots have a low level of ability to know whether the actual assembly operation was successful, i.e., such as whether the gripper grasp was "well gripped" vs. "object not found" vs. "gripped too tight" vs. "gripped on edge." In order to overcome these issues with conventional robots, including Figure 4A Some embodiments of the present disclosure are configured to provide, by way of non-limiting example, a mechanically robust universal gripper finger tip 418A, 418B that provides the full force integration of pressure, translation, rotation, and shear for use in flexible assembly lines.

[0205] Figure 4B 420C is a schematic diagram illustrating an embodiment of an elastomeric cap according to some embodiments of the present disclosure, the elastomeric cap having a pin and a ridge attached to a location on the outer impact surface 427 of the elastomeric cap. For example, the ridges 420A, 420B, 420C can extend a portion of the length of the cap 428 or the entire length of the cap 428. In addition, depending on the specific application involving a certain level of desired measurable sensitivity as predetermined by the user, the ridges 420A, 420B, 420C can be different in length, different in ridge height, or different in ridge thickness. The ridges 420A, 420B, 420C can extend along the length while the ridges extend along the width or narrow area. As described above, the ridges 420A, 420B, 420C can be constructed and designed in conjunction with the pins 423A, 423B, 423C, depending on the expected level of measurable sensitivity designed to accomplish a specific application as predetermined by the user.

[0206] Figure 4C 4 is a schematic diagram illustrating an embodiment of an elastomeric cap 438 having pins 453A-453C and cones 422A-422L attached to locations on the outer impact surface 437 of the elastomeric cap 438 according to some embodiments of the present disclosure. For example, the cones 422A-422L can cover the entire cap 438 or only areas of different portions of the cap 438. In addition, depending on the specific application involving a certain level of desired measurable sensitivity as predetermined by the user, the cones 422A-422L can have different bottom diameters, different cone heights, or different cone thicknesses. The cones 422A-422L can extend along the edge length or around the perimeter of the cap 438.

[0207] Figure 4D is a schematic diagram illustrating an embodiment of an elastomeric cap 448 having pins 464A-464G having micro-rods or micro-arms 463A-463J attached along and extending from the pins, wherein each micro-rod or micro-arm 463A-463J may include a marking attached to the end farthest from the lower surface 449, according to some embodiments of the present disclosure. Some of the pins 464A-464G having the lever arms 463A-463J may be attached to the location on the lower surface 449 of the elastomeric cap 448 at a location on the sidewall of the lower surface 449. According to experiments, such a pin location may provide an increased level of measurable motion sensitivity of the pins 464A-464G, and the level of measurable motion sensitivity is further improved when the pins have the lever arms 463A-463J nearby, compared to the pins 464A-464G attached at a location without the lever arms attached.

[0208] Reference Figure 1C , Figure 4B and Figure 4C It is contemplated that the outer impact surface may include attributes including one or a combination of at least one texture such as a smooth texture, a rough texture, symmetrical ridges, asymmetrical ridges, or a ridge pattern with or without at least one texture. Attributes may include Figure 4B The spine Figure 4C It has been recognized from experiments that a specific combination of properties with a specific structural arrangement of pins with or without lever arms can provide a higher level of measurable motion sensitivity specific to a desired mode of pressure, force, or both.

[0209] Specifically, depending on a user-specified application involving a set of measurable forces, the attributes can be constructed and arranged to include one of conical protrusions, symmetrical protrusions, or a combination of asymmetrical protrusions that provide a user-preferred increased measurable sensitivity to a pattern of forces, including detecting the amount of measurable in-plane forces, shear forces, and torque forces across an outward impact surface of the elastomeric tactile sensor.

[0210] Figure 5A is a schematic diagram illustrating an embodiment of an elastomeric cap arranged with a straight gripper cavity configuration regarding the structure of a gripper according to some embodiments of the present disclosure. For example, a gripper 520 includes a gripper cavity 522 to which an elastomeric cap 524 is attached, wherein the elastomeric cap 524 is arranged along the straight gripper cavity 522.

[0211] Figure 5Bis a schematic diagram illustrating an embodiment of an elastomeric cap arranged with a bent or curved gripper cavity configuration regarding the structure of a gripper according to some embodiments of the present disclosure. For example, a gripper 540 includes a gripper cavity 542 to which an elastomeric cap 544 is attached, wherein the elastomeric cap 544 is arranged along the bent or curved configuration of the gripper cavity 542.

[0212] Figure 5C is a photograph of an embodiment of an elastomeric cap 550 attached to a straight holder cavity configuration 552 and another elastomeric cap 560 facing upward to show an underside 570 of indicia 565 illustrating structure regarding a holder 554 in accordance with some embodiments of the present disclosure.

[0213] Figure 5D is a photograph of an embodiment of an elastomeric cap 585 attached to a bent or curved gripper cavity configuration 587 and a human-worn "teaching glove" 589 illustrating the structure of a gripper 590 according to some embodiments of the present disclosure. Some benefits / advantages of the elastomeric sensor or elastomeric cap 585 used in a human-worn "teaching glove" 589 are that the motion of the human fingers wearing the glove 589 can be recorded along with the appropriate force tensors to be applied in later robotic assembly operations. Note that for the camera in the gripper 590 (not shown), other camera angles that can be utilized include 0-10 degrees, 10-15 degrees, 10-20 degrees, and 15-30 degrees.

[0214] Figure 6 A and Figure 6 B. According to some embodiments of the present disclosure, Figure 6 A is a photograph illustrating an embodiment of an elastomeric cap 610 with an illuminated view of a pin 612 without pressure or force being applied to an outer impact surface (not shown) of the elastomeric cap 610, while Figure 6 B is the same view of pin 612 with less illumination.

[0215] Figure 6 C and Figure 6 D, according to some embodiments of the present disclosure, Figure 6 C is a photograph illustrating an embodiment of an elastomeric cap 620 with an illuminated view of a pin 622 with a force applied to an outer impact surface (not shown) of the elastomeric cap 620, and Figure 6 D is the same view of pin 622 with less illumination.

[0216] Figure 6 E and Figure 6 F, according to some embodiments of the present disclosure, Figure 6E is a photograph illustrating an embodiment of an elastomeric cap 630 with an illuminated view of a pin 632 with a torque force applied to an outer impact surface (not shown) of the elastomeric cap 630, and Figure 6 F is the same view of pin 632 with less illumination.

[0217] Figure 6 G and Figure 6 H, according to some embodiments of the present disclosure, Figure 6 G is a photograph illustrating an embodiment of an elastomeric cap 640 with an illuminated view of a pin 642 with proximal pressure applied to an outer impact surface (not shown) of the elastomeric cap 640, and Figure 6 H is the same view of pin 642 with less illumination.

[0218] Figure 6 I and Figure 6 J, according to some embodiments of the present disclosure, Figure 6 I is a photograph illustrating an embodiment of an elastomeric cap 650 with an illuminated view of a pin 652 with a lateral shear force applied to an outer impact surface (not shown) of the elastomeric cap 660, and Figure 6 J is the same view of pin 652 with less illumination.

[0219] Figure 6 K and Figure 6 L, according to some embodiments of the present disclosure, Figure 6 K is a photograph illustrating an embodiment of an elastomeric cap 660 with an illuminated view of a pin 662 with an off-center distal shear force applied to an outer impact surface (not shown) of the elastomeric cap 660, and Figure 6 L is the same view of pin 662 with less illumination.

[0220] experiment

[0221] At least one insight gained from the experiments is the inclusion of a micromechanical rod attached to the interface between the flat portion and the radius section of the underside of the cap. When the cap is depressed 0.1 mm, the radius section will be forced to change curvature and bend tighter. More precisely, the radius section will deform from a 10 mm radius to a 9.9 mm radius. Assuming the cap material is relatively incompressible and bends rather than bulges, this translates into a change in the arc length of the cap, from exactly 90 degrees and an undeformed length of 15.7079 mm to a constant length of 15.7079 mm on a 9.9 mm radius, such that 15.7079 = 9.9 mm * new angle, and the new angle is 1.5866 radians or 90.905 degrees, which is a deformation of 0.905 degrees in radius.

[0222] Since the flat top of the cap is supported by the object being pressed, the flat top will remain mostly flat, and most of the conformation will be in the side walls. Thus, the micromechanical pin will move upward (with the cap) by 0.1 mm, and bend outward by about 0.9 degrees.

[0223] Assuming our micromachined pin is 20mm long, consistent with our experimental results, the lateral displacement due to the tilt of the micromachined pin is sin(0.9 degrees)*20mm=.31mm. The pin also moves vertically, with a total movement of just under 0.4mm. This shows a gain of about 4 times.

[0224] Of course, this is an oversimplification due to the two-dimensional nature of the example. Two effects need to be addressed: first, the edge of the elastomeric cap at the base of the radius is fixed in position and angle relative to the camera, and second, the base edge of the cap has a much larger stress-bearing cross-section than the flat junction of the cap top. To a first approximation, the cross-section at the fixed root of the elastomeric cap is about twice that at the flat top, and the microrod experiences double the strain (and deformation, and tilt). Second, the edge of the cap being fixed pushes deformation into the central region of the cap, again doubling the strain, deformation, and tilt experienced at the base of the microrod.

[0225] In summary, there was tip motion of the microrod tip that experienced approximately 1.6 mm of movement, which was predicted based on rough calculations and finite element modeling and later confirmed by actual experiments using 3D printed elastomeric caps.

[0226] The observed 1.6mm motion of the fiducial mark is greater than the diameter of the pin itself, and is a much larger motion than the direct observation of a surface moving at 0.1mm. This is the reason and purpose of the free moving microrod in the present disclosure; this is due to the very large gain of 16 times in the observed position of the strain deformation of the elastomeric cap. This observed position is very obvious and can be measured using a simple camera such as a webcam.

[0227] Due to this microrod deformation gain, the images captured by the camera have larger motions of the fiducial markers and have better signal-to-noise ratio compared to traditional tactile sensor approaches, which directly image the top surface of the contact surface through grazing illumination or through a proxy of a hemispherical gel.

[0228] The use of independently moving microrods also allows different parts of the cap to experience and report vastly different local strains and forces. For example, one part of the cap could be in left-to-right shear, the other side of the cap could be in right-to-left shear, and because the microrods are free to move rather than being trapped in a continuous gel, the custom configuration of the sensor is able to correctly detect this as a surface torque, rather than being insensitive to a specific set of forces, as the hemispherical gel is insensitive to these specific sets of forces.

[0229] Several experiments were conducted to generate force signal outputs using the amplified fiducial marker motions of these microrods. Some aspects considered included structurally arranging fixed cameras at different locations within the cavity of the gripper.

[0230] Based on experiments, when having one or more cameras positioned at a viewing angle (i.e., tilt angle) between 0 and 20 degrees (0 to ~20), the one or more camera arrangements result in a low profile high sensitivity elastomeric tactile sensor. By way of non-limiting example, some benefits and advantages of the low profile elastomeric sensor of the present disclosure provide solutions to some current technical problems of conventional tactile sensors with large profiles. For example, the low profile elastomeric sensor of the present disclosure is not only applicable to the robotic gripper industry, but also, by way of non-limiting example, is applicable to applications related to installation on a "teaching glove" worn by a human being that is capable of capturing typical forces that a robot should experience when performing assembly operations.

[0231] A number of tests were conducted to determine how to design and construct an elastomeric cap based on sensitivity levels, durability, and manufacturing capabilities both through traditional injection molding and through 3D printing such as polymer-SLA. Experimental testing was conducted using arm motor torque sensors, post-grip verification sensors, wrist force sensors, gripper motor position and torque sensors. From the post-grip verification experiments, it was learned that after the robot performs a part grasping operation, the robot typically moves the part to a machine vision station. The machine vision station has a machine vision system that verifies that the gripper is holding the desired part in the desired orientation, where if any corrections are needed, they are implemented. The part is then assembled, or moved to a "retry" container for manual re-orientation and verification. This movement occurs after the part is removed from the feed fixture and before it is moved to the assembly area; it can only detect miss-grips at this point in time.

[0232] From experiments with gripper finger strain gauge type sensors, it is known that these sensors integrate strain gauges into the gripper finger and therefore only record fingertip forces, which seems equivalent to sensing only "Z" pressure. However, these sensors fail to provide sensing of other forces such as center pressure, bias pressure, lateral force X, shear force Y, torque in pitch, yaw or roll directions, squeeze lateral force X or squeeze shear force Y.

[0233] Another sensor that has been reviewed through public documents is the GelSight sensor. This sensor uses a thick block of transparent gel with a flexible opaque coating on one side. The coated side of the block is pressed against the surface, and the deformed coating is observed vertically while being illuminated under glancing illumination at several different angles. It is understood that the sensor is able to find textures (burrs) that protrude upward on the surface. However, because the opaque coating bridges over the voids and pinholes, this sensor does not seem to be useful for voids or pinholes, and the sensor has a very low level of damage force threshold and damage from surface friction is common. Another disadvantage among the many disadvantages that have been found is that this sensor requires a Z-axis margin of about 100mm for the camera to operate. Since the margin of the camera is so large, this means that the sensor certainly cannot have a low-profile configuration and seems to be better suited for robotic detection of textured surfaces rather than any type of low-profile application. In addition, the low level of force damage threshold and surface wear make this sensor unsuitable for robotic assembly.

[0234] Some aspects learned from the experiments with different types of sensors are that robots are used for assembly, but proper assembly depends on the robot always holding the part in exactly the same position and orientation. Mis-gripping leads to assembly failure, damage to the part, and damage to the gripper or the robot itself. Even if the part is gripped correctly, if the receiving part is incorrect, or the receiving part is contaminated or deformed, the robot will attempt to force assembly, resulting in poor assembly, damage to the part, or damage to the robot or robot gripper. In view of the above experimental results, it was found that the experimental sensors could not be modified or further developed to meet the desired sensitivity levels, durability, manufacturing costs, and many other factors levels of the present disclosure, and no further analysis was performed.

[0235] Based on extensive experiments and the sensitivity of the desired level, durability, manufacturing cost and other factors, many implementations are obtained for designing and constructing some embodiments of the present disclosure. Some of these implementations for some embodiments include extending the pin on the lower side of the cap in a manner that the pin extends away from the lower side. The configuration includes the following pin that must be designed, which not only meets the strength level but also meets the deformability level so that it can be deformed to record the deformation amount of the pin due to the external force tensor. Specifically, some configurations of the pin are specially designed to maintain a generally constant relative angle and position between the pins. For example, it is found from the public documents about the experiment via optical flow analysis that the pins are arranged in parallel in the experimental cap and the cap is filled with highly flexible transparent silicone. The assembly exhibits a certain sufficient level of strength, that is, the strength level that affects the pin deformation when the applied external force is applied. This experimental cap obtains an acceptable measurable level of vertical force. However, based on the image motion obtained by the vertical (z-axis) camera at the tip of the pin, this same experimental cap configuration fails to provide an acceptable level of measurable shear and torque sensing / force. It is understood from this experimental configuration that, at least in view of the measurable sensitivity goals of the desired level of the present disclosure, the pin strength level must not be too high above the strength level to limit the pin mobility level to achieve a measurable force tensor. It is also understood from this experiment that incorporating a pin with too high a strength level to keep the pins substantially parallel results in preventing the pins from tipping and tilting, which makes the experimental pin configuration insensitive to shear (X and Y pressure), insensitive to torque in X, Y and Z, and insensitive to squeeze forces in X and Y.

[0236] The number of pins to be mounted / attached / molded to the underside of the elastomeric cap can depend on many factors (as learned from experimentation), for example, recognizing that the number of pins can depend on the specific application based on user intent and objectives in terms of achieving a predetermined level of sensitivity, durability, etc.

[0237] For example, the applied external forces (i.e., all-force tensors) applied to the outer impact surface of the elastomeric cap (vertical "Z" pressure, both center and offset, lateral "X" and "Y" shear, and torque in pitch, yaw, and roll directions) can all be sensed by the elastomeric tactile sensor. The simple, compact tactile sensor design of the present disclosure can be used with robotics, prosthetic applications, computer applications, human-safe collaborative robot safety sensors, and vehicle-related applications. In addition, the high-resolution tactile sensors of some embodiments of the present disclosure can also be used in precision control devices (such as, high-density microcomputer products), for high-sensitivity robotic cap sensing; touch-sensitive virtual reality devices, such as control gloves worn by remote operators; and the fingers of robots used for patient surgery.

[0238] In order to process the data generated from the camera's images of the pin motion, finite element modeling (FEM) can be used to process the data of the cap as a metamaterial design. A metamaterial is an object whose internal structure produces a final object whose physical (electrical, magnetic, mechanical, thermal) properties are significantly different from the bulk properties of the "real" material it is composed of, i.e., negative ε, μ, almost arbitrary speed of light c, Poisson's ratio ν, or thermal conductivity κ. Basically, the FEM model can predict the deformation of the elastomer under different loads. In addition, the elastomer pin lifts the surface contour and converts the surface tilt into an XY motion that is easy to image, i.e., using a low-cost webcam image of the pin tip. In addition, the structural design of the cap with few fiducial markers provides simple image thresholding and no inter-frame tracking / ambiguity. In fact, the motion of the pin tip, which is converted into a force map, can be tracked using OpenCV. This minimizes the computational expense and maintains a high frame rate.

[0239] During the experiments, the inverse FEM proved to be computationally expensive. It was realized that the training of the model should use real data. One aspect further learned from the experiments was that using the Euclidean distance in 14-dimensional space for the label case with grasp position, angular pose, shear and torque proved to be beneficial. For example, the calculation time was very fast, i.e. the amount of computational cost proved to be very low, resulting in the ability to process in real time with excellent visualization on a 1GHz cpu (e.g., Raspberry Pi Zero).

[0240] Some of the technologies that embodiments of the present disclosure can use include FEM, 3D printing of micro-mechanical elastomeric components, small (5.5 mm diameter) webcams. Among them, webcams can be purchased cheaply (i.e., $20 or less). Embodiments can also be used for robotic gripping, which will provide verification of angles, center of gravity, force tensors, and grip posture at 30 Hz. In addition, robotic teaching, so that the control program can record what the grip should feel like, how difficult the grip is, and what slipping feels like.

[0241] Some other benefits of using FEM and 3D printing can enable the design and production of elastomeric caps with many attributes. For example, some attributes can include providing an elastomeric cap with a desired amount of sensitivity or damage threshold or a specific amount of force direction. Among them, the asymmetry of the cap, the pin height, the pin position and the mounting configuration enable different force directions to be distinguished. In addition, the low profile of the cap can enable tactile sensors to be used on both the robot gripper and the "teaching gloves" worn by the human teaching the robot, thereby capturing the correct force tensor for the task being performed. In many cases, it may not be necessary to calculate the actual force tensor, so that a simple comparison of the pin tip position with a preset library of correct and incorrect images is sufficient to determine whether the robot grip is appropriate or inappropriate, or interpolation against a known deformation library produces a good estimate of the actual force tensor on the sensor in real time.

[0242] Other embodiments of the present disclosure address today's industrial needs by providing benefits such as low cost, low mass, and low profile simple compact tactile sensors while delivering highly sensitive detection of external forces applied to elastomeric tactile sensors. Some of these benefits allow the compact tactile sensors of the present disclosure to be used in technologies where existing conventional use has proven too costly for various technical industries. For example, simply placing delicate foods such as fruit, sushi, or baked goods requires a degree of touch control that allows human employees to be replaced by cheaper robots.

[0243] feature

[0244] According to another embodiment of the present disclosure, a tactile sensor for a robot includes an elastically deformable element, which includes an outer impact surface and a lower surface having pins, ridges, or both. Each lower surface pin or ridge includes a mark, and the elastically deformable element is attached to a gripper of the robot. A camera is located in a cavity of the gripper to capture an image of the mark. The memory has stored data, which includes image data of a set of pre-learned mark positions with corresponding pre-learned force patterns, each set of pre-learned mark positions corresponding to the pre-learned force pattern. An image processor is operably connected to the camera and the memory. The image processor is configured to detect the mark position in the captured image. Based on a distance function, the mark position detected in the captured image is compared with the set of pre-learned mark positions to determine the quality of the matching value of each pre-learned mark position set. The best quality matching pre-learned force pattern is determined using a user-selected best matching function applied to the determined quality matching value to calculate the pre-learned mark position of the best matching set and the corresponding pre-learned force pattern of the best quality matching. Based on the determined best matching pre-learned force pattern, a pattern of forces acting on the elastically deformable element is identified. The identified force pattern is output to initiate a gripper action based on the identified force pattern acting on the deformable element. The following aspects below are considered to be variant embodiments of the above embodiments.

[0245] According to aspects of the present disclosure, an image processor converts each marker position detected in a captured image by using an image measurement function to identify measured image position coordinates in an image among captured images to obtain detected marker positions, wherein the captured image includes a starting position image and a final position image of each marker, wherein each detected marker position is summed as a Euclidean distance to form a matching distance between the starting position image and the final position image of the marker.

[0246] At least one benefit of this aspect may be that determining the pattern of forces acting on the elastically deformable element includes using a machine vision algorithm. In another aspect, the pattern of forces acting on the elastically deformable element includes one or a combination of: vertical pressure Z, center pressure, bias pressure, lateral shear force X, lateral shear force Y, center or bias torque in pitch, yaw or roll direction, center or bias squeeze force X, center or bias squeeze force Y, center or bias deployment lateral force X, or center or bias deployment lateral force Y. Another aspect may be that the robot is a dynamic robot and the gripper is controlled in response to the identified pattern of forces acting on the outer impact surface determined in the image processor.

[0247] Another aspect is that when a force is applied to the outer impact surface, the outer impact surface elastically deforms, causing the lower surface pins, ridges, or both to undergo flexural motion of the detection mark. In addition, one aspect may be that the pre-learned marker positions of the best matching set are determined using a matching function that includes a user-selected matching function, wherein the user-selected matching function includes: (1) the best quality match via a distance function; (2) a predetermined weighted average among the best quality match values ​​of a user-selected number via a distance function; or (3) a predetermined weighted sum of all quality match values. Another aspect is that each force pattern includes a set of forces perpendicular to the outer impact surface, torque forces, and shear forces.

[0248] One aspect may include: the outer impact surface changes between an undeformed state when no force is applied to the outer impact surface and an elastically deformed state when a force is applied to the outer impact surface, elastically deforming the outer impact surface when the force is applied to the outer impact surface, causing the lower surface pins, ridges, or both to undergo flexural movement of the detection mark, and wherein the image processor compares the position of the mark in the captured image by evaluating the relative position of the lower surface pins, ridges, or both between the undeformed state and the elastically deformed state.

[0249] Yet another aspect may be that the outer impact surface includes attributes, the attributes including one or a combination of at least one texture such as a smooth texture or a rough texture, symmetrical ridges, asymmetrical ridges, or a pattern of ridges with or without at least one texture, and constructed and arranged in conjunction with lower surface pins, ridges, or both, to provide a user-preferred measurable sensitivity to the pattern of forces, based on a user-specified application involving a set of measurable forces. One aspect may be that a user-selected best match function is applied to the determined mass match values, and a pre-learned marker location for the best match set and a corresponding pre-learned force pattern for the best mass match are calculated. Still another aspect may be that the outer impact surface includes attributes such that, based on a user-specified application involving a set of measurable forces, the attributes are constructed and arranged to include at least one area of ​​a rough texture that provides a user-preferred increased and uniform measurable sensitivity to the pattern of forces, including detecting an amount of measurable normal force across the elastically deformable element to the outer impact surface. Another aspect may be that, according to a user-specified application involving a set of measurable forces, the attribute is constructed and arranged to include a combination of one of a conical protuberance, a symmetrical protuberance, or an asymmetrical protuberance that provides a user-preferred increased measurable sensitivity to a pattern of forces, including detecting the amount of measurable in-plane forces, shear forces, and torque forces across the elastomeric tactile sensor to an external impact surface. Yet another aspect is that, according to a user-specified application involving a set of measurable forces, the attribute is constructed and arranged to include at least one region of ridged protuberances positioned perpendicular to an axis of user-preferred measurable sensitivity to a measurable force or torque to a pattern of forces. Furthermore, one aspect is that, according to a user-specified application involving a set of measurable forces, the attribute includes a user-predetermined pattern of one or a combination of ridged protuberances, rough textures, conical protuberances, symmetrical protuberances, and asymmetrical protuberances positioned perpendicular to an axis of preferred measurable sensitivity that provides a user-preferred measurable sensitivity to a pattern of forces.

[0250] The following aspects may be incorporated into any of the above devices, methods, and systems. For example, one aspect may be that the image processor converts the relative position of each fiducial tip marker in the captured image by identifying the image coordinates of the measured position in the image among the captured images using an image measurement function to obtain a set of relative positions of the fiducial tip markers, so that each fiducial marker measured in the image coordinates between the captured image and another captured image has a relative position as the sum of the Euclidean distances, thereby forming a matching distance between the captured image and another captured image.

[0251] Another aspect is to detect the marker position in the captured image by an image measurement function. The image processor implements the steps of the image measurement function, which includes retrieving a set of pre-learned marker positions with a pre-learned force pattern from a memory. The set of pre-learned marker positions corresponding to the marker positions detected in the captured image is identified from the image data. The image coordinates of the measured position in the image among the captured images are identified using the image measurement function to obtain the detected marker position, the captured image includes a starting position image and a final position image of each marker, and each detected marker position is summed as the Euclidean distance, thereby forming a matching distance between the starting position image and the final position image of the marker.

[0252] Yet another aspect is that the cameras include one or more cameras viewing the fiducial marker at an oblique angle, the oblique angle comprising a range of one of: zero degrees to up to 10 degrees, zero degrees to up to 20 degrees, or 21 degrees to up to 30 degrees.

[0253] One aspect can be that each pin includes a lever arm attached to a location on the lower surface of the elastomeric cap, including at least one location on the side wall of the lower surface, such that the pin is a mechanical motion amplification pin, wherein the pin has an increased level of motion sensitivity by utilizing the lever arm attached to the location on the lower surface as compared to when the pin is attached at a single location without the lever arm attached to the location on the lower surface of the elastomeric cap.

[0254] Yet another aspect may be to use multiple pins to amplify a measure of the sensitivity of the elastic deformation of the top surface of the elastomeric cap because the amount of measurable sensitivity of the flexural motion of the multiple pins is greater when compared to the amount of measurable sensitivity of the flexural motion of the top surface of the elastomeric cap.

[0255] In addition, an aspect may include: each fiducial marker is encoded by one of a color, a bar code, a QR-like dot code, or other type of encoding, so that a match between the fiducial marker in a position in a captured image and the fiducial marker in a nearest position in a previously learned captured image in a subset of previously learned captured images is determinable compared to the fiducial marker position of the fiducial marker in the captured image.

[0256] One aspect may include: an image measurement function determines a relative position of each of a plurality of fiducial markers in motion by identifying a final position of the fiducial marker from a captured image among a plurality of captured images. The identified final position of the fiducial marker is associated with an image coordinate measured in the captured image. The final position measurement coordinate is compared with a set of captured images of previously learned fiducial markers of an elastomeric cap from stored data to identify a subset of previously learned captured images containing corresponding previously learned fiducial markers, the corresponding previously learned fiducial markers including the fiducial markers having the final measurement image coordinates identified from the captured image. The corresponding previously stored fiducial markers are compared with a subset of previously stored captured images to identify previously learned captured images having previously stored starting positions. The identified previously stored starting positions are associated with the measured image coordinates to obtain previously stored starting position measurement image coordinates. The measured image coordinates of the fiducial markers from the previously stored starting position measurement coordinates to the final position measurement coordinates are compared to identify a set of relative positions of the fiducial markers in motion.

[0257] Yet another aspect is summing the relative positions of the fiducial markers as Euclidean distances to form matching distances between the captured image and a subset of previously learned captured images, such that the image processor then classifies the captured image as acceptably or unacceptably labeled based on the previously learned acceptable labels or previously learned unacceptable labels of the previously learned image having the closest Euclidean matching distance.

[0258] definition

[0259] According to aspects of the present disclosure, and based on experimentation, the following definitions have been established, and are certainly not exhaustive definitions of every phrase or term. The definitions provided therein are provided merely as examples based on learning from the experimentation, where other interpretations, definitions, and other aspects may be involved. However, such definitions have been provided, at least for only a basic preview of the phrases or terms presented.

[0260] Tensor : In mathematics, a tensor is an algebraic object associated with a vector space and its dual space, and can be defined in several different ways, usually as a scalar, a tangent vector at a point, a cotangent vector at a point (the dual vector), or a multilinear map from a vector space to the resulting vector space. Euclidean vectors and scalars (often used in physics and engineering applications unrelated to general relativity) are the simplest tensors. Although tensors are defined independently of any basis, physics literature often refers to them by their components in a basis associated with a particular coordinate system.

[0261] Finite Element Modeling (FEM): According to some embodiments of the present disclosure, FEM can subdivide the entire domain into simpler parts, which has several advantages: (A) accurately represents complex geometries; (B) includes different material properties; (C) easily represents the overall solution; (D) captures local effects.

[0262] The work in this method may involve (1) partitioning the domain of the problem into a set of subdomains, each of which is represented by a set of element equations of the original problem, and then (2) systematically recombining all sets of element equations into a global system of equations for the final calculation. The global system of equations has known solution techniques and can be calculated starting from the initial values ​​of the original problem to obtain a numerical answer.

[0263] Still involving FEM, in the first step above, the element equation is a simple equation that locally approximates the original complex equation to be studied, where the original equation is usually a partial differential equation (PDE). In order to explain the approximation in this process, FEM is usually introduced as a special case of the Galerkin method. In mathematical terms, this process is to construct an integral of the inner product of the residual and the weight function and set the integral to zero. In simple terms, the process minimizes the approximation error by fitting a trail function to the PDE. The residual is the error caused by the trail function, and the weight function is a polynomial approximation function of the projected residual. This process eliminates all spatial derivatives of the PDE, thereby locally approximating the PDE with a set of algebraic equations for steady-state problems and a set of ordinary differential equations for transient problems.

[0264] These systems of equations are elementary equations. They are linear if the underlying PDE is linear, and vice versa. The algebraic systems of equations arising in steady-state problems are solved using numerical linear algebra methods, while the systems of ordinary differential equations arising in transient problems are solved by numerical integration using standard techniques such as Euler's method or Runge-Kutta method.

[0265] Still referring to the FEM, in step (2) above, a global system of equations is generated from the element equations by transforming the coordinates from the local nodes of the subdomain to the global nodes of the domain. This spatial transformation includes the application of appropriate orientation adjustments relative to the reference coordinate system. This process is usually implemented by FEM software using the coordinate data generated from the subdomain.

[0266] Fig. 7A is a schematic diagram illustrating some components of an alternative robotic system according to some embodiments of the present disclosure, the robotic system including a robot having a gripper with an elastically deformable cap and various subsystems. For example, according to some embodiments of the present disclosure, Fig. 7AA sensor control computer 770 may be included that is connected to a robot 782. The sensor control computer 770 may include a hardware processor 777 connected to a memory 772 that includes stored software 774 and a pre-learned XY set database that includes labels and surface force values. The hardware processor 777 may implement or execute stored instructions stored in the memory 772. The processor 777 is connected to a control module 778 that is connected to the robot control computer 780. The robot control computer 780 may communicate back to the control module via 779. The robot control computer 780 is connected to the robot 782, where the robot 782 is connected back to the robot control computer 780.

[0267] Still refer to Fig. 7A , the robot 782 is connected to the deformation of the external impact surface 785, wherein the physical environment 784 is connected to the deformation of the external impact surface 785. For example, the physical environment 784 includes an object or something that can cause an impact on the external impact surface. In addition, the deformation of the external impact surface 785 can be an external pressure or force that constitutes a pattern of forces or a net force tensor that causes the pin to move. Wherein, at least one camera 769 captures images of the markings and pin positions to obtain video image data. For example, the deformation of the external impact surface 785 can come from the force applied to Figure 1A The external pressure, force, etc. of the outer impact surface 16. The external or external pressure, force, etc. can be a mode of force, and the embodiments of the present disclosure can determine the external pressure applied to the outer impact surface 16. Figure 1A The amount of measurable sensitivity of each pressure or force pattern of the outer impact surface 16.

[0268] Still refer to Fig. 7A Optionally, other data may be obtained from other sensors to the hardware processor 777. For example, other data may include environmental data, historical data, or data related to implementing robot actions.

[0269] Figure 7B 7 is a schematic diagram of a computing device 700 that illustrates, by way of non-limiting example, some techniques that may be used to implement methods and systems according to an embodiment of the present disclosure. Computing device or apparatus 700 represents various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers.

[0270] The computing device 700 may include a power supply 708, a processor 709, a memory 710, a storage device 711, all connected to a bus 750. In addition, a high-speed interface 712, a low-speed interface 713, a high-speed expansion port 714, and a low-speed connection port 715 may be connected to the bus 750. In addition, a low-speed expansion port 716 is connected to the bus 750. Depending on the specific application, various component configurations that can be installed on a common motherboard are envisioned by way of non-limiting example 730. In addition, an input interface 717 may be connected to an external receiver 706 and an output interface 718 via the bus 750. A receiver 719 may be connected to an external transmitter 707 and a transmitter 720 via the bus 750. Also connected to the bus 750 may be an external memory 704, an external sensor 703, a machine 702, and an environment 701. In addition, one or more external input / output devices 705 may be connected to the bus 750. Network interface controller (NIC) 721 may be adapted to connect to network 722 via bus 750 , where data or other data may be rendered on a third-party display device, third-party imaging device, and / or third-party printing device external to computer device 700 .

[0271] It is contemplated that the memory 710 may store instructions executable by the computer device 700, historical data, and any data that may be utilized by the methods and systems of the present disclosure. The memory 710 may include a random access memory (RAM), a read-only memory (ROM), a flash memory, or any other suitable memory system. The memory 710 may be one or more volatile memory units, and / or one or more non-volatile memory units. The memory 710 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0272] Still refer to Figure 7B , the storage device 758 may be suitable for storing supplementary data and / or software modules used by the computer device 700. For example, the storage device 711 may store historical data and other relevant data as mentioned above with respect to the present disclosure. Additionally or alternatively, the storage device 711 may store historical data similar to the data mentioned above with respect to the present disclosure. The storage device 711 may include a hard drive, an optical drive, a thumb drive, a drive array, or any combination thereof. In addition, the storage device 711 may include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions may be stored in an information carrier. When the instructions are executed by one or more processing devices (e.g., processor 709), one or more methods, such as those described above, are executed.

[0273] The system can be optionally linked to a display interface or user interface (HMI) 723 via bus 750, which is suitable for connecting the system to a display device 725 and a keyboard 724, wherein the display device 725 may include a computer monitor, camera, television, projector or mobile device, etc.

[0274] Still referring to 7B, the computer device 700 may include a user input interface 717 suitable for a printer interface (not shown), which may also be connected via the bus 750 and suitable for connecting to a printing device (not shown), wherein the printing device may include a liquid inkjet printer, a solid ink printer, a large commercial printer, a thermal printer, a UV printer or a dye sublimation printer, etc.

[0275] The high-speed interface 712 manages bandwidth-intensive operations of the computing device 700, while the low-speed interface 713 manages less bandwidth-intensive operations. This functional allocation is only an example. In some implementations, the high-speed interface 712 can be connected to the memory 710, the user interface (HMI) 723, the keyboard 724 and the display 725 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 714 that can accept various expansion cards (not shown) via the bus 750. In an implementation, the low-speed interface 713 is connected to the storage device 711 and the low-speed expansion port 715 via the bus 750. The low-speed expansion port 715, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be connected to one or more input / output devices 705, as well as other devices keyboard 724, pointing device (not shown), scanner (not shown), or network devices, such as switches or routers, for example, through a network adapter.

[0276] Still refer to Figure 7B , computing device 700 can be implemented in a number of different forms, as shown. For example, it can be implemented as a standard server 726, or multiple implementations in such a server group. In addition, it can be implemented in a personal computer such as a laptop computer 727. It can also be implemented as part of a rack server system 728. Alternatively, components from computing device 700 can be combined with other components in a mobile device (not shown). Each such device can contain one or more of a computing device and a mobile computing device, and the entire system can be composed of multiple computing devices that communicate with each other.

[0277] Implementation

[0278] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of exemplary embodiments will provide a sufficient description for implementing one or more exemplary embodiments for those skilled in the art. Various modifications that can be made in the function and arrangement of elements are contemplated without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0279] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be appreciated by those skilled in the art that the embodiments may be practiced without these specific details. For example, the systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary details. In other cases, known processes, structures, and techniques may be shown without non-essential details to avoid obscuring the embodiments. In addition, the same reference numerals and signs in the various drawings refer to the same elements.

[0280] Moreover, each embodiment can be described as a process depicted as a flow chart, flow diagram, data flow diagram, structure diagram or block diagram. Although a flow chart can describe an operation as a sequential process, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. When the operation of the process is completed, the process can terminate, but there may be other steps that are not discussed or included in the figure. In addition, not all operations in any particular described process can occur in all embodiments. A process can correspond to a method, function, process, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the function can correspond to the function returning to the calling function or main function.

[0281] In addition, embodiments of the disclosed subject matter may be implemented at least in part, either manually or automatically. Manual or automatic implementation may be performed, or at least assisted by a machine, hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks may be stored in a machine-readable medium. A processor may perform the necessary tasks.

[0282] In addition, the embodiments of the present disclosure and the functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, and include the structures disclosed in this specification and their equivalents, or a combination of one or more of them. In addition, some other embodiments of the present disclosure can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Further, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to an appropriate receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0283] According to an embodiment of the present disclosure, the term "data processing device" can cover all types of devices, apparatuses and machines for processing data, including, as examples, a programmable processor, a computer or a multiprocessor or a multicomputer. The device can include a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device can also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0284] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) can be written in any form of programming language (including compiled or interpreted languages) or declarative or procedural languages, and can be deployed in any form including as a standalone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A computer program may, but not necessarily, correspond to a file in a file system. A program can be stored in a portion of a file holding other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordination files (e.g., files storing one or more modules, subroutines or partial codes). A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed at multiple sites and interconnected by a communication network. Computers suitable for executing computer programs include, as examples, central processing units that can be based on general or special microprocessors or both, or any other type of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or implementing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably connected to receive data from a mass storage device or to transfer data to a mass storage device or both. However, a computer does not necessarily have such a device. In addition, a computer can be embedded in another device, such as, to name a few, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as, a universal serial bus (USB) flash drive.

[0285] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) that displays information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending documents to or receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0286] Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0287] A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between client and server is created by computer programs running on their respective computers and having a client-server relationship to each other.

[0288] Although the present disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications may be made within the spirit and scope of the present disclosure. Therefore, the aspects of the appended claims cover all such variations and modifications that fall within the true spirit and scope of the present disclosure.

Claims

1. A tactile sensor for a robot, the tactile sensor comprising: an elastically deformable element including an outer impact surface and a lower surface having pins, ridges, or both, each lower surface pin or ridge including an indicia, and said elastically deformable element being attached to the holder; a camera positioned in the cavity of the holder to capture an image of the marker; a memory having stored data including image data of a set of pre-learned marker positions with corresponding pre-learned force patterns, each set of pre-learned marker positions corresponding to a pre-learned force pattern; An image processor is operatively connected to the camera and the memory, the image processor being configured to: detecting a position of the marker in a captured image; Based on a distance function, comparing the detected marker positions in the captured image with the set of pre-learned marker positions to determine the quality of the matching value of each set of pre-learned marker positions; determining a pre-learned force pattern of best quality match using a user selected best match function applied to the determined quality match values ​​to calculate pre-learned marker positions and corresponding pre-learned force patterns of best quality match for a best match set; identifying a pattern of forces acting on the elastically deformable element based on the determined best matching pre-learned force pattern; as well as The identified pattern of forces is output to initiate a clamping action of the clamp based on the identified pattern of forces acting on the deformable element.

2. The tactile sensor according to claim 1, wherein: The image processor converts the position of each marker in the captured image by recognizing measured image position coordinates in an image among the captured images using an image measurement function.

3. The tactile sensor according to claim 2, wherein: The Euclidean distance between the detected marker position and the pre-learned marker position is summed for each marker position detected, thereby forming a matching distance between the captured image of the marker and the pre-learned image of the marker.

4. The tactile sensor according to claim 1, wherein: Determining the pattern of forces acting on the elastically deformable element includes using a machine vision algorithm.

5. The tactile sensor according to claim 1, wherein: The mode of force acting on the elastically deformable element includes one or a combination of the following: vertical pressure Z, center pressure, bias pressure, lateral shear force X, shear force Y, torque in pitch, yaw or roll direction, squeezing lateral force X, deploying lateral force X, deploying lateral force Y, or squeezing shear force Y.

6. The tactile sensor according to claim 1, wherein: The gripper is attached to a robot, which is a dynamic robot configured to move and control the gripper based on the gripper action, and the gripper is configured to accept other gripper actions determined by the image processor through other sensor data or by a user of the gripper, wherein the other sensor data is obtained from other sensors communicating with the image processor.

7. The tactile sensor according to claim 1, wherein: The distance function used to compare the detected marker positions with the set of pre-learned marker positions is based on a Euclidean distance function.

8. The tactile sensor according to claim 1, wherein: The pre-learned marker locations of the best match set are determined using the user-selected matching function comprising: (1) the best quality match via the distance function; (2) a predetermined weighted average of a user-selected number of best quality match values, or a user-specified weighting function of all quality match values ​​that is dynamically updateable; or (3) a predetermined weighted sum of all quality match values.

9. The tactile sensor according to claim 1, wherein: Each force pattern includes: a combination of forces perpendicular to the outer impact surface, torque forces and shear forces, wherein when the forces are applied to the outer impact surface, the outer impact surface elastically deforms causing the lower surface pins, ridges or both to undergo flexural motion that displaces the marker.

10. The tactile sensor according to claim 1, wherein: The outer impact surface changes between an undeformed state when no force is applied to the outer impact surface and an elastically deformed state when a force is applied to the outer impact surface to elastically deform the outer impact surface causing the lower surface pin, ridge or both to undergo a flexural motion that displaces the mark, and wherein the image processor compares the position of the mark in the captured image by evaluating the relative position of the lower surface pin, ridge or both between the undeformed state and the elastically deformed state.

11. The tactile sensor according to claim 1, wherein: The outer impact surface includes properties that, depending on a user-specified application involving a set of measurable forces, include one or a combination of: at least one of a smooth texture or a rough texture, a symmetrical ridge, an asymmetrical ridge, or a pattern of ridges with or without the at least one texture, and are constructed and arranged in combination with the lower surface pins, ridges, or both to provide a user-preferred measurable sensitivity to the pattern of forces.

12. A tactile sensor for a robot, the tactile sensor having: a deformable element including an outer impact surface and a lower surface having pins, ridges, or both, each lower surface pin or ridge including a marking, and the deformable element being attached to a gripper of the robot; and a camera positioned in the cavity of the gripper to capture an image of the marker, and a memory having stored data, the stored data including image data of a set of pre-learned marker positions with corresponding pre-learned force patterns, each set of pre-learned marker positions corresponding to a pre-learned force pattern, the tactile sensor comprising: An image processor, the image processor is operably connected to the camera and the memory, the image processor being configured to: detecting a position of the marker in a captured image; Based on a distance function, comparing the detected marker positions in the captured image with the set of pre-learned marker positions to determine the quality of the matching value of each set of pre-learned marker positions; using a user-selected best-match function to determine the best quality match to the pre-learned force pattern; identifying a pattern of forces acting on the deformable element based on the determined best matching pre-learned force pattern; The identified pattern of forces is output to initiate a clamping action of the clamper based on the identified pattern of forces acting on the deformable element.

13. The tactile sensor according to claim 12, wherein: The user selected best match function is applied to the determined quality match values ​​and a best match set of pre-learned marker positions and a corresponding best quality match pre-learned force pattern is calculated.

14. The tactile sensor according to claim 13, wherein: The outer impact surface includes properties that, depending on a user-specified application involving a set of measurable forces, include one or a combination of: at least one of a smooth texture or a rough texture, a symmetrical ridge, an asymmetrical ridge, or a pattern of ridges with or without the at least one texture, and are constructed and arranged in combination with the lower surface pins, ridges, or both to provide a user-preferred measurable sensitivity to the pattern of forces.

15. The tactile sensor according to claim 12, wherein: The outer impact surface includes properties such that, based on a user-specified application involving a set of measurable forces, the properties are constructed and arranged to include at least one region of a rough texture that provides a user-preferred increased and uniform measurable sensitivity to a pattern of forces, including detecting an amount of measurable perpendicular force across the deformable element against the outer impact surface.

16. The tactile sensor according to claim 12, wherein: The outer impact surface includes attributes that are constructed and arranged to include a combination of one of tapered protrusions, symmetrical protrusions, or asymmetrical protrusions, depending on a user-specified application involving a set of measurable forces, which provide a user-preferred increased measurable sensitivity of force patterns, including detecting an amount of measurable in-plane forces, shear forces, and torque forces across the tactile sensor against the outer impact surface.

17. The tactile sensor according to claim 12, wherein: The outer impact surface includes attributes that, based on a user-specified application involving a set of measurable forces, are constructed and arranged to include at least one region of ridges positioned perpendicular to an axis of a user-preferred measurable sensitivity to the measurable forces or torques of the mode of force.

18. The tactile sensor according to claim 12, wherein: The outer impact surface includes properties that, based on a user-specified application involving a set of measurable forces, include a user-predetermined pattern of one or a combination of ridges, rough textures, tapered ridges, symmetrical ridges, and asymmetric ridges positioned perpendicular to an axis of a preferred measurable sensitivity that provides a user-preferred measurable sensitivity for the pattern of forces.

19. A method for tactile sensing having: an elastically deformable element having an outer impact surface and a lower surface having pins, ridges, or both, the lower surface pins or ridges including indicia, and the elastically deformable element being attached to a gripper of a robot; and a camera located in the cavity of the holder to capture an image of the mark, the method comprising the steps of: capturing an image from the camera; accessing a memory having stored data, the stored data including image data of a set of pre-learned marker positions having pre-learned force patterns, each set of pre-learned marker positions corresponding to a pre-learned force pattern; detecting a position of the marker in a captured image; evaluating the captured image to detect each marker position in the captured image, and comparing the detected marker position in the captured image with the set of pre-learned marker positions of the image data based on a distance function to obtain a quality of match value for each set of pre-learned marker positions; applying a user selected best match function to the mass match values ​​to calculate a best match set of pre-learned marker positions and a pattern of pre-learned forces corresponding to the best mass matches; identifying a pattern of forces acting on the elastically deformable element based on the determined best matching pre-learned force pattern; and The identified force pattern is output to initiate a clamping action of the clamper based on the identified pattern of forces acting on the elastically deformable element.

20. An elastomeric tactile sensor for a robot, the elastomeric tactile sensor having: a deformable element including an outer impact surface and a lower surface having pins, ridges, or both, each lower surface pin or ridge including a marking, and the deformable element being attached to a gripper of the robot; and a camera positioned in the cavity of the gripper to capture an image of the marker, and a memory having stored data, the stored data comprising image data of a set of pre-learned marker positions with corresponding pre-learned force patterns, each set of pre-learned marker positions corresponding to a pre-learned force pattern, the tactile sensor comprising: An image processor is operatively connected to the camera and the memory, the image processor being configured to: detecting a position of the marker in a captured image; comparing the detected marker positions in the captured image with the set of pre-learned marker positions based on a distance function to determine a quality of a match value of each set of pre-learned marker positions; Using a user-selected best-match function, determining the best quality match of the pre-learned force pattern; identifying a pattern of forces acting on the deformable element based on the determined best matching pre-learned force pattern; and Output the identified force pattern to initiate the clamping action of the clamp based on the identified force pattern acting on the deformable element, wherein the identified force pattern includes one or a combination of the following: vertical pressure Z, center pressure, bias pressure, lateral force, i.e. lateral shear force X, lateral shear force Y, center or bias torque in pitch, yaw or roll direction, center or bias squeezing force X, center or bias squeezing force Y, center or bias deployment lateral force X, center or bias deployment lateral force Y.

Images