Elastomer Tactile Sensor

Abstract

A tactile sensor includes a camera positioned to capture an image of a marker. An elastically deformable skin includes an outer surface having an attribute and a lower surface having pins, ridges, or both. Each lower surface pin or ridge includes a marker. A processor detects displacement of the marker in the captured image and compares the displaced marker position in the captured image with a stored set of pre-learned marker positions based on a distance function to determine the quality of the matching value of each pre-learned marker position set. A pre-learned force pattern of the best quality match is determined using a user-selected best match function to calculate a best matching set of pre-learned marker positions. A force pattern acting on the elastically deformable skin is identified based on the determined best matching pre-learned force pattern.

Description

Technical Field

[0001] The present disclosure relates to tactile sensing, and more particularly to an elastomeric tactile sensor, which is an elastically deformable skin having an outer impact surface and a lower surface attached to a rigid surface through a flexible space. The lower surface includes pins, each pin includes a mark, and once an external force is applied to the outer impact surface, a force pattern representative of the external force is determined. Background Art

[0002] Conventional machines are used in manufacturing processes, including robots for product assembly, robots for transporting parts or humans, or autonomous vehicles. For example, conventional robotic systems include rigid outer surfaces composed of rigid links with actuation degrees of freedom controlled by motors. If exposed to unplanned impact forces, the rigid outer surfaces or rigid connections are very fragile, leading to catastrophic consequences in some cases and unacceptable repair costs among other major impact events. Rigid links are precision devices in which the position, velocity and / or force (or, in the case of rotary joints, angle, angular velocity and / or torque) in these degrees of freedom are controlled by advanced controllers that coordinate the motion of each joint. The controller can operate completely autonomously, or can be controlled manually in whole or in part by an operator, as in the case of a teleoperated robot.

[0003] However, if one or more components of the robot (such as the rigid surface of a rigid link) are impacted by an unexpected force, not only will the direct impact area be damaged, but other components of the robot such as mechanical components, electronic components, software components, network or communication components, etc. will also be damaged. Some of the results due to unexpected impact forces on conventional machines including conventional robots are catastrophic impact costs associated with manufacturing related processes such as machine / robot repair costs, other objects affected by machine / robot repair costs, assembly line lost production costs, and time lost (production and employee maximization) due to ordering new repair / replacement parts (if any). Possibly, one of the most important or critical results can be the creation of an unsafe working environment, which may cause harm to human life or limb loss to employees.

[0004] Robotic systems physically interact with their environment while moving through manufacturing and operating environments or performing tasks such as assembly tasks. Robotic controllers can gain knowledge of their environment and respond to expected and unexpected events. For example, with respect to autonomous systems, machine vision devices can be used to identify the location and orientation of objects, and precision equipment can guide the robot to a specific location. However, algorithms for machine vision devices are slow and prone to errors when vision is obstructed, surfaces are out of sight, shadows are present, or contrast is poor.

[0005] In some traditional systems related to remote control of semi-autonomous robots, mainly using wireless networks (such as Wi-Fi, Bluetooth, Deep Space Network, etc.) or tethered connections, human operators can interpret images from cameras or directly observe the robot and try to adjust commands to the robot as quickly and accurately as possible. However, this mental concentration can make the operator tired, and reaction time is usually slow.

[0006] Fast reaction systems require precise robots that employ rigid mechanical linkages, high quality position and force encoders, high speed feedback controllers, and powerful and heavy motors. However, these components are costly and impractical for today's technologies that require optimal cost savings relative to operability. High impact forces from unexpected collisions with objects in the environment can also have catastrophic effects on the robot or objects, including humans that may be in the workspace.

[0007] DC motors can drive industrial robotic hands or human-associated prostheses are externally powered artificial limbs that can be controlled by electrical signals naturally generated by the human's own muscles. The closing speed of the bare hand can be proportional to the voltage across the motor. In traditional prosthetic applications, this control voltage can be generated based on amplified and filtered electromyographic recordings of the prosthetic user's remaining muscles to produce a voltage proportional to the strength of muscle activation, thereby capturing the user's intention to open or close the hand. However, closing the prosthetic hand around a portion or unknown rigid object may cause the motors driving the fingertips to stall when they can no longer move forward, which may result in high stall forces.

[0008] Similar control problems can arise in preventing the robot attachments from injuring themselves or damaging external objects when colliding with or between appendages. These attachments may have hard surfaces. The impact force generated by the collision with the robot actuator may increase with the hardness of the collision surface and the relative speed of the robot attachment and the object. When the collision is detected by detecting the increased load on the actuator, serious damage may have already occurred. An example of such a collision could be an attachment of the robot accidentally hitting an elongated spear-shaped object extending in the path of the moving robot while moving, or other similarly shaped types of objects within the robot's working and operating environment. This is a real problem, especially in working environments with machines that produce steam / mist / smoke-like smoke that results in poor camera vision quality.

[0009] The ability to identify contact can become even more challenging in traditional autonomous robotic applications, where robots typically follow a prescribed trajectory at full power and only stop when an object provides a high enough resistance to stall the robot's actuators. Machine vision and ultrasonic proximity sensor approaches can help prevent unwanted collisions, but can be subject to high variability in accurately detecting objects before a collision. Machine vision is particularly prone to errors if the camera is obscured or if lighting is poor. These technologies can therefore be less than ideal in applications where safety and reliability are of high concern.

[0010] Other conventional tactile sensors have fluid-filled cavities and / or pressure sensing. Shinoda is mentioned in publication US 2010 / 0234997 by Sandini et al., describing a device that uses air pressure generated in carefully formed channels within a deformable polymer, which is transmitted to a microphone used as a pressure sensor; the differential signals from these multiple sensors are used to extract directional forces and sliding vibrations. However, most robot use occurs in noisy factories. Noise couples into the microphone, thus causing false triggering of forces and sliding when used in industrial environments.

[0011] Some traditional tactile sensing approaches have sensor arrays that do not provide much compliance and are often insensitive to contact forces applied between discrete tactile units. Patterning these sensing units on complex surfaces can be challenging and costly, and can result in areas that are insensitive to touch in areas where touch sensitivity becomes most important, such as edges and joints.

[0012] Conventional tactile sensors can identify the presence and pressure of an object, as well as the position of an object in a gripper for robotic applications including assembly. However, most conventional machine or robot sensors, including conventional tactile sensors, are based on a design similar to that of a pressure sensor and only measure compression without considering other very important forces, including shear motion or other types of force or pressure. For example, shear sensors can be used to detect aspects such as the motion of an object. In addition, the reaction time of many conventional machine or robot sensors (including conventional tactile sensors) is poor or cannot reach the response time that meets today's technological needs due to many reasons, including slow calculation, environmental factors (i.e., the camera is blocked or poorly illuminated), or in the case of remote-controlled robots due to human operator errors. Other reasons include the inability to pattern sensors on complex surfaces that are critical local areas, the inability of some conventional tactile sensors to detect / identify unplanned contact / impact that causes high stall forces, and many other reasons. Summary of the invention

[0013] The present disclosure relates to tactile sensing, and more particularly to an elastomeric tactile sensor, which is an elastically deformable skin having an outer impact surface and a lower surface attached to a rigid surface through a flexible space. The lower surface includes pins, each pin includes a mark, and once an external force is applied to the outer impact surface, a force pattern representative of the external force is determined.

[0014] Some embodiments of the present disclosure include a tactile sensor, which is an elastically deformable skin, which includes an outer impact surface that can have attributes and a lower surface with pins, ridges, or both. Each lower surface pin or ridge includes a mark, and the lower surface can be arranged on a flexible spacer from a rigid surface. The camera is positioned to capture an image of the mark. The memory includes stored data, and the stored data includes image data of a pre-learned mark position set with a corresponding pre-learned force pattern. Each pre-learned mark position set corresponds to a pre-learned force pattern. An image processor is operatively connected to the camera and the memory. The image processor can be configured to detect the displacement of the mark in the captured image. Based on a distance function, the shifted mark position in the captured image is compared with the pre-learned mark position set to determine the quality of the matching value of each pre-learned mark position set. The pre-learned force pattern of the best quality match is determined using a user-selected best match function applied to the quality of the determined matching value to calculate the best matching set of pre-learned mark positions and the corresponding pre-learned force pattern of the best quality match. Based on the determined best matching pre-learned force pattern, the force pattern acting on the elastically deformable skin is identified. The identified force pattern is output to initiate an action based on the identified pattern.

[0015] The lower surface pins or mechanical deflection amplification pins, i.e., micro-mechanical pins (hereinafter referred to as "pins"), deflect under pressure, shear and torque. Each pin carries at least one mark, logo or reference mark, such as white paint or colored paint. When the white marks are placed on a dark background, these marks can be easily tracked using machine vision algorithms. Because of the deflection movement caused by the external force on the outer impact surface, the outer impact surface is deformed under the action of pressure, bias pressure, shear force or torque force, which is recorded, and when compared with the sensitivity of traditional tactile sensors that only measure the deformation of only the outer impact surface, a higher sensitivity level is produced. For example, when the outer impact surface is deformed, the applied force transfers the same force to the lower surface, which causes the pin to be displaced, where the mark is displaced from the starting position (i.e., when no external force is applied to the outer impact surface) to the final position (i.e., after the external force is applied). Unlike traditional tactile sensors that can only record or measure compressive forces without considering shear movement or other types of forces or pressures, the present disclosure is sensitive to compressive forces, shear, torque, extrusion and expansion forces on the elastically deformable skin.

[0016] In fact, it was recognized from experimentation that the pin can be customized or modified to include lever arms extending outward from the pin (i.e., creating a tree-like graphic). Wherein, at each end of the lever arms, a white mark can be applied, which can be a traceable mark or a reference mark, providing further measurable force data. Based on experimentation, adding one or more lever arms to the pin results in a higher sensitivity amplification level (or measurable force level) than the sensitivity amplification level of only a single pin with only a single mark and without one or more lever arms. Therefore, one or more lever arms amplify the sensitivity amplification level or measurable force level per pin, resulting in a higher measurable sensitivity or a higher level of measurable force for the elastically deformable skin. Some of the reasons why these types of configurations of pins with levers can be useful is that certain areas specific to a particular application can be customized to position multiple groups of pins in areas that provide higher measurable sensitivity levels to prevent unexpected or expected impacts.

[0017] Another realization gained from the experiments is the development of flexible spacers. Specifically, one end of the flexible spacer is attached to the lower surface of the elastically deformable skin, while the other end of the flexible spacer is attached to a rigid surface, i.e., a rigid outer surface of a machine, robot, etc. The flexible spacer can be constructed with customized properties, including different amounts of elasticity, different amounts of rigidity or stiffness, and different amounts of strength. In addition, in order to achieve these customized properties, different types of materials can be used to create customized flexible spacers. Of course, depending on the location of the flexible spacer, the customized properties can include different lengths, diameters, or radii.

[0018] Unlike conventional tactile sensors, which can only record or measure pressure without considering shear movement or other types of forces or pressures. An embodiment of the present disclosure provides an elastomeric deformable skin that is configured to record and measure force patterns acting thereon and causing bending and sinking motion of an outer impact surface. For example, using one or more cameras or webcams, the force patterns recorded or measured include bending and sinking motion of the outer impact surface under pressure, bias pressure, shear force, or torque force. By way of non-limiting example, one or more cameras observe a marker or reference marker at an inclined angle (0 to ~20) degrees or at some other angle. At least one embodiment of the present disclosure includes the use of a machine vision algorithm that matches the marker motion with a pre-stored or pre-learned marker deflection set using interpolation (if necessary) to determine a force pattern or net force tensor on the outer impact surface of the elastically deformable skin.

[0019] As will be described in further detail, the elastically deformable skin, and in particular, the marked pins and lever arms, may be tailored to different types of intended measurable sensitivities, which may depend on the type of force pattern applied to the outer impact surface of the elastically deformable skin.

[0020] Due to this microrod distortion gain, the images captured by the camera have larger motion in the fiducial markers and better signal-to-noise ratio compared to traditional tactile sensor methods, which directly image the top surface of the contact surface through grazing illumination or through a hemispherical gel proxy.

[0021] The use of independently moving microrods also allows different parts of the elastically deformable skin to experience and report vastly different local strains and forces. For example, one part of the elastically deformable skin can be in shear from left to right, and the other side of the elastically deformable skin can be in shear from right to left, and because the microrods are free to move rather than being trapped in a continuous gel, the custom configuration of the sensor is able to correctly detect this as a surface torque, rather than being insensitive to a specific set of forces, just as the hemispherical gel is insensitive to these specific sets of forces.

[0022] In order to generate force signal output using the motion of the fiducial markers amplified by these micro-rods, several experiments were performed. Some aspects considered include: a fixed camera is structurally arranged at different positions within the cavity within which the robot, vehicle or machine is located. For example, the fixed camera captures an image of the fiducial markers attached to the pin. The image is sent to a computer, wherein, according to at least one experiment, a cheap webcam is connected to a computer running Debian Linux software via USB, but obviously, other experiments include different types of cameras, camera positions, camera mounting arrangements, and other types of software.

[0023] On a computer, image processing software can be used to extract the XY position of each fiducial marker. Obviously, there are many different but similar image processing software methods, but the preferred method involves using the open source package "OpenCV". At least one realization gained is that not every marker needs to be in the field of view of one camera or multiple cameras, all of which will depend on the specific application.

[0024] The next step involves using the Pythagorean metric (square root of the sum of the squares of the X and Y differences); determining the best match of the observed fiducial XY position to the pre-stored XY sets in the library. At least one surprising result is that this calculation is very fast; each observed XY position is compared to each pre-stored XY position in the pre-stored library; a "running count" is maintained for each of the approximately 50 pre-stored sets, based on the assumption that each fiducial marker is represented only once in each pre-stored set, so each pre-stored set running count is incremented by the Pythagorean distance between the observed XY fiducial position and the nearest pre-stored XY position of any fiducial in any given pre-stored set. In some applications, during experimentation, we found that there was no need to keep track of which observed XY fiducial position corresponds to which XY pin; only "closest XY, in some applications where we don't care about copying or reuse", which leads to excellent results with extremely simple coding.

[0025] More strictly, the Pythagorean metric in the preferred embodiment working for a cap with k reference pins can be expressed as follows:

[0026] For any observed XY set OBS, it contains k XY pairs OBS_X[0...k-1], OBS_Y[0...k-1]

[0027] And, any stored set of XY positions STO, consisting of STO_X[0...j-1], STO_Y[0...j-1] and using the function PDIST(arg1,arg2,arg3,arg4) defined as arg1 minus arg2 squared plus arg3 minus arg4 squared (i.e., SQRT((arg1–arg2)^2+(arg3–arg4)^2); this is the Pythagorean distance formula.

[0028] Then, the preferred distance value between any observed (OBS)XY set and any stored (STO) set is:

[0029] ·Preferred_Distance=SUM_over_all k(MIN_over_all_j(PDIST(OBS_X[k],STO_X[k],OBS_Y[k],STO_Y[k])))

[0030] As mentioned above, the surprising result from this experiment is that this calculation is exceptionally fast; with 10 fiducial markers and 100 pre-stored library sets, there are a total of 1000 XY pairwise comparisons, which totals to 2000 square operations, 1000 addition operations, and 1000 square root operations or about 4000 floating point operations. This means and recognizes that this is a very tractable method with excellent results, which even the cheapest microprocessor can perform well above the camera frame rate in terms of process configuration. Of course, more than 10 markers can be tested; the specific amount of 10 was used in the experiment.

[0031] The next step involves taking the sum of running counts of Pythagorean distances to sort the list of pre-stored XY position sets into an ordered quality of match list in order from lowest Pythagorean distance (best match) to lowest (worst match).

[0032] In some applications, a single best match is the only expectation and output of the sensor processing. For example, "No Damage Shock" vs. "Damage Shock" vs. "Possible Damage Shock" vs. "Maintenance Report Shock" might be labels for some of the stored XY position sets, and so these would be output by the sensor processor. It should be noted that there is nothing prohibited from having multiple stored XY position sets with the same label; there may be five to ten "Maintenance Report Shock" possibilities, and based on the application, it may not be necessary or even useful for the downstream system to know exactly which one it is. Different types of "Maintenance Report Shock" or "Damage Shock" possibilities may be maintenance or damage to a specific component (i.e., a spacer, a pin, a camera, a camera light aperture, an outer impact surface, an outer impact surface and an underlying surface, an attribute, etc.).

[0033] In other applications, a sensor result may be output, upon selection, that is some particular weighted, inversely weighted, or linear combination of the forces that were in effect when a particular stored set of XY positions was recorded.

[0034] It is also reasonable to limit the number of stored XY sets to be evaluated in weighting, inverse weighting, or linear interpolation; typically, the best matching (lowest Pythagorean distance) two or three stored XY position sets give the best accuracy for one-dimensional problems, 4 to 6 for two-dimensional problems, 6 to 8 for three-dimensional problems, etc. As will be appreciated from experimentation, the disclosed method is highly customizable to the user's specific application needs / needs (i.e., with respect to different types of force patterns or types of measurable sensitivity to individual forces), all while being executed at well above the camera frame rate using very inexpensive components (including use of inexpensive microprocessors).

[0035] As described above, the pin turns something that is difficult to measure (such as a force pattern or net force tensor) into something that is easy to measure. That is, the measurement is made easier based on the image motion of the tip or ends of the free micromechanical pin, i.e., the image motion of a marker at the outermost end of the pin is tracked by a camera such as a cheap webcam that observes the pin end at a certain angle. The pin end can be continuously observed by the camera before the pin end moves (i.e., before force is applied to the outer impact surface of the elastically deformable skin), during the pin end movement (i.e., while force is being applied to the external impact of the elastically deformable skin), and after the movement of the pin end stops (i.e., after the externally applied force has stopped).

[0036] Some other aspects learned while experimenting with different combinations of properties of the outer impact surface and combinations of pin configurations resulted in useful realizations, such as:

[0037] A) intentionally causing the pins of the present disclosure to become substantially non-parallel even under symmetrical forces, which was later discovered to be at least a key aspect of detecting asymmetric forces, shear, and torque on the pin ends;

[0038] B) Some configurations of the elastically deformable skin of the present disclosure are constructed with fewer pins, by way of non-limiting example, the number of pins for a thumb-sized sensor is in the range of 5 to 10 pins to obtain a substantially measurable force tensor. Larger numbers of pins, such as over 100 pins or 200 pins, do not provide measurable sensitivity levels that meet some of the goals of the present disclosure, so for some embodiments of the present disclosure, a much smaller number of pins is targeted.

[0039] C) Asymmetric Pins: Some embodiments of the present disclosure intentionally place or arrange fewer pins asymmetrically on the underside of the elastically deformable skin. In this way, some pins are placed at lower angles of the asymmetric skin or on the edge of the elastically deformable skin, between the rounded corners and the flat surface of the elastically deformable skin. Some experimental results show that the elastically deformable skin has higher sensitivity than pins located in the middle of the elastically deformable skin (i.e., not on the edge). In addition, the pin lengths can be staggered, and for highly optimized sensors, for example, when using finite element modeling (FEM), the FEM can indicate that some pins have asymmetric (highly elliptical or even buttress-like) bases. In this way, even for symmetric forces, this facilitates non-parallel motion; and

[0040] D) Oblique Camera Angle: Instead of looking along the Z axis (which requires 75-100mm vertical height), look at an oblique angle (typically around 10 degrees, and staggered pin lengths, allowing a zero degree viewing angle, parallel to the plane of the gripper end). It is therefore appreciated that this provides the opportunity to use a sensor that is only 10mm to 20mm thick without losing sensitivity to force, shear, and torque, and allows the use of the elastically deformable skin of the present disclosure for both compact spaces and compact space assemblies. By way of non-limiting example, an additional benefit / advantage is that the sensor can be used in a "teaching glove" worn by a human that records the appropriate force tensors to be applied during a robotic assembly operation. Note that other camera angles can be utilized, including 0-10 degrees, 10-15 degrees, 10-20 degrees, and 15-30 degrees.

[0041] In summary, by way of non-limiting example, some of the discovered recognized structural configurations of the elastically deformable skin may include: (1) dimensional thickness, e.g., uniform thickness or non-uniform thickness of the outer impact surface; (2) dimensional height and depth of the elastically deformable skin; (3) the type of material of the elastically deformable skin, and the locations of the different types of material that make up the elastically deformable skin; (5) how the elastically deformable skin is constructed / manufactured, including the type of three-dimensional (3D) printing, molding manufacturing process, etc. In addition, the number of pins to be mounted / attached / molded to the underside of the elastically deformable skin may depend on many factors (as learned from experimentation), for example, it is recognized that the number of pins may depend on the specific application based on the user's intent and goals in terms of achieving a predetermined level of sensitivity, durability, etc.

[0042] Another imaging processing method that has been tested and will be used with some embodiments of the present disclosure can be a finite element modeling (FEM) method. For example, in order to process data generated from a captured image of a camera including pin movement, a FEM method can be used to process the data into a metamaterial design. A metamaterial is an object whose internal structure produces a final object whose physical properties (electrical, magnetic, mechanical, thermal properties) are significantly different from the bulk properties of the "real" material of which it is composed, that is, negative ε, μ, almost any speed of light c, Poisson's ratio ν or thermal conductivity κ. Basically, the FEM model can predict the deformation of the elastomer under different loads. In addition, the elastomer pin lifts the surface contour and converts the surface inclination into an XY motion that is easy to image, that is, using a low-cost webcam image of the pin end. In addition, the structural design of the elastically deformable skin with fewer fiducial markers provides simple image thresholding without inter-frame tracking / ambiguity. In fact, OpenCV can be used to track the motion of the pin end that is converted into a force map. This minimizes computational overhead and maintains a high frame rate.

[0043] During the experiments, the inverse FEM proved to be computationally expensive. It was realized that the training of the model should use real data. One aspect further learned from the experiments was that for the case of labels with grasp position, angular pose, shear and torque, using the Euclidean distance in 14-dimensional space proved to be beneficial. For example, the computation time was very fast, i.e., the amount of computational cost proved to be very low, resulting in the ability to process in real time with excellent visualization on a 1GHz cpu (e.g., Raspberry Pi Zero).

[0044] In many cases, it may not be necessary to calculate the actual force tensor or force pattern, so that a simple comparison of the pin end position to a preset library of correct and incorrect images may be sufficient to determine the type of impact or collision to the elastically deformable skin, or to interpolate against a library of known distortions to produce a good estimate of the actual force tensor at the sensor in real time.

[0045] Other embodiments of the present disclosure also address today's industry needs by providing benefits such as low cost, low mass, and low profile simple compact tactile sensors while delivering highly sensitive sensing of external forces applied to elastomeric tactile sensors. Some of these benefits allow the compact tactile sensors of the present disclosure to be used in technologies where existing conventional tactile sensor use has proven to be too costly for various technology industries.

[0046] Some other insights gained from the experiments include configuring the camera position relative to the marker. For example, due to the deflection movement caused by the external force on the external impact surface, the external impact surface is deformed under pressure, bias pressure, shear force or torque force, and one or more cameras or webcams are located in the field of view of the marker to record the movement of the marker before, during and after the external force is applied to the external impact surface. By way of non-limiting example, one or more cameras can be positioned to observe the marker at an oblique (0 to ~20) degree angle or at some other angle. At least one embodiment of the present disclosure includes the use of a machine vision algorithm that can match the movement of the marker with pre-stored marker deflections using interpolation (if necessary) to determine the force pattern or net force tensor on the external impact surface.

[0047] Based on experiments, it is recognized that when having one or more cameras positioned with a viewing angle (i.e., tilt angle) between 0 and 20 degrees (0 to ~20), the camera arrangement results in a low profile elastomeric tactile sensor. It is also recognized that positioning the cameras to provide overlapping views can provide a complete representation of the applied force or force pattern applied to the external impact surface. By way of non-limiting example, some of the benefits and advantages of the low profile elastomeric sensors of the present disclosure provide solutions to some of the current technical problems caused by traditional tactile sensors due to their large profiles. For example, the low profile elastomeric sensors of the present disclosure are not only suitable for use in the robotics industry, but also, as a non-limiting example, are suitable for use in applications related to prosthetic applications associated with human skin, etc.

[0048] Conventional robots and conventional prosthetic applications have many problems, including having minimal tactile sensing, which creates problems for using conventional dedicated high-precision robots in work and operating robotic environments that include constant unplanned movement and object placement.

[0049] The reason is that dedicated high-precision robots cannot be used because they are not operationally capable of incorporating fast reaction times in situations where these types of objects are moving and unplanned changes of location occur. Similarly, conventional dedicated high-precision robots typically have little or no ability to know if the robot has accidentally bumped into or hit an object, which can result in a stalling effect that can result in injury or personal safety issues or expensive repairs to these types of robots. In order to overcome these problems with conventional dedicated high-precision robots, some embodiments of the present disclosure provide a robust, universal elastically deformable skin that provides, by way of non-limiting example, a full set of forces in pressure, translation, rotation, and shear that can be used to detect whether a robot, vehicle, or machine has made intentional or unintentional contact with an object, and has a fast response time, prevents damage, stalling effects, and the like.

[0050] With regard to some elastically deformable skin configurations according to some embodiments, the elastically deformable skin includes a plurality of micromechanical pins located on the underside of the elastically deformable skin. Each micromechanical pin (hereinafter referred to as a "pin") can extend outward from the underside and away from the underside of the elastically deformable skin. Wherein, the pin is located at the outermost end of the lower surface, i.e., away from the center of the bottom side, and the pin with a mark or a reference end mark (white or colored paint) has a movement damped only by air. The pin can be made of an elastomeric material, such as silicone rubber, polyurethane, thermoplastic elastomer, natural rubber, polyisoprene, polyamide (nylon), polyethylene, polypropylene polyvinyl chloride, plasticized photopolymerized acrylic, or a mixture thereof. The outer impact surface of the elastomeric cap can be made of one or more deformable materials. By way of non-limiting example, the elastomeric cap can be made of silicone rubber, polyurethane, thermoplastic elastomer, natural rubber, polyisoprene, polyvinyl chloride, plasticized photopolymerized acrylic, etc., so that it can be envisioned that the pin can be made of the same material or a different material or a combination of materials as the elastomeric cap. For example, the pin may be made of nylon, while the cap and impact surface may be made of polyurethane and manufactured by two-shot injection molding. It is thus conceivable that the pin may be made of the same material as the outer impact surface or a different material or combination of materials.

[0051] As described above, the pin converts something that is difficult to measure (such as a force pattern or net force tensor) into something that is easy to measure. That is, the measurement is made easier based on the image motion of the tip or tips of the free micromechanical pin, i.e., the image motion of a marker or a reference tip marker at the outermost end of the pin is tracked by a camera such as a cheap webcam that views the pin tip at an angle. The pin tip can be continuously viewed by the camera before the pin tip moves (i.e., before force tensors are applied to the outer impact surface of the elastically deformable skin), during the pin tip movement (i.e., while forces are being applied to the external impact of the elastically deformable skin), and after the movement of the pin tip stops (i.e., after the externally applied force pattern has stopped).

[0052] Practical Application

[0053] Certain embodiments of the present disclosure address the need for tactile sensors that address impact forces in today's technology by providing a low-cost, low-mass, and low-profile elastically deformable skin while delivering highly customized sensitivity sensing of undesired impact forces or applied external forces or force patterns on the elastically deformable skin.

[0054] For example, the attachability of the elastically deformable skin of the present disclosure (via flexible spacers to rigid surfaces) can be attached to very small highly sensitive local areas to provide customized measurable levels of sensitivity to protect areas of a robot, vehicle, or machine. In addition, these local areas include fragile components, joints, or other vulnerable local areas that can be protected from impact forces, thereby causing potential catastrophic component failure, resulting in unacceptable repair costs, reduced production costs, and other costly management of significant impact events in the robot, vehicle, or machine.

[0055] Some embodiments of the present disclosure address the needs of today's tactile sensing industry by rapidly detecting impact forces to allow for rapid action response to avoid unsafe work environments that could result in harm to human life or loss of limbs for employees, providing tailored performance measurable levels of sensitivity. Additionally, embodiments of the present disclosure assist in the safe operation of high-precision robots, vehicles, and machines in their movements and prevent damage from impact forces where position, velocity, and / or force (or, in the case of revolute joints, angle, angular velocity, and / or torque) in these degrees of freedom, or remote operation through mental concentration, is critical for specific applications.

[0056] The elastically deformable skin of the present disclosure is able to prevent the high stalling forces experienced by traditional tactile sensors. For example, the elastically deformable skin prevents high stalling forces from occurring by confirming a response in real time that stops or prevents the motor from a prescribed trajectory at full power. However, most traditional machine or robot sensors, including traditional tactile sensors, cannot provide shear movement or other types of similar forces or pressures, i.e., shear sensors detect motion that may be partially responsible for the stalling forces. Preventing these damages or failures can translate into maintaining and maintaining the operability of components such as mechanical components, electronic components, software components, network or communication components, etc. of a robot, vehicle or machine. In addition, embodiments of the present disclosure are not susceptible to harsh working or operating environments with steam, fog, smoke, etc. that can degrade traditional tactile sensing that relies on camera vision configurations.

[0057] According to an embodiment of the present disclosure, a tactile sensor includes an elastically deformable skin, the elastically deformable skin including an outer impact surface having an attribute and a lower surface having pins, ridges, or both, each lower surface pin or ridge including a mark, and the lower surface is arranged on a flexible spacer from a rigid surface. The camera is positioned to capture an image of the mark. The memory has stored data, and the stored data includes image data of a pre-learned mark position set with a corresponding pre-learned force pattern. Each pre-learned mark position set corresponds to a pre-learned force pattern. An image processor is operatively connected to the camera and the memory. The image processor is configured to detect the displacement of the mark in the captured image. Based on a distance function, the shifted mark position in the captured image is compared with the pre-learned mark position set to determine the quality of the matching value of each pre-learned mark position set. The pre-learned force pattern of the best quality match is determined using a user-selected best matching function applied to the quality of the determined matching value to calculate the best matching set of pre-learned mark positions and the corresponding pre-learned force pattern of the best quality match. Based on the determined pre-learned force pattern of the best match, the force pattern acting on the elastically deformable skin is identified. The identified force pattern is output to initiate an action based on the identified pattern.

[0058] Another embodiment of the present disclosure is a method for tactile sensing, comprising providing an elastically deformable skin, the elastically deformable skin comprising an outer impact surface having an attribute and a lower surface having pins, ridges, or both. Each lower surface pin or ridge comprises at least one mark, and the lower surface is arranged on a flexible spacer, which is attached to a rigid surface of the device. A camera arranged to capture an image of the mark is provided, and an image is captured from the camera. The displacement of the mark in the captured image is detected to obtain a shifted mark position. A memory having stored data is accessed, the stored data comprising image data of a pre-learned mark position set having a pre-learned force pattern, each pre-learned mark position set corresponding to a pre-learned force pattern. Based on a distance function, the shifted mark position in the captured image is compared with the pre-learned mark position set of the image data to obtain the quality of the matching value of each pre-learned mark position set, and the best matching function selected by the user is applied to the quality of the matching value to determine the pre-learned force pattern of the best quality match. Based on the determined best matching pre-learned force pattern, the force pattern acting on the elastically deformable skin is identified. The force pattern includes one or a combination of the following: vertical pressure Z, center pressure, offset pressure, lateral force X, shear force Y, torque in pitch, roll or roll, squeeze lateral force X or squeeze shear force Y. The identified force pattern is output to initiate an action based on the identified pattern.

[0059] According to another embodiment of the present disclosure, a system for controlling a robot includes an elastically deformable skin including an outer impact surface having an attribute and a lower surface having pins, ridges, or both. Each lower surface pin or ridge includes at least one mark, and the lower surface is arranged on a flexible spacer, which is attached to a rigid surface of the device. The camera is configured to capture an image of the mark. The memory has stored data, which includes image data of a pre-learned mark position set with a corresponding pre-learned force pattern, each pre-learned mark position set corresponding to the pre-learned force pattern. The image processor is operatively connected to the camera and the memory. The image processor is configured to detect the displacement of the mark in the captured image. Based on a distance function, the shifted mark position in the captured image is compared with the pre-learned mark position set to determine the quality of the matching value of each pre-learned mark position set. The pre-learned force pattern of the best quality match is determined using a user-selected best matching function applied to the quality of the determined matching value to calculate the best matching set of pre-learned mark positions and the corresponding pre-learned force pattern of the best quality match. Based on the determined pre-learned force pattern of the best match, the force pattern acting on the elastically deformable skin is identified. The identified force pattern is output to initiate an action based on the identified pattern.

[0060] Another embodiment of the present disclosure includes a tactile sensor for a mobile device, comprising an elastically deformable skin including an outer impact surface having an attribute and a lower surface having pins, ridges, or both. Each lower surface pin or ridge includes a mark, and the lower surface is arranged on a flexible spacer to which a rigid surface of the mobile device is attached. A camera is positioned to capture an image of the mark. The memory has stored data, the stored data including image data of a pre-learned mark position set with a corresponding pre-learned force pattern, each pre-learned mark position set corresponding to the pre-learned force pattern. An image processor is operatively connected to the camera and the memory. The image processor is configured to: detect the displacement of the mark in the captured image. Based on a distance function, the shifted mark position in the captured image is compared with the pre-learned mark position set to determine the quality of the matching value of each pre-learned mark position set. Using a user-selected best matching function applied to the quality of the determined matching value, the pre-learned force pattern of the best quality match is determined to calculate the best matching set of pre-learned mark positions and the corresponding pre-learned force pattern of the best quality match. Based on the determined pre-learned force pattern of the best match, the force pattern acting on the elastically deformable skin is identified. The identified force pattern is output to initiate an action based on the identified pattern.

[0061] The presently disclosed embodiments will be further explained with reference to the accompanying drawings. The drawings shown are not necessarily drawn to scale, with emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] [ Figure 1A ]

[0063] Figure 1A is a schematic diagram illustrating an embodiment of an elastically deformable skin attached to a rigid surface of a device such as a robot, vehicle, or machine, according to some embodiments of the present disclosure.

[0064] [ Figure 1B ]

[0065] Figure 1B is an illustration of an elastically deformable skin 8 attached to a rigid surface of an apparatus 11 (ie, a robot, vehicle, or machine) according to some embodiments of the present disclosure. Figure 1A Schematic diagram of a view of camera 5C.

[0066] [ Figure 1C ]

[0067] Figure 1C is an example of an elastically deformable skin 8 attached to a rigid surface of an apparatus 11 (ie, a robot, a vehicle or a machine). Figure 1A A schematic diagram of a view of camera 5C, and Figure 1C is camera 5C according to some embodiments of the present disclosure.

[0068] [ Figure 1D ]

[0069] Figure 1D is an example of an elastically deformable skin 8 attached to a rigid surface of an apparatus 11 (ie, a robot, a vehicle or a machine). Figure 1A A schematic diagram of the view of camera 5C, Figure 1D is a pin 2 with a mark 3 according to some embodiments of the present disclosure.

[0070] [ Figure 1E ]

[0071] Figure 1E is a schematic diagram illustrating a component of some methods and systems for implementing the present disclosure according to some embodiments of the present disclosure.

[0072] [ Figure 1F ]

[0073] Figure 1F is a schematic diagram illustrating another embodiment of an elastically deformable skin attached to a rigid surface of a device (such as a robot, vehicle, or machine) via spacers according to some embodiments of the present disclosure, wherein pins are strategically arranged in the structure to provide a large number of customized measurable sensitivities to one or more pressures, forces, or application-specific pressure and force patterns.

[0074] [ Figure 1G ]

[0075] Figure 1G is a schematic diagram illustrating another embodiment of an elastically deformable skin attached to a rigid surface of a device (such as a robot, vehicle, or machine) via spacers in accordance with some embodiments of the present disclosure, wherein the outer impact surface includes ridges and cones strategically arranged in the structure to provide a large number of customized measurable sensitivities to one or more pressures, forces, or application-specific pressure and force patterns.

[0076] [ Figure 2A ]

[0077] Figure 2A is a flowchart illustrating an implementation of some steps for implementing a method according to some implementations of the present disclosure.

[0078] [ Figure 2B ]

[0079] Figure 2B is a flowchart illustrating an implementation of some steps for implementing a method according to some implementations of the present disclosure.

[0080] [ Figure 2C ]

[0081] Figure 2C is an example of some embodiments of the present disclosure. Figure 2A 2 is a block diagram of some of the steps for pair-wise matching a set of observed fiducial marker XY positions against a pre-stored library of XY positions stored in memory in step 225 of .

[0082] [ Figure 2D ]

[0083] Figure 2D is a block diagram illustrating some steps for generating a pre-stored library of XY positions according to some embodiments of the present disclosure.

[0084] [ Figure 2E ]

[0085] Figure 2E is a block diagram illustrating some steps for using image measurement functionality according to some embodiments of the present disclosure.

[0086] [ Figure 3A ]

[0087] Figure 3A is a flow chart illustrating some steps of finite element modeling (FEM) according to some embodiments of the present disclosure.

[0088] [ Figure 3B ]

[0089] Figure 3B is an example of using according to some embodiments of the present disclosure Figure 3A Schematic diagram of aspects of the FEM.

[0090] [ Figure 3C ]

[0091] Figure 3C is an example of using according to some embodiments of the present disclosure Figure 3A Schematic diagram of aspects of the FEM.

[0092] [ Figure 4A ]

[0093] Figure 4A is a schematic diagram illustrating some assemblies for using elastically deformable skins along links of a robotic assembly according to some embodiments of the present disclosure.

[0094] [ Figure 4B ]

[0095] Figure 4B is a schematic diagram illustrating a cross-sectional view of an elastically deformable skin attached to a link of a robot according to some embodiments of the present disclosure.

[0096] [ Figure 4C ]

[0097] Figure 4C is a schematic diagram illustrating an elastically deformable skin configured at a joint of a robot according to some embodiments of the present disclosure.

[0098] [ Figure 4D ]

[0099] Figure 4D is a schematic diagram illustrating an elastically deformable skin and an articulated skin configured to another joint of two links or segments of a robot according to some embodiments of the present disclosure.

[0100] [ Figure 4E ]

[0101] Figure 4E is a schematic diagram illustrating multiple views of a camera that are overlapping views of pins associated with an elastically deformable skin attached to a rigid surface of a robot according to some embodiments of the present disclosure.

[0102] [ Figure 5A ]

[0103] Figure 5A is a schematic diagram illustrating some components of an alternative robotic system according to some embodiments of the present disclosure, the alternative robotic system including a robot having a gripper with an elastically deformable skin and various subsystems.

[0104] [ Figure 5B ]

[0105] Figure 5B is a schematic diagram illustrating an alternative processor according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0106] Although the above-identified drawings illustrate the currently disclosed embodiments, other embodiments may be envisioned as noted in the discussion. The disclosure presents exemplary embodiments by way of representation and not limitation. Those skilled in the art may devise many other modifications and embodiments that fall within the scope and spirit of the principles of the currently disclosed embodiments.

[0107] The present disclosure relates to tactile sensing, and more particularly to an elastomeric tactile sensor, which is an elastically deformable skin having an outer impact surface and a lower surface attached to a rigid surface through a flexible space. The lower surface includes pins, each pin includes a mark, and once an external force is applied to the outer impact surface, a force pattern representative of the external force is determined.

[0108] Figure 1A is a schematic diagram illustrating an embodiment of an elastically deformable skin surrounding a component 11 (such as a robot or a machine device) according to an embodiment of the present disclosure. The elastomeric tactile sensor includes an elastically deformable skin 8 having an outer impact surface 9 and a lower surface 10 attached to the component 11 via spacers 7A-7D. The lower surface 10 includes a plurality of pins 2, each of which includes a mark or a reference end mark 3. In this way, when a force pattern is applied to the outer impact surface 9, the elastically deformable skin 8 is deformed, causing the pins 2 to move, and the marks or reference end marks 3 undergo flexural motion. For example, when pressure, force, torque, extrusion, stretching, etc. are applied to the outer impact surface 9 corresponding to the movement of the pins 2 that move the marker 3, the marker 3 is in motion through the force pattern when in motion. The elastomeric tactile sensor also includes cameras 5A-5D positioned in view of the marker 3 and capturing images of the marker 3 when the marker 3 is in motion.

[0109] Figure 1B , Figure 1C and Figure 1D , according to some embodiments of the present disclosure, Figure 1B is a schematic diagram illustrating a view of a camera 5C of an elastically deformable skin 8 attached to a rigid surface of a component 11 (ie, a robot or machine), and Figure 1C It is a camera 5C, and Figure 1D It is pin 2 with marking 3. Figure 1B The camera view of is from camera 5C which observes the lower surface 10 of the elastically deformable skin 8, the pins 2, the cameras 5A, 5B and the spacers 7A, 7B.

[0110] Figure 1E 1 is a schematic diagram illustrating some components of some methods and systems for implementing the present disclosure according to some embodiments of the present disclosure. Some methods and systems of the present disclosure include an elastically deformable skin that is part of an elastomeric tactile sensor that is attached to a component having a ridged surface, that is, the component may include a robot, a vehicle, or a machine. The sensor control computer 113 includes a hardware process 120, a control module 140, a storage 130, and a memory 122, all connected via a bus 106. The hardware processor may implement or execute stored instructions stored in the memory 122 and / or the storage 130. Specifically, the hardware processor 120 may access a pre-stored XY set 131. The storage 130 may store a pre-stored XY set 131, which may include a label and a force value. Optionally, the pre-stored XY set with a label and a force value may be stored in the storage 130, the memory 122, or both, all depending on a specific application predetermined by a user.

[0111] The control module 140 connected via bus 106 is connected to the component control computer 142 so that the component control computer 142 can communicate back to the control module 140 via 143. For example, the control module 140 can process scene data via sensor data 186, which can implement a determination method to be executed in the sensor control computer 113.

[0112] Still refer to Figure 1B , component control computer 142 is connected to component 144, i.e., a robot, vehicle, or machine, wherein component 144 can communicate back to component control computer 142 via 145. For example, component control computer 142 can control the actions of a mobile robot, vehicle, or machine, or can send data about assisting in generating action commands back to the control module. Component control computer 142 can command component 144 to move or do some other action.

[0113] In addition, component 144 is connected to the distortion of the outer impact surface 148, wherein the physical environment 146 is also connected to the distortion of the outer impact surface 148. For example, the physical environment 146 includes an object or thing that may cause an impact on the outer impact surface. In addition, the distortion of the outer impact surface 148 is an external pressure or force that constitutes a force pattern or net force sensor, which causes the pin to move. Wherein at least one camera 133 captures an image of the marker and the pin position to obtain video image data. For example, the distortion of the outer impact surface 148 can come from the force being applied to Figure 1A The external pressure, force, etc. of the outer impact surface 9. The external or external pressure, force, etc. can be a force mode, and the embodiments of the present disclosure can be directed to the external impact surface 9. Figure 1AThe amount of measurable sensitivity is determined for each pressure or force of the force pattern of the outer impact surface 9.

[0114] Still refer to Figure 1B Optionally, the sensor control computer 113 may include a network interface controller 150 adapted to connect the system 100 to a network 151 via the bus 106, the network 151 connecting the control system 100 to a wireless system that can wirelessly receive the video image data 133 of the captured images of the marker and pin positions, as well as other data. Also optionally, the sensor control system via the bus 106 may include a transmitter interface 160 connected to a transmitter 165 to wirelessly send data to a remote robot control computer (not shown), a remote robot (not shown), or even a remote control module (not shown). In addition, the sensor control system 113 may be connected to a control interface via the bus 106, which is connected to one or more power sources 128. The control interface may provide data including action commands to the power source or other device. It is contemplated that the power source 128 may be one or a combination of a battery, an alternative power source, a 120 volt power source, an alternative energy source (i.e., sun, water, wind, etc.). In addition, the sensor control computer may include a receiver interface 180 connected to a receiver 185 via the bus 106 to receive the video image data or other data.

[0115] Figure 1F is a schematic diagram illustrating another embodiment of an elastically deformable skin 8 according to some embodiments of the present disclosure, wherein pins 2 attached to a lower surface 10 are strategically arranged in the structure to provide a large number of customized measurable sensitivities to one or more pressures, forces, or application-specific pressure and force patterns.

[0116] Figure 1G is a schematic diagram illustrating another embodiment of an elastically deformable skin 8 according to some embodiments of the present disclosure, wherein an outer impact surface 9 includes a pattern or non-pattern of ridges 98 and cones 99 that are strategically arranged on the structure to provide a large number of customized measurable sensitivities to one or more pressures, forces, or application-specific pressure and force patterns.

[0117] Figure 2A and Figure 2B is a flowchart illustrating an implementation of some steps for implementing a method according to some implementations of the present disclosure.

[0118] Figure 2AStep 215 includes using a camera, a video camera or a webcam to capture the image of the benchmark end mark of the motion from the starting point to the final endpoint. For example, a fixed camera captures the image of the micro-rod pin carrying the benchmark mark. As according to at least one experiment, the image is sent to a computer or processor, wherein a cheap webcam can be connected to a computer running software (such as, Debian Linux software) via USB. However, it is obvious that other experiments can include different types of cameras, camera positions, camera installation layouts and other types of software.

[0119] Figure 2A Step 220 includes identifying measured image coordinates of locations in the image of the captured image. On a computer, image processing software can be used to extract the XY position of each fiducial marker located on the lower surface of the elastically deformable skin. Obviously, there are many different but similar methods, but a preferred method includes using the open source package "OpenCV". It is recognized that depending on the specific application, not every marker needs to be in the field of view of the camera. Initially, during this experimental color, the webcam image has been reduced to a grayscale image, but this is not required and is optional.

[0120] Reference Figure 2B Optionally, step 221 may include a method step for thresholding the image. At this point, only black (background) and white (fiducial markers) are retained as black or white pixels. As a non-limiting example, a preferred threshold level is a pixel brightness value of 245 on a scale of 0 to 255. This method appears to render the fiducials as irregular white spots in black areas, as well as small white spots caused by noise in the sensor or fabric lint contamination.

[0121] In addition, refer to Figure 2B Optionally, step 222 may include a method step for denoising the white spots by replacing each pixel value with the lowest pixel value found in a 3x3 region centered around each pixel via a first method called the OpenCV "erosion" operator. This erosion method is performed 3 times, removing noisy pixels and lint contamination from the image; this is because these noise features are typically less than 6 pixels across.

[0122] In addition, refer to Figure 2BOptionally, step 223 may include a method step for re-expanding the remaining white spots by inverse erosion known as OpenCV "dilation". It replaces each pixel value with the highest pixel value found in a 3x3 region centered on that pixel. This results in expanding the remaining white spots into a relatively stable, relatively circular mask centered on the area where the fiducial marker is located. This mask image is then used with the grayscale image to essentially eliminate all noise and contamination from the grayscale image, because the areas / regions / places after the erosion / dilation process where no stable areas have been restored are black (zero) and when logically ANDed with the original image, produce a black background that makes marker detection fast and very low noise.

[0123] In addition, refer to Figure 2B Optionally, step 224 may include a method step using a standard OpenCV blob detector "Simple Blob Detector" to find the centers of these blobs, which are the XY locations of the fiducial markers on the end of the microrod pin.

[0124] Reference Figure 2A , step 225 includes determining a set of relative displacements of the fiducial markers in the captured image. For example, because the number of such XY pairs is equal to the number of fiducial markers and is small (for these experiments, only seven fiducial pins were used), the next step 225 may be to pair-wise match the set of observed fiducial marker XY positions with a pre-stored library of XY positions.

[0125] The pre-stored library of XY positions was previously generated through multiple experiments. For this particular set of experiments, less than 50 pre-stored XY sets are stored in the memory of the pre-stored library database, resulting in excellent resolution and availability. Of course, more than 50 pre-stored XY sets can be stored in the pre-stored library database, but for these sets of experiments, 50 pre-stored XY sets appear to be sufficient to achieve some goals according to some aspects of the present disclosure.

[0126] Figure 2A Step 230 includes steps for calculating a force pattern or determining a net force tensor acting on the top surface of the elastically deformable skin. As a general overview, the force pattern can be determined by matching a set of relative displacements of reference end markers to a stored set of previously learned relative displacements of reference end markers placed in motion, and then identifying the force pattern associated with the stored set of previously learned relative displacements of reference end markers.

[0127] therefore, Figure 2AStep 230 of begins with the following method steps: Using the Pythagorean metric (square root of the sum of the squares of the X and Y differences); determine the best match of the observed fiducial XY position to the pre-stored XY sets in the library. This calculation is very fast; each observed XY position is compared to each pre-stored XY position in the pre-stored library; a "running count" is maintained for each of the approximately 50 pre-stored sets, based on the assumption that each fiducial marker is represented only once in each pre-stored set, so each pre-stored set running count is incremented by the Pythagorean distance between the observed XY fiducial position and the nearest pre-stored XY position of any fiducial in any given pre-stored set. While conducting experiments for this step, it was initially assumed that a single marker was required or necessary to be tracked over time (i.e., the leftmost marker in the front row corresponds to the fourth XY coordinate in the fourth pre-stored library set, etc.). However, later experiments showed that this assumption was a completely unnecessary calculation and could be omitted without loss of accuracy.

[0128] Still refer to Figure 2A Step 230, the astounding result from this experiment is that this calculation is exceptionally fast; with 10 fiducial markers and 100 pre-stored library sets, there are a total of 1000 XY pairwise comparisons, i.e., a total of 2000 square operations, 1000 addition operations, and 1000 square root operations or about 4000 floating point operations. This means and recognizes that this is a very tractable method with excellent results, and in terms of process configuration, even the cheapest microprocessor can execute at well above the camera frame rate.

[0129] Figure 2A Step 235 may include the method step of taking a running sum of counts using a Pythagorean distance to sort the list of pre-stored XY position sets into a ranked quality of a matching list.

[0130] In some applications, a single best match is the only expectation and output of the sensor processing. For example, "non-damage impact" vs. "damage impact" vs. "possible damage impact" vs. "maintenance report impact" might be labels for some of the stored XY position sets, and therefore these would be output by the sensor processor. It should be noted that there is nothing prohibited from having multiple stored XY position sets with the same label; there may be five to ten "maintenance report impact" possibilities, and based on the application, it may not be necessary or even useful for the downstream system to know exactly which one it is. Different types of "maintenance report impact" or "damage impact" possibilities can be maintenance or damage to a specific component (i.e., a spacer, a pin, a camera, a camera light aperture, an outer impact surface, an outer impact surface and an underlying surface, an attribute, etc.).

[0131] Still refer to Figure 2A In step 230, in other applications, the sensor result may be outputted based on the selection, the sensor result being some particular weighted, inversely weighted, or linear combination of the forces that were in effect when a particular stored set of XY positions was recorded. For example, if downward forces of 1, 2, 5, and 10 Newtons were recorded, and the corresponding reference XY position sets were stored, and then the sensor was subjected to an unknown force, an image may be captured, where the image is processed as described above to obtain the Pythagorean distance, and then the following may be performed:

[0132] For the XY set stored by 1 Newton, Pythagorean distance = 350

[0133] For the XY set stored by 2 Newton, Pythagorean distance = 100

[0134] For the XY set stored by 5 Newton, Pythagorean distance = 50

[0135] For the XY set stored by 10 Newton, Pythagorean distance = 400

[0136] …which can then be calculated as follows:

[0137] The sum of the reciprocals of the weights = 1 / 350 + 1 / 100 + 1 / 50 + 1 / 400 = 0.0353

[0138] Weight of 1 Newton sample = 1*[1 / 350] / 0.0353 = 0.080

[0139] Weight of 2 Newton sample = 2*[1 / 100] / 0.0353 = 0.566

[0140] Weight of 5 Newton sample = 5*[1 / 50] / 0.0353 = 2.832

[0141] Weight of 10 Newton sample = 10*[1 / 400] / 0.0353 = 0.708

[0142] Total weighted sum: 4.186

[0143] ...shows that the indicated force on the elastically deformable skin sensing surface is 4.186 Newtons.

[0144] The same process can be extended to multiple forces operating alone or in combination, such as 1, 2, 5, 10 Newtons of downward force, -10, -5, -2, -1, 0, 1, 2, 5, and 10 Newtons of shear in X, -10, -2, 2, and 10 Newtons of shear in Y, and -10, -2, +2, and +10 Newtons of torque in Z.

[0145] Still refer to Figure 2A Step 230. It is also reasonable to limit the number of stored XY sets to be evaluated in weighting, inverse weighting or linear interpolation; typically, the best match (lowest Pythagorean distance) two or three stored XY position sets give the best accuracy for one-dimensional problems, 4 to 6 for two-dimensional problems, 6 to 8 for three-dimensional problems, etc. As can be seen, the method of the present disclosure can be highly customized to the user's specific application needs / requirements while using very inexpensive components and executing at much higher than camera frame rates using inexpensive microprocessors.

[0146] Still refer to Figure 2A In step 230, experiments have shown that for elastically deformable skins of sizes between 300 and 1000 grid points (each with three degrees of freedom), sufficient fidelity is required in a linear finite element model (more if nonlinearity is allowed). This means that the calculation phase involves solving up to 3000x3000 linear equations at the camera frame rate, which is certainly computationally expensive, if not completely impossible with current conventional tactile sensor technology.

[0147] Figure 2C is an example of some embodiments of the present disclosure. Figure 2A Step 225 of performing a pair-wise matching of the set of observed fiducial marker XY positions with a pre-stored library of XY positions stored in memory ( Figure 2A A block diagram of some steps of step 225 of the present invention is provided. The pairwise matching distance (PBPMD) of the observed set of marker XY positions and the pre-stored library of XY positions stored in memory is calculated based on the following steps:

[0148] Figure 2C Step 251 includes setting the currently stored set of XY positions to the above-mentioned stored set of XY positions.

[0149] Figure 2C Step 253 includes executing the following routine PBPMD to find the pairwise matching distance of the observed XY position set to the currently stored XY position set to obtain the pairwise matching distance of that observed XY position set to the observed XY position set.

[0150] Figure 2C Step 255 includes calculating the PBPMD of the currently stored set of XY positions and the observed set of XY positions:

[0151] Figure 2C Step 257 includes:

[0152] [--Set CURRENT_STORED_XY_SET = Current stored XY position set

[0153] --Set variable THIS_SET_TOTAL_PYTHAG_DIST=0

[0154] --For each observed XY position in the set of observed XY positions: ]

[0155] [ --set THIS_LOC_MIN_DIST=10000000.

[0156] -- Set THIS_OBS_X and THIS_OBS_Y = the X and Y values ​​of the observed XY position

[0157] --For each stored XY position in CURRENT_STORED_XY_SET: ]

[0158] [--Set THIS_STORED_X and THIS_STORED_Y = X and Y values ​​of the stored XY position

[0159] --Set THIS_DIST=SQRT((THIS_OBS_X-THIS_STORED_X)^2+(THIS_OBS_Y-THIS_STORED_Y)^2)

[0160] --if THIS_DIST <THIS_LOC_MIN_DIST:]

[0161] [ADD THIS_LOC_MIN_DIST TO THIS_SET_TOTAL_PYTHAG_DIST]

[0162] [Returns THIS_SET_TOTAL_PYTHAG_DIST as the pairwise matching distance.]

[0163] Figure 2D 260 is a block diagram illustrating some steps for generating a pre-stored library of XY positions according to some embodiments of the present disclosure. Generating a pre-stored library of XY positions 260 may include step 261, which includes applying a known force (such as pressure, shear, torque, squeeze force, or expansion force) to the outer surface of the cap.

[0164] Figure 2D Step 263 includes defining a label for the imposed force, such as "GRIP OK" or "Y shear = 30 Newtons".

[0165] Figure 2D Step 265 includes capturing an image of the markings on the interior of the cap.

[0166] Figure 2D Step 267 includes processing the image according to the same operations in the same order as steps 221 (thresholding), 222 (erosion), 223 (dilation), and 224 (blob detection) to obtain a set of XY positions.

[0167] Figure 2D Step 269 includes storing the set of XY positions with the defined labels as one of a pre-stored library of sets of XY positions stored in the memory.

[0168] Figure 2E is a block diagram illustrating some steps for using the image measurement function 270 according to some embodiments of the present disclosure. Step 271 includes identifying a starting position of a marker from one of a plurality of captured images.

[0169] Figure 2E Step 273 includes identifying a final position of the marker from another captured image of the plurality of captured images.

[0170] Figure 2E Step 275 comprises associating the start position of the identified marker with the measured image coordinates in the one captured image.

[0171] Figure 2E Step 277 comprises associating the final position of the identified marker with the measured image coordinates in the further captured image.

[0172] Figure 2E Step 279 includes comparing the measured image coordinates of the fiducial markers from the starting position measurement coordinates to the final position measurement coordinates to identify a set of relative positions of the fiducial markers in motion.

[0173] Figure 2E Step 280 includes using an image processor to convert the relative position of each marker measured in image coordinates between the one captured image and the other captured image, summing the relative positions according to the Euclidean distance to form a matching distance between the one captured image and the other captured image.

[0174] Figure 3Ais a flow chart illustrating some steps of using finite element modeling (FEM) 310 according to some embodiments of the present disclosure. FEM is an excellent place to show or illustrate that the use of multiple pins on the lower surface of the elastomeric cap can amplify and increase the sensitivity of the elastic deformation of the top surface of the elastomeric cap when compared to the amount of measurable sensitivity of the deflection motion of the "only" outer impact surface of the elastomeric cap. For example, step 315 includes creating a CAD model of the elastomeric cap, including the inner surface, outer surface, edge, micro-rods, and markers.

[0175] Figure 3A Step 320 includes converting the CAD model into a mesh suitable for use in a FEM software package.

[0176] Figure 3A Step 325 includes defining a fixed portion or portions of the mesh, typically corresponding to the edges of the elastomeric cap.

[0177] Figure 3A Step 330 includes defining a test force or forces applied to the cap, typically corresponding to pressure, shear, torque, compression or deployment forces.

[0178] Figure 3A Step 335 includes defining the Young's modulus and optionally the yield strength of each material or materials making up the cap.

[0179] Figure 3A Step 340 includes executing a FEM software package on the mesh, the fixture or fixtures, and the test force or forces, which typically involves solving three force balance equations (one each for X, Y, and Z forces) simultaneously for each mesh point.

[0180] Figure 3A Step 345 includes interpreting the results of the FEM solution, which is typically the X, Y, and Z displacements that each grid point will experience when subjected to the test force or forces.

[0181] Figure 3B and Figure 3C is an example of using according to some embodiments of the present disclosure Figure 3A of Figure 3A Schematic diagram of aspects of finite element modeling (FEM). For example, Figure 3B A CAD model of an elastomeric cap is illustrated, and the mesh of thin lines shows the mesh that will be used for finite element modeling. Within each short line in the mesh, the material behavior is assumed to be linear.

[0182] Figure 3CThe result of FEM solution of CAD model in 3B is illustrated, and material is set to elastomer, and Shore durometer stiffness is 80A (about equal to the rubber used in the sole of mountaineering boots), the fixed part of the grid is the edge of the cap, and the test force is vertically upward below the foreground micro-rod pin of the rightmost end. Note that although the foreground micro-rod end of the rightmost end has a larger movement, the adjacent left foreground micro-rod end has an even larger movement. This is due to the stress of the test force causing the tilt (tip and tilt) distortion in the cap. The base of the left micro-rod moves less vertically than the base of the right micro-rod, but the overall movement is much larger, and the movement is mainly in the-X direction. This makes it possible to eliminate the ambiguity of different forces.

[0183] Figure 4A is a schematic diagram illustrating some components for using elastically deformable skins deployed on a mobile robot assembly according to some embodiments of the present disclosure. The robot is composed of a robot base 405 having one or more links or segments 407 linked by a joint 411, which is an axial rotation joint. A plurality of elastically deformable skins 401 are attached to various locations on each link or segment 407 of the robot. The elastically deformable skins 401 can be shaped to protrude above the joint 411 without interfering with the movement of the joint 411.

[0184] Figure 4B is a schematic diagram illustrating a cross-sectional view of an elastically deformable skin 401 deployed on a link or segment 407 of a mobile robot according to some embodiments of the present disclosure. The cross-sectional view of the link 407 and the elastically deformable skin 401 illustrates the use of multiple elastically deformable skins 401 disposed on the surface of the link or segment 407 of the robot to provide spatial information about any contact point during an impact or collision with an external object.

[0185] Figure 4C is a schematic diagram illustrating an elastically deformable skin 401 configured at a joint 409 of a robot's link 407 according to some embodiments of the present disclosure. The robot's links or segments 407 are connected by hinge joints 412, wherein a plurality of elastically deformable skins 401 are configured to remain covered over the joints 412 while avoiding contact between the robot's links or segments 407 as they change relative positions during the movement of the joints 412.

[0186] Figure 4Dis a schematic diagram illustrating an elastically deformable skin 418 having an articulated skin 413 configured at another joint of two links or segments 407 of a robot according to some embodiments of the present disclosure. The robot links or segments 407 are connected by hinge joints 416, wherein a plurality of elastically deformable skins 418 and articulated skins 413 are configured to remain overlying the joints 416 while avoiding contact between the robot links or segments 407 as they change relative positions during movement of the joints 416. The articulated skin 413 includes additional articulated covering joints 419 covering the areas between the elastically deformable skins 418.

[0187] Figure 4E is a schematic diagram of multiple views of an example camera that are overlapping views of pins associated with an elastically deformable skin attached to a rigid surface of a robot according to some embodiments of the present disclosure. For example, cameras 417A-417D have overlapping views, such that camera 417A has a camera view 419A that overlaps with camera view 419B of camera 417B, and camera 417C has a camera view 419C that overlaps with camera view 419D of camera 417D. Overlapping views can provide more image data information while ensuring that image data is collected for joint motion between two links or other aspects of the robot.

[0188] experiment

[0189] The use of independently moving microrods also allows different parts of the elastically deformable skin to experience and report back significantly different local strains and forces. For example, one part of the elastically deformable skin can be in shear from left to right, and the other side of the elastically deformable skin can be in shear from right to left, and because the microrods are free to move rather than being (un)constrained in a continuous gel, the custom configuration of the elastically deformable skin can correctly detect this as a surface torque, rather than being insensitive to a specific set of forces (i.e., the continuous gel), just as the hemispherical gel is insensitive to these specific sets of forces.

[0190] Several experiments were conducted to utilize these microrod-amplified fiducial marker motions to generate force signal outputs. Some aspects considered include structurally arranging fixed cameras at different locations within the cavity within which the component, robot, vehicle, or machine is located.

[0191] Based on experiments, when having one or more cameras positioned at a viewing angle (i.e., tilt angle) between 0 and 20 degrees (0 to 20), the one or more camera arrangements result in a low profile high sensitivity elastomeric tactile sensor. By way of non-limiting example, some benefits and advantages of the low profile elastomeric sensor of the present disclosure provide solutions to some current technical problems of conventional tactile sensors with large profiles. For example, the low profile elastomeric sensor of the present disclosure is not only applicable to the robotics industry, but also, as a non-limiting example, is applicable to applications related to the vehicle industry and the machine industry.

[0192] Many tests were conducted to determine how to design and construct an elastically deformable skin based on the level of sensitivity, durability, and manufacturing capabilities. For example, a sensor that was reviewed through public documents is the GelSight sensor. This sensor uses a thick block of transparent gel with a flexible opaque coating on one side. The coated side of the block is pressed against the surface, and the distorted paint is observed vertically while being illuminated under glancing illumination at several different angles. It is understood that the sensor is able to find textures (burrs) that protrude upward on the surface. However, this Gelsight sensor does not seem to be useful for gaps or pinholes, making the sensor have a very low level of damage force threshold. Another disadvantage among the many shortcomings found is that this Gelsight sensor requires a Z-axis margin of about 100 mm for the camera to operate. Among them, because the margin of the camera is so large, it means that the sensor certainly cannot have a low-profile configuration and seems to be better suited for robotic detection of textured surfaces rather than any type of low-profile application. In addition, the low level of force damage threshold and surface wear make this Gelsight sensor unsuitable for many applications.

[0193] Based on extensive experiments and the level of desired sensitivity, durability, manufacturing cost and other factors, many implementations have been obtained for designing and constructing some embodiments of the present disclosure. Some of these implementations for some embodiments include extending the pin on the underside of the elastically deformable skin in a manner that the pin extends away from the underside. The configuration includes the following pin that must be designed to meet not only the strength level but also the deformability level so that it can be deformed to record the amount of deformation of the pin due to the external force tensor. Specifically, some configurations of the pin are specifically designed to prevent the pin from remaining parallel to other pins. For example, it is found from the public documents about experiments via optical flow analysis that the pins are arranged in parallel in the experimental elastically deformable skin. The pins exhibit a certain level of strength, that is, the strength level that affects the deformation of the pins when the applied external force is applied. This experimental elastically deformable skin obtains an acceptable measurable level of vertical force. However, based on the image motion obtained by the vertical (z-axis) camera at the end of the pin, this same experimental elastically deformable skin configuration failed to provide an acceptable level of measurable shear and torque sensing / force. It is understood from this experimental configuration that, at least in view of the measurable sensitivity goals of the desired level of the present disclosure, the pin strength level must not be too high above the strength level to limit the pin deformability level to achieve a measurable force tensor. It is also understood from this experiment that too high a strength level in conjunction with the pins to keep the pins substantially parallel results in affecting pin deformation, which makes the experimental pin configuration insensitive to shear (X and Y pressure), insensitive to torque in X, Y and Z, and insensitive to squeeze forces in X and Y.

[0194] The number of pins to be mounted / attached / molded to the underside of the elastomeric elastically deformable skin can depend on many factors (as learned from experimentation), for example, recognizing that the number of pins can depend on achieving a predetermined level of sensitivity, durability, etc. for a specific application based on user intent and objectives.

[0195] For example, applied external forces (i.e., full force tensors) applied to the outer impact surface of the elastomeric elastically deformable skin (vertical "Z" pressure, both center and offset, lateral "X" and "Y" shear, and torque in pitch, yaw and roll) can all be sensed by the elastic tactile sensor. The simple, compact tactile sensor design of the present disclosure can be used with robotics, prosthetic applications, computer applications, human-safe collaborative robot safety sensors, machine applications, and vehicle-related applications. In addition, the high-resolution tactile sensors of some embodiments of the present disclosure can also be used in precision control devices (such as, high-density microcomputer products) for high-sensitivity robotic elastically deformable skin sensing.

[0196] In order to process the data generated from the camera's images of the pin motion, finite element modeling (FEM) can be used to process the data of the elastically deformable skin as a metamaterial design. A metamaterial is an object whose internal structure produces a final object whose physical properties (electrical, magnetic, mechanical, thermal properties) are significantly different from the bulk properties of the "real" material it is composed of, i.e., negative ε, μ, almost arbitrary speed of light c, Poisson's ratio ν or thermal conductivity κ. Basically, the FEM model can predict the deformation of the elastomer under different loads. In addition, the elastomer pin lifts the surface contour and converts the surface tilt into an XY motion that is easy to image, i.e., using a low-cost webcam image of the pin end. In addition, the structural design of the elastically deformable skin with few fiducial markers provides simple image thresholding and no inter-frame tracking / ambiguity. In fact, OpenCV can be used to track the motion of the pin end converted into a force map. This minimizes the computational cost and maintains a high frame rate.

[0197] During the experiments, the inverse FEM proved to be computationally expensive. It was realized that the training of the model should use real data. One aspect further learned from the experiments was that using the Euclidean distance in 14-dimensional space for the label case with grasp position, angular pose, shear and torque proved to be beneficial. For example, the calculation time was very fast, i.e. the amount of computational cost proved to be very low, resulting in the ability to process in real time with excellent visualization on a 1GHz cpu (e.g., Raspberry Pi Zero).

[0198] Some of the technologies that embodiments of the present disclosure may use include FEM, 3D printing of micro-mechanical elastomeric components, small (5.5 mm diameter) webcams. Where webcams can be purchased cheaply (i.e., $20 or less). These embodiments may also be used for robotic protective skins, which would provide impact verification.

[0199] Some other benefits of using FEM and 3D printing can enable the design and production of elastically deformable skins with many attributes. For example, some attributes can include providing an elastically deformable skin with a desired amount of sensitivity or damage threshold or a specific amount of force direction. Among them, the asymmetry of the elastically deformable skin, pin height, pin position and mounting configuration enables different force directions to be distinguished. In addition, the low profile of the elastically deformable skin can enable tactile sensors to be used in robots, vehicles and machines at the same time, thereby capturing the correct force tensor for the task being performed. In many cases, it may not be necessary to calculate the actual force tensor, so that a simple comparison of the pin end position with a preset library of correct and incorrect images is sufficient to determine the type of impact or collision, or to interpolate against a known distortion library to generate a good estimate of the actual force tensor on the sensor in real time.

[0200] Other embodiments of the present disclosure address today's industry needs by providing benefits such as low cost, low mass, and low profile simple compact tactile sensors while delivering highly sensitive detection of external forces applied to elastomeric tactile sensors. Some of these benefits allow the compact tactile sensors of the present disclosure to be used in technologies where existing conventional use has proven to be too costly for various technology industries.

[0201] Figure 5A is a schematic diagram illustrating some components of an alternative robotic system according to some embodiments of the present disclosure, the robotic system including a robot having a gripper with an elastically deformable skin and various subsystems. For example, according to some embodiments of the present disclosure, Figure 5A A sensor control computer 570 connected to a robot 582 may be included. The sensor control computer 570 may include a hardware processor 577 connected to a memory 572, the memory 572 including stored software 574 and a pre-learned XY set database including labels and surface force values. The hardware processor 577 may implement or execute stored instructions stored in the memory 572. The processor 577 is connected to a control module 578, which is connected to the robot control computer 580. The robot control computer 580 may communicate back to the control module via 579. The robot control computer 580 is connected to the robot 582, wherein the robot 582 is connected back to the robot control computer 580.

[0202] Still refer to Figure 5A , the robot 582 is connected to the distortion of the external impact surface 585, wherein the physical environment 584 is connected to the distortion of the external impact surface 585. The physical environment 584 provides input including objects. In addition, the distortion of the external impact surface 585 can be an external pressure or force that constitutes a force pattern or net force tensor that causes the pin to move. Wherein at least one camera 569 captures images of the markers and pin positions to obtain video image data. For example, the distortion of the external impact surface 585 can come from the distortion applied to Figure 1A The external pressure, force, etc. of the outer impact surface 9. The external or external pressure, force, etc. can be a force mode, and the embodiments of the present disclosure can determine the external pressure applied to the Figure 1A The amount of measurable sensitivity of each pressure or force of the force pattern of the outer impact surface 9.

[0203] Still refer to Figure 5A Optionally, other data may be obtained from other sensors to the hardware processor 577. For example, other data may include environmental data, historical data, or data related to implementing robot actions.

[0204] Figure 5B5 is a schematic diagram of a computing device 500 that illustrates, by way of non-limiting example, some techniques that may be used to implement methods and systems according to embodiments of the present disclosure. Computing device or apparatus 500 represents various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers.

[0205] The computing device 500 may include a power supply 556, a processor 540, a memory 512, a storage device 558, all connected to a bus 556. In addition, a high-speed interface, a low-speed interface, a high-speed expansion port, and a low-speed connection port may be connected to the bus 556. In addition, the low-speed expansion port is connected to the bus 556. Depending on the specific application, various component configurations that can be installed on a common motherboard are envisioned by way of non-limiting example. In addition, the input interface 508 may be connected to an external receiver 538 and an output interface 509 via the bus 556. The receiver 546 may be connected to an external transmitter 539 and a transmitter 547 via the bus 556. Also connected to the bus 556 may be an external memory 506, an external sensor 504, a machine 502, and an environment. In addition, one or more external input / output devices 541 may be connected to the bus 556. Network interface controller (NIC) 534 may be adapted to connect to network 536 via bus 556 , where data or other information may be presented on a third-party display device, third-party imaging device, and / or third-party printing device external to computer device 500 .

[0206] It is contemplated that the memory 512 can store instructions executable by the computer device 500, historical data, and any data that can be utilized by the methods and systems of the present disclosure. The memory 512 can include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The memory 512 can be one or more volatile memory units, and / or one or more non-volatile memory units. The memory 512 can also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0207] Still refer to Figure 5B, the storage device 558 may be suitable for storing supplementary data and / or software modules used by the computer device 500. For example, the storage device 558 may store historical data and other relevant data as mentioned above with respect to the present disclosure. Additionally or alternatively, the storage device 558 may store historical data similar to the data mentioned above with respect to the present disclosure. The storage device 558 may include a hard drive, an optical drive, a thumb drive, a drive array, or any combination thereof. In addition, the storage device 558 may include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions may be stored in an information carrier. When the instructions are executed by one or more processing devices (e.g., processor 540), one or more methods, such as those described above, are executed.

[0208] The system can be optionally linked to a display interface or user interface (HMI) 557 via a bus 556, which is suitable for connecting the system to a display device 548 and a keyboard 553, where the display device 548 may include a computer monitor, camera, television, projector, mobile device, etc.

[0209] Still referring to 5B, the computer device 500 may include a user input interface 508 adapted to a printer interface (not shown), which may also be connected via a bus 556 and adapted to connect to a printing device (not shown), wherein the printing device may include a liquid inkjet printer, a solid ink printer, a large commercial printer, a thermal printer, a UV printer, or a thermal sublimation printer, etc.

[0210] The high-speed interface manages bandwidth-intensive operations of the computing device 500, while the low-speed interface manages lower bandwidth-intensive operations. This functional allocation is only an example. In some implementations, the high-speed interface can be connected to the memory 512, the user interface (HMI) 557, the keyboard 553 and the display 548 (for example, through a graphics processor or an accelerator), and can accept various expansion cards (not shown) via the bus 556. In an implementation, the low-speed interface is connected to the storage device 558 and the low-speed expansion port via the bus 556. The low-speed expansion port that can include various communication ports (for example, USB, Bluetooth, Ethernet, wireless Ethernet) can be connected to one or more input / output devices 541, and other devices keyboard 553 pointing device (not shown), scanner (not shown), or network devices, such as switches or routers, for example, through a network adapter.

[0211] Still refer to Figure 5B, computing device 500 can be implemented in a number of different forms, as shown. For example, it can be implemented as a standard server, or multiple times in such a server group. In addition, it can be implemented in a personal computer such as a laptop computer. It can also be implemented as part of a rack server system. Alternatively, components from computing device 500 can be combined with other components in a mobile device (not shown). Each such device can contain one or more of a computing device and a mobile computing device, and the entire system can be composed of multiple computing devices that communicate with each other.

[0212] feature

[0213] According to another embodiment of the present disclosure, a method tactile sensor includes: an elastically deformable skin, the elastically deformable skin includes an outer impact surface with attributes and a lower surface with pins, ridges or both, each lower surface pin or ridge includes a mark, and the lower surface is arranged on a flexible spacer from a rigid surface. A camera is positioned to capture an image of the mark. A memory has stored data, the stored data including image data of a pre-learned mark position set with a corresponding pre-learned force pattern. Each pre-learned mark position set corresponds to a pre-learned force pattern. An image processor is operatively connected to the camera and the memory. The image processor is configured to detect the displacement of the mark in the captured image. Based on a distance function, the shifted mark position in the captured image is compared with the pre-learned mark position set to determine the quality of the matching value of each pre-learned mark position set. A user-selected best matching function is applied to the quality of the determined matching value to determine the pre-learned force pattern of the best quality match to calculate the best matching set of the pre-learned mark positions and the corresponding best quality matching pre-learned force pattern. Based on the determined best matching pre-learned force pattern, the force pattern acting on the elastically deformable skin is identified. The identified force pattern is output to initiate an action based on the identified pattern. The following aspects below are considered as variant embodiments of the above embodiments.

[0214] According to aspects of the present disclosure, the attributes include one or a combination of at least one texture, symmetrical protuberances, asymmetrical protuberances, or a pattern of protuberances with or without the at least one texture, depending on the specific application of the user measurable force, and the attributes are constructed and arranged in conjunction with the lower surface pins, ridges, or both to provide a user-preferred measurable sensitivity of the force pattern. At least one benefit of this aspect may be adding a greater amount of sensitivity to the tactile sensor.

[0215] Another aspect may include: depending on the specific application of the user-measurable force, the attribute is constructed and arranged to include at least one area of ​​rough texture, which provides a user-preferred increased and uniform measurable sensitivity of the force mode, including detecting the amount of measurable perpendicular force across the outer impact surface of the elastomeric tactile sensor. Another aspect may be: depending on the specific application of the user-measurable force, the attribute is constructed and arranged to include at least one area of ​​rough texture, which provides a user-preferred increased and uniform measurable sensitivity of the force mode, including detecting the amount of measurable perpendicular force across the outer impact surface of the elastomeric tactile sensor. In addition, another aspect may include: depending on the specific application of the user-measurable force, the attribute is constructed or arranged to include one or a combination of conical protuberances, symmetrical protuberances or asymmetrical protuberances, which provides a user-preferred increased measurable sensitivity of the force mode, including detecting the amount of measurable in-plane force, shear force and torque force across the outer impact surface of the elastomeric tactile sensor. Another aspect includes: depending on the specific application of the user-measurable force, the attribute is constructed and arranged to include at least one area of ​​ridged protuberances, which is positioned perpendicular to the axis of the user-preferred measurable sensitivity to the measurable force or torque of the force mode. One aspect is that depending on the specific application of the user measurable force, the attributes include a customized pattern of one or a combination of rough textures, conical protrusions, symmetrical protrusions, asymmetrical protrusions, ridged protrusions positioned perpendicular to the axis of the preferred measurable sensitivity, which provides the user preferred measurable sensitivity of the force pattern.

[0216] In addition, another aspect is that when a force is applied to the outer impact surface, the outer impact surface elastically deforms, causing the lower surface pins, ridges, or both to undergo a flexural motion that causes the marker to shift. In addition, as an aspect, the distance function used to compare the shifted marker positions to the set of pre-learned marker positions is based on a Euclidean distance function. In addition, one aspect is that wherein the best matching set of pre-learned marker positions is determined using a user-selected matching function, the user-selected matching function comprising: (1) the best matching quality via the distance function; (2) a predetermined weighted average of the best qualities of a user-selected number of matching values ​​via the distance function; or (3) a predetermined weighted sum of the qualities of all matching values.

[0217] In yet another aspect, each pattern of forces includes a set of forces perpendicular to the outer impact surface, torque forces, and shear forces, such that the force pattern acting on the elastically deformable skin includes one or a combination of: vertical pressure Z, center pressure, bias pressure, lateral force X, shear force Y, torque in pitch, roll, or roll, squeeze lateral force X, or squeeze shear force Y. In another aspect, the outer impact surface changes between an undeformed state when no force is applied to the outer impact surface and an elastically deformed state when a force is applied to the outer impact surface, the elastically deformed state elastically deforming the outer impact surface to cause the lower surface pins, ridges, or both to undergo a flexural motion that displaces the marker, and wherein the image processor compares the displacement of the marker in the captured image by evaluating the relative displacement of the lower surface pins, ridges, or both between the undeformed state and the elastically deformed state.

[0218] Another aspect is that the flexible spacer is attached to the lower surface and the rigid surface, and the rigid surface is the outer surface of the robot, vehicle or machine. In addition, one aspect is that the identified force pattern is output to an external controller, and the external controller performs an action based on the identified pattern, so that the action is related to the robot, the vehicle or the machine, and the action includes one or a combination of the following: (A) movement of the robot, the vehicle or the machine; (B) movement of at least one device or element associated with the robot, the vehicle or the machine, such as an action related to the opening or closing of the robot's gripper, an action related to maintaining the robot's position, and an action related to the robot's gripper lifting an object or releasing an object. Wherein, the action is to slow down or stop the movement of the robot, the vehicle or the machine. In addition, the elastically deformable skin and the flexible spacer are an integral structure, and the elastically deformable skin includes perforations, grooves or ventilation holes.

[0219] According to another embodiment of the present disclosure, a method for tactile sensing includes: providing an elastically deformable skin, the elastically deformable skin including an outer impact surface having an attribute and a lower surface having pins, ridges, or both. Each lower surface pin or ridge includes at least one mark, and the lower surface is arranged on a flexible spacer, and the flexible spacer is attached to a rigid surface of a device. A camera arranged to capture an image of the mark is provided and an image is captured from the camera. The displacement of the mark in the captured image is detected to obtain a shifted mark position. A memory having stored data is accessed, the stored data including image data of a pre-learned mark position set having a pre-learned force pattern, each pre-learned mark position set corresponding to a pre-learned force pattern. Based on a distance function, the shifted mark position in the captured image is compared with the pre-learned mark position set of the image data to obtain the quality of the matching value of each pre-learned mark position set, and a user-selected best matching function is applied to the quality of the matching value to determine the pre-learned force pattern of the best quality match. Based on the determined best matching pre-learned force pattern, a force pattern acting on the elastically deformable skin is identified. Wherein, the force pattern acting on the elastically deformable skin includes one or a combination of the following: vertical pressure Z, center pressure, bias pressure, lateral force X, shear force Y, torque in pitch, roll or roll, squeeze lateral force X, or squeeze shear force Y. The identified force pattern is output to initiate an action based on the identified pattern. Wherein the following aspects are considered to constitute variant embodiments of the above embodiments.

[0220] According to an aspect of the present disclosure, using the best matching function selected by the user to determine the pre-learned force pattern of the best quality match includes: applying the best matching function selected by the user to the quality of the matching value to determine the best matching set of the pre-learned mark positions and the corresponding pre-learned force pattern of the best quality match, and wherein the device is a machine including a robot, so that the control movement of the robot is initiated in response to the identified force pattern acting on the elastically deformable skin.

[0221] According to another embodiment of the present disclosure, a method is used to control a system of a robot, which includes an elastically deformable skin, the elastically deformable skin including an outer impact surface having an attribute and a lower surface having pins, ridges or both. Each lower surface pin or ridge includes at least one mark, and the lower surface is arranged on a flexible spacer, and the flexible spacer is attached to the rigid surface of the device. The camera is configured to capture an image of the mark. The memory has stored data, and the stored data includes image data of a pre-learned mark position set with a corresponding pre-learned force pattern, each pre-learned mark position set corresponding to a pre-learned force pattern. An image processor is operatively connected to the camera and the memory. The image processor is configured to: detect the displacement of the mark in the captured image. Based on a distance function, the shifted mark position in the captured image is compared with the pre-learned mark position set to determine the quality of the matching value of each pre-learned mark position set. The pre-learned force pattern of the best quality match is determined by applying the best matching function selected by the user to the quality of the determined matching value to calculate the best matching set of the pre-learned mark position and the corresponding pre-learned force pattern of the best quality match. Based on the determined best matching pre-learned force pattern, a force pattern acting on the elastically deformable skin is identified. The identified force pattern is output to initiate an action based on the identified pattern. The following aspects are considered to constitute variant embodiments of the above embodiments.

[0222] As an aspect, the method comprises: using a user-selected best match function to determine a pre-learned force pattern of best quality match, comprising: applying the user-selected best match function to the quality of the match value to determine a best match set of pre-learned mark positions and a corresponding pre-learned force pattern of best quality match, and wherein the device is a machine including a robot, so that a controlled movement of the robot is initiated in response to an identified force pattern acting on the elastically deformable skin.

[0223] The following aspects can be combined into any of the above devices, methods and systems. For example, one aspect can be that the camera includes an illumination source that illuminates the fiducial markers so as to capture images of the fiducial markers. Another aspect is that the camera is replaced with multiple cameras spaced apart from each other at the same or different viewing angles of the fiducial markers, so that the multiple cameras can capture images of all fiducial markers at a static position before applying a force to the outer impact surface and at a final position after the force is applied to the outer impact surface, so as to capture images to determine the displacement of each marker.

[0224] Another aspect is that the mark is a fiducial mark or a fiducial mark whose material includes silicone rubber, polyurethane, plastisol, thermoplastic elastomer, natural rubber, polyisoprene, polyvinyl chloride or a mixture thereof. Wherein the mark includes a hardness between 5 and 90 Shore A hardness. In addition, one aspect may be that the relative displacement of each mark in the mark set is determined based on the image coordinates measured at the starting position of the captured image to the final position of the captured image. It is possible that one aspect may be that each pre-learned mark position set includes data for each fiducial mark based on the image coordinates measured in a static state without applying a force to the top surface and data for each fiducial mark based on the image coordinates measured in a static state after applying a force to the top surface.

[0225] Some conventional machine vision and ultrasonic proximity sensor approaches can help prevent unnecessary collisions, but can have high variability in accurately detecting objects prior to collision. Conventional machine vision systems can be erroneous if the camera is obscured or if the lighting is poor. These conventional techniques are therefore less than ideal in such applications where safety and reliability are of high concern. However, the elastically deformable skin of the present disclosure overcomes these conventional tactile sensor problems (i.e., surfaces are out of view, or are shadowed, have poor contrast, or are poorly illuminated) by not being affected by these conventional problems. Additionally, with respect to conventional remote-controlled robots, where the operator's mental concentration fatigues over time, the present disclosure of the elastically deformable skin overcomes these problems by eliminating the need for a remote-controlled robot and a human operator.

[0226] definition

[0227] According to aspects of the present disclosure, and based on experimentation, the following definitions have been established, and are certainly not complete definitions of each phrase or term. The definitions provided therein are provided merely as examples based on learning from the experimentation, where other interpretations, definitions, and other aspects may be involved. However, such definitions have been provided, at least for only a basic preview of the phrases or terms presented.

[0228] Tensor: In mathematics, a tensor is an algebraic object associated with a vector space and its dual space, and can be defined in several different ways, usually a scalar, a tangent vector at a point, a cotangent vector at a point (the dual vector), or a multilinear mapping from a vector space to a resulting vector space. Euclidean vectors and scalars (often used in physics and engineering applications unrelated to general relativity) are the simplest tensors. Although tensors are defined independently of any basis, physics literature often refers to them by their components in a basis associated with a particular coordinate system.

[0229] Finite Element Modeling (FEM): According to some embodiments of the present disclosure, FEM can subdivide the entire domain into simpler parts, which has several advantages: (A) accurately represents complex geometries; (B) includes different material properties; (C) easily represents the overall solution; and (D) captures local effects.

[0230] The work in this method may involve (1) partitioning the domain of the problem into a set of subdomains, each of which is represented by a set of element equations of the original problem, and then (2) systematically recombining all sets of element equations into a global system of equations for the final calculation. The global system of equations has known solution techniques and can be calculated starting from the initial values ​​of the original problem to obtain a numerical answer.

[0231] Still involving FEM, in the first step above, the element equations are simple equations that locally approximate the original complex equation to be studied, where the original equation is usually a partial differential equation (PDE). In order to explain the approximation in this process, FEM is usually introduced as a special case of the Galerkin method. In mathematical terms, this process is to construct an integral of the inner product of the residual and the weight function and set the integral to zero. In simple terms, the process minimizes the approximation error by fitting a trail function to the PDE. The residual is the error caused by the trail function, and the weight function is a polynomial approximation function of the projected residual. This process eliminates all spatial derivatives of the PDE, thereby locally approximating the PDE with a set of algebraic equations for steady-state problems and a set of ordinary differential equations for transient problems.

[0232] These systems of equations are elementary equations. They are linear if the underlying PDE is linear, and vice versa. The algebraic systems of equations arising in steady-state problems are solved using numerical linear algebra methods, while the systems of ordinary differential equations arising in transient problems are solved by numerical integration using standard techniques such as Euler's method or Runge-Kutta method.

[0233] Still referring to the FEM, in step (2) above, a global system of equations is generated from the element equations by transforming the coordinates from the local nodes of the subdomain to the global nodes of the domain. This spatial transformation includes the application of appropriate orientation adjustments relative to the reference coordinate system. This process is usually implemented by FEM software using the coordinate data generated from the subdomain.

[0234] experiment

[0235] During the experiments, many tests were conducted to determine how to design and construct the elastically deformable skin based on the level of sensitivity and durability. Some experimental tests were conducted using arm motor torque sensors, post-contact verification sensors, wrist force sensors, motor position and torque sensors, and contact strain gauges.

[0236] For example, the arm motor torque sensor exhibits inertial forces, frictional forces, and gravity on the robot. At least one aspect learned is that the inertial forces, frictional forces, and gravity on the robot generally completely dominate the forces actually experienced during proper mechanical assembly. It is learned from the post-contact verification experiments that after the robot performs an operation, the robot is operationally moved to a machine vision station to confirm the robot movement. The machine vision station has a machine vision system that verifies the movement, wherein if any corrections are required, they are performed after the movement is completed. However, we recognize that detecting movement after the robot has moved results in damage to the robot while creating an unsafe environment for other machines, assets, and humans. Based on the experimental results, this method and similar methods fail to meet the level of cost, safety, and acceptable technical solutions for the objectives of the present disclosure.

[0237] From experiments with wrist force sensors, it is known that these wrist force sensors appear to be very crude. For example, the wrist force sensors do not seem to be able to capture offset movements, and they also fail to capture motion failures before noticing a deviation from prior experience, and for many of the movement or assembly tasks tested, the deviation is less than the effects of inertia, gravity, and sensor noise on the wrist force sensor. Again, as indicated by other experimental tests, these approaches or similar approaches do not meet the level of cost, safety, and acceptable technical solutions that are the goals of the present disclosure.

[0238] From experiments with kinematic strain gauge sensors, it is known that these sensors integrate the strain gauges into the device and therefore only record the device force, which seems equivalent to sensing only the "Z" pressure. However, these sensors do not provide for sensing other forces such as center pressure, bias pressure, lateral force X, shear force Y, torque in pitch, yaw or roll, squeeze lateral force X or squeeze shear force Y.

[0239] Another sensor that was utilized experimentally included the GelSight sensor. This sensor uses a thick block of clear gel with a flexible opaque coating on one side. The coated side of the block is pressed against a surface and the distorted coating is viewed vertically while being illuminated under glancing illumination at several different angles. It is understood that the sensor is able to detect textures (burrs) that protrude upward on the surface. However, this sensor does not seem to be useful for voids or pinholes, making the sensor have a very low level of damage force threshold. Another disadvantage that many have found is that this sensor requires about 100 mm of Z-axis margin for the camera to operate. Since the margin of the camera is so large, this means that the sensor certainly cannot have a low profile configuration and seems to be better suited for robotic detection of textured surfaces rather than any type of low profile application, such as sensors related to elastically deformable skins. Additionally, the low level of force damage threshold and surface wear make this sensor unsuitable for robotic assembly.

[0240] Some aspects learned from experimental testing of different types of sensors are that robots are used for transportation and assembly, but correct transportation and assembly depend on the robot accurately operating the movement in the same position and orientation at all times. Incorrect movement of the robot results in costs for human safety, costs related to damage to the robot, or costs for damage to other machines or objects, etc. As another example, even if the robot movement is correct, if an object or human accidentally enters the robot's path, or some unplanned event occurs that causes an object or human to enter the robot's path, it is learned from the experiments that despite the unplanned event, the robot will still attempt to force the movement, resulting in injury to the human, damage to the robot, or damage to any object in the robot's path. In view of the above experimental conclusions, the experimental sensors cannot be modified or further developed to meet the sensitivity levels, durability, manufacturing costs, and many other factors expected by the present disclosure, and therefore, no further analysis or testing was performed.

[0241] Based on extensive experimentation and desired sensitivity levels, durability, manufacturing costs, and other factors, many insights have been gained for designing and constructing embodiments of the present disclosure. Some of these insights for some embodiments include extending the pins on the lower surface or underside of the elastically deformable skin in a manner that the pins extend away from the lower surface. This configuration includes having to design the following pins that meet the strength level and also meet the deformability level so that they can be deformed to record the amount of deformation of the pins caused by the external force tensor. Specifically, some configurations of the pins are specifically designed so that the pins are not parallel to other pins. For example, it was found from experiments via optical flow analysis that the pins were arranged in parallel in the experimental elastically deformable skin. The pins exhibit a certain level of strength, that is, the strength level that affects the deformation of the pins when the applied external force is applied. This experimental elastically deformable skin obtains an acceptable measurable level of vertical force. However, based on the image motion obtained with the vertical (z-axis) camera at the end of the pin, this same experimental elastically deformable skin configuration failed to provide an acceptable level of measurable shear and torque sensing / force. It is understood from this experimental configuration that, at least in view of the desired level of measurable sensitivity goals of the present disclosure, the pin strength level must not be too high above the strength level to limit the pin deformability level to achieve a measurable force pattern or net force tensor. It is also understood from this experiment that too high a strength level in conjunction with the pins to keep the pins substantially parallel results in affecting pin deformation, which makes the experimental pin configuration insensitive to shear (X and Y pressure), insensitive to torque in X, Y and Z, and insensitive to squeeze forces in X and Y.

[0242] Implementation

[0243] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of exemplary embodiments will provide a sufficient description for implementing one or more exemplary embodiments for those skilled in the art. Various modifications that can be made in the function and arrangement of elements are contemplated without departing from the spirit and scope of the subject matter disclosed in the appended claims.

[0244] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be appreciated by those skilled in the art that the embodiments may be practiced without these specific details. For example, the systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary details. In other cases, known processes, structures, and techniques may be shown without non-essential details to avoid obscuring the embodiments. In addition, similar reference numerals and signs in the various drawings refer to similar elements.

[0245] Moreover, each embodiment can be described as a process depicted as a flow chart, flow diagram, data flow diagram, structure diagram or block diagram. Although a flow chart can describe an operation as a sequential process, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. When the operation of the process is completed, the process can terminate, but there can be other steps that are not discussed or not included in the figure. In addition, not all operations in any particular described process can occur in all embodiments. A process can correspond to a method, function, process, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the function can correspond to the function returning to the function or main function that calls the function.

[0246] In addition, embodiments of the disclosed subject matter may be implemented at least in part manually or automatically. Manual or automatic implementations may be performed or at least assisted in implementation using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks may be stored in a machine-readable medium. A processor may perform the necessary tasks.

[0247] In addition, the embodiments of the present disclosure and the functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, and include the structures disclosed in this specification and their equivalents, or a combination of one or more of them. In addition, some other embodiments of the present disclosure can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Further, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to an appropriate receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0248] According to an embodiment of the present disclosure, the term "data processing device" can cover all types of devices, apparatuses and machines for processing data, including, by way of example, a programmable processor, a computer or a multiprocessor or a computer. The device can include dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device can also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0249] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) can be written in any form of programming language (including compiled or interpreted languages) or declarative or procedural languages, and can be deployed in any form including as a standalone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A computer program may, but not necessarily, correspond to a file in a file system. A program can be stored in a portion of a file holding other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordination files (e.g., files storing one or more modules, subroutines or partial codes). A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed at multiple sites and interconnected by a communication network. Computers suitable for executing computer programs include, as examples, central processing units that can be based on general or special microprocessors or both, or any other type of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or implementing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic, magneto-optical disks, or optical disks) for storing data, or be operationally connected to receive data from a mass storage device or to transmit data to a mass storage device or both. However, a computer does not necessarily have such a device. In addition, a computer can be embedded in another device, such as, to name a few, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as, a universal serial bus (USB) flash drive.

[0250] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) that displays information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending documents to or receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0251] Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0252] The computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between client and server is generated by computer programs running on their respective computers and having a client-server relationship to each other.

[0253] Although the present disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications may be made within the spirit and scope of the present disclosure. Therefore, the aspects of the appended claims cover all such variations and modifications that fall within the true spirit and scope of the present disclosure.

Claims

1. A tactile sensor, comprising: an elastically deformable skin including an outer impact surface having properties and a lower surface having pins, ridges, or both, each lower surface pin or ridge including at least one marking, and the lower surface being disposed on a flexible spacer attached to a rigid surface of the device; a camera positioned to capture an image of the marker; a memory having stored data, the stored data comprising image data of a set of pre-learned mark positions having a corresponding pre-learned force pattern, each set of pre-learned mark positions corresponding to a pre-learned force pattern; an image processor operatively connected to the camera and the memory, the image processor being configured to: detecting a displacement of the marker in the captured image; comparing the shifted mark positions in the captured image with the pre-learned mark position sets based on a distance function to determine the quality of the matching value of each pre-learned mark position set; Determine a pre-learned force pattern of best quality match using a user-selected best match function and applying it to the determined quality of the match value to calculate a best match set of pre-learned marker positions and a corresponding best quality match pre-learned force pattern; identifying a force pattern acting on the elastically deformable skin based on the determined best matching pre-learned force pattern; as well as The recognized force pattern is output to initiate an action based on the recognized pattern.

2. The tactile sensor according to claim 1, wherein: The image processor converts the position of each mark in the captured image by recognizing measurement image position coordinates in an image among the captured images using an image measurement function.

3. The tactile sensor according to claim 2, wherein: The detected position of each marker is summed according to the Euclidean distance between the detected marker position and the pre-learned marker position, thereby forming a matching distance between the captured image of the marker and the pre-learned image of the marker.

4. The tactile sensor according to claim 1, wherein: Depending on the specific application of the user-measurable force, the attributes include one or a combination of: at least one texture, symmetrical protrusions, asymmetrical protrusions, and the attributes are constructed and arranged in combination with the lower surface pins, ridges, or both to provide a user-measurable sensitivity of the force pattern.

5. The tactile sensor according to claim 1, wherein: The attributes include a pattern of protuberances with or without the at least one texture, depending on the specific application of the user-measurable force, and are constructed and arranged in conjunction with the lower surface pins, ridges, or both to provide a user-measurable sensitivity of the force pattern.

6. The tactile sensor according to claim 1, wherein: Depending on the specific application of the user-measurable force, the attributes are constructed and arranged to include at least one area of ​​a rough texture, a smooth texture, or both that provides increased and uniform user-measurable sensitivity of the force pattern, including detecting an amount of measurable perpendicular force across the tactile sensor against the outer impact surface.

7. The tactile sensor according to claim 1, wherein: Depending on the specific application of the user-measurable forces, the attributes are constructed and arranged to include one or a combination of conical protrusions, symmetrical protrusions, or asymmetrical protrusions that provide increased user-measurable sensitivity of the force pattern, including detecting the amount of measurable in-plane forces, shear forces, and torque forces across the outer impact surface of the tactile sensor.

8. The tactile sensor according to claim 1, wherein: Depending on the particular application of the user-measurable force, the attribute is constructed and arranged to include at least one region of a ridged protuberance positioned perpendicular to an axis of user-measurable sensitivity to a measurable force or torque of the force pattern.

9. The tactile sensor according to claim 1, wherein: When a force is applied to the outer impact surface, the outer impact surface elastically deforms causing the lower surface pins, ridges, or both to undergo a flexing motion that displaces the indicia.

10. The tactile sensor according to claim 1, wherein: The best matching set of pre-learned marker locations is determined using a user-selected matching function comprising: (1) the best matching quality via the distance function; (2) a predetermined weighted average of the best qualities of a user-selected number of matching values ​​via the distance function; or (3) a predetermined weighted sum of the qualities of all matching values, wherein the distance function used to compare the detected marker locations to the pre-learned marker location set is based on a Euclidean distance function.

11. The tactile sensor according to claim 1, wherein: The force pattern acting on the elastically deformable element includes one or a combination of the following: vertical pressure Z, center pressure, bias pressure, lateral shear force X, shear force Y, torque in pitch, roll or roll, squeezing lateral force X, deploying lateral force X, deploying lateral force Y or squeezing shear force Y.

12. The tactile sensor according to claim 1, wherein: The outer impact surface changes between an undeformed state when no force is applied to the outer impact surface and an elastically deformed state when a force is applied to the outer impact surface, wherein the elastically deformed state elastically deforms the outer impact surface, causing the lower surface pins, ridges or both to undergo a flexural motion that displaces the mark, and wherein the image processor compares the displacement of the mark in the captured image by evaluating the relative displacement of the lower surface pins, ridges or both between the undeformed state and the elastically deformed state.

13. The tactile sensor according to claim 1, wherein: The flexible spacer is attached to the lower surface and the rigid surface, and the rigid surface is an exterior surface of a robot, vehicle or machine.

14. The tactile sensor according to claim 13, wherein: The identified force pattern is output to an external controller, and the external controller performs an action based on the identified pattern, so that the action is related to the robot, the vehicle or the machine, and the action includes one or a combination of the following: (A) movement of the robot, the vehicle or the machine; (B) movement of at least one device or element associated with the robot, the vehicle or the machine, such as an action related to opening or closing the robot's gripper, an action related to maintaining the robot's position, and an action related to the robot's gripper lifting or releasing an object.

15. The tactile sensor according to claim 14, wherein: The action is to slow down or stop the movement of the robot, the vehicle or the machine.

16. The tactile sensor according to claim 1, wherein The elastically deformable skin and the flexible spacer are an integral structure, and the elastically deformable skin includes perforations, slots or ventilation-like holes.

17. A method for tactile sensing, the method comprising the steps of: providing an elastically deformable skin including an outer impact surface having properties and a lower surface having pins, ridges, or both, each lower surface pin or ridge including at least one marking, and the lower surface being disposed on a flexible spacer attached to a rigid surface of the device; providing a camera arranged to capture an image of the marking; capturing an image from the camera; detecting a displacement of the marker in the captured image to obtain a shifted marker position; accessing a memory having stored data, the stored data including image data having a set of pre-learned marker positions of a pre-learned force pattern, each set of pre-learned marker positions corresponding to a pre-learned force pattern; Based on the distance function, comparing the shifted mark positions in the captured image with the pre-learned mark position set of the image data to obtain the quality of the matching value of each pre-learned mark position set, and applying the best matching function selected by the user to the quality of the matching value to determine the pre-learned force mode of the best quality match; Based on the determined best matching pre-learned force pattern, identifying a force pattern acting on the elastically deformable skin, wherein the force pattern acting on the elastically deformable skin comprises one or a combination of the following: vertical pressure Z, center pressure, bias pressure, lateral shear force X, shear force Y, torque in pitch, roll or roll, squeeze lateral force X, deployment lateral force X, deployment lateral force Y or squeeze shear force Y; as well as The recognized force pattern is output to initiate an action based on the recognized pattern.

18. The method according to claim 17, wherein: Using the user-selected best match function to determine the pre-learned force pattern of best quality match includes: applying the user-selected best match function to the quality of the match value to determine the best match set of pre-learned mark positions and the corresponding pre-learned force pattern of best quality match, and wherein the device is a machine including a robot so that the control movement of the robot is initiated in response to the identified force pattern acting on the elastically deformable skin.

19. A system for controlling a robot, the system comprising: an elastically deformable skin including an outer impact surface having properties and a lower surface having pins, ridges, or both, each lower surface pin or ridge including at least one marking, and the lower surface being disposed on a flexible spacer attached to a rigid surface of the device; a camera configured to capture an image of the marker; a memory having stored data, the stored data comprising image data of a set of pre-learned mark positions having a corresponding pre-learned force pattern, each set of pre-learned mark positions corresponding to a pre-learned force pattern; an image processor operatively connected to the camera and the memory, the image processor being configured to: detecting a displacement of the marker in the captured image; comparing the shifted marker positions in the captured image with the pre-learned marker position sets based on a distance function to determine a quality of a match value of each pre-learned marker position set; Determine a pre-learned force pattern of best quality match using a user-selected best match function and applying it to the determined quality of the match value to calculate a best match set of pre-learned marker positions and a corresponding pre-learned force pattern of best quality match; identifying a force pattern acting on the elastically deformable skin based on the determined best matching pre-learned force pattern; as well as The identified force pattern is output to initiate an action based on the identified pattern.

20. The system of claim 19, wherein: Using a user-selected best match function to determine a pre-learned force pattern of best quality match includes: applying the user-selected best match function to the quality of the match value to determine a best match set of pre-learned mark positions and a corresponding pre-learned force pattern of best quality match, and wherein the device is a machine including a robot so that a controlled movement of the robot is initiated in response to the identified force pattern acting on the elastically deformable skin.

21. An elastomeric tactile sensor for a mobile device, the elastomeric tactile sensor comprising: an elastically deformable skin including an outer impact surface having properties and a lower surface having pins, ridges, or both, each lower surface pin or ridge including an indicia, and the lower surface being disposed on a flexible spacer attached to a rigid surface of the mobile device; a camera positioned to capture an image of the marker; a memory having stored data, the stored data comprising image data of a set of pre-learned mark positions having a corresponding pre-learned force pattern, each set of pre-learned mark positions corresponding to a pre-learned force pattern; an image processor operatively connected to the camera and the memory, the image processor being configured to: detecting a displacement of the marker in the captured image; comparing the shifted mark positions in the captured image with the pre-learned mark position sets based on a distance function to determine the quality of the matching value of each pre-learned mark position set; Determine a pre-learned force pattern of best quality match using a user-selected best match function and applying it to the determined quality of the match value to calculate a best match set of pre-learned marker positions and a corresponding pre-learned force pattern of best quality match; identifying a force pattern acting on the elastically deformable skin based on the determined best matching pre-learned force pattern; as well as The identified force pattern is output to a controller to initiate an action associated with the mobile device based on the identified pattern.

Images