Fluid tactile sensor

CN119053839APending Publication Date: 2024-11-29GELSIGHT INC

Patent Information

Application Number
CN202380034755.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-02-18
Filing Date
2023-02-18
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

虽然这样的传感器为高分辨率表面数据的基于接触的采集提供了有用的技术,但是仍然需要使用围阻流体介质的触觉传感器的改进的表面形貌测量系统

Benefits of technology

[0015]The system may include a robotic handler coupled to a fluid tactile sensor. The fluid tactile sensor may be removably and replaceably coupled to the housing of the robotic handler. The system may include a processor configured with computer-executable code to perform soft robotic gripping functions using the fluid tactile sensor. The system may include a robotic handler coupled to a fluid tactile sensor and a processor configured by computer executable code to perform the steps of: pressurizing a reservoir with an imaging medium to increasing the malleability of the fluid tactile sensor; positioning the fluid tactile sensor on the object using a robotic handler; and removing a portion of the imaging medium by depressurizing the reservoir and increasing the ductility of the fluid tactile sensor on a portion of the diaphragm engaged with the object Stiffness to hold objects. The processor may be further configured to acquire one or more images of the object through the window at one or more wavelength ranges, wherein the plurality of beads are optically transparent and refractive index matched to the imaging medium. The system may include a processor configured by computer-executable code stored in memory to acquire images from the imaging system and calculate quantitative surface topography of a target surface contacting the membrane. The imaging medium may include a gas. The imaging medium may include liquid. The illumination source may include at least one of a laser and a light emitting diode. The imaging system may use two or more imaging modalities, including at least one of photometric stereo imaging, multi-view stereo imaging, structured light imaging, and focus stacking. The system may include a pressure sensor coupled to the reservoir and configured to measure the pressure of the imaging medium within the reservoir. The system may include a processor configured to control the pump in response to signals received from the pressure sensor. The membrane may have at least one of an optically reflective surface and a patterned surface. The system may include a display that presents a visualization of at least one of a pressure field, a contact force field, and a surface topology acquired by an imaging system of an object contacting the diaphragm. The system may include a machine learning model configured to identify objects contacted by the fluid tactile sensor based on data collected from the imaging system. The window may include at least one of glass, polycarbonate, acrylic, polystyrene, polyurethane, or optically clear epoxy. The window may have a first refractive index that matches a second refractive index of the imaging medium within the reservoir. The membrane may be formed from an elastomeric polymer. The system may include an optical pattern on a portion of the membrane facing the window.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119053839A_ABST
    Figure CN119053839A_ABST
Patent Text Reader

Abstract

The topography measurement system may include a tactile sensor that uses an enclosed fluid as an imaging medium.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 311,568, filed on February 18, 2022, the entire contents of which are hereby incorporated by reference. Technical Field

[0003] The present disclosure relates generally to tactile sensing systems. Background Art

[0004] A variety of contact-based sensors are known in the art and are described, for example, in U.S. Pat. No. 8,411,140, ​​issued on April 2, 2013, U.S. Pat. No. 9,127,938, issued on September 8, 2015, U.S. Pat. No. 10,965,854, issued on March 30, 2021, and PCT Application No. US2022 / 046129P. The entire contents of each of the foregoing applications are hereby incorporated by reference. While such sensors provide useful techniques for contact-based acquisition of high-resolution surface data, there remains a need for improved surface topography measurement systems using tactile sensors that contain fluid media. Summary of the invention

[0005] A topography measurement system includes a tactile sensor using a contained fluid as an imaging medium.

[0006] In one aspect, the system described herein includes a fluid tactile sensor, the fluid tactile sensor comprising: a substrate, the substrate being formed of a rigid material and the substrate including a window of optically transparent material; a diaphragm, the diaphragm being formed of a flexible elastic material sheet and the diaphragm having a surface; a reservoir, the reservoir having a volume at least partially enclosed within the substrate and the diaphragm; and an imaging medium within the reservoir, the imaging medium comprising an optically transparent fluid; and an imaging system, comprising: an illumination source, when the diaphragm is positioned for use within the imaging volume of the imaging system, directed toward the surface of the diaphragm through the window of the substrate; and an imaging device, positioned to capture an image of the surface of the diaphragm through the window of the substrate when the diaphragm is placed for use within the imaging volume of the imaging system.

[0007] The system may include a processor configured by computer executable code stored in a memory to acquire images from an imaging system and calculate quantitative surface topography of a target surface contacting the diaphragm. The illumination source may include a structured light source. The structured light source may create a three-dimensional illumination pattern within an imaging volume of the reservoir. The system may include a box containing a fluid tactile sensor, the box being removable from the imaging system and replaceable to the imaging system. The system may include one or more containment walls that mechanically couple the diaphragm to a substrate. The window may include at least one of glass, polycarbonate, acrylic, polystyrene, polyurethane, or an optically transparent epoxy. The window may have a first refractive index that matches a second refractive index of an imaging medium within the reservoir. The imaging medium may include at least one of a gas and a liquid. The diaphragm may be formed of an elastic polymer. The system may include an optical pattern on the diaphragm visible through the window to an imaging device of the imaging system. The illumination source may include at least one of a laser and a light emitting diode. The imaging system may use two or more imaging modalities, including at least one of photometric stereo imaging, multi-view stereo imaging, structured light imaging, and focal stacking. The system may include a pressure sensor coupled to the reservoir and configured to measure a pressure of the imaging medium within the reservoir.

[0008] In another aspect, a tactile sensor disclosed herein includes a substrate including a window of a rigid optically transparent material; a diaphragm formed of a flexible material coupled to the substrate to form a reservoir visible through the window of the substrate; and an optically transparent fluid in the reservoir.

[0009] The optically transparent fluid may have a first refractive index that matches a second refractive index of the window to facilitate imaging of the membrane through the window. The membrane may have an optically reflective surface. The membrane may have a patterned surface. The membrane may have an outer surface facing away from the window, the outer surface including a friction reducing coating. The window may include one or more light shaping features for at least one of filtering, polarizing, focusing, or diffusing light passing through the window.

[0010] In another aspect, a system is disclosed herein, comprising: a fluid tactile sensor, comprising: a substrate, the substrate being formed of a rigid material and the substrate including a window of optically transparent material; a diaphragm, the diaphragm being formed of a sheet of flexible elastic material and the diaphragm having a surface; a reservoir, the reservoir having a volume enclosed within the substrate and the diaphragm; and an imaging medium within the reservoir, the imaging medium comprising an optically transparent fluid; a fluid management system, comprising: a supply of the imaging medium, and a pump coupled in a fluid path between the supply of the imaging medium to the reservoir, the pump being configured to controllably deliver the imaging medium from the supply to the reservoir; and an imaging system, comprising: an illumination source, directed toward the surface of the diaphragm through the window of the substrate when the diaphragm is positioned for use within the imaging volume of the imaging system, and an imaging device, capturing an image of the surface of the diaphragm through the window of the substrate when the diaphragm is positioned for use within the imaging volume of the imaging system.

[0011] The system may include one or more sensors for measuring the pressure of the imaging medium in the reservoir. The system may include a controller configured to operate a pump to transfer the imaging medium between the supply and the reservoir in response to signals from the one or more sensors. The controller may be configured to operate the pump to maintain the pressure at a predetermined target pressure. The volume of the imaging medium may be kept constant in the reservoir, and wherein the pressure may be measured using one or more sensors and stored with one or more images captured by the imaging system for use in processing the one or more images. The fluid tactile sensor may include one or more containment walls that couple the substrate to the diaphragm to form the reservoir. The system may include one or more illumination sources positioned in the reservoir and directed toward the diaphragm. The window may have a first refractive index that matches a second refractive index of the imaging medium. The imaging device may include one or more cameras. The illumination source may include at least one of a laser and a light emitting diode. The imaging medium may include at least one of a gas and a liquid. The window may include at least one of glass, polycarbonate, acrylic, polystyrene, polyurethane, or an optically transparent epoxy. The system may include one or more temperature sensors for monitoring the temperature of the imaging medium. The system may include a temperature controller operable by the controller to control the temperature of the imaging medium in response to signals from one or more temperature sensors. The diaphragm may have one or more optical properties that vary with the temperature of the diaphragm. The diaphragm may have one or more optical properties that vary with deformation of the diaphragm. The diaphragm may be configured to sense one or more sensing parameters at a target surface contacted by the diaphragm. The imaging system may use two or more imaging modalities, including at least one of photometric stereo imaging, multi-view stereo imaging, structured light imaging, and focal stacking. The diaphragm may have an optically reflective surface. The diaphragm may be optically clear at one or more wavelength ranges captured by the imaging system.

[0012] In another aspect, the system disclosed herein includes a fluid tactile sensor, comprising: a substrate formed of a rigid material and including a window of optically transparent material; a diaphragm formed of a sheet of flexible elastic material and having a surface; a reservoir having a volume enclosed within the substrate and the diaphragm; and an imaging medium within the reservoir, the imaging medium comprising an optically transparent fluid; a robotic handler coupled to the fluid tactile sensor; and an imaging system comprising: an illumination source directed toward the surface of the diaphragm through the window of the substrate when the diaphragm is positioned for use within the imaging volume of the imaging system, and an imaging device that captures an image of the surface of the diaphragm through the window of the substrate when the diaphragm is positioned for use within the imaging volume of the imaging system.

[0013] The fluid tactile sensor may be removably and replaceably coupled to a robotic handler. The robotic handler may be configured to automatically remove the fluid tactile sensor and replace the fluid tactile sensor with a replacement sensor. The system may include a display that presents a visualization of at least one of a pressure field, a contact force field, and a surface topology acquired by an imaging system. The system may include a machine learning model that is configured to identify an object contacted by the fluid tactile sensor based on data acquired from the imaging system. The system may include a machine learning model that is configured to automatically select an action of the robotic handler to be performed on a workpiece contacted by the fluid tactile sensor. The system may include a plurality of fluid tactile sensors arranged on a base and coupled to the robotic handler. The base may include a rigid substrate. The base may include a flexible substrate. The base may include an active substrate having a controllable shape. Each of the plurality of fluid tactile sensors may have a controllable pressure. The plurality of fluid tactile sensors are arranged in a two-dimensional array for localized measurement across a two-dimensional target surface. The base may include one or more articulated joints. The one or more articulated joints may include at least one passive articulated joint. The one or more articulation joints may include at least one active articulation joint having a computer controllable position or orientation. The system may include a processor configured by computer executable code stored in a memory to acquire images from an imaging system and calculate quantitative surface topography of a target surface contacting the diaphragm. The system may include a processor configured by computer executable code stored in a memory to acquire pressure data from a pressure sensor coupled to a reservoir, and in response to the pressure data, provide a control signal to a pump to transfer fluid between the reservoir and a supply of imaging medium. The system may include a processor configured by computer executable code stored in a memory to acquire temperature data from a temperature sensor coupled to the reservoir, and in response to the temperature data, provide a control signal to a thermal controller to heat or cool the imaging medium. The imaging system may use two or more imaging modalities, including at least one of photometric stereo imaging, multi-view stereo imaging, structured light imaging, and focal stacking. The window may have a first refractive index that matches a second refractive index of the imaging medium.

[0014] In another aspect, a system is disclosed herein, comprising: a fluid tactile sensor, the fluid tactile sensor comprising: a substrate, the substrate being formed of a rigid material and comprising a window of optically transparent material; a diaphragm, the diaphragm being formed of a sheet of flexible elastic material and having a surface; a reservoir, the reservoir having a volume enclosed within the substrate and the diaphragm; an imaging medium within the reservoir, the imaging medium comprising an optically transparent fluid, and a plurality of beads within the reservoir, the beads being formed of an optically transparent material, the optically transparent material being in contact with the imaging medium within the reservoir; a supply of imaging medium; a pump; wherein the pump is coupled in fluid communication between the supply and the reservoir, and wherein the pump is configured to transfer the imaging medium between the reservoir and the supply; and an imaging system comprising: an illumination source directed through a window in the substrate toward a surface of the diaphragm when the diaphragm is positioned for use within an imaging volume of the imaging system, and an imaging device that captures an image of the surface of the diaphragm through the window in the substrate when the diaphragm is positioned for use within an imaging volume of the imaging system.

[0015] The system may include a robotic handler coupled to a fluid tactile sensor. The fluid tactile sensor may be removably and replaceably coupled to a housing of the robotic handler. The system may include a processor configured by computer executable code to perform a soft robotic gripping function using the fluid tactile sensor. The system may include a robotic handler and a processor, the robotic handler being coupled to the fluid tactile sensor, and the processor being configured by computer executable code to perform the following steps: pressurizing a reservoir with an imaging medium to increase the ductility of the fluid tactile sensor; positioning the fluid tactile sensor on an object with the robotic handler; and gripping the object by depressurizing the reservoir to remove a portion of the imaging medium and increasing the stiffness of the fluid tactile sensor on a portion of the diaphragm that engages the object. The processor may be further configured to capture one or more images of the object at one or more wavelength ranges through a window, wherein the plurality of beads are optically transparent and index-matched to the imaging medium. The system may include a processor configured by computer executable code stored in a memory to capture images from an imaging system and calculate quantitative surface topography of a target surface that contacts the diaphragm. The imaging medium may include a gas. The imaging medium may include a liquid. The illumination source may include at least one of a laser and a light emitting diode. The imaging system may use two or more imaging modalities, including at least one of photometric stereo imaging, multi-view stereo imaging, structured light imaging, and focal stacking. The system may include a pressure sensor coupled to the reservoir and configured to measure the pressure of the imaging medium in the reservoir. The system may include a processor configured to control a pump in response to a signal received from the pressure sensor. The diaphragm may have at least one of an optically reflective surface and a patterned surface. The system may include a display that presents a visualization of at least one of a pressure field, a contact force field, and a surface topology acquired by the imaging system of an object contacting the diaphragm. The system may include a machine learning model configured to identify an object contacted by the fluid tactile sensor based on data acquired from the imaging system. The window may include at least one of glass, polycarbonate, acrylic, polystyrene, polyurethane, or an optically transparent epoxy. The window may have a first refractive index that matches a second refractive index of the imaging medium in the reservoir. The diaphragm may be formed of an elastic polymer. The system may include an optical pattern on a portion of the diaphragm facing the window. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Embodiments of the devices, systems, and methods described herein are illustrated in the following drawings. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the present disclosure.

[0017] Figure 1 An imaging system is shown.

[0018] Figure 2 is a perspective view of a tactile sensor.

[0019] Figure 3 yes Figure 2 Side view of a tactile sensor.

[0020] Figure 4 A robotic system using tactile sensors is shown.

[0021] Figure 5 An imaging system with a tactile sensor is shown.

[0022] Figure 6 A multi-sensor system is shown.

[0023] Figure 7 A robotic gripper with integrated fluid imaging media is shown.

[0024] Figure 8 A robotic gripper with integrated fluid imaging media is shown.

[0025] Fig. 9 A method for using a fluid tactile sensor is illustrated. DETAILED DESCRIPTION

[0026] Unless expressly stated otherwise or clear from the context, references to items in the singular should be understood to include items in the plural, and vice versa. Grammatical conjunctions are intended to express any and all disjunctive and conjunctive combinations of connected clauses, sentences, words, and the like, unless expressly stated otherwise or clear from the context. Thus, the term "or" should generally be understood to mean "and / or" and the like.

[0027] Unless otherwise indicated, the enumeration of the range of values ​​herein is not intended to be limiting, but instead refers to any and all values ​​falling within the range individually, and each separated value in such a range is incorporated into the specification as if it were individually listed herein. As will be appreciated by those of ordinary skill in the art, when accompanied by numerical values, the words "about" or "approximately" etc. should be interpreted as indicating deviations, to operate satisfactorily for the intended purpose. The range of values ​​and / or numerical values ​​is provided herein only as an example, and does not constitute a limitation on the scope of the described embodiments. The use of any and all examples or exemplary language ("for example," "such as," etc.) provided herein is only intended to better illustrate the embodiments, and does not constitute a limitation on the scope of the embodiments or claims. Any language in the specification should not be interpreted as indicating any unclaimed element as being essential for the practice of the disclosed embodiments.

[0028] In the following description, it should be understood that terms such as "first", "second", "top", "bottom", "upper", "lower" and the like are words of convenience and should not be construed as limiting terms unless specifically stated to the contrary.

[0029] The devices, systems, and methods described herein may include or may be used in conjunction with the teachings of U.S. Pat. No. 8,411,140, ​​issued on April 2, 2013, U.S. Pat. No. 9,127,938, issued on September 8, 2015, U.S. Pat. No. 10,965,854, issued on March 30, 2021, and PCT Application No. US2022 / 046129. In some aspects, the devices, systems, and methods described herein may be used to provide easily interchangeable tactile sensors for handheld or quantitative topography or three-dimensional measurement systems. The devices, systems, and methods described herein may also or instead be included on other systems or used in other ways with other systems. For example, the systems described herein may be useful for the following: for example, robotic end effector systems, such as for component identification and pose estimation, force feedback, robotic surgery, medical examinations, and the like, and other systems and applications in which one or more of touch, tactile sensing, surface topography, or three-dimensional measurement is necessary or helpful.

[0030] Figure 1 An imaging system is shown. In general, the imaging system 100 can be any system for quantitative or qualitative topography measurement and / or visualization, such as any of those described in the literature identified above, and modified to provide a container for a fluid imaging medium as described herein. The imaging system can collect quantitative data, such as images, surface normal maps, height maps of three-dimensional topography, force maps, elastic maps, or other metrics of the softness / hardness of the target surface, etc. It will be understood that although the term "imaging system" is used to describe some of the contemplated embodiments, the tactile sensor can also be deployed in a system that does not generate an image, for example, where the raw sensor data is provided to a neural network or other machine learning system for decision making without converting the raw data into any image or quantitative surface reconstruction. Unless otherwise expressly stated, all such permutations, combinations, or variations of the foregoing are intended to fall within the scope of the present description and within the scope of the imaging system as described herein.

[0031] In one aspect, the imaging system 100 may include a tactile sensor 102 configured as a removable and replaceable cartridge for the imaging system 100, along with a fixture 104 for holding the tactile sensor 102. The fixture 104 may have a predetermined geometric configuration relative to the imaging system 100, for example, relative to an imaging device 106 such as a camera and an illumination source 108 such as one or more light emitting diodes or other light sources, such that the tactile sensor 102, when secured in the fixture 104, has a known position and orientation relative to the camera and the light source(s). This enforced geometry advantageously permits reuse of calibration data for the tactile sensor 102, as well as reliable, repeatable positioning of the tactile sensor 102 within an optical train of the imaging system 100.

[0032] It should be appreciated that while portions of the following description emphasize the use of a removable tactile sensor 102 configured as a cartridge or the like for modular use and reuse, the tactile sensor 102 or portions thereof may also or instead be integrated into the imaging system 100 in a generally non-removable manner. Thus, the advantages of the systems and methods described herein may also be applied to imaging systems 100 that do not include a removable tactile sensor 102, but instead incorporate some or all of the components of the tactile sensor 102 into the body of the imaging system 100. Portions of the tactile sensor 102, such as a rigid substrate, may also or instead be integrated into the body of the imaging system 100, while other portions, such as portions that contact a target surface or contain a fluid imaging medium, may be removable and replaceable to permit reuse of the imaging system 100 after the contact surface has become contaminated or damaged with use.

[0033] The tactile sensor 102 may include an optical element 110 formed at least in part of a rigid, optically transparent material, such as glass, polycarbonate, acrylic, polystyrene, polyurethane, optically transparent epoxy, or any other material having suitable mechanical and optical properties for use in the systems described herein. In this context, and more generally, when the term is used herein, it will be understood that "optically transparent" can mean generally clear in the visible light range, and can also or instead mean clear in the wavelength or wavelength range of interest. Thus, for example, when imaging is performed in the infrared range, a "clear" material will transmit most of the incident light in the infrared range. As another example, different wavelength ranges can be used to usefully channelize or multiplex imaging, and the optical element 110 can be clear for these aggregated wavelength ranges, or can include multiple components, each component being clear at one or more different wavelength ranges. It should also be understood that in this context, "clear" means sufficient transmission to capture an image. This can generally be understood as, for example, greater than ninety percent transmission, and less than ten percent combined reflection and absorption. However, optical elements 110 having a transmittance of less than ninety percent may also be used, for example, due to certain material or cost constraints, provided that the optical element(s) 110 transmit sufficient light within the wavelength of interest to support imaging using the imaging system 100 as generally described herein.

[0034] In general, the optical element 110 can form a substrate of the tactile sensor 102, or the optical element 110 can be a window within a larger mechanical substrate of the tactile sensor 102, i.e., a window of optically clear material embedded in another structure for attachment to the fixture 104 or other components of the imaging system 100. In one aspect, the optical element 110 can be formed of silicone, such as hard platinum cured silicone, or any other optical quality polymer. The optical element 110 can include a first surface 112, the first surface 112 including a region having an optically clear surface for capturing an image through the optical element 110, for example, by the imaging device 106. The optical element 110 can also include a second surface 114 opposite the first surface 112, wherein a central axis 117 passes through the first surface 112 and the second surface 114.

[0035] In general, the first surface 112 can have optical properties suitable for conveying an image from the second surface 114 through the optical element 110 to the imaging device 106. To support this functionality, the first surface 112 can, for example, include a curvature that provides a lens to optically magnify, focus, or otherwise modify the image from the second surface 114. For example, the first surface 112 can include an aspheric surface that is shaped to account for spherical aberration or other optical aberrations in an image captured from the second surface 114 through the optical element 110. The first surface 112 can also or instead include a freeform surface that is shaped to reduce or otherwise mitigate geometric distortion in an image captured through the optical element 110. Imaging through thick media can generally result in spherical aberration that has a size that depends on the numerical aperture of the imaging system 100 (or more specifically, the imaging lens 106 herein). Thus, the first surface 112 of the optical element 110 may be curved or otherwise adapted to account for such spherical aberrations (and other higher order aberrations) caused by the propagation of a focused beam of rays through a thick medium. More generally, the first surface 112 may include any shape or surface treatment suitable for focusing, shaping, or modifying an image in a manner that supports capturing topographic data using the optical element 110. The second surface 114 may also or instead be modified to improve image capture. For example, the second surface 114 of the optical element 110 may include a convex surface extending from the optical element 110 (e.g., toward the target surface 130 being imaged) so as to magnify or otherwise shape the image delivered from the target surface 130 to the imaging device 106. More generally, the first surface 112 may include any light shaping features suitable for facilitating imaging as described herein, such as filters, focusing curvatures, diffusers, and the like.

[0036] As contemplated herein, the optical element 110 can generally be used for a variety of purposes in the imaging system 100. In one aspect, the optical element 110 acts as a rigid body to transmit pressure relatively uniformly across the target surface 130 when capturing an image. Specifically, the body of the optical element 110 can exert a substantially uniform pressure on the imaging medium so that the reflective diaphragm coating on the other side of the imaging medium conforms to the measured surface topography. In one aspect, the optical element 110 can provide grazing or shallow angle illumination, for example, from the illumination source 108 on its edge. The optical element 110 can also or instead provide directional dark field illumination. To this end, a sufficiently thick optical material can be used and can act as a light guide to provide controlled, uniform, and nearly collimated dark field or grazing illumination of the reflective diaphragm surface from different directions (e.g., when a single LED segment of the illumination source 108 is turned on) or from all around (e.g., when all LED segments of the illumination source 108 are turned on). The latter configuration may be useful, for example, when LEDs of different colors are used to multiplex optical channels of multispectral photometric stereo, where each color is associated with a specific illumination direction.

[0037] A fluid layer 116, such as an optically transparent fluid or other fluid imaging medium, can be enclosed within the second surface 114 of the optical element 110, the diaphragm 120 (and in the event that the diaphragm 120 is not fluidly sealed to the second surface 114), and a containment wall 121, which couples the diaphragm 120 to the second surface 114 to enclose the fluid layer 116 therebetween. In general, the fluid layer 116 can include any gas or other Newtonian or non-Newtonian fluid having suitable optical properties for imaging as described herein. The fluid layer 116 can also be sufficiently extended to permit the second surface 120 to conform to a target surface of interest. In general, a first side 118 of the fluid layer 116 adjacent to the second surface 114 of the optical element 110 can have a refractive index that matches the refractive index of the second surface 114. It will be appreciated that, as used herein, the term "matching" when referring to refractive indices does not require identical refractive indices. Instead, the term "matched" generally means having sufficiently close refractive indices to transmit an image through a corresponding interface between the two materials for capture by the imaging device 106. Thus, for example, acrylic has a refractive index of approximately 1.49, while polydimethylsiloxane has a refractive index of approximately 1.41, and these materials are sufficiently matched that they can be placed adjacent to each other and can be used to transmit images sufficient for quantitative or qualitative topographic measurements, as contemplated herein.

[0038] The diaphragm 120 containing the fluid layer 116 on a second side opposite the first side can conform to the target surface 130 while providing an inner surface (facing the imaging device 106) that facilitates topographic imaging and measurement by the imaging system 100. For example, the diaphragm 120 can include an opaque and / or reflective coating, or more generally, any optical coating having a predetermined reflectivity suitable for supporting topographic imaging as contemplated herein. In general, the coating can facilitate capturing images by the optical element 110 independently of the optical properties of the target surface 130, such as color, translucency, gloss, specular reflectivity, and the like, which might otherwise interfere with optical imaging. In one aspect, in the absence of external forces, the diaphragm 120 can form a convex surface (e.g., toward the target surface 130) that extends away from the optical element 110 and the containment wall 121. This geometric configuration may provide a number of advantages, such as facilitating imaging of surfaces having large concave shapes and mitigating accumulation of air bubbles within the field of view, for example, by forming initial contact near the center of the field of view that progresses outward and away from the center, when the tactile sensor 102 is initially placed in contact with the target surface 130, permitting air to escape as additional force is applied to increase the contact area between the diaphragm 120 and the target surface 130.

[0039] In addition to the containment wall 121 surrounding the fluid layer, a sidewall 122 may be formed around the interior 124 of the optical element 110, extending from the first surface 112 to the second surface 114. In general, the sidewall 122 may include one or more light shaping features configured to control illumination, such as from the illumination source 108, through the second surface 114 of the sidewall 122. Like the surfaces 112, 114 of the optical element 110, the sidewall 122 may assume a wide variety of geometries having useful light shaping features, such as to redirect light into and through the optical element 110 at a desired angle and uniformity. For example, the sidewall 122 may include a continuous surface forming a truncated cone between two circles formed in the first surface 112 and the second surface 114. The sidewall 122 may also or instead include a truncated hemisphere between some or all of the area between the first surface 112 and the second surface 114. In another aspect, the sidewall 122 may include two or more discrete planar surfaces arranged in a regular or irregular polygonal geometry, such as a hexagon or an octagon, about the central axis 117. In this latter embodiment having planar surfaces, each such surface may have an illumination source 108, such as one or more light emitting diodes adjacent thereto, so as to provide side illumination as desired through the optical element 110. It should be appreciated that the planar surfaces may also be used as light shaping features, wherein the planar surfaces refract light and / or otherwise control illumination in a desired manner within the imaging volume of the system 100.

[0040] Other light shaping features may also or instead be used with sidewall 122, for example, to focus or redirect incident light from illumination source 108, or to control light reflection within optical element 110 and / or optically transparent fluid layer 116. For example, the light shaping features may include a diffusing surface to diffuse point sources of incident light along sidewall 122. This may, for example, help diffuse light from individual light emitting diode elements in illumination source 108, and / or provide a more uniform illumination field from the flat surface of sidewall 122. Sidewall 122 may also or instead include a polished surface to refract incident light into optical element 110. It will be appreciated that diffusing and reflective surfaces may also be used in various combinations to generally shape the illumination within optical element 110. Sidewall 122 may also or instead include a curved surface, for example, to form a lens within sidewall 122 to focus or redirect incident light into optical element 110 as desired.

[0041] In another aspect, the sidewall 122 can include a neutral density filter with a gradual attenuation to compensate for the distance from the sidewall 122. More specifically, to avoid over-illumination of areas of the second surface 118 close to the sidewall 122, and / or under-illumination of areas of the second surface 118 far from the sidewall 122 (and closer to the central axis 117), the sidewall 122 can provide a broadband attenuation with a neutral density filter that provides greater attenuation in areas of the sidewall 122 closer to the second surface 114, and less attenuation in areas of the sidewall 122 closer to the first surface 112. In this manner, light rays that directly illuminate the second surface 114 adjacent to the sidewall 122 at a downward angle can be attenuated more than other light rays that leave the illumination source 108 toward the center of the second surface 114. The attenuation can be continuous, discrete, or otherwise gradual, for example, to provide generally greater attenuation closer to the sidewall 122, or to otherwise balance the illumination within the field of view.

[0042] In another aspect, the light shaping features can include one or more color filters that can be usefully employed, for example, to associate a particular color with a particular illumination direction within the optical element 110, or to otherwise control the use of colored illumination from the illumination source 108. In the case where the imaging system uses wavelength multiplexing imaging, the color filters on the sidewalls can also reduce stray illumination within the tactile sensor by selectively reflecting or transmitting a frequency range of interest. In another aspect, the light shaping features can include a non-normal angle of the sidewall 122 to the second surface 114. For example, as Figure 1, the sidewall 122 is angled away from the second surface 114 to form an obtuse angle therewith. This approach can advantageously support indirect illumination of the second surface 118, for example, by total internal reflection of light from the first surface 112 and entering the optical element 110. In another aspect, the sidewall 122 can be angled toward the second surface to provide an acute angle therewith, for example, to support greater direct illumination of the second surface 118. These approaches can be used alone or in combination to redirect light into and through the optical element 110 as desired.

[0043] The light shaping features may also or instead include geometric features, such as focusing lenses, planar areas, etc., to direct incident light as desired. Other optical elements may also or instead be usefully formed on or in the sidewall 122. For example, the light shaping features may include optical films, such as any of a variety of commercially available films for filtering, attenuating, polarizing, or otherwise shaping incident light. The light shaping features may also or instead include a microlens array, etc., to turn or focus incident light from the illumination source 108. The light shaping features may also or instead include a plurality of micro-replicated and / or diffractive optical features, such as lenses, gratings, etc. For example, the microstructured sidewall 122 may include, for example, micro-imaging lenses, convex mirrors, micro-prisms, etc. as light shaping features to turn light from the illumination source 108 into the optical element 110 to improve the manner in which the topographic changes are imaged on the imaging surface of the tactile sensor 102 on the second side 120 of the optically transparent fluid layer 116. For example, the microstructure features can facilitate shaping the illumination pattern to provide a uniform light distribution across the measured field, reduce light reflections back into or out of the optical element 110, etc. The microstructures can be applied, for example, during injection molding of the optical element 110, or by applying an optical film having the desired microstructure to a side surface. For example, commercially suitable optical films can include Vikuiti TM , an advanced light management film (ALCF) sold by 3M.

[0044] As noted above, one or more of these light shaping features may also or instead be integrated into other surfaces 112, 118 of the imaging system 100, where they may be deployed to filter, focus, modulate, or otherwise modify illumination along the optical path from the illumination source 108 to the diaphragm 120 and / or back to the imaging device 106.

[0045] A mechanical key 126 may be disposed on the exterior of the optical element 110 for enforcing a predetermined positioning of the optical element 110 (and more generally, the tactile sensor 102) within the fixture 104 of the imaging system 100. The mechanical key 126 may, for example, include at least one radially asymmetric feature about the central axis 117 for enforcing a unique rotational orientation of the optical element 110 within the fixture 104 of the imaging system 100. The mechanical key 126 may include any number of mechanical elements, etc., suitable for holding the optical element 110 in a predetermined orientation within the imaging system 100. The mechanical key 126 may also or instead include a matching geometry between the optical element 110 and the fixture 104. For example, the mechanical key 126 may include a cylindrical structure extending from the optical element 110, or an elliptical prism, etc., which may be useful to enforce rotational orientation simultaneously with positioning. Mechanically enforced positioning may be particularly advantageous in the context of fluidic tactile sensors, where the shape, thickness, elasticity, and optical properties of the containment diaphragm may vary among sensors and, for a particular sensor, across the fluidic layer 116 .

[0046] In one aspect, the mechanical key 126 may include one or more magnets 128 or other mechanisms (one or more) to secure the optical element 110 in the fixture 104 of the imaging system 100. The magnets 128 may be further encoded via positioning and / or polarity to ensure that the optical element 110 is only inserted in a specific rotational orientation about the central axis 117. The mechanical key 126 may also or instead include a plurality of protrusions, including at least one protrusion having a shape different from other protrusions in the plurality of protrusions, for enforcing a unique rotational orientation of the optical element 110 about the central axis 117 within the fixture 104 of the imaging system 100. The mechanical key 126 may also or instead include at least three protrusions (e.g., exactly three protrusions) shaped and sized to form a kinematic coupling with the fixture 104 of the imaging system 100. Mechanical key 126 may also or instead include features such as flanges, dovetails, or any other mechanical shape or feature to securely mate optical element 110 to fixture 104 in a predetermined position and / or orientation. A number of specific mechanical keying systems are discussed herein with reference to specific optical element designs and configurations.

[0047] The surfaces of the tactile sensor 102 may be further processed as necessary or to facilitate use in the imaging system 100 as contemplated herein. For example, regions of the top, side, and bottom surfaces of the optical element 110 or other portions of the tactile sensor 102 may be covered with a light absorbing layer, such as black paint, to, for example, block light from the illumination source 108 or to reduce penetration of ambient light.

[0048] In one aspect, the optical element 110 and fluid layer 116 of the tactile sensor 102 can be formed as a box that is provided to the end user as an integral, removable and replaceable device that can be quickly and easily replaced by the end user as required, for example due to wear in the diaphragm 120, or to replace the tactile sensor 102 with different optical properties, for example for different imaging applications, resolutions, etc.

[0049] In another aspect, the fluid layer 116 can have a variety of different surface shapes based on the shape and mechanical properties of the membrane 120, along with the volume of fluid in the fluid layer 116. For example, the membrane can form a convex surface shaped to provide a sensor extending outwardly from the imaging system 100, and more specifically, from the optical element 110 of the imaging system 100, which can advantageously facilitate imaging of relatively concave surfaces and can also advantageously mitigate bubble formation when the fluid layer 116 is placed on a target surface for image capture.

[0050] Figure 2 is a perspective view of a tactile sensor with a fluid imaging medium. The tactile sensor 202 can, for example, have a generally rectangular configuration and can include one or more flanges 204, etc., so that the tactile sensor 202 can be linearly slid into engagement with a fixture of the housing. This type of engagement mechanism can be particularly suitable for robotic applications, etc., such as where the tactile sensor 202 is removed from an end effector of a robotic handler and replaced to an end effector of a robotic handler. The tactile sensor 202 can, for example, be any of the tactile sensors described herein. A fluid layer 206, such as any of the fluid layers described herein, can provide an optically transparent medium that is partially enclosed by a diaphragm 208 and partially by a substrate 210 of a rigid optically clear material.

[0051] Figure 3 yes Figure 2 Side view of a tactile sensor.

[0052] Figure 4A robot system using a tactile sensor is shown. Generally speaking, the system 400 may include a robot handler 402 having a housing 404 on one end thereof, the housing 404 being configured to removably and replaceably receive a tactile sensor 406, such as a box or other tactile sensor or any of the other optical devices described herein. Generally speaking, the robot handler 402 may include any robot assembly or assembly combination suitable for positioning and manipulating an object. For example, the robot handler 402 may include a robot arm, a gantry, a SCARA robot, a Cartesian robot, a delta arm, or any combination of these or other positioning controllers, together with suitable sensors, actuators, and the like to control its movement. The robot handler 402 may also include any suitable manipulator, gripper, end effector, etc., for grasping or otherwise handling and manipulating an object. The system 400 may also include a processor or other controller, etc., for providing programming or a user interface to control the operation of the robot handler 402.

[0053] The robotic handler 402 can be configured to position the tactile sensor 406 in contact with the target surface 408 so as to capture a topographic image of the target surface 408 using, for example, a camera or other imaging device in the housing 404. It will be appreciated that the components of such an imaging device can generally be within the housing 404, or positioned remotely and optically coupled to the tactile sensor 406, for example, by optical fiber, etc., or some combination of these. In one aspect, the system 400 can be configured, for example, by computer executable code stored in a memory of the system 400 and executed by a processor of the system 400, to automatically remove the tactile sensor 406 from a fixture of the system 400 (e.g., in the housing 404) and insert a second cartridge 410 having a replacement sensor into the housing 404. The second tactile sensor 410 can be identical to the tactile sensor 406, e.g., to provide replacement after normal wear, or the second tactile sensor 410 can have a different optical configuration than the first tactile sensor 406, e.g., to provide greater magnification, greater field of view, better feature resolution, deep feature illumination, different aggregate surface shapes, different shape tolerances of the target surface 408, etc. The second tactile sensor 410 can be stored in a box or other container accessible to the robotic handler 402 of the system 400. In general, the system 400 can include one or more magnets, electromechanical latches, actuators, etc. within the housing 404, or more generally within the system 400, to facilitate removal and replacement of the tactile sensor 406 as described herein. More generally, the system 400 can include any gripper, fixture, or other electromechanical end effector, etc. suitable for removing and replacing the tactile sensor 406 and positioning the tactile sensor 406 for use in the imaging process.

[0054] In one aspect, the robotic handler 402 can be manually operated by a human technician from a console, etc. The robotic handler 402 can also or instead be programmed to operate automatically, for example in a test or manufacturing facility. In this context, the robotic handler 402 can automatically position the tactile sensor 406 on the workpiece of interest, for example using a sensing network, machine learning algorithms, and other techniques, and can control contact force, fluid pressure, temperature, or other parameters, for example, in preparation for measurement and / or when collecting image data. After correct positioning, the robotic handler 402 can control an imaging system (also in the housing 404, or accessible from it) to collect data for a three-dimensional reconstruction of the target surface of the workpiece. This general technology can be used for component inspection, metrology, and the like, for example.

[0055] In another aspect, the robotic handler 402 may use tactile feedback to guide decision making. For example, the robotic handler 402 may determine whether a workpiece meets a particular physical requirement, and may then classify the workpiece as acceptable, unacceptable, and / or requiring manual inspection. In another aspect, the robotic handler 402 may use tactile feedback from the tactile sensor 406 to control, for example, the gripping strength of a robotic hand, gripper, or other end effector, or to control the amount of instantaneous contact force or torque, etc., applied to a workpiece manipulated by the robotic handler 402. In another aspect, the tactile sensor 406 or an array of tactile sensors 406 may be used to create a visualization of a contact force field, pressure field, or surface topology, which may be presented to a human operator in a display 410 to assist the operator in controlling actions by the robotic handler 402 with respect to the workpiece.

[0056] The system 400 may include a computing device 412 that may be used, for example, to process data from the tactile sensor 406, control the operation of the robotic handler 402, provide a user interface for the robotic handler 402, and the like. For example, the computing device 412 may be configured, for example, by code stored in a memory and executed on a processor of the computing device 412, to identify an object or surface contacted by the tactile sensor 406 of the robotic handler 402, generate an alert to a user based on tactile feedback collected from the tactile sensor 406, make decisions (including decisions recommended to a user, as well as decisions automatically performed by the robotic handler 402) on actions with respect to the workpiece contacting the tactile sensor 406, and the like. In one aspect, the code may usefully employ, for example, a machine learning model or the like for identification, decision making, and other intelligent sensing and / or data-driven operations.

[0057] Figure 5An imaging system with a tactile sensor is shown. In general, the imaging system 500 can include a tactile sensor 502, such as any of the fluid tactile sensors described herein, etc. The imaging system 500 can also include a light source 503, an imaging device 506, a controller 516, and an imaging volume 508 that at least partially spatially intersects a fluid layer 509 of the tactile sensor 502 when the tactile sensor 502 is positioned for use relative to the imaging device 506. The optical element 513 can be positioned to control the illumination of the imaging volume 508 by the light source 503.

[0058] The tactile sensor 502 may include any of the fluid tactile sensors described herein for contacting a target surface to facilitate three-dimensional imaging. The tactile sensor 502 may, for example, include a fluid imaging medium enclosed within a diaphragm 519, a substrate 518 (such as an optically clear sheet of rigid material), and one or more containment walls 520. The imaging system 500 may have an axis 517, such as an imaging axis or an optical axis, that passes through the imaging volume 508. When the tactile sensor 502 is placed for use in the imaging system 500, the sensing region of the imaging system 500 (where the fluid layer 509 and the imaging volume 508 intersect) may therefore intersect the axis 517 of the imaging system 500 so that the imaging device 506 can capture an image of the sensing region. In one aspect, the tactile sensor 502 can be removably and replaceably coupled to the imaging system 500 and can be mechanically bonded or otherwise coupled to the imaging system 500 in a manner that aligns a portion of the fluid layer 509 of the tactile sensor 502 with the imaging volume 508 of the imaging system 500 to form a sensing region.

[0059] The light source 503 can be any illumination source suitable for providing illumination through the optical element 512 and into the imaging volume 508. When the tactile sensor 502 is placed for use in the imaging system 500, the light source 504 can illuminate the sensing area, such as the imaging volume 508, and permit the imaging device 506 to capture images. These images can then be processed by the controller 516 to resolve the three-dimensional surface information of the object in contact with the diaphragm 519 in the sensing area, together with any of the other information described herein. In one aspect, the light source 503 can be a laser or other device with coherent, fixed focus and / or providing collimated illumination. In this context, it will be understood that the fixed focus can include light focused at infinity, that is, collimated or formed by parallel ray trajectories, and light with any other fixed focus that can be used to create the illumination pattern described herein. In another aspect, with appropriate modifications to the optical element 512 and other optical features, the light source 503 can provide non-focused illumination.

[0060] The imaging device 506 may be any other combination of a camera or optical device, lens, filter, optical fiber, and other hardware suitable for capturing an image of the imaging volume 508 for use by the controller 516 in resolving a three-dimensional image. In general, the imaging device 506 may have an imaging axis that passes through the imaging volume 508, such as the axis 517 of the imaging system 500, so as to capture an image thereof. The controller 516 may include any processor, microcontroller, or other circuitry, or combination of the foregoing, suitable for controlling the operation of the imaging system 500 to acquire three-dimensional information as described herein. In one aspect, the controller 516 may be physically coupled to the imaging system 500 and may be configured to acquire data for transmission to a separate processor for processing. In another aspect, the controller 1208 may include one or more microprocessors, field programmable gate arrays, graphics processing units, and / or other processors configured to process images and resolve the image data into three-dimensional data of a surface within the imaging volume 508. In one aspect, the controller 516 may include a processor configured by instructions stored in a memory to receive one or more images from the imaging device 506, including a pattern created by the optical element 512 and reflected by the diaphragm 519 (or a coating on the diaphragm, such as a thin reflective coating) when the diaphragm 519 is deformed to a target surface of an object within the imaging volume 508. The processor, or another processor integrated into the imaging system 500 or communicatively coupled to the imaging system 500, may be further configured by instructions stored in the memory to calculate a quantitative surface topography of the diaphragm 519 based on the image(s) captured by the imaging device 506. As described herein, the surface may include, for example, a deformable surface of the diaphragm 519 that intersects the imaging volume 508 and conforms to the target surface of the object to be measured. It will be understood that the measured surface may include an outer surface of the diaphragm that contacts the target surface, an inner surface of the diaphragm that contacts the fluid layer 509, or some combination of these layers or some other layers, such as an inner layer therebetween.

[0061] The imaging volume 508 may generally define a three-dimensional field of view for the imaging device 506. As described above, the imaging device 506 may have an imaging axis that passes through the imaging volume 508, such as the axis 517 of the imaging system 500. A plane may intersect the imaging volume 508 and lie perpendicular to the axis 517 of the imaging system 500 (and the imaging device 506). The plane may also lie perpendicular to Figure 5 The plane in which the image is taken lies and will appear in the figure as a horizontal line passing through the imaging volume.

[0062] The optical element 512 may include any optical element or combination of optical elements, including a diffraction grating, a lens, a filter, a micro-textured surface, a metasurface, or any other optical device suitable for controlling illumination and / or creating a desired illumination pattern within the imaging volume 508. In one aspect, the optical element 512 may create a pattern including a plurality of features, such as points, lines, or polygons, that may be identified within an image of the imaging volume 508 captured by the imaging device 506. For example, the pattern may usefully include a first plurality of features closely spaced within a plane and a second plurality of features that are visually distinguishable from the first plurality of features and spaced further apart within the plane. In the pattern, the more distantly spaced features may provide fiducials or landmarks within the imaging volume 508 to assist in processing, while the more closely spaced features support higher resolution sensitivity to surface topography. The pattern may also or instead include a first plurality of features and a second plurality of features that together form a regular geometric pattern within a plane, wherein the second plurality of features form visually distinguishable anchor points within the pattern. The anchor points or landmarks may be spaced sufficiently far apart that they are unlikely to intersect (or physically unable to intersect) in the imaging plane due to deflection along axis 517. In these embodiments, the pattern may generally include a first plurality of features that are closely spaced to provide high resolution detection of depth within the imaging volume, and a second plurality of features that are placed sufficiently far apart in a plane through the imaging volume 508 to avoid intersection along the imaging axis (e.g., axis 517) within the imaging volume 508 during the maximum expected deformation of the diaphragm 519 when it contacts the target surface. It will be understood that in this context, the expected deformation may include z-axis displacement, as well as any x-axis or y-axis displacement resulting from shearing, wrinkling, and the like of optical elements when the imaging system 500 is placed against the target surface and manipulated by the user.

[0063] It will be appreciated that the membrane 519 may also or instead include a similar arrangement of visual patterning or texturing and for similar purposes. However, optically creating these patterns may provide additional flexibility, for example, by facilitating changes in the pattern during imaging, or by more generally permitting changes in the visual pattern independent of the optical properties and physical construction of the membrane 519.

[0064] In one aspect, the optical element 512 can include a diffractive optical element positioned to receive illumination from the light source 503 (e.g., a coherent light source such as a laser) on a first surface (e.g., a surface facing the light source 503) and to create a three-dimensional illumination pattern within the imaging volume 508 from a second surface opposite the first surface (e.g., a surface facing the imaging volume 508). Where a diffractive optical element is used, the diffractive optical element can include micro-patterned structures, for example on either or both of these surfaces, optionally together with an additional lens, that cooperate to create a desired illumination pattern when a suitable light source is directed at the optical element 512. Various types of diffractive optical elements are known in the art and can be used to create illumination patterns that vary in intensity in a far field plane, as well as illumination patterns that vary in intensity and / or focus along an imaging axis. As a significant advantage, these properties can be exploited to create a three-dimensional illumination pattern within the imaging volume 508 of the imaging system 500 to facilitate resolving three-dimensional information from a target surface of the membrane 519 contacting the tactile sensor 502. Any number of additional optical components may also or instead be included to create illumination patterns as described herein. For example, interfaces between layers or components of the optical system may incorporate light shaping features such as lenses, filters, and the like to, for example, control optical power, compensate for distortion or wavefront error, control exit angles, and the like.

[0065] In one aspect, the optical element 512 may include a diffractive optical element or other optical device that generates a three-dimensional illumination pattern or structure that varies with depth or distance from the optical element 512. For example, the three-dimensional illumination pattern may include a divergent illumination projection, such as a grid, dot array, cone, or pyramid pattern that diverges (e.g., becomes larger in the imaging plane) as the distance from the optical element 512 increases, or more generally, a three-dimensional pattern that varies along an imaging axis (e.g., axis 517) within the imaging volume 508. In another aspect, the three-dimensional illumination pattern may include a pattern having one or more features that vary along a projection line from the optical element 512. For example, as the distance of the projected image from the optical element 512 increases, the circles, dots, or other images may change in intensity or focus (with or without a change in size), or may appear or disappear at different focal distances.

[0066] In many illumination modes, steeper angles of incidence (e.g., sharper angles relative to a plane) can provide greater sensitivity to three-dimensional displacements. As such, it may be advantageous to include one or more additional light sources 503 and / or optical elements 512 to provide illumination from different directions around the axis 517 of the imaging device 500 so that different areas of the imaging volume 508 can benefit from steep side illumination. In one aspect, these additional light sources 503 may also use different spectral bands so that different patterns may be captured simultaneously, for example, in a single image frame, where visual features may be associated with a particular light source 503 and illumination mode based on wavelength. The method may also advantageously improve sensing of occluded areas of a surface and / or steep or sharp surface features. Thus, in one aspect, illumination from different light sources 503 and / or optical elements 512 may be used to calculate three-dimensional data for different portions of the imaging volume 508. While in such embodiments the images captured by the imaging device 506 may be divided and processed strictly in this manner (e.g., where one side of the imaging volume 508 is processed using illumination from an opposite side of the imaging volume 508), image data from different illumination directions may also or instead be combined or weighted in a variety of ways, where such combinations may be shown to improve accuracy or repeatability for particular regions of the imaging volume 508 or for particular imaging applications, or where such combinations permit analysis of occluded areas, deep valleys, and the like.

[0067] In another aspect, different illumination sources can be multiplexed in such a manner that, for example, multiple images from multiple illumination directions can be captured and / or processed simultaneously by using light of different wavelength ranges (or different specific wavelengths) to illuminate the imaging volume 508 from different directions, and by processing the images from these different wavelength ranges separately. In accordance with the foregoing, the imaging system 500 can usefully include a second optical element 512 positioned and configured to create a second pattern within the imaging volume 508 for a different location with respect to the perimeter of the imaging volume 508 than the first optical element 512. More generally, two or more additional light sources 503 and / or optical elements 512 can be incorporated into the imaging system 500 to improve imaging under various imaging conditions with various surface topography.

[0068] In another aspect, additional imaging techniques may be incorporated into the imaging system 500, for example, to improve the accuracy and robustness of the imaging system 500, to support higher speed, lower resolution processing for specific imaging contexts (image preview, sparse three-dimensional processing, etc.), or for other reasons. Thus, in one aspect, the imaging system 500 may include a multi-view imaging system (e.g., a stereoscopic imaging system or a photometric stereo system, etc.) configured to compute quantitative surface topography of a surface within the imaging volume 508 based on images of the surface from two or more different viewing angles. In this context, the multi-view imaging system may include a stereoscopic imaging system, a photometric stereo system, etc., and / or an imaging system that multiplexes using fluorescence, different visible and / or infrared wavelengths, etc. In another aspect, a gradient-based system may resolve three-dimensional surface information using unfocused illumination from a variety of directions. In general, these alternative imaging modalities may be optically multiplexed (e.g., channelized into different wavelengths or wavelength ranges) for simultaneous operation with the systems described above. For example, these alternative systems may resolve the three-dimensional shape of the target surface using light from a second light source in a second spectral band having wavelengths that do not overlap with the first spectral band of light source 503 and / or one or more other light sources used by imaging system 500 .

[0069] More generally, any of a variety of complementary imaging modalities may be used to measure absolute depth with greater accuracy, and / or to adapt to a variety of topologies and imaging applications. For example, imaging system 500 may use a multi-viewing system based on stereo disparity. Figure 3 dimensional imaging, or a system with an optical pattern that converts depth directly into XY displacement, or any other triangulation-based or other depth measurement technique. As a significant advantage, these complementary techniques for measuring absolute depth can support improved measurement of low spatial frequency three-dimensional features, such as macroscopic large-scale features of the target surface, which are preferably removed prior to measuring micron-scale surface features using gradient-based depth calculations, etc. Further, these alternative depth measurements can provide information about the amount of compression of the fluid layer 509 to provide real-time guidance and user feedback for optimal compression, support higher speed rendering (e.g., using sparser data arrays), support measurement of high-frequency forces (e.g., using finite element models of the fluid layer 509), and the like.

[0070] The diaphragm 519 can be a deformable diaphragm that can be deformed when placed in contact with the target surface for measurement. The diaphragm 519 can be opaque, reflective, or some combination of these. For example, the diaphragm 519 can include a thin reflective coating on the outer surface opposite to the imaging device 506, or on the inner surface facing the imaging device 506. The diaphragm 519 can also or instead be made of a suitably reflective material. The diaphragm 519 can be formed by any elastic polymer or other material sheet that is suitably flexible to conform to the surface of interest and, where useful, appropriately elastically expands or contracts according to the volume of the medium in the fluid layer 509 and / or the surface pressure on the diaphragm 519. In one aspect, the outer surface of the diaphragm 519 can be coated with a functional coating, for example, to reduce friction or mitigate trapped bubbles, etc. The diaphragm 519 can also include any suitable optical coating useful for the imaging modality used to collect surface data. For example, this may include an opaque surface, a colored surface, a partially transparent surface (e.g., transmissive at some wavelengths and reflective at other wavelengths), a patterned surface (applied to or inherent to the material of diaphragm 519), and the like.

[0071] In one aspect, the imaging system 500 may include a supplemental depth measurement mode that is used to measure the distance to the target surface, estimate the compression of the fluid layer 509, and provide feedback to the user to guide the user to an optimal range of contact forces. For example, this may include user feedback via multiple LEDs on a handheld imaging device, etc., indicating whether and how the user should reposition the device to capture an image or improve image quality. This may also or instead include other user feedback that guides the user to properly position the device, such as an auditory output from an audio output device, or a display in a user interface of the imaging system 500, for example, on a computer coupled to the handheld device, etc.

[0072] In one aspect, the imaging system 500 can include a lens 530 for variably focusing the imaging device 506 on a surface within the imaging volume 508, such as a reflective surface of the membrane 519 or a plane within the imaging volume 508. For example, the lens 530 can include a liquid lens that uses a combination of an optical fluid and a polymer membrane to change focus by changing shape, or any other adaptive lens, etc. The liquid lens advantageously provides a compact mechanism for controlling focus without mechanical moving parts and without mechanically moving the lens along the imaging axis to change the focus distance. However, other lenses can also or instead be used to focus the imaging device 506 at various depths or z-axis locations through the imaging volume 508 and along the imaging axis, and can be adapted for use in the imaging system 500 as described herein, such as a lens system that focuses using a piezoelectric focus drive, a voice coil motor, or any other electromechanically controlled lens or lens system suitable for z-stack image acquisition.

[0073] Lens 530 can advantageously include one or more high-resolution lenses with narrow depth of field. To avoid low-pass filtering that might otherwise be imposed by a locally out-of-focus lens, lens 530 can be variably focused to scan through a range of depths (e.g., along the z-axis or imaging axis) to provide a partial, locally focused image at each desired depth. The image stack can be assembled into a single image with greater depth of field for subsequent three-dimensional processing, such as using photometric stereo, or directly measuring quantitative depth information by finding the best focus among various focal depths of localized regions within the imaging field. This single image with improved depth of field also permits recovery of texture, etc., and can be combined with other imaging modalities (such as photometric stereo) to provide more accurate and high-resolution surface measurements across the imaging field without distortion artifacts.

[0074] In one aspect, the system can use photometric stereo imaging to measure surface orientation, for example, as a surface normal vector based on pixel intensity, which can be integrated to resolve three-dimensional surface data. This reconstruction method can be sensitive to small changes in surface orientation that cause low-frequency distortion, resulting in small-scale distortion across the measurement field. Therefore, the system can supplement photometric stereo imaging with triangulation-based 3D reconstruction, which advantageously permits direct depth measurement at each location, thereby providing distortion-free 3D measurements at lower resolution. The combined method advantageously supports high-resolution 3D measurements with consistent resolution and accuracy across the entire imaging field.

[0075] For example, a pattern projection system for the device may create a pattern of dots projected at a highly oblique angle to the target surface (and / or diaphragm 519). A suitable pattern may be created using laser illumination of a diffractive optical element (DOE), which may be micropatterned to suppress and amplify specific diffraction orders (using the coherence of the laser) to create an optical pattern with desired locations of dots or other objects, shapes, symbols, etc. The DOE may also be configured (e.g., by micropatterning its (one or more) surfaces) to adjust the focus of the variation across the imaging volume due to the highly oblique projection angle relative to the imaging plane within the imaging volume. In general, the projected pattern may be imaged by the imaging device 506 to provide triangulation for 3D imaging. When the object for measurement is pressed into the contact surface of the diaphragm 519, the dot pattern will bend in the imaging volume 508 according to the local depth change. Thus, the motion of the dots encodes the 3D shape of the object in a manner that can be captured and resolved into 3D data using the imaging device 506 and an associated processor (which may be the controller 516 or some other processing device).

[0076] More generally, an imaging system as described herein may use any suitable combination of different three-dimensional imaging modalities within or in addition to a retrographic sensor or other imaging device with a fluid imaging medium. For example, in one aspect, a device is disclosed herein, comprising: an imaging volume within a fluid imaging medium, the imaging volume defining a three-dimensional field of view for capturing images; together with an imaging system, configured to calculate quantitative surface topography of a target surface intersecting the imaging volume within the three-dimensional field using two or more three-dimensional imaging modalities. For example, one or more imaging modalities may include photometric stereo and multi-view stereo imaging. Photometric stereo may, for example, use a single camera and directional illumination provided from two or more directions. Depth is encoded in the shadow changes (e.g., intensity gradients) between the captured images. The modality supports spectral multiplexing, for example, using red, green, blue (RGB) or hyperspectral imaging to capture images with multiple illumination directions in a single image frame. The one or more imaging modalities may also or instead include a multi-view stereo imaging modality, which employs any of a variety of techniques to obtain depth information from multiple cameras or views.

[0077] The one or more imaging modalities may also or instead employ single camera triangulation. In this modality, the imaging volume is illuminated with structured light from one or more directions (different from the viewing direction of the camera), and the depth is determined based on the imaging pattern relative to a reference image of the structured light captured during calibration. Alternatively, the pose of a single camera may be moved to different positions in order to capture different images of the target surface. In the case where multiple light directions are used for illumination, these different illumination directions are preferably separated in time or spectrum in order to avoid visual interference among overlapping illumination patterns.

[0078] In another aspect, multi-view stereo or triangulation may be used to obtain depth information from two or more cameras under structured lighting. In another aspect, multi-view stereo or triangulation may be used to obtain depth information from two or more cameras based on surface texture.

[0079] In another aspect, one of the imaging modalities may include depth from defocus or focus stacking, in which the focus / defocus along the optical axis through the imaging volume is used to infer depth. This may be used instead of or in addition to the multi-view stereo technique described above. The focus stacking system may use uniform natural light, as long as the target surface encloses sufficient natural texture to assess focus. In another aspect, structured light (typically coaxial with the optical axis) may be used, particularly where the target surface does not provide suitable features for assessing focus. In either case, different colors may be focused at different depths to support increased depth resolution using spectral multiplexing. In another aspect, one of the imaging modalities may include time-of-flight imaging, in which distances are captured directly in known directions and are used to reconstruct the measured surface.

[0080] In one aspect, the window 505 can be a rigid substrate of optically clear material, and the membrane 519 can be a balloon filled with a fluid and coupled to the window 505 to form an expandable reservoir. In this embodiment, a camera or the like can be mechanically coupled to the window in a fixed relationship, and the camera can be moved around manually or automatically, allowing the balloon to deform while the pose of the camera is displaced across the target surface contacting the membrane 519 to facilitate shape-from-motion based three-dimensional reconstruction.

[0081] In one example embodiment, the imaging system may use photometric stereo and multi-view stereo, with a visible textured surface or the like on the contact surface of the diaphragm 519 (or the inner surface of the diaphragm 519 facing the imaging device 506). The texture may generally be a physical texture that provides optically visible features, an optical texture created using suitable optical processing, or some combination of these. For example, the texture may include a random texture that is invisible unless a specific illumination is used. Such a texture may be created using fluorescent pigments that are only visible when illuminated by UV light. On the other hand, the diaphragm 519 may use IR absorbing pigments to create a random texture that may be illuminated using infrared light to make the texture visible. In this combination, the random texture may be imaged only by a camera dedicated to multi-view stereo, while the photometric stereo camera (a single camera) captures an image of the field of view in the imaging volume 508 based on illumination in different spectral bands provided from different illumination directions, without texture. It will be understood that other arrangements of photometric stereo and the various multi-view imaging techniques described above may also or instead be used.

[0082] Other ranging or three-dimensional imaging techniques may also or instead be used, including but not limited to confocal imaging, interferometric imaging, light detection and ranging (LiDAR), and the like.

[0083] In one aspect, the imaging system 500 may include a fluid management system 538 to manage the amount and / or pressure of the fluid imaging medium 508 within a reservoir 540 formed by the diaphragm 519, the substrate 518, and one or more containment walls 520 that collectively retain the fluid imaging medium 508 of the fluid layer 509. It will be appreciated that while the reservoir 540 may include, for example, Figure 5502, but the reservoir 540 may instead be formed of only a flexible membrane, or of a membrane 519 and a substrate 518, or more generally of any combination of flexible, resilient and / or rigid membranes, walls, and other structures suitable for containing a fluid medium for use in imaging as described herein. The fluid management system 538 may generally include a supply 542 of fluid imaging medium 508 coupled in fluid communication with the reservoir 540, a pump 544, or the like, to control the volume and / or pressure of the fluid imaging medium in the reservoir 540, and a sensor 546 for detecting, for example, the pressure within the reservoir 540. The fluid management system 538 may also include a controller 516 for controlling the pump 544 and / or other hardware associated with the tactile sensor 502 (which may be the same controller 516 that operates the imaging system 500, or a separate controller for operating the fluid management system 538, or some combination of these). In general, the tactile sensor 502, also referred to herein as a fluid tactile sensor, can be used in addition to or in place of the elastomeric sensors or other sensors or sensor cartridges described in the literature cited herein, such as when enclosing a fluid imaging medium 508, unless otherwise specifically noted. It will also be understood that a system using the tactile sensor 502 can include any of the other components described herein, including, for example, cameras or other imaging hardware, lenses and other optical components, light sources, robotic actuators and controllers, etc., to capture and analyze images when the tactile sensor 502 is placed in contact with a target surface.

[0084] The containment wall(s) 520 may be formed of a suitably rigid material to mechanically support the reservoir 540 and the imaging medium 508 contained therein, as well as to support mechanical coupling with an imaging device, such as any of the devices or systems described herein. This may include mechanical keys formed in or coupled to the containment wall(s) 520 to enforce a predetermined position and / or orientation within the handheld imaging system.

[0085] One or more windows 505 may be formed by or within the substrate 518 supporting the diaphragm 519 and the containment wall(s) 520, e.g., as necessary or useful to support imaging functions such as illumination and optical imaging. The window(s) 505 may be formed of an optically clear material and / or a material index matched to the imaging medium 508 to facilitate optical functions of the tactile sensor 502, such as illumination and imaging. The optical and other properties of the window(s) 505 may depend, for example, on the imaging modality used to capture topographic information, and may vary depending on the wavelength(s) used, the technique(s) used (e.g., multi-view structured light, time of flight, ultrasound, etc.), etc. In one aspect, the inner surface of the substrate 518 (e.g., the surface facing the reservoir 540) may be index matched to the imaging medium 508 to mitigate optical artifacts due to the interface. In another aspect, the surface of the containment wall(s) 520 may integrate light shaping features such as filters, lenses, etc. to support improved imaging, which may be deployed within the wall and / or as additional films or surface treatments on or between other elements. In one aspect, the window 505 may usefully be formed of a rigid, optically clear material to facilitate a consistent imaging environment for the imaging device 506, and consistent transmission of forces from the housing to the windowed surface of the reservoir 540. However, in one aspect, the window 505 may also or instead be formed of a flexible and / or elastic membrane that is (a) suitable for retaining the imaging medium 508 within the reservoir 540, and / or (b) optically transparent to facilitate illumination and image acquisition through the window 505.

[0086] The window(s) 505 (and any other optically functional surfaces of the device) may generally be enhanced for imaging with various additional structures, configurations, and / or additional components. For example, components of the tactile sensor 502 may support light shaping, light pipes, etc., and / or may be configured with microtexturing or surface treatments to control optical properties. In one aspect, rigid structural support may be added to facilitate manipulation of the tactile sensor 502, particularly where the tactile sensor 502 is formed by the diaphragm 519 without the containment wall(s) 520. In this latter aspect, a light guiding film may be disposed between the diaphragm 519 and the rigid retainer to control illumination therethrough. Additionally, the rigid retainer of the diaphragm 519 may have an optically driven surface(s), such as a freeform lens surface, which may evenly distribute light over the deformable sensing surface of the diaphragm 519 to form a deformable sensing surface.

[0087] In another aspect, the substrate 518 or the window 505 therein may include multiple optical layers at the interface with the fluid layer 509. These additional layers may be used to control light transmission at the interface. For example, the surfaces of these layers may be designed to maintain internal reflections within the interface, except for light incident at a particular angle. This permits control over the distribution and directionality of light leaving the window 505, either toward the camera 506, or toward the fluid layer 509, or both, which may be used to diffuse or distribute illumination, or to control the exit angle, for example, to improve glancing illumination of the diaphragm 519 for the purpose of three-dimensional surface reconstruction. A variety of techniques may be used to usefully control the interface in this manner, such as varying the refractive index, incorporating biconvex lenses or other surface structures, adding filters or diffusers, and the like.

[0088] In general, the membrane 519 can include any surface coating or treatment described herein for use with a tactile sensor. For example, the membrane 519 can include an inner and / or outer coating to impart desired optical properties. For example, this can include an opaque coating, a reflective coating, a colored coating, an optically patterned coating, a filter, an anti-glare treatment, a refractive index matching coating, and the like. The membrane 519 can be formed from a variety of materials, including flexible and / or elastic sheet materials such as elastomers, or any other film, foil, etc. suitable for retaining the imaging medium 508 within the reservoir 540 and conforming to the target surface with sufficient fidelity to support the desired z-axis resolution of the target surface through the tactile sensor 502, for example, as generally described herein.

[0089] The diaphragm 519 can be adapted for various types of imaging. For example, in one aspect, it may be desirable to observe a target surface, in which case an optically clear or otherwise optically transmissive material suitable for imaging may be used. In another aspect, visual information from the target surface may interfere with a three-dimensional reconstruction of the target surface. In this case, an optically opaque material may be preferred. In another aspect, the diaphragm may be opaque at some wavelengths and clear at other wavelengths, thereby permitting the use of a wavelength multiplexing imaging system to capture visual information from the target surface at one wavelength range while permitting three-dimensional reconstruction of the opaque surface at another wavelength range (e.g., using shaping from shadows, etc.).

[0090] The diaphragm 519 can use a variety of surface treatments for different imaging modes or techniques. For example, the diaphragm 519 can have a physical or optical textured surface to allow recovery of shape from motion or stereoscopic imaging, etc. In one aspect, the texture can be spectrally dependent, that is, the texture can be visible only in the visible light spectrum, only in the infrared spectrum, only in the short-wave infrared spectrum, or some combination of these. In this way, the diaphragm 519 can have different optical properties at different wavelengths. For example, the diaphragm 519 can be transparent in one or more wavelength ranges, opaque in one or more other wavelength ranges, and / or textured in one or more other wavelength ranges to facilitate a range of multiplexed imaging techniques. The diaphragm 519 can also or instead include a physical texture, such as an external texture, to control friction or allow trapped air at the contact interface to flow out, or to support texture-based imaging internally. More generally, the diaphragm 519 can be incorporated into a variety of functional coatings useful in different imaging contexts.

[0091] In one aspect, the diaphragm 519 can include a reflective surface or coating to permit imaging of light reflected from the region where the diaphragm 519 contacts the target surface back into the reservoir. It should be noted that reflectivity, i.e., the ratio of incident light to reflected light, can vary with wavelength. Thus, the diaphragm 519 can be completely or nearly completely reflective at one or more wavelengths or wavelength ranges, while completely absorbing and / or transmitting light at other wavelengths. Thus, reflectivity in this article should be understood to occur at a wavelength or wavelength range that matches the other imaging components of the system. In general, within such wavelengths, this can be any degree of reflectivity sufficient to facilitate image capture in support of three-dimensional reconstruction as described herein.

[0092] The diaphragm 519 can be structurally and / or functionally adapted to enhance imaging in a variety of ways. In one aspect, the diaphragm 519 can include a temperature-sensitive material or coating to permit direct optical observation of temperature or temperature changes across the contact surface. For example, the temperature-sensitive material can have one or more optical properties (e.g., color, opacity, transmittance, etc.) that vary with the temperature of the diaphragm 519. In another aspect, the diaphragm 519 can have internal and / or external treatments to facilitate wall thickness measurements of the diaphragm 519, which permits inferences about deformation, fluid pressure, and the like. For example, the diaphragm 519 can have one or more properties that vary as a function of wall thickness, including optical properties (color, opacity, etc.), electrical properties (e.g., conductivity), and the like. Other sensing modalities can also or instead be integrated into the diaphragm 519. For example, the diaphragm 519 can be configured to sense temperature, moisture, radiant heat, conductivity / resistance, electrostatic charge, and the like. For example, this can include optical properties that vary as a function of the sensed parameter of interest. This may also or instead include sensors or sensing materials that permit direct electrical measurement of the sensing parameter of interest, such as using a grid of sensing nodes embedded within the diaphragm. For example, with respect to contact force, the diaphragm 519 may infer the contact force based on the pressure within the reservoir and the contact area, or the diaphragm 519 may use a conductive material that changes resistance or conductive properties when exposed to pressure or strain. In another aspect, other techniques may be used to directly measure contact force at locations along the diaphragm 519.

[0093] In another aspect, the diaphragm 519 (and / or the containment wall(s) 520) can be used as a light guide to facilitate imaging. For example, the diaphragm 519 can include a light guide to guide light through the diaphragm 519 in a manner that the light enters at a light source and exits at any desired zone for directional illumination of the imaging medium. The interior surface of the diaphragm 519 can also or instead incorporate light extraction features or other light management features to filter light, introduce light, guide light, shape light, and / or expose light from an illumination surface for capture and processing. In addition, the diaphragm 519 can have multiple layers having refractive indices that facilitate light guiding in one layer and illumination in different layers.

[0094] More generally, fluid tactile sensors offer great flexibility in terms of shape (of the balloon or diaphragm enclosing the fluid), flexibility (by controlling the pressure in the reservoir 540 and / or the viscosity of the fluid medium), volume (by expanding or contracting the reservoir bounded by the elastic diaphragm), and the like. Similarly, the functional properties of the contact surface can be controlled by the choice of materials used to make and / or coat the diaphragm 519. Suitable materials for the flexible diaphragm include latex, silicone rubber, or any other material or combination of materials that can be manufactured as a thin coating on a substrate (which can have a shape selected according to the intended use of the sensor). Optical and mechanical properties, such as elasticity, can also be usefully controlled by controlling the thickness of the diaphragm 519. For example, a layer having a thickness of 1-10 microns can be readily manufactured using many common elastomers that have suitable elasticity and compliance for high resolution imaging.

[0095] The imaging medium 508 can be any fluid, such as a liquid or gas having suitable optical properties for imaging as contemplated herein. For example, this can include an optically transparent material that allows the transmission of one or more wavelengths of light waves used by an imaging system associated with the tactile sensor 502. In one aspect, this can include a clear gas such as air or an inert gas, or a clear liquid such as water or oil. However, it will be understood that in the case of envisioning alternative imaging techniques, an imaging medium with corresponding transmission properties may be suitable instead. For example, for ultrasound imaging, a sound-transmitting material can be used, even if the material is optically opaque. It will also be understood that the fluid can include any suitable combination of liquids, gases, and other phases, and / or materials in various transition states between gases and liquids. For example, steam can include evaporated water (e.g., primarily oxygen and nitrogen) in a gaseous state within air, which can be used as a fluid as described herein. Steam can also condense to form a mist or aerosol, which also behaves as a fluid suitable for use with the imaging techniques described herein. In one aspect, water condensation can be used to provide texture to the imaging surface, regulate temperature control pressure within the tactile sensor, or otherwise enhance the functionality of the tactile sensor as described herein. More generally, the gas can contain moisture such as water vapor that does not significantly affect the optical clarity of the gas, and the fluid can contain dissolved gases that similarly do not significantly affect the optical clarity. All such combinations are intended to fall within the scope of the fluid imaging medium as described herein.

[0096] The use of fluid as an imaging medium in the fluid layer 509 can provide many advantages over elastomers and / or gels. For example, the use of a fluid with suitable properties such as mechanical properties or physical properties can allow the imaging medium 508 to conform more closely to the target surface, especially when used in combination with a sufficiently soft diaphragm 519. This can facilitate the acquisition of surface data of high aspect ratio features such as beams and deep grooves or valleys. Fluid-based sensors can also penetrate deeper into the structure and allow access to negative angles or other occluded areas. In one aspect, the entire imaging device can be deflated, inserted into the chamber through the channel, and then inflated for use in imaging, as described in addition herein. This allows the use of corresponding contact-based imaging techniques in body cavities, containers, or other internal spaces that can only be accessed through smaller openings. In one aspect, the exterior of the diaphragm 519 can be coated with graphite or other low-friction surface treatments to facilitate access to such spaces, and / or improve the compliance of the diaphragm 519 with the target surface.

[0097] As another advantage, fluid can be added to or removed from the reservoir 540, thereby controlling the mechanical properties of the contact surface formed between the diaphragm and the target surface. For example, the pressure in the fluid layer 509 can be increased to support inverted use, or reduced to enhance compliance with the target surface. On the other hand, when imaging a non-rigid target such as a soft or flexible material, the pressure of the reservoir 540 can be changed between measurements, thereby providing information about the mechanical properties of the target surface. As a further advantage, the force applied to the diaphragm 519 by the target surface can be measured, for example, by measuring the pressure in the reservoir 540. This information can be used, for example, to control the movement of a robotic system associated with the tactile sensor 540, to derive an inference about the modulus of the target surface, or for any other purpose. On the other hand, pressure waves can be propagated through the fluid layer 509 and used, for example, to measure the dynamic mechanical properties of the target surface based on the way in which the target surface deforms in response to the propagated pressure waves.

[0098] The supply 542 may be a reservoir, canister, syringe, or other vessel that holds additional imaging medium 508 for addition to the reservoir 540. The supply 542 may also optionally recover the imaging medium 508 from the reservoir 540 when the tactile sensor 502 is deflated or depressurized. Although a single supply 542 is illustrated, it will be appreciated that the system 500 may include two or more supplies 542. This may be useful, for example, where different fluids or fluid types are exchanged within the fluid layer 509, for example, to change the refractive index of the fluid layer 509 relative to the window 505, in order to control illumination of the diaphragm 519 (e.g., by changing the difference between the refractive indices at the interface between the fluid layer 509 and the window 505).

[0099] The pump 544 can be any pump or other device for transferring the imaging medium 508 to and from the reservoir 540. This can include a syringe plunger, a rotary pump, a piston, a hydraulic pump, a peristaltic pump, and the like. In the case where the imaging medium 508 is a gas, flow control can be managed in other ways, such as by providing a pressurized gas tank as the supply 542, and using one or more valves to discharge the pressurized gas from the supply 542 into the reservoir 540 (to pressurize) and to discharge the gas from the reservoir 540 to the environment (to depressurize). In one aspect, the pump 544 can include two separate pumps and / or fluid paths, including a first fluid path for transferring the imaging medium 508 to the reservoir and a second fluid path for transferring the imaging medium 508 to the supply 542. In general, the pump 544 can control fluid delivery based on sensed pressure, for example, to maintain a target pressure, or to controllably increase or decrease the pressure over time. In another aspect, the pump 544 can control fluid delivery based on fluid volume. This may include managing fluid transfer to achieve a predetermined volume within the reservoir 540, which may be estimated based on the operation of the pump 544, or measured using a volumetric flow rate sensor or the like within the fluid path of the imaging medium 508. In another aspect, the volume within the reservoir 540 may remain substantially constant, and the pressure within the reservoir 540 may be measured over time and used as an input to an imaging process or other analysis of data acquired using the imaging system 500.

[0100] The one or more sensors 546 may include any sensor or combination of sensors suitable for monitoring the imaging medium 508 within the reservoir 540. For example, this may include a liquid or gas pressure sensor positioned to measure the pressure of the fluid within the reservoir 540 (or along a supply line to the reservoir 540). In another aspect, the pressure may be inferred by monitoring the pump 544, for example, to determine the amount of work associated with the transfer of fluid to (or out of) the reservoir 540. In another aspect, the one or more sensors 546 may include a temperature sensor for measuring the temperature of the fluid, the surface temperature of the reservoir, or any other temperature. This may be used, for example, to estimate heat transfer through the imaging medium, the temperature of a contact surface, and the like. In another aspect, the temperature of the imaging medium may be controlled, for example, to control the viscosity of the imaging medium, to apply heating or cooling to a target surface, or for any other useful thermal function.

[0101] The controller 516 may be used to collect data from and / or control the operation of other active components of the system, such as a pump 544 and one or more sensors 546. The controller 516 may include, for example, a microcontroller, a microprocessor, or other processor, and may be configured by computer executable code to perform the various functions described herein.

[0102] In another aspect, the one or more sensors 546 may include a temperature sensor that can be used, for example, to monitor the temperature of the imaging medium 508 within the reservoir 540 and / or within the supply 542. This can be useful for a variety of purposes, such as, for example, controlling the temperature of the fluid when used in contact with a temperature sensitive surface, or for estimating heat transfer to and from a target surface contacting the diaphragm 519. Where temperature control is desired, the system may also include a temperature controller 548, such as a heater, cooler, or a combination of these that can be controlled, for example, by the controller 516 to increase and / or decrease the temperature of the imaging medium 508. A variety of heat transfer systems are known in the art, including resistive heating systems, inductive heating systems, Peltier devices, evaporative cooling systems, and heat pumps, among others. These and other thermal management systems can be adapted by one of ordinary skill to heat and / or cool the imaging medium and provide a thermal controller 548 as described herein, with the selection of a particular system depending on a range of factors, such as the rate of heat transfer, the need for cooling and / or heating, the desired temperature range for temperature control, and the like. It will also be appreciated that a temperature controller may be used to control the temperature of the imaging media within the reservoir 508 , within the supply 542 of imaging media, and / or within the pump 544 or other hardware fluidly coupling the reservoir 508 to the supply 542 .

[0103] In one aspect, imaging hardware 550 may be positioned within imaging medium 508. For example, this may include an illumination source (such as a light emitting diode, laser, or other broadband or narrowband illumination source), a camera, a rangefinder, and the like, any of which may be advantageously positioned within reservoir 540 to provide improved positioning for imaging functions, optical distance measurements, and the like. Where multiple cameras are used, imaging hardware 550 may be channelized or multiplexed, for example, by wavelength, for improved resolution, multi-function imaging, and the like. In one aspect, diaphragm 519 may be optically transparent at a set of wavelengths, and imaging hardware 550 may be configured to capture images of the environment outside of diaphragm 519 in that wavelength range, permitting optical assessment of the background surrounding diaphragm 519 while collecting quantitative topological data at other wavelengths where diaphragm 519 is optically opaque, such as when in use on a robotic end effector, and the like. Optical fibers, electrical conductors, and the like may be used to control and / or collect data from each item of imaging hardware 550. In another aspect, some or all of the imaging hardware 550 may be self-powered, with a wireless data / control system, such that the imaging hardware 550 may operate within the reservoir 540 without requiring a physical connection for power and data.

[0104] The imaging hardware 550 may also support forming from motion or similar imaging techniques. For example, with the membrane 519 extending significantly from the containment wall(s) 520 along the axis 517, the membrane 519 may be placed in contact with the target surface, and the tactile sensor 502 may be moved around, e.g., laterally along the target surface, while the imaging hardware 550 captures images to facilitate extraction of surface information using forming from motion techniques, etc. This may also advantageously facilitate a combination of forming from motion measurements and photometric stereo measurements of the target surface.

[0105] In addition to the advantages described above, fluid tactile sensors can also provide many practical benefits in different contexts. In one aspect, the use of fluid as an imaging medium provides a wider range of material options and manufacturing options in situations where, for example, elastomeric gels are inappropriate or impractical. It is also possible to change the volume of the sensor, which allows use in a wide range of volume-sensitive imaging applications. The fluid tactile sensor can also control the normal contact force along a non-planar target surface in a more consistent and / or dynamic manner, which can be useful in a variety of situations, particularly where the target surface can include flexible or pliable features. More generally, the fluid tactile sensor allows a clear and dynamic compromise between contact force and signal quality for tactile sensing applications, for example by controlling the pressurization of the reservoir 540 in a manner that is generally impossible using gels.

[0106] In one aspect, fluid tactile sensors can be usefully applied in a medical context. For example, the ability to change volume, such as by reducing the volume of entry through a restricted access channel, permits use in minimally invasive medical procedures such as endoscopy or endoscopic examinations, or in medical imaging applications where relatively large internal volumes with concave surfaces are to be measured, such as the stomach, bladder, heart chambers, etc. The use of a highly flexible fluid imaging medium can enable improved soft tissue imaging, and with appropriate accompanying processing, a robotic alternative to palpitations or other manual exploration can be provided by measuring not only shape but also deformation in response to applied pressure. In this latter case, telemedicine can be improved by providing tactile information to medical professionals during remote health care sessions, or by providing a more complete force / shape description for remote surgical robots, etc.

[0107] Figure 6 A multi-sensor system 600 is shown. A plurality of fluid tactile sensors 602, such as any of the fluid tactile sensors described herein, can be attached to a base 604 and used as a polymeric surface sensor. The base 604 can be a rigid substrate, a flexible substrate that conforms to the polymeric shape of the contact surface, or an active substrate such as a robotic gripper with controllable shape and positioning. In one aspect, the base 604 can have a general shape such as a plane or a curved surface. In another aspect, the base 604 can be shaped to match a target surface of interest, such as the shape of a manufactured object to be inspected or measured. In this type of multi-sensor configuration, each liquid tactile sensor 602 can advantageously detect local pressure and / or shape, and can optionally control pressurization to support improved imaging performance and / or acquisition of dynamic surface data, such as a response to changing localized pressure.

[0108] The multi-sensor system 600 may include sensors 602 of any suitable shape, size, and arrangement. In one aspect, the sensors 602 may be arranged in a two-dimensional array for localized measurement across an extended two-dimensional target surface. The multi-sensor system 600 may also include one or more articulation joints 606 to allow the base 604 to bend and conform to the target surface. This may include, for example, a passive articulation joint 606 in a glove or other manually operated sensing system 600. This may also or instead include a passive articulation joint 606 that is spring-biased to a specific orientation or positioning so that the base 604 or a portion of the base 604 returns to a predetermined starting position without external force. This may also or instead include one or more actively controlled articulation joints 606 that may be actuated and controlled by a processor or other controller, etc. For example, the active articulation joint 606 may be driven by a rotary motor, a linear actuator, a pneumatic actuator, and the like together with a control wire or other force transmission mechanism for it, and may be used to computerize the positioning of one or more of the articulation joints 606. For example, in one aspect, the multi-sensor system 600 can be used as a robotic hand that can detect touch and contact force while also measuring the shape of the object being contacted and the accompanying contact force(s) to facilitate improved grasping and manipulation of the target object.

[0109] In one aspect, the base 604 can include a piece of clothing, such as a glove or shoe, and individual fluid tactile sensors 602 can be arranged on the surface of the piece of clothing to permit measurement of pressure and / or shape of a contact object. In one aspect, this can include the exterior of a glove, where the liquid tactile sensors 602 can be used to recover shape information. In another aspect, this can include internal sensors on a helmet or shoe to measure internal contact forces and / or deformations for use in managing safety or comfort. In one aspect, the piece of clothing can be equipped to monitor the positioning of each sensor 602 to permit acquisition of aggregated object shapes as well as detailed surface shapes on a sensor-by-sensor basis.

[0110] In another aspect, the base 604 may be a detachable base. In this embodiment, the base 604 may be able to be hinged, folded, bent, rolled up or otherwise reduced in shape and volume for deployment through a fixed-size access channel. For example, this may include a base 604 with one or more fluid tactile sensors 602, which may be flattened and rolled into a small cylinder for insertion through a minimally invasive surgical access channel or other small access routes. The reservoir of the one or more fluid tactile sensors 602 may then be pressurized, for example, using a remote supply of imaging media coupled to the reservoir by a pipeline, etc. and any one of the pumps described herein to expand (one or more) fluid tactile sensors 602, and cause the base to unfold or otherwise expand into a deployed sensor for use in capturing images. A variety of techniques for folding and expanding surgical hardware are known in the prior art, and these techniques may be used to create a fluid tactile sensor 602 that can be deployed through a minimally invasive access port or other similar constrained environment.

[0111] It will be appreciated that each fluid tactile sensor 602 (for single or multiple sensor configurations) can have a different three-dimensional shape for contacting a target surface, including different two-dimensional surface projections shaped as desired for surface coverage, and three-dimensional profiles selected for desired measurement ranges and consistency with the intended target surface. In one aspect, different sensors 602 can have different three-dimensional shapes to facilitate different types of uses and measurements. In another aspect, different bases 604 can have articulation joints 606 of different shapes and arrangements depending on the intended target surface, including, for example, a planar surface, a spherical surface, a hemispherical surface, or a composite surface adapted to a particular target object, and lenses and cameras arranged accordingly to provide optical access to the contact surface for illumination and imaging. It will be appreciated that, as described herein, the articulation joint 606 can include any moving or movable mechanical device or combination of devices to provide sufficient flexibility and freedom for the intended use.

[0112] Figure 7 and Figure 8A soft robotic gripper with an integrated fluid imaging medium is shown. In general, any of the fluid tactile sensors described herein can use a reservoir with a fluid imaging medium 704 as described herein to form a gripper 702. In addition to the fluid imaging medium 704, the reservoir can contain a plurality of beads 706 of optically clear transmissive material. The beads 706 can also advantageously be index matched to the fluid imaging medium 704, such that the reservoir is transparently clear through the beads 706 and the fluid imaging medium 704 during image capture. The beads 706 can be formed of, for example, any of the rigid optically clear materials described herein or any other suitable clear rigid index matching material.

[0113] In this general configuration, the reservoir can be reversely pressurized by extracting some of the fluid imaging medium 704, so that the beads 706 move together to frictionally engage with each other, so that the reservoir is transformed from a more ductile state to a more rigid state. The degree of ductility and rigidity will vary according to the type of diaphragm, the size, shape and distribution of the beads, the amount of pressurization, etc. However, the ductility of the clamp 702 can generally be increased by increasing the pressure until the beads 706 are disengaged from each other, and can generally be reduced by reducing the pressure until the beads 706 are mechanically engaged to a greater extent with each other. Conversely, the rigidity of the clamp 702 can generally be increased by reducing the pressure until the beads 706 are engaged with each other, and can generally be reduced by increasing the pressure until the beads 706 are disengaged from each other. Depending on the desired size and strength of the clamp and the shape of the object interacting with the clamp 702, any of a variety of shapes and sizes (and / or size distributions) of granular materials can be used as beads 706 to perform the blocking function.

[0114] Generally speaking, if Figure 8 As shown in , when the reservoir is pressurized, the gripper 702 can be placed against the object 710, for example, in a manner that permits low friction movement and flow of beads 706 relative to each other. In this configuration, the polymeric shape of the gripper 702 is generally malleable, and the gripper can conform to and surround the object 710. The shape of the object 710 can be imaged using any of the techniques described herein with the fluid imaging medium 704 and beads 706 (which are index matched to the medium 704 and optically transparent at the imaging wavelength of interest). Once the gripper 702 is positioned in contact with the object 710 (and optionally, any desired surface measurements are taken), some of the fluid imaging medium 704 can be extracted from the reservoir to reverse-pressurize the reservoir and move the beads ( Figure 8710) are pressed into mechanical engagement with each other so that the gripper 702 is transformed from a relatively soft, flowable state to a relatively fixed state. In this latter fixed state, the gripper 702 will tend to maintain its shape and can resist deformation sufficiently to support the load of a partially enclosed object (such as object 710), especially when the gripper 702 fully surrounds the object 710 to block a portion of the exit path of the object 710 from the gripper 702. The gripper 702 can then hold the object 710 and manipulate the object 710, for example, by picking up the object and moving it to a new location, or otherwise interacting with the object. In this context, the load bearing capacity will depend on the shape of the object 710, the frictional engagement between the object 710 and the gripper 702, the degree of reverse pressure, and other mechanical parameters, such as the shape and number of beads 706 and the strength of the diaphragm 712 of the gripper 702.

[0115] In accordance with the foregoing, in one aspect, the system described herein may include a processor configured by computer executable code to perform the steps of pressurizing a reservoir, such as any of the reservoirs described above, with an imaging medium to increase the ductility of the fluid tactile sensor, positioning the fluid tactile sensor on an object with a robotic handler, and clamping the object by depressurizing the reservoir to remove a portion of the imaging medium and increase the stiffness of the fluid tactile sensor on the surface engaging the object. In one aspect, the processor may pressurize and pressurize to a predetermined pressure target. In another aspect, the processor may sense volume, object movement, shape and degree of engagement between an object and a diaphragm, or any other useful property, and variably control pressurization and depressurization to achieve improved clamping and release.

[0116] The reservoir can then be re-pressurized to release the object 710. In general, throughout these transitions from a rigid to a flowable state, images and surface reconstructions can be obtained. However, where larger beads are used, reverse pressurization may pull regions of the diaphragm away from the target surface and into the beads, interfering with imaging resolution. More generally, the distribution of shapes and sizes of beads 706 can affect gripping performance, flowability, and imaging resolution, and can be optimized in various ways depending on the intended use.

[0117] It will be appreciated that other soft gripping techniques with pliable membranes are also known in the art and can be geometrically formed, for example, into cups, rings, etc. These other techniques can also be adapted for use with fluid tactile sensors as described herein to provide an actuator with a combination of three-dimensional, contact-based surface imaging and controlled gripping. In addition to capturing surface shape information, evaluation of the contact surface of the soft gripper can provide information about slippage (e.g., based on changes in the shape, position, or pressure field of the gripped object) or loss of suction / seal (based on vacuum measurements of the trapped air volume), etc., which can be used as feedback to control robotic actuation and determine, for example, when to release and reacquire an object.

[0118] In accordance with the foregoing, a system is disclosed herein, the system comprising a reservoir having a surface bounded by a flexible membrane; an optically transparent imaging medium within the reservoir; a plurality of particles within the reservoir, the particles being formed of an optically transparent material that is index matched to the optically transparent imaging medium within the reservoir over at least one wavelength range; and a pressurizing system configured to control the amount of the optically transparent imaging medium within the reservoir. The pressurizing system may be controllable, for example, using a processor configured by computer executable code to operate the pressurizing system and / or a robotic handler coupled to the reservoir to perform a soft robotic gripping function utilizing the flexible membrane. This may include automatically or manually manipulating the flexible membrane to contact a target object using the robotic handler, and then automatically or manually removing a portion of the optically transparent imaging medium from the reservoir using the pressurizing system so as to mechanically secure the target object using the surface of the flexible membrane. The system may also or instead include an imaging system configured to capture an image of the flexible membrane through the reservoir over at least one wavelength range.

[0119] Fig. 9 Methods for using a fluid tactile sensor are shown. Although the fluid tactile sensor can be used in fully manual tactile sensing applications, as described in the examples above, the increased flexibility permits various forms of data acquisition, computer control, and manipulation that are not possible with other sensors. Fig. 9, a variety of useful applications of fluid tactile sensors are described by way of non-limiting examples. It will be understood that in the following method 900, steps may be omitted, repeated, modified, or rearranged in various ways, depending on the specific context of use of the fluid tactile sensor. It will also be understood that the various steps described below may be performed or caused by computer executable code stored in a non-transitory computer-readable medium when executed on one or more computing devices. In this context, it will also be appreciated that different steps may be performed by different processors, such as where control of the fluid tactile sensor is maintained by a local processor or microcontroller, and where image processing is performed remotely by one or more computing instances in a remote cloud-based computing environment. Similarly, the system described herein may include one or more processors or other computing devices configured by such computer executable code to perform corresponding steps. All such variations are intended to fall within the scope of the present disclosure.

[0120] As shown in step 902, method 900 can begin by engaging the fluid tactile sensor with a target surface. In one aspect, this can include manually placing the fluid tactile sensor into contact with the object, for example using a handheld sensing system. In another aspect, this can include computer-assisted positioning of the sensor, such as by manually steering the fluid tactile sensor into engagement with the object using a robotic handler. This can also or instead include directing the robotic handler to automatically position the object and / or steering the fluid tactile sensor into appropriate engagement with the object, for example by identifying and contacting a suitable target surface for additional action.

[0121] As shown in step 904, the method may include sensing parameters associated with imaging the target surface. This may include collecting image data. This may also or alternatively include using any of the other sensors described herein to collect data. For example, this may include sensing the surface temperature (or temperature field) of the target surface, or the temperature of the imaging medium of the reservoir filling the diaphragm adjacent to the fluid tactile sensor. The latter temperature may be measured, for example, in the reservoir, in the supply of imaging medium outside the reservoir, and / or at one or more other locations. In another aspect, the sensing parameter may include sensing the pressure of the imaging medium in the reservoir. This may be useful for extracting pressure or normal force data, which may characterize the amount and distribution of force applied to the target surface of the object. This may also or alternatively be used to control the volume elasticity of the fluid tactile sensor, which may affect the resolution of the surface data, the deformation of the target surface, or other aspects of the imaging environment and data acquisition. In another aspect, this may include measuring the volume of the imaging medium in the form of a fluid, etc. In addition to temperature, pressure, and volume, sensors or sensing systems may be deployed to measure, for example, the thickness of the diaphragm, the deformation of the diaphragm, and the electrical properties of the diaphragm, among others, any of which may be used to enhance the use of the fluid tactile sensor as described herein.

[0122] As shown in step 906, method 900 may include controlling parameters associated with imaging the target surface. This may generally include controlling temperature, pressure, volume, and the like as described above. Other features may also be usefully observed and controlled, such as the viscosity of the imaging medium. This may also or instead include controlling illumination, data acquisition, focusing, and other properties associated with image capture for three-dimensional reconstruction. This may also or instead include controlling the positioning of a robotic handler, or any other parameter or output that may be usefully controlled for the imaging application described herein.

[0123] As shown in step 908, method 900 may include acquiring image data. This may generally include imaging using any of the techniques described herein, and may include controlling, for example, focusing, illumination, and the like as described herein to support such imaging. In another aspect, this may include selecting imaging parameters, such as data rate, imaging modality, and the like, which may be automatically controlled based on known or observed properties of the target surface (e.g., z-axis range, feature size, etc.). In the case where non-optical data (e.g., temperature field, etc.) is available, the data may also be captured and stored with the image data to support additional processing. For example, surface temperature may provide useful information in detecting specific electrical or mechanical faults, performing medical examinations, and the like, and may be used to enhance visual data for smart sensing applications using fluid tactile sensors.

[0124] In one aspect, the image can provide location-specific data of the target surface. For example, in the case where the membrane is processed to have a surface with optical properties, this can include temperature data, moisture data, pressure data, etc., and the optical properties vary based on the corresponding physical properties of the target surface contacted by the membrane. Thus, for example, using a suitably configured membrane, the image data can include a two-dimensional temperature field, a pressure field (e.g., based on optically detectable local deformations of the membrane, humidity or moisture contact, etc.) of the target surface. These supplementary data sets can also or instead be collected by supplementary hardware. For example, an infrared camera can be pointed at the membrane and used to measure the spatial temperature field. Other types of measurements can also or instead be captured.

[0125] In one aspect, surface friction can be measured. To obtain this type of measurement, a shear or sliding force can be applied to the imaging system in a direction tangential to the target surface (after contact is made with the target surface). The amount of displacement and shear that occurs locally in the diaphragm (e.g., in the case of an optical change such as a deformation of the diaphragm or based on a pattern on the diaphragm), along with the normal force measurement, can provide data to calculate the coefficient of friction of the target surface. Similarly, an elastic diaphragm can be pressurized or expanded, and similar techniques can be used to detect sliding along an edge, where the elastic diaphragm expands outward along the perimeter of the target surface contact area.

[0126] As shown in step 910, method 900 may include processing the image data. In general, this may include processing any data collected by the system described herein, and extracting information such as quantitative topological data. This may also or instead include extraction of flexibility data or other mechanical properties of the target surface, which may be measured or inferred based on the response of the target surface to contact forces of varying amounts and directions.

[0127] As shown in step 912, method 900 may include taking additional actions. This may include presenting data, such as by displaying an image of the target surface based on quantitative surface data collected, for example, by one or more fluid tactile sensors. In another aspect, this may include adjusting imaging parameters or other controllable parameters of the imaging system based on the measurements. This may also or instead include changing the imaging modality, selecting a different imaging medium, selecting a different fluid tactile sensor with a different diaphragm, etc. This may also or instead include performing additional processing, such as object identification using machine learning, decision making for example for a manufacturing or testing facility, or controlling the operation of a robotic actuator that grips an object based on tactile feedback from one or more fluid tactile sensors in contact with the object.

[0128] These and other steps may be repeated as necessary or helpful for a particular imaging or control application.More generally, unless specifically stated otherwise, the various features and techniques described herein may be used alone or in any suitable combination without departing from the scope of the present disclosure.

[0129] The above systems, devices, methods, processes and the like can be implemented in hardware, software or any combination of these suitable for a specific application. Hardware can include general-purpose computers and / or special computing devices. This can include implementation in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices or processing circuits together with internal and / or external memory. This can also or instead include one or more application-specific integrated circuits, programmable gate arrays, programmable array logic components, or any other (one or more) devices that can be configured to process electronic signals. It will be further appreciated that the implementation of the process or device described above can include computer executable code created using a structured programming language such as C, an object-oriented programming language such as C++, or any other high-level or low-level programming language (including assembly language, hardware description language, and database programming language and technology), which can be stored, compiled or interpreted to run on one of the above devices and a heterogeneous combination of processors, processor architectures, or combinations of different hardware and software. In another aspect, the method can be embodied in a system that performs its steps, and can be distributed across devices in multiple ways. At the same time, processing can be distributed across devices such as various systems described above, or all functionality can be integrated into dedicated stand-alone devices or other hardware. In another aspect, means for performing the steps associated with the processes described above may include any of the hardware and / or software described above. All such permutations and combinations are intended to fall within the scope of the present disclosure.

[0130] Embodiments disclosed herein may include a computer program product including computer executable or computer usable code that, when executed on one or more computing devices, performs any and / or all of the steps thereof. The code may be stored in a non-transitory manner in a computer memory, which may be a memory from which a program is executed (such as a random access memory associated with a processor), or a storage device such as a disk drive, flash memory, or any other optical, electromagnetic, magnetic, infrared or other device or combination of devices. In another aspect, any of the systems and methods described above may be embodied in any suitable transmission or propagation medium that carries the computer executable code and / or any input or output from the computer executable code.

[0131] It will be appreciated that the apparatus, systems, and methods described above are set forth by way of example and not limitation. Without departing from the scope of the present disclosure, the disclosed steps may be modified, supplemented, omitted, and / or reordered if there is no clear indication to the contrary. Many variations, additions, omissions, and other modifications will be apparent to one of ordinary skill in the art. Furthermore, the order or presentation of the method steps in the above description and accompanying drawings is not intended to require that the enumerated steps be performed in that order unless a particular order is expressly required or otherwise clear from the context.

[0132] The method steps of the implementation described herein are intended to include any suitable method for causing such method steps to be performed, consistent with the patentability of the following claims, unless a different meaning is explicitly provided or otherwise clear from the context. Thus, for example, performing step X may include any suitable method for causing another party such as a remote user, a remote processing resource (e.g., a server or cloud computer), or a machine to perform step X. Similarly, performing steps X, Y, and Z may include any method for directing or controlling any combination of such other individuals or resources to perform steps X, Y, and Z to obtain the benefits of such steps. Therefore, the method steps of the implementation described herein are intended to include any suitable method for causing one or more other parties or entities to perform the steps consistent with the patentability of the following claims, unless a different meaning is explicitly provided or otherwise clear from the context. Such a party or entity does not need to be directed or controlled by any other party or entity, and does not need to be located in a specific jurisdiction.

[0133] It should be further appreciated that the above methods are provided by way of example. Without departing from the scope of the present disclosure, if there is no explicit indication to the contrary, the disclosed steps may be modified, supplemented, omitted and / or reordered.

[0134] It will be appreciated that the methods and systems described above are set forth by way of example and not limitation. Many variations, additions, omissions, and other modifications will be apparent to one of ordinary skill in the art. In addition, the order or presentation of the method steps in the above description and accompanying drawings is not intended to require that the steps recited be performed in that order unless a particular order is expressly required or otherwise clear from the context. Therefore, although specific embodiments have been shown and described, it will be apparent to those skilled in the art that various changes and modifications may be made in form and detail without departing from the spirit and scope of the present disclosure, and that the changes and modifications are intended to form a part of the present invention as defined by the following claims, which will be interpreted in the broadest sense permissible by law.

Claims

1. A system comprising: Fluid tactile sensor, including: a substrate formed of a rigid material and comprising a window of optically transparent material, a diaphragm formed from a sheet of flexible, resilient material and having a surface, a reservoir having a volume at least partially enclosed within the substrate and the diaphragm, and an imaging medium within the reservoir, the imaging medium comprising an optically transparent fluid; and Imaging system, including: an illumination source directed toward a surface of the diaphragm through the window of the substrate when the diaphragm is positioned for use within an imaging volume of an imaging system, and An imaging device is positioned to capture an image of a surface of the diaphragm through the window of the substrate when the diaphragm is positioned for use within an imaging volume of an imaging system.

2. The system of claim 1, further comprising a processor configured by computer executable code stored in the memory to acquire images from the imaging system and calculate quantitative surface topography of the target surface contacting the membrane.

3. The system of any one of claims 1-2, wherein the illumination source comprises a structured light source.

4. The system of claim 3, wherein the structured light source creates a three-dimensional illumination pattern within an imaging volume of the reservoir.

5. The system of any one of claims 1-4, further comprising a cartridge housing the fluid tactile sensor, the cartridge being removable from and replaceable to the imaging system.

6. The system of any one of claims 1-5, further comprising one or more containment walls mechanically coupling the diaphragm to the substrate.

7. The system of any one of claims 1-6, wherein the window comprises at least one of glass, polycarbonate, acrylic, polystyrene, polyurethane, or optically clear epoxy.

8. The system of any one of claims 1-7, wherein the window has a first refractive index that matches a second refractive index of the imaging medium within the reservoir.

9. The system of any one of claims 1-8, wherein the imaging medium comprises at least one of a gas and a liquid.

10. The system of any one of claims 1-9, wherein the diaphragm is formed from an elastic polymer.

11. The system of any one of claims 1-10, further comprising an optical pattern on the membrane, the optical pattern visible to an imaging device of the imaging system through the window.

12. The system of any one of claims 1-11, wherein the illumination source comprises at least one of a laser and a light emitting diode.

13. The system of any one of claims 1-12, wherein the imaging system uses two or more imaging modalities, including at least one of photometric stereo imaging, multi-view stereo imaging, structured light imaging, and focal stacking.

14. The system of any one of claims 1-13, further comprising a pressure sensor coupled to the reservoir and configured to measure a pressure of the imaging medium within the reservoir.

15. A tactile sensor, comprising: a substrate comprising a window of rigid optically transparent material; a diaphragm formed of a flexible material, the diaphragm being coupled to the substrate to form a reservoir visible through a window of the substrate; and Optically clear fluid in reservoir.

16. The tactile sensor of claim 15, wherein the optically transparent fluid has a first refractive index that matches a second refractive index of the window to facilitate imaging of the membrane through the window.

17. The tactile sensor according to any one of claims 1 to 16, wherein the membrane has an optically reflective surface.

18. The tactile sensor according to any one of claims 1 to 17, wherein the membrane has a patterned surface.

19. The tactile sensor according to any one of claims 1 to 18, wherein the membrane has an outer surface facing away from the window, the outer surface comprising a friction reducing coating.

20. The tactile sensor of any one of claims 1-19, wherein the window comprises one or more light shaping features for at least one of filtering, polarizing, focusing or diffusing light passing through the window.

21. A system comprising: Fluid tactile sensor, including: a substrate formed of a rigid material and comprising a window of optically transparent material, a diaphragm formed from a sheet of flexible, resilient material and having a surface, a reservoir having a volume enclosed within the substrate and the diaphragm, and an imaging medium within the reservoir, the imaging medium comprising an optically transparent fluid; Fluid management systems, including: supply of imaging media, and a pump coupled in a fluid path between the supply of imaging medium and the reservoir, the pump being configured to controllably deliver the imaging medium from the supply to the reservoir; and Imaging system, including: an illumination source directed toward a surface of the diaphragm through the window of the substrate when the diaphragm is positioned for use within an imaging volume of an imaging system, and An imaging device is positioned to capture an image of the surface of the diaphragm through the window of the substrate when the diaphragm is positioned for use within an imaging volume of an imaging system.

22. The system of claim 21, further comprising one or more sensors for measuring the pressure of the imaging medium within the reservoir.

23. The system of claim 22, further comprising a controller configured to operate a pump to transfer imaging media between a supply and a reservoir in response to signals from the one or more sensors.

24. The system of any one of claims 21-23, wherein the controller is configured to operate the pump to maintain the pressure at a predetermined target pressure.

25. The system of any one of claims 21-24, wherein the volume of imaging medium is maintained constant within the reservoir, and wherein the pressure is measured using the one or more sensors and stored with one or more images captured by the imaging system for use in processing the one or more images.

26. The system of any one of claims 21-25, wherein the fluid tactile sensor comprises one or more containment walls coupling a substrate with a diaphragm to form a fluid reservoir.

27. The system of any one of claims 21-26, further comprising one or more illumination sources positioned within the reservoir and directed toward the diaphragm.

28. The system of any of claims 21-27, wherein the window has a first refractive index that matches a second refractive index of the imaging medium.

29. The system of any one of claims 21-28, wherein the imaging device comprises one or more cameras.

30. The system of any one of claims 21-29, wherein the illumination source comprises at least one of a laser and a light emitting diode.

31. The system of any one of claims 21-30, wherein the imaging medium comprises at least one of a gas and a liquid.

32. The system of any of claims 21-31, wherein the window comprises at least one of glass, polycarbonate, acrylic, polystyrene, polyurethane, or optically clear epoxy.

33. The system of any one of claims 21-32, further comprising one or more temperature sensors for monitoring the temperature of the imaging medium.

34. The system of claim 33, further comprising a temperature controller operable by the controller to control the temperature of the imaging medium in response to signals from the one or more temperature sensors.

35. The system of any one of claims 21-34, wherein the membrane has one or more optical properties that vary with the temperature of the membrane.

36. The system of any one of claims 21-35, wherein the diaphragm has one or more optical properties that vary with deformation of the diaphragm.

37. The system of any one of claims 21-36, wherein the diaphragm is configured to sense one or more sensed parameters at a target surface contacted by the diaphragm.

38. The system of any one of claims 21-37, wherein the imaging system uses two or more imaging modalities, including at least one of photometric stereo imaging, multi-view stereo imaging, structured light imaging, and focal stacking.

39. The system of any one of claims 21-38, wherein the diaphragm has an optically reflective surface.

40. The system of any one of claims 21-39, wherein the diaphragm is optically clear at one or more wavelength ranges collected by an imaging system.

41. A system comprising: Fluid tactile sensor, including: a substrate formed of a rigid material and comprising a window of optically transparent material, a diaphragm formed from a sheet of flexible, resilient material and having a surface, a reservoir having a volume enclosed within the substrate and the diaphragm, and an imaging medium within the reservoir, the imaging medium comprising an optically transparent fluid; a robotic handler coupled to the fluid tactile sensor; and Imaging system, including: an illumination source directed toward a surface of the diaphragm through the window of the substrate when the diaphragm is positioned for use within an imaging volume of an imaging system, and An imaging device is positioned to capture an image of a surface of the diaphragm through the window of the substrate when the diaphragm is positioned for use within an imaging volume of an imaging system.

42. The system of claim 41, wherein the fluid tactile sensor is removably and replaceably coupled to a robotic handler.

43. The system of any one of claims 41-42, wherein the robotic handler is configured to automatically remove the fluid tactile sensor and replace the fluid tactile sensor with a replacement sensor.

44. The system of any one of claims 41-43, further comprising a display that presents a visualization of at least one of a pressure field, a contact force field, and a surface topology acquired by the imaging system.

45. The system of any one of claims 41-44, further comprising a machine learning model configured to identify an object contacted by the fluid tactile sensor based on data acquired from the imaging system.

46. ​​The system of any one of claims 41-45, further comprising a machine learning model configured to automatically select an action for the robotic handler to perform on a workpiece contacted by the fluid tactile sensor.

47. The system of any one of claims 41-46, further comprising a plurality of fluid tactile sensors disposed on the base and coupled to the robotic handler.

48. The system of claim 47, wherein the base comprises a rigid substrate.

49. The system of claim 47, wherein the base comprises a flexible substrate.

50. The system of claim 47, wherein the base comprises an active substrate having a controllable shape.

51. The system of any one of claims 47-50, wherein each of the plurality of fluid tactile sensors has a controllable pressure.

52. The system of any one of claims 47-51, wherein the plurality of fluid tactile sensors are arranged in a two-dimensional array for localized measurements across a two-dimensional target surface.

53. The system of any one of claims 47-52, wherein the base comprises one or more articulation joints.

54. The system of claim 53, wherein the one or more articulation joints include at least one passive articulation joint.

55. The system of any one of claims 53-54, wherein the one or more articulation joints include at least one active articulation joint having a computer-controllable position or orientation.

56. The system of any one of claims 41-55, further comprising a processor configured by computer executable code stored in the memory to acquire images from the imaging system and calculate quantitative surface topography of the target surface contacting the membrane.

57. The system of any one of claims 41-56, further comprising a processor configured by computer executable code stored in the memory to acquire pressure data from a pressure sensor coupled to the reservoir and, in response to the pressure data, provide a control signal to a pump to transfer fluid between the reservoir and a supply of imaging media.

58. The system of any one of claims 41-57, further comprising a processor configured by computer executable code stored in the memory to acquire temperature data from a temperature sensor coupled to the reservoir and, in response to the temperature data, provide a control signal to the thermal controller to heat or cool the imaging medium.

59. The system of any one of claims 41-58, wherein the imaging system uses two or more imaging modalities, including at least one of photometric stereo imaging, multi-view stereo imaging, structured light imaging, and focal stacking.

60. The system of any of claims 41-59, wherein the window has a first refractive index that matches a second refractive index of the imaging medium.

61. A system comprising: Fluid tactile sensor, including: a substrate formed of a rigid material and comprising a window of optically transparent material, a diaphragm formed from a sheet of flexible, resilient material and having a surface, a reservoir having a volume enclosed within the substrate and the diaphragm, an imaging medium within the reservoir, the imaging medium comprising an optically transparent fluid, and a plurality of beads within the reservoir, the beads formed of an optically transparent material having a refractive index matched to the imaging medium within the reservoir over at least one wavelength range; Supply of imaging media; Pumps; wherein the pump is coupled in fluid communication between the supply and the reservoir, and wherein the pump is configured to transfer imaging media between the reservoir and the supply; and an imaging system comprising: an illumination source directed toward a surface of the diaphragm through the window of the substrate when the diaphragm is positioned for use within an imaging volume of an imaging system, and An imaging device is positioned to capture an image of the surface of the diaphragm through the window of the substrate when the diaphragm is positioned for use within an imaging volume of an imaging system.

62. The system of claim 61, further comprising a robotic handler coupled to the fluid tactile sensor.

63. The system of claim 62, wherein the fluid tactile sensor is removably and replaceably coupled to a housing of a robotic handler.

64. The system of any one of claims 61-63, further comprising a processor configured by computer executable code to perform soft robotic gripping functions utilizing a fluid tactile sensor.

65. The system of any one of claims 61-64, further comprising a robotic handler and a processor, the robotic handler being coupled to the fluid tactile sensor, and the processor being configured by the computer executable code to perform the steps of: pressurizing the reservoir with imaging media to increase the ductility of the fluid tactile sensor; positioning the fluid tactile sensor on the object using a robotic handler; and The object is clamped by depressurizing the reservoir to remove a portion of the imaging media and increase the stiffness of the fluid tactile sensor on the portion of the diaphragm that engages the object.

66. The system of claim 65, wherein the processor is further configured to acquire one or more images of the object through a window at the one or more wavelength ranges, wherein the plurality of beads are optically transparent and refractive index matched to an imaging medium.

67. The system of claim 61 further comprising a processor configured by computer executable code stored in the memory to acquire images from the imaging system and calculate quantitative surface topography of the target surface contacting the membrane.

68. The system of claim 61, wherein the imaging medium comprises a gas.

69. The system of claim 61, wherein the imaging medium comprises a liquid.

70. The system of any one of claims 61-69, wherein the illumination source comprises at least one of a laser and a light emitting diode.

71. The system of any one of claims 61-70, wherein the imaging system uses two or more imaging modalities, including at least one of photometric stereo, multi-view stereo, structured light imaging, and focal stacking.

72. The system of any one of claims 61-71, further comprising a pressure sensor coupled to the reservoir and configured to measure a pressure of the imaging medium within the reservoir.

73. The system of claim 72, further comprising a processor configured to control the pump in response to a signal received from the pressure sensor.

74. The system of any one of claims 61-73, wherein the diaphragm has at least one of an optically reflective surface and a patterned surface.

75. The system of any of claims 61-74, further comprising a display that presents a visualization of at least one of a pressure field, a contact force field, and a surface topology acquired by the imaging system of an object contacting the diaphragm.

76. The system of any one of claims 61-75, further comprising a machine learning model configured to identify an object contacted by the fluid tactile sensor based on data acquired from the imaging system.

77. The system of any one of claims 61-76, wherein the window comprises at least one of glass, polycarbonate, acrylic, polystyrene, polyurethane, or optically clear epoxy.

78. The system of any one of claims 61-77, wherein the window has a first refractive index that matches a second refractive index of the imaging medium within the reservoir.

79. The system of any one of claims 61-78, wherein the diaphragm is formed from an elastic polymer.

80. The system of any one of claims 61-79, further comprising an optical pattern on a portion of the diaphragm facing the window.

Citation Information

Patent Citations

  • Surface topography measurement systems

    US10965854B2

  • Intent analysis for call center response generation

    US20220046129A1

  • Tactile sensor using elastomeric imaging

    US8411140B2

  • High-resolution surface measurement systems and methods

    US9127938B2

Cited By

  • Self-adaptive visual tactile sensor and adjusting method

    CN120264120A