Tactile sensor device and electronic appratus

Abstract

A tactile sensor device according to a mode of one embodiment of the present disclosure comprises: a substrate on which one or a plurality of lenses are formed; and an EVS element facing the one or the plurality of lenses. This tactile sensor device further comprises: a deformation layer disposed at a position on the reverse side from the EVS element in terms of a positional relationship to the substrate; a plurality of markers disposed on the surface of the deformation layer or inside the deformation layer; and a plurality of light sources that irradiate light on the plurality of markers. Among the plurality of markers, a plurality of first markers that is at least a portion of the plurality of markers has a pattern of in-plane anisotropy.

Description

Tactile sensor device and electronic device

[0001] The present disclosure relates to a tactile sensor device and an electronic device.

[0002] Many sensors are used in robots to control the handling of objects by the robot. Sensors that can be used in robots are disclosed in, for example, Patent Documents 1 to 3 listed below.

[0003] JP 2019-509494 A JP 2007-518966 A JP 2021-154412 A

[0004] To address the slow responsiveness of robots due to the sensor's frame rate, an event-based vision sensor (EVS) camera is sometimes used as the camera built into the sensor. However, when the position of a marker built into the sensor changes, an EVS camera loses track of the marker because it is unable to determine the correspondence between the marker before and after the change. This problem is particularly prevalent when only positive events (on events) are used. It would be desirable to provide a tactile sensor device that can reduce the possibility of losing track of the marker built into the sensor, and an electronic device equipped with such a tactile sensor device.

[0005] A tactile sensor device according to a first aspect of the present disclosure includes a substrate on which one or more lenses are formed and an EVS element facing the one or more lenses. The tactile sensor device further includes a deformation layer disposed on the opposite side of the substrate from the EVS element, a plurality of markers disposed on the surface of the deformation layer or inside the deformation layer, and a plurality of light sources that irradiate the plurality of markers with light. At least some of the plurality of markers, namely, a plurality of first markers, have a pattern with in-plane anisotropy.

[0006] An electronic device according to a second aspect of the present disclosure includes one or more first tactile sensor devices. The one or more tactile sensor devices include a substrate on which one or more lenses are formed and an EVS element facing the one or more lenses. The tactile sensor device further includes a deformation layer disposed on the opposite side of the substrate from the EVS element, a plurality of markers disposed on the surface of or within the deformation layer, and a plurality of light sources that irradiate the plurality of markers with light. At least some of the plurality of markers, the plurality of first markers, have a pattern with in-plane anisotropy.

[0007] FIG. 1 is a diagram illustrating an example cross-sectional configuration of a tactile sensor device according to a first embodiment of the present disclosure. FIG. 2 is a diagram illustrating light reception in the tactile sensor device of FIG. 1. FIG. 3 is a diagram illustrating an example functional block of the light-receiving element of FIG. 1. FIG. 4 is a diagram illustrating an example functional block of the light-receiving pixel of FIG. 3. FIG. 5 is a diagram illustrating an example planar layout of the marker of FIG. 1. FIG. 6 is a diagram illustrating another example planar layout of the marker of FIG. 1. FIG. 7 is a diagram illustrating an example of dimensions of multiple markers included in the marker of FIG. 1. FIGS. 8(A) to 8(C) are diagrams illustrating an example of marker positions and EVS output (event image data) in a tactile sensor device according to a comparative example. FIGS. 9(A) to 9(C) are diagrams illustrating an example of marker positions and EVS output (event image data) in a tactile sensor device according to an example. FIGS. 10(A) to 10(C) are diagrams illustrating an example of marker positions and EVS output (event image data) in a tactile sensor device according to an example. FIG. 11 is a diagram showing a modified planar layout of the marker in FIG. 1 . FIG. 12 is a diagram showing a modified planar layout of the marker in FIG. 1 . FIG. 13 is a diagram showing a modified planar layout of the marker in FIG. 1 . FIG. 14 is a diagram showing a modified planar layout of the marker in FIG. 1 . FIG. 15 is a diagram showing a modified cross-sectional configuration of the tactile sensor device in FIG. 1 . FIG. 16 is a diagram showing light reception in the tactile sensor device in FIG. 15 . FIG. 17 is a diagram showing an example cross-sectional configuration of a tactile sensor device according to a second embodiment of the present disclosure. FIG. 18 is a diagram showing a modified cross-sectional configuration of the tactile sensor device in FIG. 1 . FIG. 19 is a diagram showing a modified cross-sectional configuration of the tactile sensor device in FIG. 15 . FIG. 20 is a diagram showing an example external appearance of a game controller in which the tactile sensor devices in FIGS. 1 and 15 are applied to a gripper. FIG. 21 is a diagram showing an example external appearance of a robot device in which the tactile sensor devices in FIGS. 1 and 15 are applied to the tip portion of a robot arm device. Figures 22(A) and 22(B) are diagrams showing an example of a procedure for calibrating a force sensor provided at the tip of the robot arm device in Figure 21. Figure 23 is a diagram showing another example of calibration of a force sensor provided at the tip of the robot arm device in Figure 21. Figure 24 is a diagram showing an example of the appearance of a 3D model controller to which the tactile sensor device in Figure 19 is applied.Fig. 25 is a diagram showing an example of the appearance of a cylindrical controller to which the tactile sensor device of Fig. 1 and Fig. 15 is applied. Fig. 26 is a diagram showing an example of the appearance of a head-mounted display that can be used instead of the information processing device of Fig. 24 and Fig. 25.

[0008] Hereinafter, embodiments for carrying out the present disclosure will be described in detail with reference to the drawings. The following description is one specific example of the present disclosure, and the present disclosure is not limited to the following embodiment. Furthermore, the present disclosure is not limited to the arrangement, dimensions, dimensional ratios, etc. of each component shown in each drawing. The description will be given in the following order.

[0009] 1. First embodiment (tactile sensor device) Example of receiving light through a lens (Figs. 1 to 10) 2. Modified examples (tactile sensor device) Modified example A: Example of configuring multiple markers in a non-repeated pattern (Fig. 11) Modified example B: Example of configuring multiple markers with multiple types of markers (Figs. 12 to 14) Modified example C: Example of a configuration capable of stereo distance measurement (Figs. 15 and 16) Modified example D: Example of receiving light through a lens array (Figs. 12 and 13) 3. Second embodiment (tactile sensor device) Example of sharing one deformation layer with multiple light receiving devices (Fig. 17) 4. Modified examples common to all embodiments (tactile sensor device) Example of minimizing air gaps (Figs. 18 and 19) 5. Application Examples Application Example 1: An example of applying the tactile sensor device to a game controller (Fig. 20) Application Example 2: An example of applying the tactile sensor device to a robot device (Figs. 21 to 23) Application Example 3: An example of applying the tactile sensor device to a 3D model controller (Fig. 24) Application Example 4: An example of applying the tactile sensor device to a cylindrical controller (Fig. 25) Application Example 5: An example of using an HMD instead of an information processing device in Application Examples 4 and 5 (Fig. 26)

[0010] 1. First Embodiment [Configuration] A tactile sensor device 1 according to a first embodiment of the present disclosure will be described. FIG. 1 illustrates an example cross-sectional configuration of the tactile sensor device 1. FIG. 2 illustrates light reception in the tactile sensor device 1. As shown in FIG. 1 , the tactile sensor device 1 includes, for example, a support substrate 10, a plurality of light-emitting elements 20, a light-receiving element 30, a light-emitting driver 21, a light-receiving driver 31, a lens substrate 40, a controller 50, a wiring substrate 60, a deformation layer 70, a light-shielding layer 80, and a marker 90. The plurality of light-emitting elements 20 correspond to a specific example of "plurality of light sources" in the present disclosure. The light-receiving element 30 corresponds to a specific example of "EVS element" in the present disclosure. The deformation layer 70 corresponds to a specific example of "deformation layer" in the present disclosure.

[0011] (Support substrate 10) The support substrate 10 has supports 11 and 12 and spacers 13 and 14. The support 12 corresponds to a specific example of "substrate" in the present disclosure. The support 11 is a substrate that supports the deformation layer 70. The support 12 is a substrate that supports the plurality of light-emitting elements 20 and the lens substrate 40. The spacer 13 is a frame that ensures a gap G between the support 11 and the support 12. By providing the spacer 13, a gap G is provided between the support 11 and the lens 41 (described below) and each of the light-emitting elements 20 (described below). The spacer 14 is a frame or support that ensures a gap between the support 12 and the wiring substrate 60.

[0012] The support 11 is a substrate, such as a glass substrate, made of a material (transparent material) that is at least transparent to the light emitted from the light emitting element 30. The support 11 only needs to have enough light transparency to realize the functions of the tactile sensor device 1.

[0013] The support 12 is a circuit board on which wiring connected to the plurality of light-emitting elements 20 is formed. The support 12 is made of a material (opaque material) that has a light-blocking property against at least the light emitted from the light-emitting elements 30. The support 12 is fixed to the support 11 via a spacer 13. The support 12 has an opening H at a location facing the lens substrate 40 (particularly the lens 41 described below). The opening H has, for example, a circular shape. The spacer 13 is made of a material (opaque material) that has a light-blocking property against at least the light emitted from the light-emitting elements 30. The spacer 14 may be made of a material (opaque material) that has a light-blocking property against at least the light emitted from the light-emitting elements 30.

[0014] (Light-emitting elements 20) The light-emitting elements 20 are, for example, semiconductor light-emitting diodes (LEDs) or organic light-emitting diodes (OLEDs). The plurality of light-emitting elements 20 are mounted on the surface of the support 12 facing the support 11. The plurality of light-emitting elements 20 are arranged around the opening H on the surface of the support 11. The plurality of light-emitting elements 20 are electrically connected to the light-emitting driver 21 via the support 12.

[0015] The light-emitting elements 20 irradiate the markers 90 and the light-shielding layer 80 with light through the support 11 and the deformation layer 70. The light-emitting elements 20 emit light at a predetermined divergence angle with respect to the normal direction of the support 11. The light-emitting elements 20 are, for example, elements that emit light in the visible region. The light-emitting elements 20 may also be, for example, elements that emit light outside the visible region (for example, in the infrared region). The light-emitting driver 21 is an IC chip that controls the emission and extinction of the multiple light-emitting elements 20. The light-emitting driver 21 is, for example, mounted on the wiring board 60.

[0016] (Light-Receiving Element 30) The light-receiving element 30 is disposed opposite the lens substrate 40 (particularly the lens 41) with a predetermined gap therebetween. The light-receiving element 30 is disposed at a position opposite the opening H with a predetermined gap therebetween. The light-receiving element 30 detects light incident through the lens substrate 40 (particularly the lens 41), for example, as indicated by the dashed line in FIG. 2 . The light-receiving element 30 is, for example, an event-based vision sensor (EVS) element that generates event image data Ie by receiving image light formed by the lens substrate 40 (particularly the lens 41). The EVS element includes, for example, a charge-coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. The EVS element is sensitive, for example, to at least the wavelength range of light emitted from the light-emitting element 20. The EVS element is sensitive, for example, to light in the visible range. Note that the EVS element may also be sensitive, for example, to light outside the visible range (for example, in the infrared range). The light receiving element 30 is mounted on, for example, a wiring substrate 60 .

[0017] The light receiving element 30 is disposed in a position where it can receive image light formed by the lens substrate 40 (particularly the lens 41). The light receiving surface of the light receiving element 30 is disposed in an area facing the lens 41. The light receiving surface of the light receiving element 30 may be, for example, a collection of multiple light receiving surfaces that are separated from one another.

[0018] 3 shows an example of functional blocks of the light receiving element 30 and the light receiving driver 31. The light receiving element 30 has a plurality of light receiving pixels P arranged two-dimensionally in a matrix, for example, as shown in Fig. 3. In the light receiving element 30, for example, a vertical signal line is arranged for each pixel column, and a horizontal signal line is arranged for each pixel row.

[0019] The light receiving driver 31 is an IC chip that controls light reception in each light receiving pixel P and controls the readout of the event signal Sev from each light receiving pixel P. The light receiving driver 31 has, for example, a vertical scanning circuit 32 and a horizontal scanning circuit 33, as shown in FIG. 3 . The vertical scanning circuit 32 selects a plurality of light receiving pixels P row by row via a plurality of horizontal signal lines, and outputs the event signal Sev generated by each of the light receiving pixels P in one row to the horizontal scanning circuit 33 via a plurality of vertical signal lines. The horizontal scanning circuit 34 outputs the event signal Sev output from the plurality of light receiving pixels P row by row. The light receiving driver 31 is mounted on, for example, a wiring board 60.

[0020] 4 shows an example of functional blocks of the light-receiving pixel P. Each light-receiving pixel P has, for example, a photoelectric conversion circuit 34, a subtraction circuit 35, and a quantization circuit 36, as shown in FIG.

[0021] The photoelectric conversion circuit 34 includes, for example, a photodiode and performs photoelectric conversion on light incident on the light-receiving surface of the light-receiving element 30 to generate an electric charge corresponding to the amount of light received. The subtraction circuit 35 includes, for example, a buffer circuit and a sample-and-hold circuit. The buffer circuit holds a voltage signal corresponding to the level of the electric charge output from the photodiode. The sample-and-hold circuit samples the signal supplied from the buffer circuit, holds the sampled signal, and then outputs a signal corresponding to the difference between the signal supplied from the buffer circuit and the signal held in the sample-and-hold circuit as a detection signal.

[0022] The quantization circuit 37 generates the event signal Sev by performing a quantization process based on the detection signal output from the subtraction circuit 36. Specifically, for example, the quantization circuit 37 sets the event signal Sev to a low level when the change in the amount of received light is less than a predetermined amount, and sets the event signal Sev to a high level when the amount of received light increases or decreases by more than a predetermined amount. That is, the quantization circuit 37 outputs a pulse signal as the event signal Sev when the amount of received light changes by more than a predetermined amount. In this way, the quantization circuit 37 determines that an event has occurred when the amount of received light changes by more than a predetermined amount, and outputs a pulse signal.

[0023] The quantization circuit 37 is capable of generating an event signal Sev with a positive polarity when the signal level changes to a positive value, and generating an event signal Sev with a negative polarity when the signal level changes to a negative value. The event signal Sev with a positive polarity is called a positive event (on event). The event signal Sev with a negative polarity is called a negative event (off event).

[0024] The light receiving element 30 is capable of generating, for example, event image data Ie that includes the polarity of the event signal Sev. In the event image data Ie that includes the polarity of the event signal Sev, a region where an event signal Sev with positive polarity is generated is a positive event region (on-event region), and a region where an event signal Sev with negative polarity is generated is a negative event region (off-event region). An example of a positive event region is the positive event region PE1 shown in Figure 8(B) (described later).

[0025] (Lens Substrate 40) As shown in FIG. 1, the lens substrate 40 includes a lens 41, a support 42, and a spacer 43. The lens 41 is an optical element that can condense a portion of the light reflected by the marker 90 and irradiate the light-receiving surface of the light-receiving element 30 with the condensed light. The optical element is, for example, a lens module including multiple lenses and an aperture. The optical element may be, for example, a lens module array in which two or more of the above-described lens modules are arranged. The support 42 supports the lens 41. The flat surface of the convex lens of the lens 41 that protrudes toward the marker 90 is in contact with the support 42. The support 42 is made of a material (transparent material) that is transparent to at least the light emitted from the light-emitting element 30. The support 42 only needs to have sufficient optical transparency to realize the functions of the tactile sensor device 1. The support 42 is fixed to the wiring substrate 60 via the spacer 43. The spacer 43 is made of a material (opaque material) that has a light-blocking property against at least the light emitted from the light emitting element 30 .

[0026] (Controller 50) The controller 50 is an IC chip that controls the emission and extinction of the light-emitting element 30 via the light-emitting driver 21, and also controls the light-receiving and readout of the light-receiving element 30 via the light-receiving driver 31. The controller 50 is mounted on, for example, a wiring board 60. The controller 50 is electrically connected to, for example, the light-emitting driver 21 and the light-receiving driver 31 via the wiring board 60. The controller 50 outputs control signals to the light-emitting driver 21 and the light-receiving driver 31 via, for example, the wiring board 60.

[0027] The controller 50 outputs, for example, the event signal Sev obtained from each light-receiving pixel P of the light-receiving element 30 to an external circuit as event image data Ie.

[0028] Assume that the external circuit is, for example, a control circuit in a robotic device that controls the operation of the robot. In this case, the control circuit of the robotic device estimates the three-dimensional deformation of each portion of the surface of the tactile sensor device 1 (the surface of the light-shielding layer 80) and the force (vector) acting on each portion of the surface of the tactile sensor device 1 (the surface of the light-shielding layer 80) based on data such as multiple image data sets detected at different times obtained from the tactile sensor device 1. The control circuit of the robotic device controls the actuator unit of the robot based on the information obtained by such estimation, for example.

[0029] Assume that the external circuit is, for example, a control circuit in a game controller that controls a specific function of a game. In this case, the control circuit of the game controller estimates the three-dimensional deformation of each portion of the surface of the tactile sensor device 1 (the surface of the light-shielding layer 80) and the force (vector) acting on each portion of the surface of the tactile sensor device 1 (the surface of the light-shielding layer 80) based on data such as multiple image data sets detected at different times obtained from the tactile sensor device 1. The control circuit of the game controller controls the specific function of the game based on information obtained by such estimation, for example.

[0030] (Wiring Board 60) The wiring board 60 is a circuit board on which wiring connected to the light-emitting driver 21, the light-receiving driver 31, the light-receiving elements 30, and the controller 50 is formed. The wiring board 60 is made of a material (opaque material) that has a light-blocking property against at least the light emitted from the light-emitting elements 30. The wiring board 60 is fixed to the support body 12 via the spacers 14.

[0031] (Deformation layer 70) The deformation layer 70 is disposed on the opposite side of the light receiving element 30 in terms of its positional relationship with the supports 11 and 12. The deformation layer 70 deforms in response to an external physical stimulus (for example, vertical displacement (pressure) or shear displacement (rubbing)). The deformation layer 70 is disposed in contact with a portion of the surface of the support 11 that faces the multiple light emitting elements 20 and the lens substrate 40 (particularly the lens 41). The deformation layer 70 is disposed so as to cover the multiple light emitting elements 20 in a planar view.

[0032] The deformation layer 70 is made of an elastic material (transparent elastic material) that is transparent to at least the light emitted from the light emitting element 30. The deformation layer 70 is made of, for example, a rubber material or a gel material. The deformation layer 70 only needs to have enough light transparency to realize the functions of the tactile sensor device 1. The deformation layer 70 is, for example, dome-shaped, as shown in FIG. 1 . When the deformation layer 70 is dome-shaped, it becomes possible to measure the initial slippage when an external object comes into contact with the deformation layer 70.

[0033] (Light-shielding layer 80) The light-shielding layer 80 is disposed in contact with the surface of the deformation layer 70. The light-shielding layer 80 is disposed to cover the deformation layer 70 and the markers 90 and functions to prevent light (light from the light-emitting element 20) propagating through the deformation layer 70 from leaking to the outside. The light-shielding layer 80 also functions to prevent changes in the amount of light incident on the light-receiving element 30 due to changes in the external environment. The light-shielding layer 80 is made of a material (opaque material) that has light-shielding properties at least for the light emitted from the light-emitting element 30 and has a refractive index greater than that of the support 11. If the support 11 is made of a glass substrate (refractive index 1.46), the deformation layer 70 is made of a material with a refractive index of 1.5, for example. This allows light to be incident on the lens 41 at a wide angle of view. The same applies to the interface between the support 11 (refractive index 1.46) and the gap G (refractive index 1.0). That is, since the surface of the support 11 on the lens 41 side is in contact with the gap G, it is possible to allow light to be incident on the lens 41 at a wide angle of view.

[0034] The light-shielding layer 80 may be formed, for example, from a film made of a material (opaque material) that blocks at least the light emitted from the light-emitting element 30, and may be attached to the surface of the deformation layer 70. The light-shielding layer 80 may be formed, for example, by incorporating a powder that blocks at least the light emitted from the light-emitting element 30 into the surface of the deformation layer 70 or its vicinity. The light-shielding layer 80 may be formed, for example, by printing ink that blocks at least the light emitted from the light-emitting element 30 onto the surface of the deformation layer 70. The light-shielding layer 80 may be formed, for example, from rubber (for example, rubber with a black pigment mixed in) that blocks at least the light emitted from the light-emitting element 30. The surface of the light-shielding layer 80 facing the light-receiving element 30 is a color (for example, black) that provides high contrast with the marker 90.

[0035] (Marker 90) The marker 90 is disposed, for example, between the deformation layer 70 and the light-shielding layer 80. The marker 90 is disposed, for example, on the surface of the deformation layer 70 or inside the deformation layer 70. The marker 90 is disposed, for example, in contact with the surface of the deformation layer 70 (the surface in contact with the light-shielding layer 80), and is formed, for example, by printing on the surface of the deformation layer 70 or by embedding beads therein. The marker 90 functions as a reflective layer that reflects at least the light of the light-emitting element 20. The marker 90 is made of a material that is reflective to at least the light emitted from the light-emitting element 20.

[0036] The marker 90 is composed of a plurality of markers MK1 arranged two-dimensionally, for example, as shown in FIGS. 5 and 6 . The marker MK1 corresponds to a specific example of a “first marker” in the present disclosure. The marker MK1 has a pattern having in-plane anisotropy. The marker MK1 may have, for example, a + shape as shown in FIG. 5 , or a * shape as shown in FIG. 6 . The marker MK1 may have, for example, a shape including a + or an * sign, or a shape obtained by rotating a + sign. The surface of the marker MK1 facing the light receiving element 30 has a color (e.g., white) that provides high contrast in relation to the light-shielding layer 80. The marker 90 may include one or more markers having a pattern having in-plane isotropy.

[0037] 7 is a diagram for explaining the size, pitch, etc., of the marker MK1. The length of the marker MK1 in the left-right direction (X direction) on the paper is defined as Lx1, and the length in the up-down direction (Y direction) on the paper is defined as Ly1. The interval (pitch) between two adjacent markers MK1 in the X direction is defined as ΔX, and the interval (pitch) between two adjacent markers MK1 in the Y direction is defined as ΔY. When an external circuit that acquires event image data Ie from the controller 50 uses marker track as the tracking algorithm for the marker MK1, the length of the tracker TR in the X direction is defined as Lx2, and the length in the Y direction is defined as Ly2.

[0038] It is assumed that the area of ​​the tactile sensor device 1 (marker 90) where displacement can be detected is approximately φ10 mm. In this case, the size and pitch of the marker MK1 are as follows: When ΔX = ΔY = 1 mm, the marker MK1 is projected onto the light receiving surface of the light receiving element 30 with a size of approximately 200 μm square. Lx1 = Ly1 = 0.25 mm Line width of marker MK1 = 0.05 mm ΔX = ΔY = 1 mm Lx2 = Ly2 = 0.85 mm

[0039] It is assumed that the area of ​​the tactile sensor device 1 (marker 90) where displacement can be detected is approximately φ60 mm. In this case, the size and pitch of the marker MK1 are as follows: When ΔX = ΔY = 4 mm, the marker MK1 is projected onto the light receiving surface of the light receiving element 30 with a size of approximately 200 μm square. Lx1 = Ly1 = 1 mm Line width of marker MK1 = 0.2 mm ΔX = ΔY = 4 mm Lx2 = Ly2 = 3.4 mm

[0040] From the viewpoint of reducing the possibility of losing sight of marker MK1, it is preferable that ΔX and ΔY are four or more times Lx1 and Ly1, and it is preferable that Lx2 and Ly2 are approximately 0.85 times ΔX and ΔY.

[0041] 8A to 8C show an example of the position of the marker MKc and the EVS output (event image data Ie) in a tactile sensor device according to a comparative example. In the comparative example, for example, as shown in FIG. 8A, when the circular marker MKc is not moving, the event image data Ie does not include an ON-event region PE1. On the other hand, for example, as shown in FIG. 8B, when the circular marker MKc is displaced to the left, only one ON-event region PE1 in the shape of a crescent moon with a missing right side appears in the event image data Ie. Furthermore, for example, as shown in FIG. 8C, when the circular marker MKc moves to the right from the position shown in FIG. 8B, only one ON-event region PE1 in the shape of a crescent moon with a missing left side appears in the event image data Ie.

[0042] In the comparative example, as shown in Figures 8(B) and 8(C), when the circular marker MKc is displaced, the gap D1 between the on-event region PE1 before the displacement and the on-event region PE1 after the displacement is much larger than the displacement of the circular marker MKc. As a result, no matter what method (e.g., marker track or optical flow) is used as the tracking algorithm for the marker MK1, there is a high possibility that the marker MKc will be lost. Note that when estimating the gap D1, the tip of the crescent shape, which corresponds to the minute region of the on-event region PE1, was omitted.

[0043] 9(A) to 9(C) and 10(A) to 10(C) show examples of the position of the marker MK1 and the EVS output (event image data Ie) in the tactile sensor device 1 according to the embodiment. As shown in FIGS. 9(A) to 9(C), when a +-shaped marker is used as the marker MK1, multiple (three) on-event regions PE1 appear. On the other hand, as shown in FIGS. 10(A) to 10(C), when a *-shaped marker is used as the marker MK1, multiple (seven) on-event regions PE1 appear.

[0044] In each embodiment, the marker MK1 has a shape that allows multiple on-event regions PE2 to be formed when the marker MK1 is displaced to an extent that the pre-displacement marker MK1 and the post-displacement marker MK1 overlap with each other as the deformation layer 80 is displaced. Examples of such shapes include a + shape, an * shape, a star shape, a shape including a +, a shape including an *, a star, and a rotated + shape. Here, for example, as shown in FIGS. 9B and 9C, when the marker MK1 is displaced, the gap D2 between the pre-displacement on-event region PE2 and the post-displacement on-event region PE2 is equal to the displacement amount of the marker MK1. Note that in FIGS. 10B and 10C, when the marker MK1 is displaced, the gap D2 between the pre-displacement on-event region PE2 and the post-displacement on-event region PE2 does not exist. Therefore, no matter what method (for example, marker track or optical flow) is used as the tracking algorithm for the marker MK1, the possibility of losing sight of the marker MK1 is extremely low.

[0045] [Operation] Next, the operation of the tactile sensor device 1 according to this embodiment will be described.

[0046] The controller 50 outputs a control signal to the light-emitting driver 21 to cause each light-emitting element 20 to emit light. The light-emitting driver 21 then drives each light-emitting element 20, causing each light-emitting element 20 to emit light. The light emitted from each light-emitting element 20 reaches the marker 90 and the light-shielding layer 80 through the deformation layer 70 and is mainly reflected by the marker 90. A portion of the light reflected by the marker 90 (reflected light) is refracted at the interface between the deformation layer 70 and the support 11, as well as at the interface between the support 11 and the gap G, and enters the lens 41. The light incident on the lens 41 illuminates the light-receiving surface of the light-receiving element 30 at a predetermined angle of view. As a result, an image (marker image) including a portion of the marker 90 is formed on the light-receiving surface of the light-receiving element 30. The controller 50 generates event image data Ie based on the event signal Sev obtained from each light-receiving pixel P of the light-receiving element 30 and outputs the event image data Ie to an external device.

[0047] [Effects] Next, the effects of the tactile sensor device 1 according to this embodiment will be described.

[0048] To address the slow responsiveness of robots caused by the sensor's frame rate, EVS cameras are sometimes used as cameras built into sensors. However, when the position of a marker built into the sensor changes, the EVS camera loses track of the marker because it is unable to determine the correspondence between the marker before and after the change. This problem is particularly prevalent when only on-events (positive events) are used.

[0049] On the other hand, in this embodiment, each marker MK1 in the marker 90 has a pattern with in-plane anisotropy. As a result, when the marker MK1 is displaced, the gap D2 between the on-event region PE2 before the displacement and the on-event region PE2 after the displacement becomes equal to or less than the displacement amount of the marker MK1. As a result, the possibility of losing sight of the marker MK1 can be greatly reduced.

[0050] In this embodiment, the marker MK1 has a shape that allows multiple on-event regions PE2 to be formed when the marker MK1 is displaced to an extent that the marker MK1 before displacement and the marker MK1 after displacement overlap with each other as the deformation layer 80 is displaced. The marker MK1 has, for example, a shape including a + or *, or a shape obtained by rotating a +. As a result, when the marker MK1 is displaced, the gap D2 between the on-event region PE2 before displacement and the on-event region PE2 after displacement becomes equal to or less than the displacement amount of the marker MK1. As a result, the possibility of losing track of the marker MK1 can be significantly reduced.

[0051] In this embodiment, a light-shielding layer 80 is provided to cover the deformation layer 70 and the markers 90. This prevents light (light from the light-emitting element 20) that has propagated through the deformation layer 70 from leaking to the outside, and also prevents the amount of light incident on the light-receiving element 30 from changing due to changes in the external environment.

[0052] In this embodiment, the marker MK1 is white, and the light-shielding layer 80 is black. This makes it possible to obtain a high-contrast ON event region PE1, thereby enabling the displacement of the ON event region PE1 to be detected with high accuracy.

[0053] In this embodiment, a gap G is provided between the lens 41 and the deformation layer 70. This allows light with a wide angle of view to be incident on the lens 41. As a result, the area in the tactile sensor device 1 (marker 90) where displacement can be detected can be expanded.

[0054] In this embodiment, the deformation layer 70 is made of an elastic material (transparent elastic material) that is transparent to at least the light emitted from the light emitting element 30. This increases the restoring force of the deformation layer 70 compared to when the deformation layer has a hollow structure, allowing the deformation layer 70 to quickly return to its original size and shape after being deformed by an external force. As a result, even if the external force changes over a short period of time, the change can be detected with high accuracy.

[0055] 2. Modifications [Modification A] In the above embodiment, the multiple markers MK1 may be arranged in a non-repeating pattern in any direction, including the left-right direction (X direction) and the up-down direction (Y direction), as shown in FIG. 11 . This allows the external device to assign a code to each marker MK1 included in the event image data Ie and grasp the positional relationship between the markers MK1. As a result, if the external device detects a marker MK1 that moves in a way that deviates from the positional relationship between the markers MK1, it can, for example, stop tracking the marker MK1, thereby reducing the tracking load. After stopping tracking of the marker MK1, if the external device detects a marker MK1 that moves in a way that deviates from the positional relationship between the markers MK1 in an area where it should have been detected, it resumes tracking the marker MK1. In this way, this modification allows a lost marker MK1 to be re-tracked while reducing the tracking load of the marker MK1.

[0056] [Variation B] In the above embodiment and variations thereof, at least some of the markers included in marker 90, namely, markers MK2 (second markers) different from marker MK1, may have a pattern that differs from marker MK1 in at least one of shape, orientation, and size, as shown in Figures 12, 13, and 14. "The size of marker MK2 differs from that of marker MK1" refers to a difference in the degree to which tracking of the displacement of the projected image of marker MK2 (on-event region PE1) is possible even when tracking of the displacement of the projected image of marker MK1 (on-event region PE1) fails when markers MK1 and MK2 are projected onto the light-receiving surface of light-receiving element 30.

[0057] Fig. 12 illustrates an example in which the marker MK2 has a * shape. Fig. 13 illustrates an example in which the marker MK2 has a shape obtained by rotating a + sign by 45 degrees. Fig. 14 illustrates an example in which the marker MK2 has a + shape and is larger than the size of the marker MK1. Figs. 12, 13, and 14 also illustrate an example in which the distances between multiple markers MK2 are wider than the distances between multiple markers MK1.

[0058] In addition, in this modified example, the marker MK2 has a shape that allows for the formation of multiple on-event regions PE2 when the marker MK2 is displaced to the extent that the marker MK2 before displacement and the marker MK2 after displacement overlap each other as the deformation layer 80 is displaced.

[0059] As a result, when the markers MK1 and MK2 are displaced, the gap D2 between the on-event region PE2 before the displacement and the on-event region PE2 after the displacement becomes equal to or smaller than the displacement of the markers MK1 and MK2, thereby greatly reducing the possibility of losing sight of the markers MK1 and MK2.

[0060] [Variation C] In the above-described embodiment and its variations, for example, as shown in FIGS. 15 and 16 , two lenses 41a and 41b may be formed on the support 42. The lens 41a corresponds to a specific example of a “first lens” in the present disclosure. The lens 41b corresponds to a specific example of a “second lens” in the present disclosure. The lenses 41a and 41b are disposed at positions where parallax image data can be obtained from first event image data Ie1 obtained by light incident on a first region of the light receiving element 30 via the lens 41a and second event image data Ie2 obtained by light incident on a second region of the light receiving element 30 via the lens 41b. The first event image data Ie1 and the second event image data Ie2 may be included in a single event image data Ie generated by an event signal Sev obtained from the light receiving element 30, for example. This enables distance measurement by triangulation using the event image data Ie including the first event image data Ie1 and the second event image data Ie2.

[0061] 3. Second embodiment [Configuration] Next, a tactile sensor device 2 according to a second embodiment of the present disclosure will be described. Fig. 17 shows an example cross-sectional configuration of the tactile sensor device 2. As shown in Fig. 17, the tactile sensor device 2 includes, for example, a plurality of light-emitting devices 210, a plurality of light-receiving devices 220, a controller 230, a core unit 240, a deformation layer 250, markers 260, a light-shielding layer 270, and a communication unit 280.

[0062] The light-emitting device 210 is, for example, a light-emitting module including a light-emitting element 20, a light-emitting driver 21, and a support body that forms a gap G. The light-receiving device 220 is, for example, a camera module including a light-receiving element 30, a light-receiving driver 31, a lens substrate 40, and a support body that forms a gap G. The controller 230 is a chip including a controller 50 that controls the light-emitting driver 21 of each light-emitting device 210 and the light-receiving driver 31 of each light-receiving device 220. The controller 230 outputs event image data Ie generated by the controller 50 to the communication unit 280. The communication unit 280 outputs the event image data Ie input from the controller 230 to an external device. The core unit 240 is, for example, a block-shaped support member having multiple flat surfaces whose normal directions are different from each other. For example, the light-emitting device 210 and the light-receiving device 220 are mounted on each flat surface of the core unit 240.

[0063] Deformation layer 250 is made of the same material as deformation layer 70, and is disposed in contact with and covers core portion 240 and the plurality of light-emitting devices 210 and the plurality of light-receiving devices 220 mounted on each flat surface of core portion 240. The appearance of deformation layer 250 is, for example, ball-shaped, and the surface of deformation layer 250 is, for example, spherical.

[0064] The light-shielding layer 270 is disposed in contact with the surface of the deformation layer 250. The light-shielding layer 270 is disposed so as to cover the deformation layer 250 and the markers 260, and has the function of blocking light (light from the light-emitting element 20) that has propagated through the deformation layer 250 from leaking to the outside. The light-shielding layer 270 also has the function of preventing changes in the amount of light incident on the light-receiving element 30 due to changes in the external environment. The light-shielding layer 270 is made of the same material as the light-shielding layer 80.

[0065] Marker 260 is disposed, for example, between deformation layer 250 and light-shielding layer 270. Marker 260 is disposed, for example, on the surface of deformation layer 250 or inside deformation layer 250. Marker 260 is disposed, for example, in contact with the surface of deformation layer 250 (the surface in contact with light-shielding layer 270), and is formed, for example, by printing on the surface of deformation layer 250 or by embedding beads therein. Marker 260 functions as a reflective layer that reflects at least the light of light-emitting element 20. Marker 260 has the same configuration as marker 90.

[0066] [Effect] In this embodiment, the deformation layer 250 is disposed in contact with and covers the plurality of light-emitting devices 210 and the plurality of light-receiving devices 220. At this time, each marker MK1 in the marker 90 has a pattern with in-plane anisotropy. As a result, when the marker MK1 is displaced, the gap D2 between the on-event region PE2 before the displacement and the on-event region PE2 after the displacement becomes equal to or less than the displacement amount of the marker MK1. As a result, the possibility of losing track of the marker MK1 can be significantly reduced.

[0067] In this embodiment, the marker MK1 has a shape that allows multiple on-event regions PE2 to be formed when the marker MK1 is displaced to an extent that the marker MK1 before displacement and the marker MK1 after displacement overlap with each other as the deformation layer 80 is displaced. The marker MK1 has, for example, a shape including a + or *, or a shape obtained by rotating a +. As a result, when the marker MK1 is displaced, the gap D2 between the on-event region PE2 before displacement and the on-event region PE2 after displacement becomes equal to or less than the displacement amount of the marker MK1. As a result, the possibility of losing track of the marker MK1 can be significantly reduced.

[0068] In this embodiment, a light-shielding layer 80 is provided to cover the deformation layer 70 and the markers 90. This prevents light (light from the light-emitting element 20) that has propagated through the deformation layer 70 from leaking to the outside, and also prevents the amount of light incident on the light-receiving element 30 from changing due to changes in the external environment.

[0069] In this embodiment, the marker MK1 is white, and the light-shielding layer 80 is black. This makes it possible to obtain a high-contrast ON event region PE1, thereby enabling the displacement of the ON event region PE1 to be detected with high accuracy.

[0070] In this embodiment, a gap G is provided between the lens 41 and the deformation layer 70. This allows light with a wide angle of view to be incident on the lens 41. As a result, the area in the tactile sensor device 1 (marker 90) where displacement can be detected can be expanded.

[0071] In this embodiment, the deformation layer 70 is made of an elastic material (transparent elastic material) that is transparent to at least the light emitted from the light-emitting element 30. This minimizes optical loss, allowing the light-receiving element 30 to accurately detect a response to an external input.

[0072] 4. Modifications common to all embodiments In the first embodiment and its modifications, the tactile sensor device 1 may be configured, for example, as shown in Figures 18 and 19, so that the surfaces of the lens 41 and each light-emitting element 20 are in contact with the back surface of the support body 11. In the second embodiment, the light-receiving device 220 included in the tactile sensor 2 may be configured, for example, as shown in Figures 18 and 19, so that the surface of the lens 41 is in contact with the back surface of the support body 11. By using a configuration that eliminates the gap G in this way, the proportion of light from each light-emitting element 20 that is reflected by the support body 11 and enters the light-receiving element 30 can be reduced.

[0073] 5. Application Examples Next, application examples of the tactile sensor device 1 according to the first embodiment and its modifications (hereinafter simply referred to as the "tactile sensor device 1") and application examples of the tactile sensor device 2 according to the second embodiment and its modifications (hereinafter simply referred to as the "tactile sensor device 2") will be described. The application examples correspond to specific examples of "electronic devices" according to an embodiment of the present disclosure.

[0074] [Application Example 1] FIG. 20 shows an example of the appearance of a game controller 3 equipped with multiple tactile sensor devices 1. For example, as shown in FIG. 20 , the game controller 3 has two gripping portions (grip portions 310) that the user grips. Each grip portion 310 is provided with one or more tactile sensor devices 1. The controller 50 in the game controller 3 can detect the manner and strength of the user's grip of the grip portion 310 based on the event image data Ie output from the tactile sensor device 1. As a result, the controller 503 in the game controller can output a command to the game console corresponding to the detected manner and strength of the user's grip of the grip portion 310. In this way, this application example makes it possible to provide new commands using the tactile sensor device 1.

[0075] 20, the game controller 3 may further include a power supply 320 that supplies power to one or more tactile sensor devices 1. This allows the game controller 3 to operate without receiving power from an external source.

[0076] 21 shows an example of the appearance of a robot device 4 equipped with a plurality of tactile sensor devices 1. As shown in FIG. 21 , the robot device 4 includes, for example, a main body 410, two robot arm devices 420, a movement mechanism 430, a sensor 440, and a sensor 450.

[0077] The main body 410 includes, for example, a power unit and a control unit of the robot device 4, and is a central portion to which each unit of the robot device 4 is attached. The control unit controls the two robot arm devices 420, the movement mechanism 430, the sensor 440, and the sensor 450 provided in the robot device 4. The main body 410 may have a shape that resembles the upper half of a human body, including a head, neck, and torso.

[0078] The main body 410 may further include, for example, a power source 411. Each robot arm device 420 has one or more tactile sensor devices 1 at its tip portion (for example, a finger).

[0079] Each robot arm device 420 is, for example, an articulated manipulator attached to the main body 410. One robot arm device 420 is attached, for example, to the right shoulder of the main body 410, which resembles the upper half of a human body. The other robot arm device 420 is attached, for example, to the left shoulder of the main body 410, which resembles the upper half of a human body. One or more tactile sensor devices 1 are attached to the tip of each robot arm device 420.

[0080] The movement mechanism 430 is provided, for example, at the bottom of the main body 410, and is a component responsible for moving the robot device 4. The movement mechanism 430 may be a two-wheeled or four-wheeled wheeled movement device, or a two-legged or four-legged legged movement device. Furthermore, the movement mechanism 430 may be a hover-type, propeller-type, or caterpillar-type movement device.

[0081] The non-contact sensor 440 is provided, for example, on the main body 410 or the like, and is a sensor that detects (senses) information about the environment (external environment) surrounding the robot device 4 in a non-contact manner. The non-contact sensor 440 outputs sensor data obtained by detection (sensing). The non-contact sensor 440 is, for example, an imaging device such as a stereo camera, a monocular camera, a color camera, an infrared camera, or a polarization camera. The non-contact sensor 440 may be an environmental sensor for detecting weather or meteorological conditions, a microphone for detecting sound, or a depth sensor such as an ultrasonic sensor, a Time of Flight (ToF) sensor, or a Light Detection and Ranging (LiDAR) sensor. The non-contact sensor 450 may be a position sensor such as a Global Navigation Satellite System (GNSS) sensor.

[0082] 22(A) and 22(B) show an example of a procedure for calibrating a force sensor provided at the tip portion 421 of a robot arm device 420. As shown in FIG. 22(A), this force sensor has, for example, two tactile sensor devices 1 and a force calculation unit 421c that calculates force based on event image data Ie obtained from each tactile sensor device 1. One tactile sensor device 1 is provided on the first finger portion 421a, and the other tactile sensor device 1 is provided on the second finger portion 421b. The first finger portion 421a and the second finger portion 421b are positioned facing each other with a predetermined gap between them.

[0083] The control unit of the main body 410 controls the tip portion 421 of the robot arm device 420 to move the first finger portion 421a and the second finger portion 421b so that the two tactile sensor devices 1 are spaced apart from each other, as shown in Fig. 22(A), for example. Next, the control unit of the main body 410 drives each tactile sensor device 1. At this time, each tactile sensor device 1 acquires event image data Ie.

[0084] The force sense calculation unit 421c calculates an initial force sense value based on the event image data Ie obtained by each tactile sensor device 1 when the two tactile sensor devices 1 are spaced apart from each other. As a result, the force sense initial value is stored in the force sense calculation unit 421c. The initial force sense value is a value calculated based on the event image data Ie obtained from each tactile sensor device 1 when no object is touching each tactile sensor device 1.

[0085] Next, the control unit of the main body 410 controls the tip portion 421 of the robot arm device 420 to move the first finger portion 421a and the second finger portion 421b so that the two tactile sensor devices 1 come into contact with each other, as shown in FIG. 22(B), for example. This causes the two tactile sensor devices 1 to come into contact with each other with a predetermined pressure. Next, the control unit of the main body 410 drives each tactile sensor device 1. At this time, each tactile sensor device 1 acquires event image data Ie.

[0086] The force sense calculation unit 421c calculates the force sense based on the event image data Ie obtained by each tactile sensor device 1 when the two tactile sensor devices 1 are in contact with each other at a predetermined pressure. As a result, the force sense under the predetermined pressure is stored in the force sense calculation unit 421c. The force sense under the predetermined pressure is a value calculated based on the event image data Ie obtained from each tactile sensor device 1 when the tactile sensor devices 1 are in contact with each other at the predetermined pressure. The force sense calculation unit 421c uses these two types of force sense as reference values ​​to calculate the force sense when an object is grasped by the first finger 421a and the second finger 421b.

[0087] The control unit of the main body 410 controls the tip portion 421 of the robot arm device 420 to move the first finger portion 421a and the second finger portion 421b so that the two tactile sensor devices 1 sandwich the jig J, for example, as shown in FIG. 23 . As a result, the two tactile sensor devices 1 come to rest in a state where they sandwich the jig J with a predetermined pressure. Next, the control unit of the main body 410 drives each tactile sensor device 1. At this time, each tactile sensor device 1 acquires event image data Ie.

[0088] The force sense calculation unit 421c calculates the force sense based on the event image data Ie obtained by each tactile sensor device 1 when the two tactile sensor devices 1 are stationary and clamping the jig J with a predetermined pressure. At this time, the force sense calculation unit 421c corrects the calculated force sense using the two types of force sense described above as reference values. The control unit of the main body 410 uses the force sense obtained in this way to control the tip portion 421 of the robot arm device 420.

[0089] In this application example, one or more tactile sensor devices 1 according to the first embodiment and its modified examples are attached to a robot arm device 420 in a robot device 4. This allows the tip portion 421 of the robot arm device 420 to be controlled with high precision.

[0090] 24 shows an example in which the tactile sensor device 2 is used as a 3D model controller. Here, the 3D model refers to a 3D model 520 displayed on a display screen 510 of an information processing device 500. The 3D model 520 is displayed on the display screen 510 of the information processing device 500 by executing a 3D model program in an arithmetic processing unit (e.g., an arithmetic processing circuit configured by a CPU, GPU, etc.) of the information processing device 500.

[0091] The arithmetic processing unit of the information processing device 500 can detect, based on the event image data Ie input from the tactile sensor device 2 via the communication unit 530, how the user 600 grips the tactile sensor device 2, and the position and strength of the pressing, pinching, or stroking. As a result, the tactile sensor device 2 can output to the information processing device 500 a command corresponding to the detected way the user 600 grips the gripping unit 310, and the position and strength of the pressing, pinching, or stroking. The information processing device 500 operates the 3D model 520 based on the command obtained from the tactile sensor device 2. For example, when the tactile sensor device 2 is stroked, the information processing device 500 displays an image of the 3D model 520 being stroked on the display screen 510. In this way, in this application example, new commands can be provided using the tactile sensor device 2 as a 3D model controller.

[0092] The tactile sensor device 2 can also be applied to, for example, a console of a vehicle.

[0093] [Application Example 4] Fig. 25 shows an example of the appearance of a cylindrical controller 5 equipped with multiple tactile sensor devices 1. The cylindrical controller 5 has a cylindrical housing that can be held by users 610, 620, as shown in Fig. 25, for example. Multiple tactile sensor devices 1 are provided on the surface of this housing, and a communication unit 710 is further provided inside this housing. Fig. 25 shows an example in which a tactile sensor device 1 is provided on each end of the cylindrical housing. The communication unit 710 outputs event image data Ie input from each tactile sensor device 1 to the information processing device 500.

[0094] For example, a game screen is displayed on the display screen 510 of the information processing device 500. For example, a relay race 540 is displayed on the game screen as shown in Fig. 25. The relay race 540 is displayed on the display screen 510 of the information processing device 500 by executing a relay race game program in an arithmetic processing unit (for example, an arithmetic processing circuit configured by a CPU, a GPU, etc.) of the information processing device 500.

[0095] The arithmetic processing unit of the information processing device 500 can detect the manner and strength of gripping of the cylindrical controller 5 by the users 610, 620, based on the event image data Ie input from the cylindrical controller 5 via the communication unit 530. The arithmetic processing unit of the information processing device 500 can also detect, for example, movements such as twisting the cylindrical controller 5 or rubbing it with fingers. As a result, the cylindrical controller 5 can output commands corresponding to the detected manner and strength of gripping of the cylindrical controller 5 by the users 610, 620 to the information processing device 500. The information processing device 500 operates a relay race 540 game based on the commands obtained from the tactile sensor device 1. For example, when the cylindrical controller 5 is handed over from the user 610 to the user 620, the information processing device 500 displays on the display screen 510 an image of the baton being handed over from one runner to the other in the relay race 540 game. In this way, in this application example, new commands can be provided by using the tactile sensor device 1 for the cylindrical controller 5. In this manner, in this application example, new commands using the cylindrical controller 5 can be provided.

[0096] In addition, in Application Example 3 and Application Example 4, for example, a head-mounted display 700 shown in Fig. 26 may be used instead of the information processing device 500. The head-mounted display 700 has a configuration similar to that of the information processing device 500, and includes, for example, a glasses-type display unit 720 and a support unit 710. The display unit 720 is equipped with a display screen 510.

[0097] Note that the effects described in this specification are merely examples. The effects of the present disclosure are not limited to the effects described in this specification. The present disclosure may have effects other than the effects described in this specification.

[0098] Furthermore, for example, the present disclosure can take the following configurations: (1) A tactile sensor device comprising: a substrate on which one or more lenses are formed; an EVS (Event-Based Vision Sensor) element facing the one or more lenses; a deformation layer positioned on the opposite side of the substrate from the EVS element; a plurality of markers positioned on the surface of or inside the deformation layer; and a plurality of light sources that irradiate the plurality of markers, wherein at least some of the plurality of markers, i.e., a plurality of first markers, have a pattern with in-plane anisotropy. (2) The tactile sensor device described in (1), wherein the first markers have a shape that can form a plurality of on-event regions when the first markers are displaced in accordance with displacement of the deformation layer to an extent that the first markers before displacement and the first markers after displacement overlap each other. (3) The tactile sensor device described in (2), wherein the first markers have a shape including a + or *, or a shape obtained by rotating a +. (4) The tactile sensor device according to any one of (1) to (3), wherein at least some of the plurality of markers, a plurality of second markers different from the plurality of first markers, have a pattern that differs from the first markers in at least one of shape, orientation, and size. (5) The tactile sensor device according to (4), wherein the second markers have a shape that allows the formation of a plurality of on-event regions when the second markers are displaced in accordance with the displacement of the deformation layer to an extent that the second markers before displacement and the second markers after displacement overlap each other. (6) The tactile sensor device according to (1) or (5), wherein the spacing between the plurality of second markers is wider than the spacing between the plurality of first markers. (7) The tactile sensor device according to any one of (1) to (3), wherein the plurality of markers have a non-repeating pattern. (8) The tactile sensor device according to any one of (1) to (7), further comprising a light-shielding layer that covers the deformation layer and the plurality of markers. (9) The tactile sensor device according to (8), in which the plurality of markers are white, and the light-shielding layer is black.(10) The tactile sensor device according to any one of (1) to (9), wherein a gap is provided between the one or more lenses and the deformation layer. (11) The tactile sensor device according to any one of (1) to (9), further comprising a support body supporting the deformation layer, wherein the one or more lenses and each of the light sources are in contact with the support body. (12) The tactile sensor device according to any one of (1) to (11), wherein the deformation layer is formed of an elastic material that is transparent to light emitted from the light source. (13) The tactile sensor device according to any one of (1) to (12), wherein a first lens and a second lens are formed on the substrate as the multiple lenses, and the first lens and the second lens are arranged at positions where parallax image data can be obtained from first event image data obtained by light incident on the EVS element through the first lens and second event image data obtained by light incident on the EVS element through the second lens. (14) An electronic device comprising one or more tactile sensor devices, the one or more tactile sensor devices having: a substrate on which one or more lenses are formed; an EVS (Event-Based Vision Sensor) element facing the one or more lenses; a deformation layer arranged on the opposite side of the EVS element relative to the substrate; a plurality of markers arranged on the surface of the deformation layer or inside the deformation layer; and a plurality of light sources that irradiate light onto the plurality of markers, wherein at least some of the plurality of markers, i.e., a plurality of first markers, have a pattern with in-plane anisotropy.

[0099] This application claims priority based on Japanese Patent Application No. 2024-015011, filed on February 2, 2024, in the Japan Patent Office, the entire contents of which are incorporated herein by reference.

[0100] Those skilled in the art will recognize that various modifications, combinations, subcombinations, and variations may occur depending on design requirements and other factors, and are intended to be within the scope of the appended claims and their equivalents.

Claims

1. A tactile sensor device comprising: a substrate on which one or more lenses are formed; an EVS (Event-Based Vision Sensor) element facing the one or more lenses; a deformation layer positioned on the opposite side of the substrate from the EVS element; a plurality of markers positioned on the surface of or inside the deformation layer; and a plurality of light sources that irradiate light onto the plurality of markers, wherein at least some of the plurality of markers, i.e., a plurality of first markers, have a pattern with in-plane anisotropy.

2. The tactile sensor device of claim 1, wherein the first marker has a shape that allows it to form multiple on-event areas when the first marker is displaced in accordance with the displacement of the deformation layer to an extent that the first marker before displacement and the first marker after displacement overlap each other.

3. The tactile sensor device according to claim 2, wherein the first marker has a shape including a + or *, or a shape of a rotated +.

4. The tactile sensor device according to claim 2, wherein at least a portion of the plurality of markers, namely, a plurality of second markers different from the plurality of first markers, have a pattern that differs from the first markers in at least one of shape, orientation, and size.

5. The tactile sensor device described in claim 4, wherein the second marker has a shape that allows the formation of multiple on-event areas when the second marker is displaced in accordance with the displacement of the deformation layer to an extent that the second marker before displacement and the second marker after displacement overlap each other.

6. The tactile sensor device according to claim 4, wherein the distance between the plurality of second markers is wider than the distance between the plurality of first markers.

7. The tactile sensor device according to claim 1, wherein the plurality of markers are in a non-repeating pattern.

8. A tactile sensor device according to any one of claims 1 to 7, further comprising a light-shielding layer that covers the deformation layer and the plurality of markers.

9. The tactile sensor device according to claim 8, wherein the plurality of markers are white, and the light-shielding layer is black.

10. A tactile sensor device according to any one of claims 1 to 7, wherein a gap is provided between the one or more lenses and the deformation layer.

11. A tactile sensor device according to any one of claims 1 to 7, further comprising a support that supports the deformation layer, wherein the one or more lenses and each of the light sources are in contact with the support.

12. A tactile sensor device according to any one of claims 1 to 7, wherein the deformation layer is formed from an elastic material that is transparent to the light emitted from the light source.

13. A tactile sensor device as described in any one of claims 1 to 7, wherein a first lens and a second lens are formed on the substrate as the plurality of lenses, and the first lens and the second lens are arranged in positions that enable parallax image data to be obtained from first event image data obtained by light incident on the EVS element through the first lens and second event image data obtained by light incident on the EVS element through the second lens.

14. An electronic device comprising one or more tactile sensor devices, the one or more tactile sensor devices having: a substrate on which one or more lenses are formed; an EVS (Event-Based Vision Sensor) element facing the one or more lenses; a deformation layer positioned on the opposite side of the substrate from the EVS element; a plurality of markers positioned on the surface of or inside the deformation layer; and a plurality of light sources that irradiate light onto the plurality of markers, wherein at least some of the plurality of markers, i.e., a plurality of first markers, have a pattern with in-plane anisotropy.

Images