Retrofit touchless interface for contact-type input devices

Abstract

The retrofit interface device interfaces with the target device to provide touchless user input to the target device. The touchless sensing system includes one or more sensors responsive to touchless input provided by a human user, thereby generating one or more corresponding sensor input signals. The retrofit controller is connected to receive the one or more sensor input signals from the touchless sensing system, thereby generating corresponding control signals based on the one or more sensor input signals. The retrofit controller is connectable to the target device, thereby providing the control signals as inputs to the target device's existing control system, bypassing the target device's touch-based input, thereby causing the target device to operate based on the control signals via the target device's control system.

Description

[Technical field]

[0001] The present technology relates to touchless interaction with machines and the like. Certain embodiments provide retrofit touchless interfaces and / or inputs, and methods of installation and / or operation thereof, for use with existing target devices (target systems, target equipment) to provide touchless functionality in lieu of contact-based human machine interfaces (HMIs) and / or contact-based inputs for such target devices. Other embodiments provide custom and / or modular touchless interfaces and / or inputs, and methods of installation and / or operation thereof, for facilitating human interaction with machines or other target devices. [Background technology]

[0002] Touch-based interfaces are everywhere: touch screens, keypads, light switches, elevator buttons, etc. These interfaces are not only cheap to produce but also rudimentarily intuitive to use. Touchless types of human-machine interfaces (HMIs) are currently limited to niche applications, mostly industry-specific, such as gaming, entertainment, automotive panels, home automation, and clinical sterile interaction.

[0003] This paradigm is changing in light of the COVID-19 pandemic and the growing awareness of avoiding common means of infection transmission, for example, those that have high traffic touch surfaces commonly exposed to a large number of individuals. There is an increasing need for touchless and / or contactless solutions for human machine interface HMI systems. As a specific example, and without limiting the generality of the general need, there is a particular need for touchless and / or contactless solutions for human machine interface HMI systems used in hospitals, clinics, surgical environments, and / or the like. However, current touchless human machine interface HMI systems are inferior to touch-based systems in many applications. Current touch-based human machine interface HMI systems not only often have haptic feedback, but can also be faster than touchless solutions and easier to use and / or more intuitive than touchless solutions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2014 / 0327645 Summary of the Invention [Problem to be solved by the invention]

[0005] Thus, there is a need for improved touchless human machine interface HMIs that provide an intuitive and practical user experience, haptic feedback, and / or the like. For the most common touch-based human machine interface HMIs and input interfaces present in today's society, it is desirable to provide touchless interaction (e.g., retrofit the touch-based human machine interface HMIs and input interfaces of target devices with touchless solutions). [Means for solving the problem]

[0006] Aspects of the present invention provide methods and apparatus for providing touchless user input to a target device by retrofitting an existing target device equipped with a touch-based user input device. The retrofit method and apparatus according to some embodiments includes a capacitive sensor for touchless sensing and a touch actuated component (TAC) that presses, slides, rotates, or otherwise interacts with the touch-based input (e.g., buttons, latches, door handles, and / or the like) of the target device. The retrofit method and apparatus may also include user feedback that may be embodied, by way of non-limiting examples, using sonic wave transmission (SWT) for touchless haptic feedback (THF) (i.e., tactile), a display screen (e.g., LCD or LED) for visual feedback, sound cues for auditory feedback, and / or the like. User feedback may be desirable to provide an intuitive user experience that is easy to learn and quick to use. It may be desirable for the timing of use of retrofit methods and devices to be on the same (or comparable) time scale as the target touch-based system, or not much longer than the target touch-based system.

[0007] Another aspect of the present invention provides a retrofit interface device for interfacing with a control system of a target device. The retrofit interface device (the device) includes a touchless sensing system for detecting touchless inputs corresponding to gestures made by a human user and generating one or more corresponding input signals. The device also includes a retrofit controller (controller, interface controller) connected to receive the input signals from the touchless sensing system and to generate corresponding control signals and corresponding display signals based on the input signals. The retrofit controller is connected to provide the display signals to the control system of the target device and / or to a display of the target device. The display signals can cause a corresponding visual indicator to be displayed by the display based on the display signals. The retrofit controller is connected to provide the control signals to the control system of the target device, thereby causing the control system of the target device to operate the target device based on the control signals.

[0008] Further aspects and example embodiments are illustrated in the accompanying drawings and / or described in the following description. The accompanying drawings illustrate non-limiting exemplary embodiments of the present invention. [Brief description of the drawings]

[0009] [Figure 1] 1 illustrates a retrofit interface device in accordance with an illustrative embodiment. [Figure 1A] 13 illustrates a retrofit interface device according to another exemplary embodiment. [Figure 1B] 13 illustrates a retrofit interface device according to another exemplary embodiment. [Figure 1C] 13 illustrates a retrofit interface device according to yet another exemplary embodiment. [Diagram 2]1 illustrates a schematic diagram of an array of capacitive sensors that may be used in the touchless sensing system of any of the add-on interface devices described herein, according to an exemplary embodiment. [Figure 3A] 1 illustrates generally a touchless sensing system including one or more capacitive sensors and an optical sensing system including multiple (e.g., two) optical sensors that may be used in any of the add-on interface devices described herein in accordance with certain exemplary embodiments. [Figure 3B] 1 illustrates generally a touchless sensing system including one or more capacitive sensors and an optical sensing system including multiple (e.g., two) optical sensors that may be used in any of the add-on interface devices described herein in accordance with certain exemplary embodiments. [Figure 3C] 1 illustrates generally a touchless sensing system including one or more capacitive sensors and an optical sensing system including multiple (e.g., two) optical sensors that may be used in any of the add-on interface devices described herein in accordance with certain exemplary embodiments. [Figure 4A] 2 shows a schematic representation of a touch screen of a target device. [Figure 4B] FIG. 1C is a schematic diagram illustrating a touch emulator that may be used with the add-on interface device of FIG. 1B in accordance with certain exemplary embodiments. [Figure 5A] 2 is a front view of a panel of the retrofit user interface device of FIG. 1 (or any of the other retrofit user interface devices described herein) displaying virtual buttons according to an exemplary embodiment. [Figure 5B] 2 is a side view of a panel of the retrofit user interface device of FIG. 1 (or any of the other retrofit user interface devices described herein) displaying virtual buttons according to an exemplary embodiment. [Figure 6] FIG. 13 is a schematic diagram illustrating a method for determining a touchless input selection using any of the retrofit interface devices described herein in accordance with certain exemplary embodiments. [Figure 6A] 1 is a schematic depiction of a method for determining a touchless input selection using any of the retrofit interface devices described herein according to another exemplary embodiment. [Figure 7A] FIG. 13 is a schematic diagram illustrating an acoustic transmission feedback mechanism and a touchless sensing mechanism (touchless sensing system) that may be used with any of the retrofit interface devices described herein according to certain embodiments. [Figure 7B] FIG. 13 is a schematic diagram illustrating an acoustic transmission feedback mechanism and a touchless sensing mechanism that may be used with any of the retrofit interface devices described herein in accordance with certain embodiments. [Figure 8] FIG. 13 illustrates an exploded view of a touchless sensing system including one or more capacitive sensors and, optionally, one or more optical sensors arranged in a ring around a display that may be used as part of a feedback mechanism of any of the add-on interface devices described herein in accordance with certain embodiments. [Figure 8A] 1 is a schematic diagram of a capacitive sensor including one or more receiving electrodes and one or more transmitting electrodes arranged in a ring around an aperture in accordance with certain embodiments. [Figure 8B] 9 is a schematic depiction of the retrofit interface device of FIG. 8 applied to a target device having an elevator panel. [Figure 9] 1B is a schematic exploded view of the retrofit interface device of FIG. 1A including multiple linear actuators in accordance with certain embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Throughout the following description, specific details are set forth to provide a more complete understanding of the invention. However, the invention may be practiced without these specific details. In other instances, well-known elements have not been shown or described in detail to avoid unnecessarily obscuring the invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

[0011] Aspects of the present invention provide methods and apparatus for retrofitting existing target devices with target touch-based user input devices (e.g., touch-based human-machine interfaces HMIs) to provide touchless user input to such target devices.

[0012] Methods and apparatus are provided for providing touchless user input capabilities to a target device by retrofitting the target device with a target touch-based interface (or other target touch-based user input). Since such target inputs are typically integrated into a given device (target device), according to some embodiments, the retrofit interface device may be adjacent to or otherwise positioned relative to the existing (target) touch-based input such that the retrofit interface device can interact with at least a portion of the target touch-based input (e.g., button, knob, handle, slider, etc.) when a user makes a touchless gesture (e.g., hover, point / tap, wave, etc.) thereby providing input to the target device. In some embodiments, the retrofit interface device is connected to communicate directly with the target device's control system, bypassing the target device's existing target touch-based input. In some embodiments, the retrofit interface device is configured to emulate the target device's existing target touch screen interface.

[0013] In general, retrofit interface devices according to certain embodiments may be retrofitted to any suitable target device to provide touchless user input capabilities to the target device. Exemplary target devices and their target touch-based inputs include, but are not limited to, pedestrian controls for street light signals, light switches and panels, intercom system keypads, payment (POS) terminals, ATMs (individual and in-bank units), elevator panels, parking meters / terminals, keypad locks, kiosks with touch screens (e.g., airport check-in kiosks, retail self-check-out kiosks, advertising kiosks, hospital check-in kiosks, and / or the like), computers, faucets, water dispensers, lunchroom equipment, vending machines, gaming machines, game controllers, casino slots and other gambling equipment, door handles, latches, knobs, industrial equipment, medical equipment, and the like.

[0014] FIG. 1 illustrates a schematic of a retrofit interface device 10 according to an exemplary embodiment of the present invention. The retrofit interface device 10 is retrofitted to a target device 50 having a touch-based input 50A to provide the target device 50 with touchless input functionality that can be used by a user (in addition to or instead of the touch-based input 50A) to provide input to the target device 50. In the illustrated embodiment of FIG. 1, the retrofit interface device 10 interfaces with a target control system 51 of the target device 50 using a touchless control signal 16, and optionally with a target display 22 of the target device 50 using a touchless display signal 18. The retrofit interface device 10 includes a touchless sensing system 12 for detecting touchless input (e.g., input corresponding to a gesture made by a human user) and a retrofit controller (interface controller) 20 connected to receive a touchless input signal 14 generated by the touchless sensing system 12. In some embodiments, the touchless input signal 14 may be generated in response to the touchless sensing system 12 detecting a touchless input from a user or otherwise determining that a user has made a touchless input. That is, the touchless sensing system 12 may configure its own controller or internal logic circuitry (not shown) that interprets signals from the sensors of the touchless sensing system 12 and provides the touchless input signal 14 indicative of a particular user input to the retrofit controller 20. In some embodiments, such as in the illustrated embodiment of FIG. 1, the touchless input signal 14 may be received by the retrofit controller 20, which may determine whether the touchless input signal 14 corresponds to a particular touchless user input. Appropriate signal conditioning circuitry (not shown) known to those skilled in the art, such as amplifiers, filters, multiplexers MUX, and / or the like, may be provided between the touchless sensors of the touchless sensing system 12 and the retrofit controller 20.

[0015] The retrofit controller 20 is connected to receive (accept) the touchless input signal 14 from the touchless sensing system 12. The retrofit controller 20 may be configured (e.g., suitably programmed) to generate a touchless control signal 16 and / or a touchless indication signal 18 for the target device 50 based on the touchless input signal 14 received from the touchless sensing system 12. The retrofit controller 20 may be operatively connected to a target control system 51 of the target device 50 to provide the touchless control signal 16 to the target control system 51, thereby causing the target control system 51 to operate the target device 50 based on the touchless control signal 16. For example, the retrofit controller 20 may be connected to the target control system 51 to deliver the touchless control signal 16 to the target control system 51, thereby causing the target control system 51 to operate the target device 50 based on a touchless user input detected by the touchless sensing system 12, such as a movement or estimated location of a part of a user's hand (e.g., a finger). The target control system 51 of the target device 50 can use touchless control signals 16 from the add-on controller 20 of the add-on interface device 10 in addition to or instead of touch-based control signals 19 from the touch-based interface (50A) of the target device 50 to control the operation of the target device 50.

[0016] The retrofit interface device 10 may optionally include a retrofit display 60. The retrofit display 60 may be part of a retrofit feedback mechanism 55 (discussed further below). The retrofit controller 20 may provide a video signal (61) through the retrofit display 60 to cause the retrofit display 60 to display a corresponding image, video, visual indicator, and / or the like. In this manner, the retrofit display 60 may provide some feedback to the user regarding the status of the retrofit interface device 10 (e.g., whether the retrofit interface device 10 detected a touchless input) and / or the status of the target device 50. In some embodiments, the target device 50 may constitute its own display or some other form of target output device (22) through which feedback regarding the status of the target device 50 may be provided to the user. For example, a target device 50 that is an elevator control panel may include a target output device (22) that is a flat screen display (e.g., for indicating the next floor at which the elevator car will stop) or may simply include a target output device (22) that illuminates the corresponding floor button (or portion of a button) that is pressed. For simplicity, the target output device (22) of the target device 50 may be referred to herein as the target display 22, and unless the context dictates otherwise, such reference should be understood to include any suitable form of target output device (22).

[0017] The target control system 51 of the target device 50 may control the target display 22 of the target device 50 via an appropriate target display signal 21. In some embodiments, the rear controller 20 of the rear interface device 10 may optionally be additionally or alternatively connected to the target display 22 of the target device 50 to provide a touchless display signal (output signal) 18 to the target display 22, thereby causing the target display 22 to display a corresponding image, video, visual indicator, and / or the like based on the touchless display signal 18. In some embodiments, the rear controller 20 of the rear interface device 10 may provide a touchless control signal 16 to the target control system 51 of the target device 50, which may cause the target display 22 to display a corresponding image, video, visual indicator, and / or the like (e.g., via the target display signal 21).

[0018] The touchless sensing system 12 includes one or more sensors for detecting touchless user input. In some embodiments, the touchless sensing system 12 of the retrofit interface device 10 and / or other retrofit interface methods and devices described herein includes one or more capacitive sensors 30 that are sensitive (i.e., have a capacitance that changes in response to an electric field) to an electric field in the vicinity of the sensor. For example, a gesture (e.g., a tap, finger movement, etc.) made by a user may cause a corresponding change in the electric field proximate to the capacitive sensor 30, thereby causing a detectable change in the capacitance of the capacitive sensor 30. For simplicity, a single capacitive sensor 30 is shown in FIG. 1, but it will be understood that the touchless sensing system 12 may be configured with multiple capacitive sensors 30 arranged appropriately. Advantageously, the capacitive sensor 30 can facilitate accurate short-range detection of touchless user interaction events (eg, hand gestures, finger gestures, and / or the like).

[0019] In some embodiments, the capacitive sensor 30 is fabricated or otherwise supported on a printed circuit board (PCB) (not shown). The printed circuit board PCB may be configured with a multi-layer printed circuit board PCB. In some such embodiments, one surface or layer of the printed circuit board PCB (known as the receiving surface or receiving electrode) may be exposed to the object to be sensed (e.g., a user's finger or hand). In some such embodiments, this printed circuit board PCB receiving surface or layer is not directly exposed, but rather is covered by a protective layer of non-conductive material. Each capacitive sensor 30 may be configured with a pair of electrodes (a transmitting electrode and a receiving electrode) between which an electric field is established. In some embodiments, multiple capacitive sensors 30 may be provided with a single transmitting electrode and multiple receiving electrodes (or vice versa). The transmitting electrodes may be located relatively farther from the user than the receiving electrodes (e.g., on a layer relatively farther from the user than the first layer of the printed circuit board PCB). A layer of non-conductive material may be disposed between the transmitting electrode(s) and the receiving electrode(s) to electrically insulate the transmitting and receiving electrodes.

[0020] In operation, an object (e.g., a human body part) desired to be sensed by the touchless sensing system 12 causes a disturbance in the electric field between the transmitting and receiving electrodes of the capacitive sensor 30 when the object is located at or in close proximity to the capacitive sensor 30, thereby changing the capacitance of the capacitive sensor 30. Each capacitive sensor 30 may be configured with suitable circuitry (not explicitly shown) that causes the capacitive sensor 30 to output a signal m (e.g., touchless input signal 14) that depends on the capacitance between its receiving and transmitting electrodes, and thus on the location of the user's body part (e.g., hand or finger). The capacitive sensor 30 of the touchless sensing system 12 may thereby be used (e.g., by a suitably configured controller, such as the retrofit controller 20 of the retrofit interface device 10, or an internal controller (not shown) of the capacitive sensor 30 or the touchless sensing system 12, etc.) to detect and track the movement of the user's body part (e.g., hand or finger). With a suitably positioned array of capacitance sensors 30 and corresponding output signals, a machine learning algorithm may be trained to infer location characteristics regarding a portion of a user's body. It will be appreciated that the output signals from the sensors (including capacitance sensors 30) may be interpreted by any suitable combination of one or more controllers, which may include the retrofit controller 20 of the retrofit interface device 10, an internal controller (not shown) of the capacitance sensors 30 or touchless sensing system 12, and / or the like.

[0021] For some human-machine interface HMI applications, the capacitive sensor 30 may provide the following non-limiting advantages over other types of sensors (e.g., 3D camera sensors) to facilitate touchless input detection: the detection range of the capacitive sensor 30 is centered around a few centimeters; firmware-level algorithms can be tuned to detect a single point of interest (e.g., the center of mass of a conductive object such as a user's finger); ambient factors (e.g., light, background motion, etc.) have limited effect on functionality; and the capacitive sensor 30 does not raise the same privacy concerns as camera-based sensors. These advantages, when compared to other types of sensors, allow the capacitive sensor 30 to be used (e.g., by a suitably configured controller, such as the retrofit controller 20 of the retrofit interface device 10 or an internal controller (not shown) of the capacitive sensor 30 or touchless sensing system 12) to provide more consistent and / or more accurate estimation of the types of user gestures that would typically be used to touchlessly interact with the retrofit interface device 10.

[0022] The layout arrangement, size, and / or shape of the receiving and / or transmitting electrodes of the capacitive sensor 30 can be customized (e.g., on a printed circuit board PCB). In some embodiments, the receiving electrodes of the capacitive sensor 30 are spaced apart, and the signal of each receiving electrode is based on the proximity (proximity, closeness) of a part of the human body (e.g., a finger). The capacitive sensor 30 and / or the touchless sensing system 12 may include an internal controller suitably configured (e.g., programmed) with signal processing algorithms that utilize these signals (from each receiving electrode) to determine the location (e.g., three-dimensional x, y, z coordinates; a suitable subset or representation of these coordinates; and the like) of a conductive part of the user's body (e.g., a fingertip). In some embodiments, the retrofit controller 20 is suitably configured (e.g., programmed) with these signal processing algorithms to determine the location of the conductive part of the user's body (e.g., a fingertip). The shape and / or size and / or location of each receiving electrode of the capacitive sensor 30 can be designed based on, for example, whether the touchless sensing system 12 is wired or battery powered, the desired detection range, the desired detection accuracy, and / or mechanical constraints.

[0023] In some embodiments, the layer of the printed circuit board PCB supporting the transmit electrode(s) comprises a copper expansion layer and / or a copper hatch layer (e.g., "criss-cross" copper traces) having dimensions (e.g., x- and y-dimensions in the plane of the printed circuit board PCB) that are slightly larger than the corresponding x- and y-dimensions of the receive electrodes.

[0024] In some embodiments, the receive and transmit electrodes of the capacitance sensor 30 are arranged (e.g., paired) in a concentric configuration in a rectangular cell, a schematic diagram of which is shown in Figure 2. In the embodiment of Figure 2, the capacitance sensor 30 comprises an array of individual capacitance sensor cells 30A, each of which is configured with a receive electrode 36 and a transmit electrode 34. For each capacitance sensor cell 30A, the receive electrode 36 surrounds the transmit electrode 34, or in an alternative arrangement, the transmit electrode 34 surrounds the receive electrode 36. In such an embodiment, each capacitive sensor cell 30A (i.e., each individual capacitive sensor cell 30A) is connected to a controller (e.g., an internal controller of the capacitive sensor 30 or touchless sensing system 12 (not shown), a retrofit controller 20 of the retrofit interface device (system) 10, and / or the like) via appropriate signal conditioning circuitry (e.g., amplifiers, filters, digital-to-analog converters, multiplexers, and / or the like) to output a unique capacitive detection signal (e.g., a signal from each receiving electrode 36). In such an embodiment, the voltage applied to each capacitive sensor cell 30A (e.g., to the transmitting electrode) may be configured to control the electric field output corresponding to the capacitive sensor cell 30A. Independently controlling the electric field of each individual capacitive sensor cell 30A in this manner allows precise control of the shape and boundaries of the overall sensor field of the capacitive sensor 30. For example, each receiving electrode 36 of each capacitive sensor cell 30A of the capacitive sensor 30 may be individually scanned to obtain a series of precise measurements. Advantageously, such a modular design (with independent control of each capacitive sensor cell 30A) offers good scalability due to the modular cell design (i.e., such a design can not only be expanded to larger sizes (e.g., by increasing the number of capacitive sensor cells 30A), but can also be shaped (e.g., by suitable positioning of the capacitive sensor cells 30A) while maintaining an acceptable detection / sensitivity volume).

[0025] In some embodiments, the capacitive sensor 30 is positioned and / or oriented to detect one or more positional and / or motional characteristics (e.g., velocity, acceleration, and / or direction associated with such velocity and / or acceleration) of a user's body part or other conductive object, typically the center or mass or centroid of the body part or other conductive object. It will be appreciated that suitable digital or analog signal processing (e.g., taking differentiation) may be used to derive motional characteristics from positional characteristics. Thus, the positional and / or motional characteristics of the centroid of the body part or other conductive object may be determined by the capacitive sensor 30 in cooperation with a suitably configured controller and / or signal processing hardware, such as the retrofit controller 20 of the retrofit interface device 10, the internal controller (not shown) of the capacitive sensor 30 or the touchless sensing system 12, and / or the like. For brevity, the remainder of this disclosure will refer to location characteristics, with the understanding that such location characteristics comprise movement characteristics that may be derived from the location characteristics or may be detected independently, unless the context dictates otherwise. For brevity, this disclosure may refer to the touchless sensing system 12 and / or its sensors (e.g., capacitive sensor 30 or any other sensor) performing a particular operation (e.g., determining a characteristic, detecting a characteristic, sensing a characteristic, generating a signal and / or information, and / or the like). Unless the context dictates otherwise, it should be understood that such sensors may perform such operations in cooperation with one or more appropriately configured controllers and / or signal processing hardware, such as the retrofit controller 20 of the retrofit interface device 10, a sensor or internal controller (not shown) of the touchless sensing system 12, and / or the like.

[0026] The position characteristic may correspond to a gesture. For example, the position characteristic may comprise a location of a user's body part (e.g., a centroid of a finger, hand, fist, wrist, forearm, etc.) when the user uses the body part to perform a gesture. Unless the context dictates otherwise, the term "gesture" (as used herein) refers to a location, movement, and / or configuration made by a user's body part and should not be construed as being limited to hand gestures. As a non-limiting example, the capacitive sensor 30 may be configured to detect hand gestures, finger gestures, an angle (e.g., proximity) of the body part movement relative to the sensor 30, a position of the body part, and / or the like. Such position characteristics may be used (e.g., by the add-on controller 20) to determine a particular touchless input to the target device 50. For example, if the touch-based interface (50A) includes a series of input buttons (e.g., such as an elevator control panel), such position characteristics may be used to determine a touchless input representing a selection (e.g., pressing) of one such button.

[0027] In some embodiments, the retrofit controller 20 is configured to detect, based on the touchless input signal 14, a position characteristic comprising a circular motion made by a portion of the user's hand, and generate a control signal (e.g., a touchless control signal 16, an actuator control signal 39 (described further below), or an emulator control signal (emulator signal) 43 (described further below)) that affects operation of the target device 50 based on the detected circular motion. In some embodiments, the retrofit controller 20 is configured, based on the touchless input signal 14, to detect a position characteristic comprising a portion of the user's hand that is lacking in movement for a threshold period (within a threshold), and generate a control signal (e.g., a touchless control signal 16, an actuator control signal 39 (described further below), or an emulator control signal 43 (described further below)) that affects operation of the target device 50 based on the detected lack of movement. In some embodiments, the retrofit controller 20 is configured to detect, based on the touchless input signal 14, a position characteristic comprising the proximity (e.g., for an appropriate period) of a portion of a user's hand to a portion of the retrofit display 60 or the target display 22, and thereby generate a control signal (e.g., a touchless control signal 16, an actuator control signal 39 (described further below), or an emulator control signal 43 (described further below)) that affects operation of the target device 50 based on the proximity of a portion of the user's hand to a portion of the retrofit display 60 or the target display 22. In some embodiments, the retrofit controller 20 is configured to not only detect, based on the touchless input signal 14, a position characteristic comprising the proximity of a portion of a user's hand to a portion of some other generally planar surface, such as a panel showing icons, virtual buttons, etc. (e.g., for an appropriate period), but also to generate a control signal (e.g., a touchless control signal 16, an actuator control signal 39 (described further below), or an emulator control signal 43 (described further below)) that affects operation of the target device 50 based on the proximity of a portion of the user's hand to a portion of the other planar surface.

[0028] The touchless sensing system 12 may optionally include one or more additional sensors 33 (e.g., a second set of one or more capacitive sensors 35, one or more optical sensors (40), described in more detail elsewhere herein, etc.) positioned and / or oriented to detect one or more secondary characteristics. Such secondary characteristics may be different (i.e., in addition to or instead of) the position characteristics detected by the capacitive sensors 30, but this is not required. Such secondary characteristics may include secondary position characteristics (and / or secondary motion characteristics) of the user's body part. Such body part may, but need not, be different from the body part associated with the primary position characteristics. The secondary position characteristics may correspond to a secondary gesture of the body part. Such additional sensors 33 may generate secondary position sensor signal(s) upon detecting the secondary position characteristics of the user's body.

[0029] Examples of secondary characteristics include, but are not limited to, the volume, shape, position, and / or angle of movement (e.g., approach) of a part of the user's body performing a gesture that may provide touchless input to the add-on interface device 10 that may in turn be used to effect control of the target device 50.

[0030] The additional sensor 33 may be disposed adjacent to the first capacitive sensor (30). The first capacitive sensor (30) and the additional sensor 33 may have different detection ranges or other detection characteristics different from each other. For example, the additional sensor 33 may have a detection range that is larger (e.g., in the xy plane corresponding to the printed circuit board PCB on which the capacitive sensor 30 is embodied) and / or more distant (e.g., in the z direction orthogonal to the xy plane) than the detection range of the first capacitive sensor (30). In some embodiments, a significant portion (e.g., 75%, 90%, or 100%) of the detection range of the first capacitive sensor (30) is disposed within the detection range of the additional sensor 33. In other embodiments, the detection ranges of the first capacitive sensor (30) and the additional sensor 33 are non-overlapping.

[0031] In some embodiments, the touchless input signal 14 (from the touchless sensing system 12 to the retrofit controller 20) comprises both a position sensor signal generated by the first capacitive sensor (30) and a secondary sensor signal generated by the additional sensor 33. In some embodiments, the retrofit controller 20 receives the touchless input signal 14 from the first capacitive sensor (30) and the additional sensor 33, and the retrofit controller 20 is configured to determine a position (location) characteristic and / or a secondary position characteristic. The retrofit controller 20 may be configured to generate the touchless control signal 16 and / or the touchless indication signal 18 based on either or both of the position sensor signal from the capacitive sensor 30 and the secondary sensor signal from the additional sensor 33.

[0032] In some embodiments, the retrofit controller 20 is configured to determine a position characteristic of a part of the user's body based on the touchless input signal 14 from both the capacitive sensor 30 and the additional sensor 33. For example, the retrofit controller 20 may be configured to process (e.g., scale, filter, weight, otherwise modify, interpret, calibrate, and / or the like) the information from the capacitive sensor 30 based on the information from the additional sensor 33. In some embodiments, the retrofit controller 20 is configured to determine the presence of a user located in the vicinity of the target device 50 based on the secondary sensor signal (from the additional sensor 33). The retrofit controller 20 may be configured to calibrate the position sensor signal (touchless input signal 14) from the capacitive sensor 30 based on the determination of the presence or absence of a user in the vicinity of the target device 50. The retrofit controller 20 may be otherwise configured to calibrate the position sensor signal (touchless input signal 14) from the capacitive sensor 30 based on the position characteristic detected by the additional sensor 33 and / or based on the output signal generated by such additional sensor 33.

[0033] The retrofit interface device 10 and / or other retrofit interface methods and devices described herein may be configured with voice detection technology (e.g., Alexa®, Google®, and / or the like) that includes a microphone (e.g., as an additional sensor 33) and a suitable voice detection algorithm (which may be programmed into the retrofit controller 20 and / or another controller by training suitable machine learning or artificial intelligence software). Such voice detection technology may be used in addition to or instead of the touchless capacitive sensor 30. Such voice detection technology may generally be included as part of the touchless sensing system 12 (e.g., as an additional sensor 33) and / or as part of the retrofit interface device 10. Such voice detection may be used to input, by way of example only, a buzzer number, a name, a license plate, an elevator floor, a door lock code, and the like.

[0034] In some embodiments, the retrofit interface device 10 and / or other retrofit interface methods and devices described herein may include an optical sensing system 40. The optical sensing system 40 may be embodied as part of the touchless sensing system 12 (e.g., as additional sensor 33) as shown in FIG. 1, but this is not required and the optical sensing system 40 may be a stand-alone component of the retrofit interface device 10. The optical sensing system 40 may be configured with one or more optical sensors (e.g., thermal camera, infrared camera, RBG camera, time-of-flight camera, stereoscopic camera, structured light camera, 3D camera, etc.). Such optical sensors may be sensitive to changes in electromagnetic radiation reflected from a portion of the user's body (e.g., changes caused by a gesture made by the user).

[0035] Advantageously, the optical sensing system 40 may be configured to enable the retrofit controller 20 to calibrate and thereby obtain more accurate position characteristics of a user's body part (e.g., a finger) as compared to using only the capacitance sensor 30. The optical sensing system 40 may detect any of a variety of characteristics of the body part (e.g., size and shape, left / right hand, fist / forearm position, hand angle, etc.) that may be used (e.g., by the retrofit controller 20) to calibrate or otherwise compensate for information from the capacitance sensor 30. By way of non-limiting example, the optical sensor (40) may be used to determine: the user being particularly tall (and thus may tilt their fingers downward when interacting with the capacitive sensor 30); the user's forearm being relatively close to the capacitive sensor 30 (and thus may affect the amplitude of the signal received from a particular capacitive sensor 30, or an estimate of the center of gravity of an object within the sensing volume may be biased or should be compensated for to the user's forearm rather than the user's fingertip); and the user approaching or interacting with the sensing volume of the capacitive sensor 30 with their left or right hand (which may cause the retrofit controller 20 to use different left / right machine learning inference engines to predict the position characteristics of the user's fingers), and / or the like.

[0036] In some embodiments, the retrofit controller 20 (and / or additional controllers (not shown) associated with the touchless sensing system 12) are configured with a machine learning data training model (e.g., a neural network model) that is trained to determine a position characteristic of a portion of a user's body by using the position sensor signal (e.g., a signal generated by the capacitance sensor 30) and the secondary sensor signal (e.g., a signal generated by the optical sensing system 40, and / or the second capacitance sensor 35, and / or some other additional sensor 33). Such position sensor signal and secondary sensor signal may be part of the touchless input signal 14. The retrofit controller 20 may be configured to generate the touchless control signal 16 and / or the touchless indication signal 18 based on the position characteristics determined from the position sensor signal and / or the secondary sensor signal. For example, the retrofit controller 20 may be configured to select the touchless control signal 16 from among a plurality of potential control signals based on the position characteristics determined from such position sensor signal and / or the secondary sensor signal.

[0037] In some embodiments, the optical sensing system 40 is configured to detect or otherwise confirm the presence (or lack of presence) of a user in the vicinity of the retrofit interface device (system) 10 and / or the target device 50. In such embodiments, after detecting the presence of a user, the capacitive sensor 30 may be configured to record a measurement of the noise level. The retrofit controller 20 can perform a system calibration (e.g., of the capacitive sensor 30 and / or of other sensors of the touchless sensing system 12 in general) based on the measured background noise.

[0038] The optical sensing system 40 may optionally include one or more lasers for emitting laser radiation (e.g., infrared laser light, near infrared laser, or other visible light) and one or more detectors (e.g., photodetectors) for detecting the emitted laser radiation. For example, the optical sensing system 40 may be configured with a detector positioned to receive laser radiation reflected from a user's body part (e.g., a body part making a gesture). The optical sensing system 40 may include suitable optics positioned to receive the emitted (e.g., reflected) laser radiation. In some embodiments, the one or more lasers of the optical sensing system 40 may be adapted for use as a primary sensor of the retrofit interface device 10 - for example, the optical sensing system 40 may be used in place of the capacitive sensor 30 described herein.

[0039] In some embodiments, the optical sensing system 40 is configured to generate a two-dimensional plane of laser radiation, sometimes referred to herein as a detection plane. This two-dimensional detection plane may have a normal vector that is substantially parallel (e.g., within 10° or 15°) to a normal vector of a plane of the rear display 60 (of the rear interface device 10) or the target display 22 (or the target device 50), a plane tangent to an external surface of the rear display 60 or the target display 22, and / or a plane tangent to some other surface of the target device 50 and / or the rear interface device 10 (e.g., a plane on which icons, virtual buttons, etc. are displayed). This two-dimensional detection plane may be positioned adjacent to (but spaced from) the plane of the rear display 60 or the target display 22, a plane tangent to the rear display 60 or the target display 22, and / or a plane tangent to a surface of the target device 50 and / or the rear interface device 10. In some embodiments, this two-dimensional detection plane may be positioned adjacent to (but spaced apart from) some other generally planar surface (e.g., a panel decorated with images such as icons). A location on this two-dimensional detection plane may be characterized by a pair of suitable coordinates (e.g., a Cartesian set of orthogonal x and y coordinates). The retrofit controller 20 may be configured to estimate the x and y coordinates of the user's body part based on a secondary sensor signal (e.g., a signal generated by the optical sensing system 40) indicative of the location where the body part intersects the two-dimensional detection plane.

[0040] The optical sensing system 40 may be disposed adjacent to the capacitive sensor 30 (or at another suitable location relative to the capacitive sensor 30). The optical sensing system 40 may be configured to detect the crossing of a part of the user's body (e.g., a finger) at a position (e.g., an x- and y-position on a detection plane) at a configurable distance (i.e., a z-direction orthogonal to the xy-directions) from the capacitive sensor 30, the rear display 60, the target display 22, and / or some other generally planar surface. The optical sensing system 40 may be configured to sense the lateral x- and y-coordinates of the user's finger as it crosses the detection plane as it approaches the capacitive sensor 30 and / or the rear display 60, the target display 22, and / or some other generally planar surface, and the capacitive sensor 30 may sense the orthogonal z-coordinate of the finger as it approaches the detection plane (e.g., before and / or after the finger crosses the detection plane of the optical sensor (40)). The capacitive sensor 30 may also detect the lateral (xy) coordinates of a finger (or other body part), although the lateral coordinates measured by the capacitive sensor 30 may be relatively coarse relative to the lateral coordinates detected by the optical sensing system 40. Such x and y coordinates detected by the optical sensing system 40 may be associated with the selection of a particular input (e.g., a virtual button) that may be displayed at a particular x and y coordinate on the rear display 60, the target display 22, and / or some other general plane. Detection of the z coordinate may help the touchless sensing system 12 anticipate the user's next selection. As a non-limiting example, the z coordinate may be used by the rear controller 20 as part of a "virtual click" detection algorithm (e.g., to ascertain whether the user is attempting to touchlessly press or click a virtual button or whether a body part of the user is simply dwelling within the sensing volume of the sensing system 12). Such detection of user interaction with a virtual button is discussed further below.As another example, the z coordinate may be used to provide feedback to the user (e.g., via a suitable retrofit feedback mechanism 55) to indicate that a part of the user's body is being detected. The capacitive sensor 30 and its ability to detect x, y and z coordinates may be used to determine an approach angle to the detection plane. This approach angle may be used in conjunction with more accurate lateral (xy) information from the optical sensing system 40 as part of a "virtual click" detection algorithm. Such detection of user interaction with a virtual button is discussed further below.

[0041] The optical sensing system 40 may comprise a number of optional features that may be used to control the spatial location and / or orientation of the two-dimensional detection plane of the optical sensing system 40. For example, the optical sensing system 40 may comprise one or more laser emitters that may be positioned next to the plane (or a tangential plane) of the target display 22 or touch-based input 50A or some other suitable surface of the target device 50 and / or add-on interface device 10, and may be oriented to direct radiation parallel with respect to a normal to such plane. In such implementations, the optical sensing system 40 may optionally comprise one or more mirrors (optimized for reflection of emitted radiation) oriented to reflect the laser radiation in a direction at an angle of about 45 degrees relative to the emitted laser radiation direction and at an angle of about 90 degrees relative to the emission direction to form a detection plane that is parallel to the plane (or a tangential plane) of the target display 22 or touch-based input 50A, or to the plane (or a tangential plane) of some other suitable surface of the target device 50 and / or add-on interface device 10. The optical sensing system 40 may include different combinations of mirror shapes / sizes, and / or different placement angles, and / or different combinations of other optical elements (e.g., lenses, waveguides, and / or the like) to provide a desired detection plane at a desired location and / or orientation. The optical sensing system 40 may be configured with commercially available laser-based optical sensors, such as a Neonode® or a zForce sensor®.

[0042] 3A-3C illustrate a touchless sensing system 12 including one or more capacitive sensors 30 and an optical sensing system 40 including multiple (e.g., two) optical sensors 42A, 42B (collectively, optical sensors 42). The optical sensing system 40 can be used with the retrofit interface device 10 (FIG. 1) and / or other retrofit interface methods and devices described herein, according to certain exemplary embodiments. In the exemplary embodiment of FIGS. 3A-3C, the optical sensors 42A, 42B are optically oriented to direct radiation toward (or across) the sensing area 32 of the capacitive sensor 30. In the exemplary embodiment shown in FIGS. 3A-3C, each of the optical sensors 42 includes a radiation source for emitting radiation and a radiation receiver (e.g., a photodetector, a receiving diode, etc.) for receiving radiation reflected from any object in the path of the emitted radiation. Unless the context dictates otherwise, the term "optically oriented" (as used herein) should be interpreted to mean that the optical sensor 42 described herein (or other sensor of the optical sensing system 40) may comprise any number (e.g., 0, 1, 2, 3, etc.) of suitable optical elements (e.g., lenses, mirrors, waveguides, etc.) arranged to shape, align, and / or direct radiation from a radiation source in an optically oriented direction.

[0043] As shown with the Cartesian axes shown in FIG. 3A, the radiation sensors (optical sensors 42) may be oriented or otherwise configured to together detect the x- and y-positions of an object (e.g., a finger) relative to the coordinate system of the capacitive sensor 30. For example, in the illustrated embodiment shown in FIG. 3A, not only can the optical sensor 42B detect the x′- and y′-positions (see x′, y′, z′ axes in FIG. 3A), but the optical sensor 42A can detect the x″- and y″-positions (see x″, y″, z″ axes in FIG. 3A). Using either or both of these detected positions (x′, y′) and (x″, y″), appropriate geometric calculations may be used (e.g., by the retrofit controller 20 and / or by a controller associated with the optical sensor 42 or the touchless sensing system 12) to determine the x- and y-positions in the coordinate frame of the capacitive sensor 30 (see X, Y, Z-axes in FIG. 3A, FIG. 3B). That is, the optical sensing system 40 may be configured to determine the lateral (xy) position of an object relative to the capacitive sensor 30 based on the lateral (x'-y') and / or (x''-y'') position detected by the optical sensor 42.

[0044] As described above, the optical sensing system 40 (e.g., the optical sensor 42) can accurately determine the x- and y-positions of an object (e.g., a finger) relative to the capacitive sensor 30, while the capacitive sensor 30 can accurately detect the distance of the object relative to the capacitive sensor 30 (i.e., the z-position of the object in the coordinate frame of the capacitive sensor 30). In this way, the optical sensing system 40 and the capacitive sensor 30 can complement each other. When an object such as a human finger enters a detection area 5 of the optical sensing system 40 defined by the optical sensor 42 (see FIG. 3B) (e.g., intersects one or more touchless detection planes 6 of the touchless sensing system 12 shown in FIG. 3B or otherwise enters the detection area 5 of the optical sensor), the optical sensing system 40 can be configured to determine the x- and y-positions of the object relative to the capacitive sensor 30 based on information from the radiation emission sensor (the optical sensor 42).

[0045] Since the sensing area (detection area 5) of the optical sensor 42 may be different from the sensing area 32 of the capacitive sensor 30, in some cases the x- and y-positions of the object may be determined first and tracked or temporarily stored until the object enters within the sensing area 32 of the capacitive sensor 30 and / or until the object intersects the touchless detection plane 6 of the touchless sensing system 12. In some embodiments, the z-position of the object is detected by the capacitive sensor 30 after the x- and y-positions of the object are determined by the optical sensing system 40. In other embodiments, the z-position of the object is detected by the capacitive sensor 30 before the x- and y-positions of the object are determined by the optical sensing system 40. In some embodiments, the touchless sensing system 12 is configured to register the x- and y-positions of the object (detected by the optical sensing system 40) after the z-position of the object reaches a configurable threshold distance from the capacitive sensor 30 and / or from some other reference surface, such as the surface of the target display 22 (or a plane tangent to the target display 22). For example, touchless sensing system 12 may be configured to register an x-,y-position when an object crosses touchless detection plane 6. This threshold z-distance at which an x-,y-position is registered may be evaluated by capacitance sensor 30. In some embodiments, retrofit controller 20, or some other controller associated with touchless sensing system 12, may be configured to make a determination (e.g., detection of a particular gesture, or detection of a gesture based on the registered x-,y-position).

[0046] In some embodiments, the working plane (6) is defined at a fixed position relative to the capacitive sensor 30 and / or at a fixed position from some other reference plane, such as the surface of the target display 22 or a plane tangent to the target display 22--(e.g., at a distance corresponding to the location where the rays 5 emitted by the optical sensors 42A, 42B intersect)--. In other embodiments, the position of the working plane (6) may be adjusted by appropriate programming (or user setting) of the retrofit controller 20 or of some other controller associated with the touchless sensing system 12.

[0047] FIG. 3C is a schematic diagram of an exemplary arrangement of a capacitive sensor 30 and an optical sensing system 40 according to certain embodiments. In the example shown in FIG. 3C, the touchless sensing system 12 includes a plurality (e.g., two) optical sensors 42A, 42B (collectively, optical sensors 42) optically oriented to emit radiation from a position that may be coplanar with the capacitive sensor 30. In some embodiments, the optical sensor 42 and the capacitive sensor 30 are formed as part of an integrated module (e.g., as part of a thin slate having a shape like a tablet device). In some embodiments, the optical sensor 42 and the capacitive sensor 30 are encapsulated in a medium 44, such as a resin, to fix the relative positions of the optical sensor 42 and the capacitive sensor 30. The medium 44 may be transparent at the wavelength of the optical sensor 42. In some embodiments, the medium 44 may provide optical elements in the path of radiation emitted (and / or received) by the optical sensor 42 to shape, position, and / or direct (orient) such radiation. The medium 44 may comprise glass or quartz. In some embodiments, the capacitive sensor 30 is formed using a transparent conductive material such as indium tin oxide (ITO) and is located within the glass medium (44), with the optical sensor 42 oriented at a desired angle behind the glass. Advantageously, the glass medium (44) may be used to encapsulate the capacitive sensor 30 and the optical sensing system 40 to form a flat (non-protruding) design that can be flush mounted to another surface (e.g., a wall or horizontal surface). This flat mounting contrasts with prior art optical designs that protrude away from the surface (the wall or horizontal surface that is mounted in or on). This flat design may prevent tampering, for example, with a retrofit interface device.

[0048] 3A-3C, in some embodiments, a second capacitive sensor 35 (not shown in FIGS. 3A-3C) may be positioned adjacent to the first capacitive sensor (30) and oriented to sense the user's hand interacting with the first capacitive sensor (30). The second capacitive sensor 35 may then detect a second distance estimate (coordinates) that provides another degree of information about the object (hand / fingers) relative to the first capacitive sensor (30). This information, along with the known orientation of the second sensor, may help determine the orientation of the hand / fingers, which can provide additional information useful in identifying the particular input (e.g., a particular virtual button) that the user is attempting to interact with.

[0049] Additional optical sensors (not shown in FIGS. 3A-3C) can be deployed by stacking them on the existing optical sensors 42A, 42B and can provide more coverage and accuracy of detected objects. The additional optical sensors can be utilized to determine the orientation of the object as well as provide additional depth information for further analysis. One can imagine optical sensors at different angles being triggered at different points of the finger's movement towards a virtual button or other touchless input. This can not only determine a "tap-vector" or trajectory, but also use this information to determine which virtual button the user intended to select. For example, the finger may be placed over the virtual button corresponding to "2" but pointing to the right, so the retrofit controller 20 can be configured (by appropriate software) to determine that the user intended to click "3" if the angle of the tap vector is sufficient. In addition to or as an alternative to providing more optical sensors (e.g., in addition to the optical sensors 42A, 42B), other sensors and techniques can be used to determine the "tap-vector". Such additional or alternative techniques may include, but are not limited to, determining a tap-vector based on first and second three-dimensional position estimates based on information from a capacitive sensor (e.g., upon detecting the crossing of a first and second threshold distance (z-coordinate distance) away from the plane of the virtual button). It is noted in this regard that the capacitive sensor may detect lateral (x, y) coordinates, but these lateral coordinates may be less accurate than the coordinates detected by the optical sensor (40). Such additional or alternative techniques may include, but are not limited to, determining a tap-vector based on a first three-dimensional detection based on information from the capacitive sensor 30 (e.g., upon detecting the crossing of a first threshold (z-coordinate) away from the plane of the virtual button) and a second three-dimensional detection based on the lateral information (x, y) coordinates based on information from the optical sensor (40) and based on the z-coordinate determined by the capacitive sensor 30.In any of these tap vector estimation techniques, a suitable time threshold may be used as another criterion related to tap decision making.

[0050] The optical sensor (40) (such as sensors 42A, 42B) can detect multiple points at a time. The user's intent to work with either multiple points or a single point can be determined within a threshold time window of an object entering the detection field of the optical sensor. Depending on how many points are detected within the threshold time window, the optical sensor (40) can enter into a single point or multi-point mode of operation. Multi-point accuracy can be better performed by additional optical sensor(s) (i.e., in addition to sensors 42A, 42B shown in Figures 3A-3C), as described above. Multi-point detection can be used for two-finger gestures, such as pinch-to-zoom and pinch-to-pan.

[0051] Further improvement in the range and accuracy of the optical sensor (40) (e.g., optical sensors 42A, 42B) can be achieved by oscillating the detection plane of the optical sensor through a range of angles (e.g., by using a rotary actuator, etc.). Such oscillation of the detection plane allows the optical sensor itself to carry depth (z-direction) information in addition to the z-direction information obtained from the capacitive sensor 30. Such an oscillating detection plane also allows for lateral xy position to be obtained at different object depths (i.e., different z coordinates) as opposed to a fixed depth.

[0052] In some embodiments, the retrofit controller 20 is configured to estimate a line-of-sight vector of a human user and / or a distance between the human user and the retrofit display 60 based on the touchless input signal 14 from the touchless sensing system 12. The retrofit controller 20 may be configured to estimate the line-of-sight vector based on an estimate of a position of a portion of the user's hand (e.g., fist, wrist, forearm, etc.) and / or an estimate of an approach angle of a portion of the user's hand. The line-of-sight vector may also be estimated based on a position of the user's head / torso in some cases. Information for making such line-of-sight estimation may be provided by the touchless sensing system 12.

[0053] The user's gaze and / or position may be detected, in some cases, by using additional gaze sensors (e.g., capacitance sensors, optical sensors, cameras, and / or others (not shown)) that may be provided as part of the touchless sensing system 12. For example, suitable optical sensors may be used to detect the user's head pose, gaze, and hand position (i.e., left or right hand). In some embodiments, the capacitance sensor 30 may be used to detect the position and / or approach angle of a portion of the user's hand (e.g., by using machine learning algorithms). In some embodiments, the gaze sensor comprises a height or elevation sensor (e.g., a laser-based sensor) configured to determine the height of the retrofit interface device 10 (or a particular portion thereof). In such embodiments, the user's gaze may be estimated based on the average human height along with the height of the retrofit interface device 10. This height information may be used in conjunction with the other detections described above to estimate the user's gaze.

[0054] Returning to FIG. 1 , the retrofit interface device 10 (e.g., the retrofit controller 20) may be connected to communicate directly with the target control system 51 of the target device 50 by using the touchless control signal 16. In this manner, the retrofit interface device 10 may replace or complement (i.e., provide an alternative or additional) input functionality of the existing touch-based input 50A and touch-based control signal (input signal) 19 of the target device 50. In some embodiments, the retrofit interface device 10 is provided with wiring having one or more suitable communication protocols (e.g., USB, SPI, I2C, CAN, RS-232, serial, etc.). The wiring of the retrofit interface device 10 may be provided in an easy plug package for easy integration with the target device 50, and the target device 50 may be provided with wiring having one or more corresponding communication protocols. For example, if the existing touch-based input 50A of the target device 50 is a USB mouse or similar USB input device, the retrofit interface device 10 may be directly plugged into a USB jack of the target device 50 to provide the touchless input functionality to the target device 50. The connection of the retrofit interface device 10 with the target device 50 may additionally or alternatively be achieved by coupling to an existing electrical node (not shown) of the target device 50. For example, such an electrical node may comprise the same node to which the touch-based input 50A is connected to the target control system 51. In some embodiments, the connection of the touch-based input 50A may be disconnected from such electrical node, whereby the retrofit interface device 10 replaces the functionality of the touch-based input 50A with a touchless input, although this is not required.

[0055] The design of the retrofit interface device 10 can take several scalable forms. Some embodiments of the retrofit interface device 10 have a modular design that allows manufacturers, and OEM third parties, or end users themselves, to connect a single "block" unit to a larger unit to cover a larger touch-based interface. A single unit may be applicable for a target device 50 that is a street light signal (which has only a single button), while 12 units combined in a 3×4 grid may be used if the target device 50 has an intercom system with a keypad. This modular design would mean that each module block may comprise a capacitive sensor, a method for visual or haptic feedback, and a touch actuation component TAC, although some such implementations may share some such components between the module blocks.

[0056] FIG. 1A is a schematic depiction of a retrofit interface device 10A according to another exemplary embodiment of the present invention. Similar to the retrofit interface device 10 described above, the retrofit interface device 10A retrofits a target device 50 (having a touch-based input 50A) to provide touchless input to the target device 50. Instead of (or optionally in addition to) interfacing with the target device 50 using touchless control signals 16 (as in the case of the retrofit interface device 10 - see FIG. 1), the retrofit interface device 10A includes one or more actuators 37 that are positioned and movable to physically interact with the touch-based input 50A of the target device 50. The retrofit interface device 10A can also optionally communicate with the target display 22 of the target device using touchless display signals 18. The actuators 37 may be referred to herein in the singular or plural, and it is understood that the retrofit interface device 10A may be configured with one or more actuators 37. The actuator 37 may be connected to the retrofit controller 20 of the retrofit interface device 10A to receive an actuator control signal 39 from the retrofit controller 20. The actuator control signal 39 may be used by the retrofit controller 20 to control the movement of the actuator 37. The retrofit interface device 10A may include suitable actuator drive circuitry (not explicitly shown), such as power supplies, amplifiers, and / or the like, that may be used to drive the actuator 37 in response to the actuator control signal 39. In response to the actuator control signal 39, the actuator 37 may move to physically interact with a touch-based input 50A of the target device 50, thereby providing a touchless (from a user's perspective) input to the target device 50.

[0057] The actuator 37 of the retrofit interface device 10A can physically interact with and / or exert a force on the touch-based input 50A of the target device 50 by pushing, pulling, sliding, rotating, or otherwise interacting with the touch-based input 50A of the target device 50. The touch-based input 50A of the target device 50 may have many forms, and as a result, the actuator 37 may have many different types of actuators 37 that are desirable to interact with a particular touch-based input 50A. As non-limiting examples, the actuator 37 may be used to press a keypad button, turn, pull, or push a dial or knob, rotate a door handle, slide a latch or slider input, touch a touch sensor, and / or the like. The actuator 37 may be suitably positioned with respect to the target device 50 to interact with the touch-based input 50A of the target device 50 in a suitable manner.

[0058] In some embodiments, while the touch-based input 50A of the target device 50 comprises a touch screen interface, the actuator 37 may be used to touch the touch screen interface (e.g., with a conductive stylus, tip, and / or the like). In some such embodiments, the actuator 37 may be positioned, moved into position, and / or sized to interact with a portion of the touch screen where input may be provided (e.g., a portion where touch buttons, a keypad, or a keyboard typically appear) such that the actuator 37 does not obstruct visibility of other portions of the touch screen interface.

[0059] In some embodiments, the actuators 37 may comprise electromechanical actuators driven by electrical power, such as motors, solenoids, linear actuators, hydraulic actuators, piezoelectric actuators, and / or the like. Some embodiments of the retrofit interface device 10A may be configured with an assortment of actuators 37 for retrofitting to a target device 50 comprising an assortment of touch-based inputs 50A. One exemplary embodiment of the actuators 37 comprises a discrete matrix array of solenoids or motors arranged in a grid and positioned or configurable to press corresponding buttons of the touch-based inputs 50A, such as keypad buttons, keyboard buttons, touch screen virtual buttons, and / or the like. In some such embodiments, there may be a one-to-one correspondence between the actuators 37 and individual inputs (e.g., buttons, virtual buttons, touch screen positions, and / or the like) of the touch-based inputs 50A, although this is not required. In some embodiments, one actuator 37 may be used to position, move into position, size, and / or otherwise interact with two or more individual inputs (e.g., buttons, virtual buttons, touch screen locations, and / or the like) of touch-based input 50A.

[0060] In some embodiments, the actuator 37 is supported for movement relative to the touch-based input 50A. For example, the actuator 37 may be supported on a movable gantry or on a framework with a movable head such that the actuator 37 may be moved into proximity with the touch-based input 50A (or a portion thereof) to facilitate interaction with the touch-based input 50A. For example, the actuator 37 may comprise a grid arrangement of solenoids supported on a "gantry," on a framework with a movable head, or the like, such that the grid of solenoids may be positioned adjacent to keypad buttons to depress the keypad buttons. In some embodiments, a suitable system comprising the use of magnetic and / or electric fields may be used to position one or more appropriate actuators 37 for interaction with individual inputs of the touch-based input 50A.

[0061] The retrofit interface device 10A may include solenoid actuators (37). The solenoid actuators (37) include one or more solenoids that utilize a change in electrical current to generate a magnetic field within a respective coil, thereby creating linear motion. When actuated, a metal rod located inside the solenoid is either pushed out of the coil or pulled into the coil. The retrofit interface device 10A may be configured with a matrix of discrete (e.g., independently controllable) solenoid actuators (37) in some embodiments. As a non-limiting example, a grid arrangement of solenoid actuators (37) located adjacent to buttons (or virtual buttons) of the touch-based input 50A of the target device 50 could be used to press a given button. The solenoid actuators (37) may have a compact mechanical design when compared to the motor-based actuators 37.

[0062] The retrofit interface device 10A may additionally or alternatively include a motor-based actuator 37. For example, the motor-based actuator 37 may be configured with one or more servo motors, DC motors, or stepper motors for pushing, pulling, linearly actuating, or rotating the touch-based input 50A of the target device 50. The motor-based actuator 37 may be equipped with or equipped with a suitable mechanism for manipulating the direction, power, speed, and / or other characteristics of the force generated by each motor. A motor-based actuator 37 configured with or equipped with such a mechanism may be utilized, for example, to generate a linear force to press a button of the touch-based input 50A or to rotate a set of gears (e.g., a gear train) to generate sufficient power and / or range to turn a door handle of the touch-based input 50A. The motor-based actuator 37 may be more power efficient when compared to a solenoid-based actuator 37.

[0063] As mentioned above, the actuators 37 are not limited to solenoids and / or motors. Other suitable actuators 37 can include, by way of non-limiting examples, linear actuators, hydraulic / pneumatic actuators, piezoelectric actuators, thermal actuators, spring-based actuators, magnetic actuators, electrostatic actuators, and / or the like. Figure 9 is a schematic exploded perspective view of the retrofit interface device 10A of Figure 1A including multiple linear actuators (37) according to certain embodiments.

[0064] In other respects, retrofit interface device 10A may be similar to retrofit interface device 10 described herein. Unless the context indicates otherwise, references to features of retrofit interface device 10 should be understood to be applicable to retrofit interface device 10A.

[0065] FIG. 1B is a schematic diagram of a retrofit interface device 10B according to another exemplary embodiment. Similar to the retrofit interface devices 10, 10A described herein, the retrofit interface device 10B retrofits a target device 50 (having a touch-based input 50A) to provide touchless input to the target device 50. In the case of the retrofit interface device 10B of the embodiment of FIG. 1B, the touch-based input 50A of the target device 50 is configured with a target touch screen (input) 50B. Instead of (or optionally in addition to) interfacing with the target device 50 by using touchless control signals 16 (as was the case with the retrofit interface device 10 - see FIG. 1) or using moving actuators 37 (as was the case with the retrofit interface device 10A - see FIG. 1A), the retrofit interface device 10B is configured with one or more touch emulators 41 that are controllable to electrically interact with the target touch screen (touch panel) 50B of the target device 50 by emulating the touch of a user's body. The retrofit interface device 10B can also optionally communicate with a target display 22 of the target device using touchless display signals 18. In some embodiments, the retrofit interface device 10B may optionally include an actuator (such as actuator 37 of the retrofit interface device 10A) for interacting with touch-based input 50A.

[0066] The touch emulator 41 may be referred to herein in the singular or plural, and it is understood that the retrofit interface device 10B may be configured with one or more touch emulators 41. The touch emulator 41 may be connected to the retrofit controller 20 of the retrofit interface device 10B to receive emulator control signals 43 from the retrofit controller 20. The emulator control signals 43 may be used by the retrofit controller 20 to control electrical characteristics of the touch emulator 41. In response to the emulator control signals 43, the touch emulator 41 may emulate human touch by electrically interacting with the touch-based input 50A (target touch screen 50B) of the target device 50, thereby providing touchless (from a user's perspective) input to the target device 50.

[0067] 4A shows a schematic diagram of an exemplary capacitive target touch screen 50B of an exemplary target device 50. The target touch screen 50B is typically coated with a layer of transparent conductive material (e.g., indium tin oxide (ITO)). When the target touch screen 50B is touched by a conductive element (such as a user's finger), the local electrical properties (specifically the local charge, local capacitance, and / or local electric field) change at the location of the touch, and the touch event is observed by the target touch screen 50B and / or by the target control system 51 of the target device 50 (see FIG. 1B).

[0068] FIG. 4B illustrates a schematic of a touch emulator 41 according to a particular exemplary embodiment. The touch emulator 41 of the FIG. 4B embodiment includes one or more elements (e.g., cells) of a transparent conductive film 45, which in turn may include indium tin oxide (ITO), conductive carbon nanotubes, graphene, polymeric materials, thin metal (e.g., silver) nanowires, and / or the like. The transparent conductive film 45 may be disposed adjacent to (e.g., within suitable interaction proximity of) one or more regions of the target touch screen 50B. For example, the transparent conductive film 45 may directly overlay (contact) the target touch screen 50B. The transparent conductive film 45 may be controllably and selectively connectable to ground (i.e., to the ground of the target touch screen 50B or to a ground sufficiently close to the potential of the earth of the target touch screen 50B) to change the local charge, local capacitance, and / or local electric field, thereby emulating a "touch" at that region of the target touch screen 50B. For example, in the exemplary illustrated embodiment of Figure 4B, the transparent conductive film 45 is divided into a plurality of cells 52, and each cell 52 may be in direct physical contact (e.g., via a conductive adhesive or applied pressure) with a corresponding area of ​​the target touch screen 50B. It will be understood that the number of cells 52, and the number of corresponding areas of the target touch screen 50B shown in Figure 4B are for illustration purposes, and that the number and / or layout of the cells 52 may vary in different embodiments.

[0069] Each cell 52 may be controllably and selectively connectable to ground via electrical conductors 53 (e.g., wires, printed circuit board PCB traces, and / or the like) and suitable switches (shown in FIG. 4B as switches 54A-54L (collectively switches 54)). Each cell 52 of the transparent conductive film 45 may be electrically isolated from the other cells 52. Each electrical conductor 53 may be electrically isolated from the other electrical conductors 53. The cells 52 may be laid out in a suitable grid arrangement or other suitable arrangement (which may depend on the location of the virtual buttons on the target touch screen 50B). The number, size, and / or arrangement of the cells 52 may determine the touch emulation resolution of the touch emulator 41 and the retrofit interface device 10B. In some embodiments, the electrical conductors 53 are transparent (e.g., made of one or more transparent conductive materials) or made of thin (narrow) strands of conductive material (e.g., metal nanowires) that appear transparent to the user. The electrical conductors 53 may be electrically connected to the cells 52 with a transparent conductive adhesive or solder. The electrical conductors 53 may be positioned to minimize the amount of wiring between the target touch screen 50B and the user.

[0070] A switch 54 (e.g., a MOSFET transistor, a switching relay, and / or the like) is electrically connected between each cell 52 and electrical ground. The retrofit controller 20 of the retrofit interface device 10B may control the switches 54 by using the emulator control signal 43 (see FIG. 1B ). Specifically, the retrofit controller 20 may be configured to switch a particular switch 54 between an ON configuration in which the switch 54 is conductive and thereby connects the corresponding cell 52 to ground, and an OFF configuration in which the switch 54 is non-conductive. When the switch 54 is in the ON configuration and thus conductive, the target touch screen 50B of the target device 50 may interpret this (ON configuration) as a touch event at the location of the corresponding cell 52. When the touchless sensing system 12 detects a touchless input from a user (e.g., a finger tapping gesture), the retrofit controller 20 may switch one or more switches 54 to the ON position by using the emulator control signal 43. The retrofit controller 20 may be configured to control the switch (electrical switch) 54 in real time (i.e., with little or no time delay between a touchless tap detected by the touchless sensing system 12 and a touch-emulated tap).

[0071] In some embodiments, the retrofit interface device 10B is configured to establish a spatial correspondence between the touchless sensing system 12 and the cell 52 of the touch emulator 41. Advantageously, the spatial correspondence may allow a user to activate the cell 52 of the touch emulator 41 by inputting a touchless input (e.g., a finger tapping gesture) directly adjacent to the cell 52 (e.g., a touchless tap may be detected by the touchless sensing system 12, and the retrofit controller 20 may use the emulator control signal 43 to turn on a switch 54 corresponding to the cell 52 adjacent to the location of the user's touchless tap). Preferably, the switch (electrical switch) 54 has low stray / parasitic capacitance to establish a strong connection to electrical ground when switched on. The retrofit controller 20 may be located adjacent to the target touch screen 50B or behind the target touch screen 50B. In some embodiments, the retrofit controller 20 may be integrated with the touchless sensing system 12.

[0072] The retrofit interface device 10B (via the touchless sensing system 12) may be configured to detect a wide variety of touchless inputs as well as actuate the target touch screen 50B in a variety of ways. For example, a touchless input gesture detected by the touchless sensing system 12 may be transient (e.g., in the case of a tap or double tap gesture), so the retrofit interface device 10B may be configured to temporarily electrically ground the cell 52 and simulate a corresponding gesture on the target touch screen 50B. As another example, a touchless input detected by the touchless sensing system 12 may be held for a period of time, so the retrofit interface device 10B may be configured to electrically ground the cell 52 for a corresponding period of time to simulate a longer "touch hold" on the target touch screen 50B. As another example, because touchless inputs detected by the touchless sensing system 12 may include hover, drag, swipe, pinch, etc., the retrofit interface device 10B may be configured to electrically ground one or more of the cells 52 simultaneously, over a period of time, or sequentially to emulate such movements across the conductive surface of the target touch screen 50B.

[0073] In some embodiments, the transparent conductive film 45 of the touch emulator 41 and the capacitive sensor 30 of the touchless sensing system 12 are provided on the same printed circuit board PCB, although this is not required. In some such embodiments, the transparent conductive film 45 and the capacitive sensor 30 may be electrically isolated from each other. For example, a printed circuit board PCB (e.g., a transparent printed circuit board PCB) may include a transmitting electrode and a receiving electrode of the capacitive sensor 30 located on two of the layers of the printed circuit board PCB that are distal from the target touch screen 50B of the target device 50. The cells 52 of the transparent conductive film 45 may be located on one of the layers of the printed circuit board PCB so as to be proximate (e.g., closest to) the target touch screen 50B for contact with the target touch screen 50B.

[0074] In some embodiments, the retrofit interface device 10B and / or the touch emulator 41 may include a coating or other layered material to provide waterproofing or water resistance. In some embodiments, the retrofit interface device 10B and / or the touch emulator 41 may be constructed to withstand various ranges of temperature and humidity (i.e., to match the target touch screen 50B of the target device 50 to which it is being retrofitted).

[0075] Advantageously, the retrofit interface device 10B can be easily integrated into an existing target touch screen 50B (i.e., a common type of touch-based interface in public spaces) without requiring (although such connection is optional) a direct connection of the retrofit interface device 10B to the target control system 51 of the target device 50. The retrofit interface device 10B can be configured to allow a user to interact with the existing target touch screen 50B of the target device 50 in a touchless manner.

[0076] In other respects, retrofit interface device 10B may be similar to retrofit interface device 10 described herein. Unless the context dictates otherwise, references to features of retrofit interface device 10 should be understood to be applicable to retrofit interface device 10B.

[0077] 1C is a schematic diagram illustrating an embodiment of an additional aspect of the present invention relating to the attachment of a retrofit interface device 10C to a target mechanical device 150. The target mechanical device 150 does not have its own controller, but instead has a touch-based input 50A (which may be described as a self-actuated object 150A), in which a user's interaction with the self-actuated object 150A (in a touch-based context) affects the operation of a corresponding target mechanism 150B of the target mechanical device 150. Non-limiting examples of target mechanical devices 150 include door handles / knobs (the self-actuated object 150A is a handle / knob mechanism while the target mechanism 150B is a tongue / bolt), mechanical locking mechanisms (the self-actuated object 150A is a lock handle while the target mechanism 150B is a lock shaft), light switches (the self-actuated object 150A is a light switch lever while the target mechanism 150B is a switch contact), and / or the like. Retrofit interface device 10C may be configured with its own actuator 37 to interact with each self-actuated object 150A. In other respects, retrofit interface device 10C may be similar to retrofit interface device 10A (FIG. 1A) described herein, and unless the context dictates otherwise, the features of retrofit interface device 10C should be considered similar to the features of retrofit interface device 10A described herein.

[0078] In some embodiments, the retrofit interface device 10C and the target mechanical device 150 may be manufactured or otherwise provided as a single device. In the original case, the self-actuated object 150A is manufactured to operate like a normal touch-based device, while the retrofit interface device 10C is provided to enable the self-actuated object 150A to be operated touchlessly. In some such embodiments, the retrofit interface device 10C may be provided with a retrofit display 60 for user interaction or instruction. For example, the target mechanical device 150 may be configured with a light switch that is pressed or flicked with a hover tap or swipe without causing a physical touch of the light switch. As another example, the target mechanical device may be configured with a door handle that is made rotatable by hovering the FOB over an appropriate virtual button, etc.

[0079] Any of the retrofit interface devices described herein (e.g., retrofit interface devices 10, 10A, 10B, 10C) may facilitate interaction with a user's smartphone or handheld computing device (portable computing device) 47 (e.g., communication between the retrofit controller 20 and the handheld computing device 47 via mobile device signals 49). As a non-limiting example, the retrofit interface device 10 may be equipped with a QR code, NFC / RFID tag, WIFI / Bluetooth connectivity, and / or the like, to enable a user to not only wirelessly provide input (e.g., button selections, alphanumeric characters, slide gestures, and / or the like) to the retrofit controller 20 via the mobile device signals 49, but also utilize the keyboard / keypad input of the handheld computing device 47 to subsequently provide corresponding input to the target device 50. This allows a user to not only utilize their handheld computing device 47 to interact with the retrofit interface device 10, but then to interact with the target device 50, such that the user need only physically touch their handheld computing device 47 to communicate (e.g., provide input) with the target device 50. In some embodiments, the handheld computing device 47 may be used by the user in addition to or instead of the touchless sensing system 12 to provide input signals (mobile device signals 49) to the retrofit controller 20.

[0080] The retrofit interface device 10 may be configured (e.g., by appropriate programming of the retrofit controller 20) with a software application (e.g., a smartphone app) that facilitates quick connection to and quick automatic disconnection from the retrofit interface device 10. The retrofit interface device 10 may be configured (e.g., by appropriate programming of the retrofit controller 20) with a software application that facilitates quick connection to and / or quick disconnection from a handheld computing device 47.

[0081] In some embodiments, the connection between the handheld computing device (appliance) 47 and the retrofit interface device 10 is limited to a particular user's handheld computing device 47 or to one particular handheld computing device at a time to prevent the target device 50 from receiving user input from multiple handheld computing devices 47 or from receiving input from multiple handheld computing devices 47 at one time. Such an embodiment may be suitable for secure applications (e.g., for entering pin codes at POS terminals and / or the like) and / or personal use applications (e.g., parking meters and / or the like). One exemplary implementation of such an embodiment is configured to use a timer configured to disconnect each handheld computing device (user device) 47 from the retrofit interface device 10 after a threshold period (e.g., after a threshold period of inactivity) and / or to use an event detection algorithm that disconnects each handheld computing device (user device) 47 from the retrofit interface device 10 to allow a new user to connect after an event is detected or completed. Another additional or alternative implementation of such an embodiment includes configuring the retrofit interface device 10 to require that the user device (handheld computing device 47) maintain a QR code or NFC tag within the field of view of a camera or NFC sensor of the handheld computing device 47 during use in order to communicate with the retrofit interface device 10. Another additional or alternative implementation of such an embodiment includes providing the retrofit interface device 10 with appropriate communication hardware and / or software (e.g., a WIFI and / or Bluetooth antenna) that can be configured to estimate the proximity of the handheld computing device 47 such that the connection between the handheld computing device 47 and the retrofit interface device 10 can be revoked as soon as the proximity of the handheld computing device 47 reaches a threshold distance away from the retrofit interface device 10.

[0082] The retrofit interface device 10 can include a variety of methods for accepting communication of input selections from the handheld computing device 47. Exemplary methods include, but are not limited to, coded audio signals, LED light signals, viewing alphanumeric text or QR codes displayed on a smartphone screen and / or the like (e.g., a piece of paper) using a camera system, NFC or RFID.

[0083] Any of the retrofit interface devices described herein (e.g., retrofit interface devices 10, 10A, 10B, 10C) may include one or more retrofit feedback mechanisms 55 for providing feedback to a user interacting with a target device 50 via the retrofit interface device 10. Such retrofit feedback mechanisms 55 may be controlled by the retrofit controller 20 through the use of appropriate feedback control signals 57. As discussed elsewhere herein, the retrofit display 60 of the retrofit interface device 10 may form part of the retrofit feedback mechanisms 55. A retrofit interface device 10 according to certain embodiments may include one or more of the following retrofit feedback mechanisms 55:

[0084] The retrofit interface device 10 may include a retrofit feedback mechanism 55 that includes an ultrasonic and / or acoustic wave transmission (collectively referred to herein as sonic wave transmission (SWT)) feedback mechanism. The sonic wave transmission SWT feedback may be used to create touchless haptic feedback by simulating a button click sensation that can be felt by human skin (e.g., the sensation typically felt when a finger strikes a key on the surface of a conventional keyboard, or the left / right buttons of a touchpad mouse, or other suitable sensation), thereby indicating to a user that an input has been made (e.g., a selection has been made, a button has been pressed, and / or the like).

[0085] A typical off-the-shelf printed circuit board PCB with a capacitive sensor 30 does not support sonic transmission SWT. A sonic transducer located behind such a printed circuit board PCB would not be felt by a finger present on the other side of the printed circuit board PCB. The retrofit interface device 10 (and / or other retrofit interface devices 10A, 10B, 10C described herein) may have a design configuration that allows a capacitive sensor and a sonic transmission SWT to coexist in a single form factor. Touchless haptic feedback may be achieved by positioning the touchless sensing system 12 (e.g., capacitive sensor(s) 30) and sonic transducer 55A (e.g., on a printed circuit board PCB or otherwise) to allow acoustic energy from the sonic transmission SWT transducer (sonic transducer 55A) to reach the sensing area (sensing zone) of the touchless sensing system 12 while not impeding (or unduly interfering) the sensing operation of the sensor of the touchless sensing system 12.

[0086] For example, the retrofit interface device 10 and / or the retrofit interface device 10A may include a "perimeter" arrangement in which a circular, elliptical, or rectangular assortment of sonic-transmitting SWT transducers are arranged around or partially around the periphery of the capacitive sensor printed circuit board PCB, possibly directed towards the center of the sensing zone on the user-facing side of the capacitive sensor printed circuit board PCB. This arrangement of the sonic-transmitting SWT transducers around or partially around the capacitive sensor printed circuit board PCB provides a free path for sound waves to strike a human finger to feel a button click sensation. The sonic-transmitting SWT transducers may be configured to direct sonic energy to the location of the user's finger to simulate a button click sensation and / or the like on the user's finger. In some embodiments, the location of the finger detected by the touchless sensing system 12 is processed by a suitably configured controller (e.g., the retrofit controller 20) to enable the sonic transducers to be controlled (via feedback control signals 57) to direct sound waves towards the detected location of the finger.

[0087] 7A and 7B are schematic diagrams of an acoustically-transmitted retrofit feedback mechanism 55 that may be used with any of the retrofit interface devices 10, 10A, 10B, 10C described herein according to certain embodiments. The acoustically-transmitted retrofit feedback mechanism 55 includes an acoustic transducer (sonic-transmitted SWT transducer) 55A and a touchless sensing system (touchless sensing mechanism) 12. The touchless sensing system 12 includes a capacitance sensor(s) 30. In the illustrated embodiment of FIG. 7A, FIG. 7B, the acoustically-transmitted SWT retrofit feedback mechanism 55 includes a layout of acoustic transducers 55A that may be positioned on a surface (which may be formed to be concave or flat). In the illustrated embodiment of FIG. 7A, the first support surface 38A supporting the sonic transducer 55A is generally flat, and the sonic transducer 55A is supported on the first support surface 38A such that it is located behind (i.e., further from the user) the touchless sensing system 12 (e.g., behind one or more capacitive sensors 30). In the embodiment of FIG. 7A, the capacitive sensor(s) 30 or some of their electrodes (or the touchless sensing system 12 in general) are perforated at openings 31 through which the sonic wave transmitting SWT transducer (sonic transducer 55A) may direct sonic waves to the sensing zone of the capacitive sensor 30 (or the touchless sensing system 12 in general). In some embodiments, a small number of openings, such as a single large opening 31, may be provided. In some embodiments, a matrix comprising a plurality of small openings 31 may be provided. In the illustrated embodiment of FIG. 7B, not only does the second support surface 38B supporting the acoustic transducer 55A have a recess, but the acoustic transducer 55A is positioned at the periphery of the touchless sensing system 12 (e.g., at the periphery of one or more capacitance sensor(s) 30) such that the acoustic transducer 55A is oriented to generate acoustic waves at the user-facing side of the touchless sensing system 12 (e.g., generally at the capacitance sensor(s) 30 and / or sensing area of ​​the touchless sensing system 12).In some embodiments, the configuration of FIG. 7B may be reversed. That is, the capacitive sensor 30 (or a set of electrodes thereof) may be arranged in a ring, and the acoustic transducer 55A may be placed in the central opening of the ring of the capacitive sensor 30. In some embodiments, the aspects of the feedback system (post-feedback mechanism 55) shown in FIG. 7A and FIG. 7B may be combined. For example, the first support surface 38A supporting the acoustic transducer in the embodiment of FIG. 7A may have a concave shape. In some embodiments, the acoustic transducer 55A may be made (at least primarily) of a non-conductive material.

[0088] The retrofit interface device 10 may include a retrofit feedback mechanism 55 that includes a visual feedback device. In some embodiments, the visual feedback device may be implemented via the target display 22 of the target device 50 using the touchless display signal 18 (and / or indirectly via the touchless control signal 16 and the target display signal 21). In some embodiments, the visual feedback device of the retrofit feedback mechanism 55 of the retrofit interface device 10 may be configured with its own retrofit display 60 controlled by the retrofit display signal 61. In some embodiments, the target device 50 does not have a target display 22, while the retrofit feedback mechanism 55 of the retrofit interface device 10 includes only a display or visual indicator(s). In some embodiments, the retrofit feedback mechanism 55 provides visual feedback via a display screen, such as an LCD screen, an e-ink screen, or any other suitable display screen. The retrofit display 60 may be incorporated into an external panel (e.g., a panel disposed adjacent to a printed circuit board PCB that houses the touchless sensing system 12). The retrofit interface device 10 may use the retrofit display 60 and / or the target display 22 to provide an interactive touchscreen-like user experience with the touchless sensing system 12. For simplicity, the remainder of this description regarding visual feedback refers to the retrofit display 60 that is part of the retrofit interface device 10 controlled by the retrofit display signal 61. Without loss of generality, such visual feedback may additionally or alternatively be provided by the touchless display signal 18 or by the target display 22 that is part of the target device 50 that is indirectly controlled by the touchless control signal 16 and the target display signal 21.

[0089] 8 is an exploded view and schematic illustration of a touchless sensing system 12 that may be used with a retrofit feedback mechanism 55 of any of the retrofit interface devices 10, 10A, 10B, 10C described herein in accordance with certain embodiments. The touchless sensing system 12 includes one or more capacitive sensors 30 arranged in a "ring" configuration around a retrofit display 60, and optionally one or more optical sensors (40) and / or additional sensors 33 (not explicitly shown in FIG. 8). For clarity, a retrofit controller 20 is not shown in FIG. 8 but may be housed with the touchless sensing system 12 and / or the retrofit display 60. As shown in FIG. 8 , since the capacitive sensor 30 of the touchless sensing system 12 may be located adjacent to the rear display 60, the printed circuit board PCB supporting the capacitive sensor 30 may be sized, shaped, and / or positioned such that the receiving electrodes 36 of the capacitive sensor 30 are located outside the perimeter of the rear display 60 (e.g., partially around, completely around, or around) to provide a sensing area around and in front of the rear display 60 (i.e., on the user-facing side). That is, the printed circuit board PCB supporting the capacitive sensor 30 may be designed to form a “ring” around or partially around the rear display 60 such that the sensing area of ​​the capacitive sensor 30 is located on the user-facing side of the rear display 60. In some embodiments, the printed circuit board PCB supporting the capacitive sensor 30 may be slightly displaced toward the user (relative to the rear display 60) to mitigate interference that may be caused by the rear display 60 to the electric field of the capacitive sensor 30. The ring center 46 of the capacitive sensor 30 may generally correspond in size and shape to the active portion of the rear display 60, i.e., the portion of the rear display 60 that actually displays content. Figure 8A shows a schematic depiction of a capacitive sensor 30 with one or more receiving electrodes 36 and one or more transmitting electrodes 34 arranged in a ring around an opening (46) according to certain embodiments.In some embodiments, the capacitive sensor 30 may be disposed around (or partially around) an opening (46) that may be used for purposes other than the rear display 60. For example, the target display 22 or some other surface may be viewed through the opening (46).

[0090] In some embodiments, the center 46 of the ring of the capacitance sensor 30 is an empty opening. In some embodiments, this center 46 of the ring of the capacitance sensor 30 may be comprised of a suitable transparent material (e.g., glass, plexiglass (poly(methyl methacrylate)), and / or the like). In some embodiments, the transparent material of the center 46 of the ring may be comprised of a transparent conductive material (e.g., indium-tin-oxide (ITO) or any other suitable transparent material). In some such embodiments, the transparent conductive material may form one of the electrodes (e.g., a transmitting electrode) of the capacitance sensor 30 such that only one of the electrodes (e.g., a receiving electrode) of the capacitance sensor 30 is disposed in a ring around the rear display 60. Having an electrode in the central region (46) may help to increase the accuracy of the capacitance sensor 30 by providing a stronger electric field in the center of the printed circuit board PCB, thereby providing uniformity of the electric field of the capacitance sensor 30. Providing a strong electric field at the center of the printed circuit board PCB may help the touchless sensing system 12 to better detect a finger rather than a larger fist / forearm located further away - thus increasing the accuracy of the touchless sensing system 12. In some such embodiments, both the transmitting and receiving electrodes of the capacitance sensor 30 may be made from a transparent conductive material, in which case the capacitance sensor 30 may be placed directly on the user-facing surface of the rear display 60 - i.e., the ring-based layout of the capacitance sensor 30 around the aperture (46) shown in FIG. 8 may be modified to a layout that covers the "aperture" (46), since the transmitting and receiving electrodes 34 and 36 are both transparent. In the illustrated embodiment of FIG. 8, the ring of the capacitance sensor 30 may be partially or completely covered (on the user-facing surface) by a suitable sheet 48 of transparent material (e.g., glass, plexiglass (poly(methyl methacrylate)), and / or the like). A transparent sheet or panel (48) can be used to encase the capacitive sensor 30 to form a flat, non-protruding design that sits flush with a wall or other surface to which the retrofit interface device 10 may be mounted.Such a flat design may meet industry safety requirements in some cases, such as when the target device 50 is an elevator panel, by mitigating tampering with the retrofit interface device 10. Such an example of the retrofit interface device 10 applied to a target device comprising an elevator panel is shown in FIG.

[0091] Advantageously, such a "ring" design of the capacitive sensor 30 may minimize (at least to an acceptable extent) interference that may be caused to the electric field of the capacitive sensor 30 by the rear display 60. Additionally, the touchless sensing system 12 of FIG. 8 (and shown in FIGS. 3A-3C) advantageously includes a fusion of the capacitive sensor 30 and the optical sensor (40) that is located away from the rear display 60, thereby providing precise lateral (xy) detection provided by the optical sensor (40), precise z coordinate detection and lateral detection trigger provided by the capacitive sensor 30, and useful user feedback provided by the combination of the touchless sensing system 12 and the rear display 60.

[0092] In some embodiments, the retrofit controller 20 is configured to generate a retrofit display signal 61 that causes the retrofit display 60 to display indicia (e.g., a cursor, pointer, and / or the like) on the retrofit display 60. The position of the displayed indicia on the retrofit display 60 may correspond to or be based on an estimated position of a portion of the user's hand (i.e., a position estimated by the retrofit controller 20 based on the touchless input signal 14 received from the touchless sensing system 12). The position of the displayed indicia on the display screen (retrofit display 60) may additionally or alternatively correspond to or be based on a proximity of a portion of the user's hand to the display screen (retrofit display 60) (e.g., a proximity estimated by the retrofit controller 20 based on the touchless input signal 14 received from the touchless sensing system 12). For example, in some embodiments, the rear controller 20 may cause the rear display 60 to display indicia indicative of an estimated location of a portion of the user's hand in a plane parallel to the plane of the rear display 60 (e.g., the xy plane), in a plane tangential to the surface of the rear display 60, or in a plane tangential to some other suitable surface of the target device 50 and / or the rear interface device 10, and may then cause the rear display 60 to display indicia features (e.g., indicia size, brightness, color, and animation feature(s), and / or the like) indicative of, for example, the proximity (e.g., in the z direction) of the portion of the user's hand to said plane. In some embodiments, indicia are always displayed on the rear display 60 to inform the user of the location of the user's fingers.

[0093] In some embodiments, the retrofit controller 20 may generate a retrofit display signal 61 that causes the display screen (retrofit display 60) to change the appearance of the displayed indicia based on an estimated proximity (e.g., in the z-direction) of a portion of the user's hand and / or an estimated position of a portion of the user's hand (e.g., relative to the plane of the retrofit display 60, relative to a plane tangent to the surface of the retrofit display 60, or relative to a plane tangent to some other suitable surface of the target device 50 and / or the retrofit interface device 10). The change in the appearance of the displayed indicia may comprise at least one of a change in color of the displayed indicia, a size of the displayed indicia, a change in brightness (intensity) of the displayed indicia, a change in the shape of the displayed indicia, a change in the color gradient of the displayed indicia, an animation function of the displayed indicia, the addition of other indicia to the displayed indicia, and / or the like. For example, the displayed indicator (indicia) may be a "ring cursor" and the size of the ring cursor may decrease (i.e., the diameter of the ring cursor may decrease) as the user's hand approaches a threshold distance away from the retrofit display 60.

[0094] In some embodiments, the rear display 60 is configured (via the rear display signal 61 from the rear controller 20) to display one or more virtual inputs (e.g., virtual buttons). In these embodiments, the rear display signal 61 may cause the rear display 60 to change the appearance of the virtual inputs based on an estimated position of a portion of the user's hand and / or an estimated proximity of a portion of the user's hand (estimated proximity, estimated proximity).

[0095] In some embodiments, the retrofit controller 20 is configured to detect a circular motion made by a portion of the user's hand based on the touchless input signal 14 from the touchless sensing system 12. In some such embodiments, the retrofit controller 20 may be configured to generate a retrofit display signal 61 that causes the retrofit display 60 to provide some visual feedback (e.g., change the appearance of one or more displayed virtual inputs) based on the detected circular motion.

[0096] In some embodiments, the retrofit controller 20 is configured to detect a lack of (or a lack of) movement of a portion of the user's hand for (up to) a threshold period of time based on the touchless input signal 14. In these embodiments, the retrofit controller 20 may be configured to generate a retrofit display signal 61 that causes the retrofit display 60 to provide some visual feedback (e.g., change the appearance of one or more displayed virtual inputs) based on the detected lack of movement.

[0097] In some embodiments, the retrofit controller 20 is configured to generate a retrofit display signal 61 that, in turn, causes the retrofit display 60 to vary the size of the displayed indicia in a manner that correlates with the proximity of a portion of the user's hand to the retrofit display 60. In some such embodiments, the retrofit controller 20 may be configured to modify the touchless control signal 16 when the size of the displayed indicia is the same (to within a suitable threshold) as the size of one of the multiple displayed virtual inputs and when the position of the portion of the user's hand lacks movement (within a suitable threshold) for a threshold period of time, thereby affecting the operation of the target device 50 (e.g., making a selection). That is, the retrofit controller 20 is enabled to confirm selection of the displayed virtual input when the user brings a portion of their hand in proximity (within a threshold) to the displayed virtual input and thereafter does not move (for the threshold period of time) and to feed back to the user a proximity factor of this selection based on the size of the indicia being the same as the size of the virtual input when the user brings a portion of their hand in sufficient proximity.

[0098] The rear display 60 may optionally include one or more visible light sources in addition to or as an alternative to a flat screen display. The rear controller 20 may be configured to generate a rear display signal 61 that causes the rear display 60 to illuminate or change the color of illumination of at least one of the visible light sources in response to, for example, an estimated position of a portion of the user's hand in a plane (e.g., an xy plane) corresponding to the plane of the rear display 60, a tangent plane to the surface of the rear display 60, or a tangent plane to some other suitable surface of the target device 50 and / or the rear interface device 10, and / or an estimated proximity of a portion of the user's hand to such plane.

[0099] In some embodiments, the retrofit controller 20 is configured to generate a retrofit display signal 61 that causes the retrofit display 60 to provide feedback (or otherwise modify the output of the retrofit display 60) based at least in part on the user's estimated gaze vector. In such embodiments, the retrofit display signal 61 can cause the retrofit display 60 to display an image that is skewed toward or away from the user based on the estimated gaze vector. Advantageously, causing the retrofit display 60 to display a "distorted" image can help the user avoid inaccurate touchless interactions with the retrofit interface device (retrofit interface system) 10 that may otherwise be caused by, for example, the user's viewing angle and viewpoint relative to the retrofit display 60 and / or the virtual inputs displayed on the retrofit display 60. By skewing the location of the virtual input (or the detection zone of the virtual input) based on the user's estimated gaze vector, the probability that the user will accurately interact with the touchless sensing system 12 (e.g., select a particular virtual input) will be increased. In some embodiments, the retrofit controller 20 may generate a retrofit display signal 61 that causes the retrofit display 60 to display a distorted indicator (e.g., a cursor) to correspond (accommodate) to the user's viewpoint (as determined by an estimate of the user's line of sight).

[0100] The retrofit interface device 10 may optionally include additional visual indicators (as part of the retrofit feedback mechanism 55) to help inform the user of a selection, finger / hand presence, etc. For example, the retrofit interface device may include a light (e.g., a light emitting diode) that turns on to indicate a successful finger / hand tap / wave. The light may be synchronized with the retrofit display 60 to together provide visual feedback to the user. For example, the retrofit display 60 may be configured to display various icons, images, and / or text for various keypad numbers or other enabled selections, and the light may be configured to indicate finger presence and / or successful selection.

[0101] The retrofit interface device 10 may additionally or alternatively include a retrofit feedback mechanism 55 that supports auditory feedback. The auditory feedback mechanism may be configured (e.g., with an appropriate feedback signal 57 from an appropriately programmed retrofit controller 20) to provide an audio cue indicating a successful touchless selection or other interaction between the user and the retrofit interface device 10 (e.g., the touchless sensing system 12). For example, when a touchless input corresponding to alphanumeric text is entered by the user (e.g., when an appropriate finger press gesture is detected by the touchless sensing system 12), an audio cue may be generated to simply indicate a button press or to repeat the entered input (e.g., "9", "m"). The auditory feedback mechanism may include a speaker. The speaker can be used to inform the user of system states and / or changes in system states of the retrofit interface device 10 and / or target device 50, such as, by way of non-limiting example, when user presence is detected, when the system is waiting for user input, when the user has reached the "Main Menu" or "Submenu X", and / or the like.

[0102] As discussed elsewhere herein, the retrofit interface device 10 not only receives (receives, accepts) touchless input from the user (via the touchless sensing system 12), but also uses such touchless input to provide corresponding input to the target device 50. A number of non-limiting exemplary embodiments are now described of how the retrofit interface device 10 may determine a particular touchless input. These examples may be configured to include additional or alternative techniques to those described elsewhere herein. In some exemplary embodiments, the capacitive sensor 30, the optical sensing system 40, and / or other sensors forming part of the touchless sensing system 12 may detect the movement of a part of the user's body (e.g., a finger). In some embodiments, the retrofit feedback mechanism 55 provides visual feedback via a display (which may include a retrofit display 60 that is part of the retrofit interface device 10 and / or a target display 22 that is part of the target device 50). Such visual feedback may comprise displaying a visual indication corresponding to position characteristics detected and / or made detectable by the capacitive sensor 30, the optical sensing system 40, and / or other sensors forming part of the touchless sensing system 12.

[0103] 5A-5B show front and side views of a panel 100 of a retrofit (user) interface device 10 (or any of the other retrofit (user) interface devices 10A, 10B, 10C described herein) displaying virtual buttons 104 according to an exemplary embodiment. As shown generally in FIG. 5A-5B, the retrofit interface device 10 may include a panel 100 labeled with icons, graphics, or alphanumeric text 102 that may be used to represent, for example, buttons on a keypad or other selectable virtual buttons (or inputs) 104. Such icons, graphics, or alphanumeric text 102 may be considered virtual buttons (or inputs) 104 that are touchlessly operated (as opposed to being physical buttons) and / or that exist only on the panel 100. Such a panel 100 displaying virtual buttons (or inputs) 104 may comprise a display (e.g., a display that is part of a retrofit feedback mechanism 55 that comprises a retrofit display 60 that is part of a retrofit interface device 10 and / or a target display 22 of a target device 50). Such a panel 100 displaying virtual buttons (or inputs) 104 may additionally or alternatively comprise a static panel or surface that displays static virtual buttons (or inputs) 104. In the exemplary touchless input selection embodiments described below, it is assumed without loss of generality that there is a panel 100 labeled with icons and / or alphanumeric text 102 such that the various virtual buttons 104 are visible to the user. FIGS. 5A-5B also show orthogonal axes x, y, and z that are used to describe directions in the following description of the exemplary touchless input selection, where x and y are orthogonal lateral directions and the z direction is generally perpendicular to the panel 100.

[0104] For the sake of brevity, the exemplary touchless input selection embodiments described herein refer to virtual buttons 104 shown on the rear display 60 of the rear interface device 10 that may be controlled by the rear controller 20 (via the rear display signal 61). Without loss of generality, the control by the rear controller 20 may additionally or alternatively be provided by a target display 22 that is part of a target device 50 that may be controlled by the rear controller 20 (via the touchless display signal 18 and / or indirectly via the touchless control signal 16 and the target display signal 21), or such virtual buttons may be displayed on a static panel or surface. The exemplary touchless input selection embodiments described below provide techniques by which the rear interface device 10 may determine (or conclude) that a particular virtual button 104 has been touchlessly selected by a user. As described elsewhere herein, the rear interface device 10 may use the determination of such a selection of the virtual button 104 to effect a corresponding operation of the target device 50.

[0105] 6 is a schematic depiction of a method 200 of determining a touchless input selection using a retrofit interface device 10 (or any other retrofit interface device 10A, 10B, 10C described herein), according to a particular exemplary embodiment. For purposes of illustrating the method 200 of FIG. 6, it is assumed that the touchless sensing system 12 comprises a combination of an array of capacitive sensors 30 and an optical sensing system 40 similar to those shown in the embodiment of FIGS. 3A-3C, as well as the selection of the input comprises interaction with a virtual button 104 as described elsewhere herein. The method 200 may be implemented by the touchless sensing system 12 in combination with a retrofit controller 20 and, optionally, a retrofit feedback mechanism(s) 55.

[0106] Method 200 begins at block 202, where the touchless sensing system 12 waits for a part of the user's body (assumed to be a finger, without loss of generality, for purposes of describing method 200) to enter a detection range of the touchless sensing system 12. In some embodiments, block 202 comprises detecting the user's finger entering a sensing area 32 of a capacitive sensor 30, which may be a z distance away from a plane displaying one or more virtual buttons 104, and the z coordinate may be perpendicular to the plane displaying the virtual buttons 104. Once the touchless sensing system 12 detects the user's finger in its sensing area 32, method 200 proceeds to block 204, where the touchless sensing system 12 (e.g., capacitive sensor 30) tracks the position of the user's finger as it moves across the sensing area 32. In some embodiments, block 204 may comprise tracking only the z coordinate of the finger. In some embodiments, block 204 may comprise tracking the lateral (xy) coordinate of the user's finger in addition to the z coordinate. In block 204A, the add-on controller 20 optionally provides some type of feedback by using the add-on feedback mechanism 55 when the user's finger is detected in the sensing area 32. For example, the add-on controller 20 may cause the add-on display 60 (or the target display 22) to not only track the lateral (xy) position of the user's finger, but also display a cursor or the like having some indicator (e.g., size, color, etc.) that indicates the z coordinate of the user's finger.

[0107] In block 206A, the method 200 queries whether the user's finger has been pulled out of the sensing area 32. If so, the method 200 returns to block 202. If the user's finger is still present within the sensing area 32, the method 200 proceeds to optional block 206B. Optional block 206 comprises querying whether the user's finger has crossed a detection plane. The detection plane may be implemented using a threshold z coordinate. That is, if the touchless sensing system 12 determines that the z coordinate of the user's finger has gone from above a configurable threshold to below a configurable threshold, a detection plane crossing is detected in block 206B. If a detection plane crossing is not detected in block 206B, the method 200 returns to block 204. However, if a detection plane crossing is detected in block 206B, the method 200 proceeds to optional block 208. In block 208, the lateral (xy) position of the user's finger is determined as it crosses the detection plane. This determination in block 208 may be performed by an optical sensor (40), which may allow for more accurate determination of the lateral (xy) coordinates (compared to capacitive sensor 30), as described elsewhere herein.

[0108] Whether arrived at via optional blocks 206B and 208 or directly from block 206A, method 200 comprises ascertaining whether a particular selection criterion is met in block 210. As described elsewhere herein, various techniques may be used to determine whether the selection criterion is met in block 210. Such techniques may generally be based on a tracked position of a part of the user's body (e.g., the user's finger) as determined by the touchless sensing system 12 (e.g., the capacitive sensor 30 and / or the optical sensing system 40). In certain embodiments, block 210 comprises determining that a selection is made when the user's finger crosses another detection plane (similar to the detection plane described above in connection with blocks 206B and 208). In other particular embodiments, the selection criterion in block 210 may comprise detection of a tap gesture, a double tap gesture, a hover gesture, and / or the like, some examples of which are described below. If the selection criteria of block 210 are met, the method 200 proceeds to block 212, which comprises determining which selection was made (e.g., which particular virtual button 104). Depending on the nature of the selection criteria satisfied in block 210, the selection determination of block 212 may be made as part of the query of block 210.

[0109] In some embodiments, since each virtual button 104 has a corresponding range of lateral (x,y) coordinates, block 212 comprises selecting the particular virtual button 104 that corresponds to the tracked x,y position of the user's finger when the selection criteria of block 210 are met, or corresponds to the determined x,y position of the user's finger when it crosses the detection plane in block 208. In some embodiments, block 212 may take into account an approach angle of the user's finger. Such an approach angle may be determined, for example, by the capacitive sensor 30 as it tracks the user's finger between blocks 204 and 210. Such an approach angle may be determined between the x,y position of the first detection plane crossing (as determined in block 208) and the x,y position of the second detection plane crossing (as determined in block 210 when the selection criteria are met). If the approach angle (relative to the z direction) is greater than the threshold angle, block 212 may comprise determining that the user was attempting to select an adjacent virtual button 104 (i.e., a virtual button adjacent to a virtual button corresponding to the tracked xy position of the user's finger when the selection criteria of block 210 were met, or a virtual button adjacent to a virtual button corresponding to the determined xy position of the user's finger when it crossed the detection plane 208). In some embodiments, block 212 comprises calculating a tap vector based on the (x,y,z) detection pairs (x1,y1,z1) and (x2,y2,z2). By utilizing these two separate detections, a vector in space can be obtained that can then predict the virtual button the user was attempting to select. As a non-limiting example, such an (x,y,z) detection pair can comprise a laser optical sensor for each detection. A detection occurs when the detection plane of sensor A is hit, while another detection occurs when the detection plane of sensor B is hit. The z coordinate can be calculated by the angle of the optical sensor or by the capacitive sensor 30. In general, any combination of information from the capacitive sensors 30 and / or optical sensors (40) described herein may be used to perform one or both of the (x,y,z) sensing pairs.In some embodiments, a suitable time threshold may be used between (x,y,z) detection pairs to identify whether the user made the gesture intentionally.

[0110] In block 212A, the retrofit controller 20 may optionally cause the retrofit feedback mechanism 55 to display some type of feedback indicating that the virtual button 104 has been selected by the user. For example, the retrofit controller 20 may cause the color of an icon associated with the virtual button to change, display or change an animation, and / or the like. The method 200 then proceeds to block 214, which includes a query as to whether the retrofit interface device (system) 10 is waiting for additional input from the user. For example, if the target device 50 is an elevator panel and the user presses the virtual button corresponding to the eighth floor, no further input is required. However, if the target device 50 is an airport kiosk and the user enters the first letter of their last name, the retrofit interface device (system) 10 may request further input. If the query of block 214 is negative, the method 200 ends. Otherwise, the method 200 returns to either block 202 or block 210, via optional blocks 218 and 220. In optional block 218, the retrofit controller 20 may provide a feedback mechanism to indicate that the retrofit interface device is waiting for further input. Optional block 220 comprises a query as to whether the user's finger has been withdrawn beyond the detection plane (away from the virtual button) evaluated in blocks 206B and 208. If the query in block 220 is negative, the method 200 returns to block 210. If the query in block 220 is positive, the method 200 returns to block 202.

[0111] Returning to the inquiry of block 210, it is possible that the selection criteria have not been met, in which case method 200 proceeds to block 216. Block 216 may comprise an inquiry as to whether a restart is desired. If a restart is not desired, method 200 returns to block 210. If a restart is desired, appropriate restart feedback may be displayed (block 216A) and method 200 may return to block 204. The restart criteria evaluated in block 216 may be configured to comprise, for example, an appropriate time threshold, a determination that an anti-crossing of the detection plane has been detected in blocks 206B, 208, and / or the like.

[0112] As discussed elsewhere herein, the touchless sensing system 12 (e.g., the capacitive sensor 30 and / or the optical sensing system 42) is enabled to detect the x, y, and z position (see FIGS. 3A, 5A, 5B) of a user's body part (e.g., the user's finger). An exemplary method of touchless input selection criteria that may be used, for example, in block 210 of method 200, comprises a user hovering a finger over (e.g., in sufficient proximity (within a threshold z-distance (FIGS. 3A, 5A, 5B)) a particular virtual button 104 and in the correct lateral (xy) position relative to the particular virtual button 104 without changing the finger's x, y, z position by more than a threshold value (e.g., 0.5 cm or 1 cm) for a threshold period (e.g., 0.5 seconds or 1 second). In other embodiments, virtual button (or icon) 104 selection criteria that may be used in block 210 include determining that a user first moves a finger within a threshold distance (e.g., z-distance) from the virtual button 104 (determined by the lateral (xy) position of the finger) for a threshold period of time, and then detecting an abrupt change in the finger's z-position (e.g., moving the finger towards or away from the virtual button 104) with a threshold change in z-position within the threshold period of time (e.g., a z-change of 1 cm or more in less than 0.25 seconds) and / or a z-position change rate greater than an appropriate threshold (e.g., a z-change rate greater than 4 cm / sec).

[0113] The rear display 60 may display a visual indicator or animation (e.g., a load bar / circle) or other types of feedback (e.g., including those described elsewhere herein) by the rear feedback mechanism 55 to indicate that a selection has been made (e.g., in block 212A) and that the finger may be moved to a new position (e.g., in block 218). In some embodiments, the rear feedback mechanism 55 of the rear interface device 10 used in block 212A is configured with, for example, a sound wave emitter that directs sound waves toward the user's finger to indicate (provide feedback corresponding to) the selection.

[0114] The example hover selection touchless input selection techniques may be used in addition to or instead of (eg, in block 210) any of the other touchless input techniques described herein.

[0115] Another exemplary method of touchless input selection criteria that may be used, for example, in block 210 of method 200 for determining selection of a virtual button 104, comprises a user not only hovering over the virtual button (or icon) 104 (e.g., in close enough proximity (within a threshold z-distance) and in the correct relative lateral (xy) position) for a threshold period of time, but then tapping their finger further (in the z direction) towards the virtual button 104. The tapping action may comprise a user not only moving their finger further towards the virtual button 104 (e.g., to within a close threshold distance in the z direction), but then moving or lifting their finger away from the virtual button 104. The touchless sensing system 12 (e.g., capacitive sensor 30 and / or optical sensing system 40) may detect hovering over a particular virtual button that may be a qualifying event for a subsequent tap event. A "tap" is then identified when it is detected that the xy position does not change (by more than a threshold) and the z direction decreases (e.g., by a threshold compared to the z value during hover detection).

[0116] This exemplary method may also comprise tapping and dragging the finger laterally after an initial hover. As a non-limiting example, the first identified tap may be used to perform a drag (e.g., of an icon), and subsequent xy movements of the finger may be used to perform the drag until a subsequent tap, increasing z coordinate, and / or the like is used to release the drag.

[0117] For example, another exemplary method of touchless input selection criteria that may be used in block 210 of method 200 for determining selection of virtual button 104 comprises a user double tapping a virtual button or icon. This selection method potentially allows for a quicker interaction with the retrofit interface device 10 (when compared to the hover selection discussed above). The double tap selection method comprises a user tapping a finger toward and away from the virtual button 104 (the lateral position of the virtual button) twice within a time from the threshold period to input a selection. The touchless sensing system 12 (e.g., the capacitive sensor 30 and / or the optical sensing system 40) may detect such a double tap in a similar manner to detecting a first tap (discussed above), but may then detect an increase in z-position following a second tap within the threshold period. A double tap advantageously allows for a reduction in false positives or accidental clicks of the virtual button, as well as increased click robustness, as compared to the hover-type selection criteria discussed above, since some users may naturally hover over a particular virtual button while trying to decide which button to select. In some embodiments, the selection criteria of block 210 may comprise a combination of the double-tap method and the hover selection method described above. For example, a user may first use the hover selection method to pre-select an icon and then use the double-tap method (or alternatively the single-tap method) to confirm the selection.

[0118] For example, another exemplary method of touchless input selection criteria that may be used in block 210 of method 200 to determine selection of a virtual button 104 comprises a user pointing a finger toward a virtual button or icon and making a circling motion (i.e., drawing a circle) around the virtual button or icon they wish to select. In some such embodiments, the touchless sensing system 12 may track the lateral xy position of the user's finger as the finger comes within a threshold z-distance proximity to the virtual button. During tracking, the touchless sensing system 12 may check whether the lateral position of the finger changes by a threshold amount and then returns to within a threshold vicinity of the same position within a configurable time threshold (e.g., 1 second). If "true," the touchless sensing system 12 analyzes the shape created while tracking the lateral position. One example and non-limiting method for determining whether a movement is a circle or some other "closed" shape includes checking whether the movement sequentially traverses four Cartesian quadrants during the movement (e.g., by not only detecting four "corner" (angle) points that may correspond to the extremes of a particular coordinate, but also by checking whether these corner points are in four separate quadrants) and checking that there is a minimum threshold radius or threshold separation between opposing corner points. It will be appreciated that other techniques may be used to check whether the user's finger is traversing a closed shape. Before confirming the selection of a particular virtual button, the touchless sensing system 12 may additionally check whether the xy coordinates of the virtual button are fully or partially enclosed within the detected circle. In some embodiments, the rear display 60 is configured to provide feedback indicating a circle being drawn around the selected virtual button or icon. Such feedback may be provided, for example, prior to a YES conclusion that the selection criteria have been met in block 210.

[0119] Another example method of touchless input selection comprises a user positioning their finger over a “ring” indicator (or other suitable “selector” indicator) that may be displayed on the rear display 60, and then “dragging” the selector indicator over the virtual button 104 they wish to select. In some such embodiments, the touchless sensing system 12 may first detect a hover gesture in proximity to the ring indicator, but may also interpret this action as activating the ring to move with the finger. The touchless sensing system 12 may then track the lateral xy position of the finger as it moves over the virtual buttons. The touchless sensing system 12 may then detect a hover gesture, a move away gesture (z-distance increases by a certain threshold within a threshold time period), or other suitable gesture, to select the virtual button corresponding to the xy position of the ring. Block 210 selection criteria may then determine selection of the virtual button 104 by the user holding a selector indicator over the virtual button (or icon) 104 for a threshold period of time or by removing a finger (e.g., in the z-direction) from the virtual button 104 after the selector indicator has been dragged over the virtual button (or icon) 104. Advantageously, because this ring selection technique requires a deliberate user action similar to the double-tap method described above, this ring selection technique can reduce false positives and user errors similar to the double-tap method compared to hover-based selection.

[0120] Another exemplary method of touchless input selection criteria that may be used, for example, in block 210 of method 200 to determine selection of virtual button 104, comprises a user waving or swiping a hand across the surface of touchless sensing system 12. In some such embodiments, touchless sensing system 12 first ensures that the user's body part is close enough (within a z-distance threshold), and then tracks the lateral (xy) position of the user's body part (e.g., finger, hand, forearm) over a threshold period (e.g., 1 second). A swipe is detected if the change in the lateral position of the user's body part is greater than a configurable threshold within the threshold time (threshold period) or if the lateral change rate during the threshold lateral movement is higher than a configurable threshold. In some implementations, the directionality of the swipe may be understood by touchless sensing system 12 by detecting that the swipe movement is closest to one of the following directions: left (negative x) / right (positive x) / up (positive y) / down (negative y). The hand must be within the detection range of the sensor at all times. A threshold change-distance is required for a valid swipe to be identified. In some embodiments, the detection of a swipe may be used as a navigation gesture (e.g., to change between sets of virtual buttons) in addition to or as an alternative to being used as a technique for selecting a virtual button in block 210 of method 200.

[0121] For example, another exemplary method of touchless input selection criteria that may be used in block 210 of method 200 for determining selection of a virtual button 104 comprises a user making a circular motion with their hand to cycle or scroll through selections, manipulating a slider widget, selecting a numerical value, or adjusting a setting (e.g., the temperature of a thermostat). In some such embodiments, the touchless sensing system 12 first verifies that a part of the user's body is close enough (within a z-distance threshold), and then tracks the lateral xy position of the finger / hand to verify that a consistent circular motion is occurring. This circular motion can be captured in a similar manner to the circular selection described above. In some embodiments, the radius and velocity of the motion can be tracked to verify the speed of cycling through a selection, etc. In some embodiments, tracking these rotations can comprise: assuming a circular motion; detecting and calculating a velocity vector of the instantaneous motion; and then calculating a cross product vector based on the current and previous velocity vectors, where the direction of these cross products provides directional information, while the magnitude of these cross products provides velocity information. In some embodiments, detection of circular motion may be used as a navigation gesture (e.g., to change between sets of virtual buttons) in addition to or as an alternative to being used as a technique for selecting a virtual button in block 210 of method 200.

[0122] Selection of a virtual button or icon may comprise selection of an interactive virtual icon, such as a slider button and / or the like. A virtual slider button may be configured with a slider icon that can be "dragged" through a range of motion. In one particular embodiment implementing a virtual slider button, interaction with the virtual slider button may comprise the following: the touchless sensing system 12 detects a finger that meets the selection criteria over the slider icon (hover, hover tap, double tap, etc.) and, as long as it detects that the z-distance does not increase (by more than a threshold amount), the touchless sensing system 12 tracks the lateral xy position, thereby affecting the slider to drag left / right or up / down on the screen. If it detects that the finger's z-distance has increased beyond a threshold amount, the interaction is terminated.

[0123] In some embodiments, the rear display 60 is configured to display a keyboard (e.g., the virtual buttons 104 correspond to keys of the keyboard), and the touchless sensing system 12 is configured to detect a user swiping his / her finger across (but at intervals from) the surface of the rear display 60 between various alphanumeric icons to input (type) a word. The touchless sensing system 12 may detect such a swipe as a series of swipes and may track the endpoints of the swipe as pressed keyboard keys. After a threshold period of inactivity, or after detection of some other suitable gesture (e.g., moving away in the z-direction and / or the like), the detected endpoints may form an alphanumeric string. The rear controller 20 may support a predictive algorithm that may be employed to predict what word is desired to be input based on the proximity and endpoints of the user's finger movements.

[0124] In some embodiments, the touchless sensing system 12 is configured to detect finger movements made by a user, and the retrofit controller 20 is configured to interpret the movements. For example, the user can use finger movements to draw an entire word or code, such as "2" or "d," or "open" or "1234." Once the user's finger is within a threshold z-distance, the touchless sensing system 12 tracks the lateral (xy) position of the user's finger, and when a moving away gesture (or other suitable gesture) is detected, either character recognition (CR) or a similar algorithm can be employed to identify which alphanumeric input was drawn. Multiple numbers / letters can be detected together to form a word or passcode. In some embodiments, the user's finger position and finger movements (i.e., what the user has written) are displayed on a display screen (the retrofit display 60).

[0125] In some embodiments, the retrofit controller 20 is configured to cause the retrofit display 60 to display a warning message if the estimated proximity of a portion of the user's hand (e.g., a finger) is too close to the panel 100. This can discourage users who are unfamiliar with the touchless sensing system 12 from touching the surface of the panel 100 (to prevent infection transmission from contact-based virus infection). This can also inhibit the finger from getting too close to the panel 100 to reduce inaccuracies in the sensor's detection capabilities that may occur, for example, when the retrofit controller 20 is tuned for detection of relatively thin fingers, and the user's hand or forearm comes within detection range, resulting in potentially inaccurate results.

[0126] The sensors (e.g., capacitance sensors 30) of the touchless sensing system 12 may be configured to determine 3D (x, y, z) coordinates corresponding to the position of a body part (e.g., a finger). Using such 3D coordinates, the retrofit controller 20 may be configured to detect and interpret different custom gestures as inputs to the retrofit interface device 10. Examples of custom gestures include, but are not limited to, any shape such as a figure eight, a triangle, a letter, a circular loop, and the like.

[0127] FIG. 6A is a schematic diagram of a method 300 for determining a touchless input selection by using any of the add-on interface devices 10, 10A, 10B, 10C described herein, according to another exemplary embodiment. The method 300 is a specific embodiment in which the touchless sensing system 12 is configured with the capacitive sensor 30 and the optical sensors 42A, 42B described above in connection with FIGS. 3A-3C. The flow chart diagram of the method 300 is logically decomposed into two parts: a capacitive portion 310 performed by the capacitive sensor 30, and an optical portion 320 performed by the optical sensor (40). The method 300 begins with a block 302 similar to block 202 described above, and comprises waiting for a user's body part (e.g., a finger or hand) to move into the sensing area 32. For the sake of brevity, and without limiting the generality of the description, the remainder of the method 300 is described with reference to the user's finger being the relevant body part.

[0128] When the user's finger enters the detection area (sensing area 32), the central capacitive sensor 30 in the capacitive portion 310 detects and tracks the z coordinate of the finger (block 312) and can optionally indicate to the user (block 312A) when the finger approaches or leaves the plane of the virtual button 104 or the rear display 60 or the target display 22. If the user's finger moves out of the detection area (sensing area 32) (YES branch of block 314), the method 300 returns to block 302. If the user's finger moves through the detection plane (crosses from above the configurable z coordinate to below the configurable z coordinate), the method proceeds to block 330 via the YES branch of block 316. In this sense, block 316 may be similar to the determination of whether the selection criterion is met in block 210, with the selection criterion of method 300 being the crossing of the detection plane. In this manner, the detection plane of block 316 may be referred to as the actuation plane. In other embodiments, block 316 may additionally or alternatively utilize any of the selection criteria of the other blocks 210 described herein.

[0129] Meanwhile, in the optical portion 320, the optical sensors 42A, 42B (see FIG. 3A) located next to the plane of the virtual button 104 or the add-on display 60 or the target display 22 emit two detection planes of laser light at an angle to the plane of the virtual button 104 or the add-on display 60 or the target display 22. When the user's finger crosses the detection plane (block 322), as determined by the finger crossing the detection plane of one or both of the optical sensors 42A, 42B and / or by comparing the z coordinate measured by the capacitive sensor 30 to a suitable threshold, the optical sensors 42A, 42B are triggered and the lateral (xy) position of the user's finger is determined as described elsewhere herein (blocks 324, 326). The detection plane of block 322 may be the same as the actuation plane of block 316 or may be different from the actuation plane of block 316. For example, the actuation plane of block 316 may be farther away from the user than the intersection of the detection planes of optical sensors 42A, 42B (see intersection 56 in FIG. 3A). In this configuration, optical detection (at blocks 322, 324, 326) will occur before method 300 reaches block 316. If the user moves their finger away from the detection plane of block 322 (YES branch of block 328), method 300 returns to block 322.

[0130] If the inquiry of block 316 is positive (i.e., the user's finger crosses the working plane), the method 300 proceeds to block 330. Block 330 may determine an input (e.g., virtual button) selection based on the xy position grasped in block 326. In this sense, block 330 may be similar to block 212 of FIG. 6. Feedback regarding the selection of block 330 may be provided by the retroactive feedback mechanism 55 in block 330A. If the user removes his / her hand from the sensing area 32 (YES branch of block 332), the method 300 returns to block 302. Otherwise, the z coordinate is displayed (block 334) and the method returns to block 332.

[0131] Certain embodiments described and claimed herein provide a retrofit (touchless) interface device 10 that is used to provide touchless user input to a target device 50 having a touch-based input 50A by being retrofitted to the target device 50. This is not required. Those skilled in the art will appreciate that some embodiments may be suitably modified to provide a custom and / or modular touchless interface for directly facilitating human interaction with a machine or other device. That is, the retrofit interface device 10 and the target device 50 may be integrated into a newly manufactured device. In some such newly manufactured devices, some aspects of the retrofit interface device 10 and the target device 50 described as independent herein may be integrated. For example, the retrofit controller 20 of the retrofit interface device 10 and the target control system 51 of the target device 50 may be integrated into one functional controller. As another example, the retrofit display 60 of the retrofit interface device 10 and the target display 22 of the target device 50 may be integrated into one display. Some of such newly manufactured devices may be equipped with a touchless sensing system 12 similar to that described herein.

[0132] <Interpretation of terms> Throughout this specification and claims, unless the context clearly requires otherwise, - "comprise", "comprising" and the like shall be construed in their inclusive sense, i.e. "including but not limited to," as opposed to their exclusive or exhaustive sense.

[0133] "Connected," "coupled," or variations thereof means a direct or indirect connection or coupling between two or more elements, where the coupling or connection between the elements can be physical, logical, or a combination thereof.

[0134] -The words "herein," "above," "below," and words of similar import, as used to describe this specification, shall refer to this specification as a whole and not to any particular portions of this specification.

[0135] - "Or" in reference to a list of two or more items covers all interpretations of this word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

[0136] -The singular forms "a" (one), "an" and "the" have the appropriate plural meanings. Orientational terms such as "vertical," "lateral," "up," "down," "forward," "backward," "inward," "outward," "orthogonal," "side," "left," "right," "front," "rear," "top," "bottom," "directly below," "above," "below," and the like, used in this specification and any appended claims (if any), are dependent on the particular orientation of the device being described and illustrated. The subject matter described herein can assume a variety of alternative orientations. Thus, these directional terms are not precisely defined, nor should they be narrowly construed.

[0137] Some components of various embodiments of the present invention (including, as non-limiting examples, the retrofit controller 20 of the retrofit interface device 10, 10A, 10B, 10C, the target control system 51 of the target device 50, or any other controller described herein) may be implemented using specially designed hardware, configurable hardware, a programmable data processor configured by providing software (which may optionally comprise "firmware") executable on the data processor, a special purpose computer or data processor that is specially programmed, configured, or constructed to perform one or more steps of the methods detailed herein, and / or a combination of two or more of these. Examples of specifically designed hardware are logic circuits, application specific integrated circuits ("ASICs"), large scale integrated circuits ("LSIs"), very large scale integrated circuits ("VLSIs"), etc. Examples of configurable hardware include one or more programmable logic devices, such as programmable array logic ("PALs"), programmable logic arrays ("PLAs"), field programmable gate arrays ("FPGAs"). Examples of programmable data processors include microprocessors, digital signal processors ("DSPs"), embedded processors, graphics processors, mathematical co-processors, general purpose computers, server computers, cloud computers, mainframe computers, computer workstations, etc. For example, one or more data processors in the control circuitry of a device may perform the methods described herein by executing software instructions in program memory accessible to the processors.

[0138] Processing may be centralized or distributed. When processing is distributed, information, comprising software and / or data, may be maintained centrally or distributed. Such information may be exchanged between different functional units by way of a communications network, such as a Local Area Network (LAN), a Wide Area Network (WAN), or the Internet, wired or wireless data links, electromagnetic signals, or other data communications channels.

[0139] For example, while processes or blocks are presented in a certain order, alternative examples may perform routines having steps or employ systems having blocks in a different order, and some processes or blocks may be removed, moved, added, sub-divided, combined, and / or modified to provide alternative or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks may be shown as being performed in series, these processes or blocks may instead be performed in parallel or may be performed at different times.

[0140] Moreover, while elements may be shown as being performed sequentially, they may instead be performed simultaneously or in different orders, and it is therefore intended that the following claims be interpreted to include all such variations that are within their intended scope.

[0141] The software and other modules may reside on servers, workstations, personal computers, tablet computers, image data encoders, image data decoders, PDAs, color grading tools, video projectors, audiovisual receivers, displays (such as televisions), digital cinema projectors, media players, and other devices suitable for the purposes described herein. Those skilled in the relevant art will appreciate that aspects of the system can be implemented in other communication, data processing, or computer system configurations. These aspects include Internet appliances, handheld devices (including personal digital assistants (PDAs)), wearable computers, cellular or mobile phones of any kind, multiprocessor systems, microprocessor-based or programmable appliances (e.g., video projectors, AV receivers, displays such as televisions, etc.), set-top boxes, color grading tools, network PCs, minicomputers, mainframe computers, and the like.

[0142] In some embodiments, the invention may be implemented partially in software. For clarity, "software" comprises any instructions executed on a processor, and may comprise (but is not limited to) firmware, resident software, microcode, etc. Both the processing hardware and software may be, in whole or in part, centralized or distributed (or a combination thereof), as known to those skilled in the art. For example, the software and other modules may be accessible via local memory, over a network, via a browser or other application in a distributed computing context, or via any other means suitable for the purposes described above.

[0143] When a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including reference to a "means") should be interpreted to include equivalents (i.e., functionally equivalent) of that component, including components that are not structurally equivalent to the disclosed structures that perform that function in exemplary embodiments of the invention.

[0144] Specific examples of systems, methods, and apparatus are described herein for illustrative purposes. These are merely examples. The techniques provided herein may be applied to systems other than the exemplary systems described above. Many changes, modifications, additions, omissions, and permutations are possible within the practice of this invention. The present invention provides for variations of the described embodiments that will be apparent to those skilled in the art, and includes variations obtained by at least one of replacing features, elements, and / or acts with equivalent features, elements, and / or acts; mixing and matching features, elements, and / or acts from different embodiments; combining features, elements, and / or acts from the embodiments described herein with features, elements, and / or acts of other technologies; and omitting combinations of features, elements, and / or acts of the described embodiments.

[0145] Various features are described herein as being present in "some embodiments." Such features are not required and may not be present in all embodiments. An embodiment of the invention may comprise any combination of none, any one, or more than two of such features. This is limited only to the extent that some of such features are incompatible with other of such features in the sense that it would be impossible for one of ordinary skill in the art to construct a working embodiment combining such incompatible features. Thus, a statement that "some embodiments" have feature A, but also that "some embodiments" have feature B, should be construed as an explicit indication that the inventors also contemplate an embodiment combining feature A and feature B (unless otherwise stated in the description or feature A and feature B are fundamentally incompatible).

[0146] It is therefore intended that the following appended claims and the claims which follow from them be interpreted with all modifications, permutations, additions, omissions, and subcombinations which may reasonably be imagined. The scope of the claims should not be limited by the preferred embodiments set forth in the examples, but should be accorded the broadest interpretation consistent with the specification as a whole.

Claims

1. A retrofit interface device for interfacing with a target device to provide touchless user input to the target device, the retrofit interface device comprising: a touchless sensing system including one or more sensors responsive to touchless input by a human user and generating one or more corresponding sensor input signals; a retrofit controller coupled to receive one or more of the sensor input signals from the touchless sensing system and configured to generate corresponding control signals based on the one or more sensor input signals; Equipped with the add-on controller is connectable to the target device to bypass touch-based input of the target device and provide the control signals as inputs to an existing control system of the target device, thereby causing the control system of the target device to operate the target device based on the control signals; the touchless sensing system comprising one or more capacitive sensors sensitive to disturbances of an electric field caused by a part of a human body proximate to the one or more capacitive sensors; at least one electrode of each of the one or more capacitive sensors is located on a user-facing surface of a display of the target device; at least one of the electrodes of each of the one or more capacitive sensors is fabricated from a transparent conductive material, such that a human user can view the display of the target device through the at least one of the electrodes of each of the one or more capacitive sensors; Retrofit interface device.

2. The add-on controller is configured to estimate a position of a portion of a hand of a human user based on one or more of the sensor input signals; the add-on controller is configured to generate a display signal based on an estimated position of a portion of a user's hand to cause the display to display an indicator (at least one of a cursor, a pointer, and the like) on a display based on the estimated position of the portion of the user's hand; the add-on controller is configured to estimate a proximity of a portion of a user's hand to the display based on the one or more sensor input signals. The retrofit interface device according to claim 1 .

3. The display signal causes the display to change an appearance of a displayed indicia based on an estimated proximity of a portion of the user's hand; the change in appearance of the displayed indicator comprises at least one of: a change in color of the displayed indicator, a change in size of the displayed indicator, a change in brightness (intensity) of the displayed indicator, a change in shape of the displayed indicator, a change in color gradient of the displayed indicator, and the addition of other indicators to the displayed indicator. The retrofit interface device according to claim 2 .

4. The add-on controller is configured to detect a circular movement made by a portion of the user's hand based on one or more of the sensor input signals. The retrofit interface device according to claim 2 .

5. The display signal causes the display to vary a size of a displayed indicia in a manner correlated with a proximity of a portion of the hand to the display; the add-on controller is configured to modify the control signal, thereby modifying the operation of the target device, when a size of the displayed indicator is the same (to within a suitable threshold) as one of the virtual inputs and a position of the hand portion lacks movement for a threshold period (having a suitable threshold); The retrofit interface device according to claim 2 .

6. The retrofit interface device includes a retrofit display. The retrofit interface device according to claim 1 .

7. The touchless sensing system and the add-on display are manufactured as an integrated module.

7. The retrofit interface device according to claim 6.

8. The touchless sensing system comprises one or more optical sensors disposed about a periphery of the display, the one or more optical sensors being sensitive to changes in electromagnetic radiation reflected from the user's body on a user-facing surface of the display; the one or more sensor input signals comprise one or more capacitive sensor signals from one or more of the capacitive sensors and one or more optical sensor signals from one or more of the optical sensors; and the retrofit controller is configured to generate the control signal based on both the capacitive sensor signal or signals and the optical sensor signal or signals. The retrofit interface device according to claim 1 .

9. A retrofit interface device for interfacing with a target device to provide touchless user input to the target device, the retrofit interface device comprising: a touchless sensing system including one or more sensors responsive to touchless input by a human user to generate one or more corresponding sensor input signals; a retrofit controller coupled to receive one or more of the sensor input signals from the touchless sensing system and configured to generate corresponding control signals based on the one or more sensor input signals; Equipped with the add-on controller is connectable to the target device to bypass touch-based input of the target device and provide the control signals as inputs to an existing control system of the target device, thereby causing the control system of the target device to operate the target device based on the control signals; the one or more sensors comprise one or more capacitive sensors sensitive to changes in an electric field caused by a part of a human body proximate the one or more capacitive sensors, the one or more capacitive sensors generating one or more corresponding capacitive sensor signals forming at least a part of the sensor input signal; at least a portion of the one or more capacitive sensors are disposed around a periphery of a display of the target device and are sensitive to electric field disturbances caused by a part of a human body proximate a user-facing surface of the display at a center of the periphery of the display; Retrofit interface device.

10. The touchless sensing system includes one or more optical sensors disposed about the periphery of the display, the one or more optical sensors being sensitive to changes in electromagnetic radiation reflected from the user's body on a user-facing surface of the display; the one or more sensor input signals comprise one or more capacitive sensor signals from one or more of the capacitive sensors and one or more optical sensor signals from one or more of the optical sensors; and the retrofit controller is configured to generate the control signal based on both the capacitive sensor signal or signals and the optical sensor signal or signals.

10. The retrofit interface device according to claim 9.

11. Each of the one or more optical sensors comprises one or more lasers for emitting laser radiation and a detector positioned to receive the laser radiation reflected from a portion of a user's body. The retrofit interface device of claim 10.

12. each said optical sensor comprising suitable optics arranged to receive the emitted laser radiation and generate a two-dimensional plane of the laser radiation; the add-on controller is configured to estimate two orthogonal spatial coordinates of the body part based on the body part intersecting the two-dimensional plane; The retrofit interface device of claim 11.

13. The add-on controller is configured to estimate a third orthogonal spatial coordinate of the body part based on the capacitance sensor signal.

13. The retrofit interface device of claim 12.

14. The third orthogonal spatial coordinate is a z-coordinate along a normal to a tangent plane of the display of the target device so as to reflect a distance of the body part from the tangent plane; the first and second orthogonal spatial coordinates being lateral x and y coordinates reflecting distances of the body part in directions orthogonal to the normal to the tangent plane and orthogonal to each other; 14. A retrofit interface device according to claim 13.

15. The method of claim 1, wherein at least one electrode of each of the one or more capacitive sensors is located on a user-facing surface of a display of the target device; at least one of the electrodes of each of the one or more capacitive sensors is fabricated from a transparent conductive material, such that a human user can view the display of the target device through the at least one of the electrodes of each of the one or more capacitive sensors; 10. The retrofit interface device of claim 9.

16. The retrofit interface device includes a retrofit display.

10. The retrofit interface device of claim 9.

17. A retrofit interface method for interfacing with a target device to provide touchless user input to the target device, the retrofit interface method comprising: detecting touchless input corresponding to a gesture made by a human user and generating one or more corresponding sensor input signals in response to the detection; providing a retrofit controller coupled to receive one or more of the sensor input signals and generating corresponding control signals by the retrofit controller based on the one or more sensor input signals; and connecting the add-on controller to the target device such that touch-based input of the target device is bypassed and the control signal is provided as an input to an existing control system of the target device, whereby the control system of the target device operates the target device based on the control signal; It is equipped with detecting the touchless input comprises providing a touchless sensing system including one or more sensors responsive to the touchless input; the one or more sensors comprise one or more capacitive sensors sensitive to disturbances of an electric field caused by a part of a human body proximate to the one or more capacitive sensors, the one or more capacitive sensors generating one or more corresponding capacitive sensor signals forming at least a part of the sensor input signal; at least one electrode of each of one or more of the capacitive sensors is located on a user-facing surface of a display of the target device, and at least one electrode of each of one or more of the capacitive sensors is fabricated from a transparent conductive material, such that a human user can view the display of the target device through at least one electrode of each of one or more of the capacitive sensors; Retrofit interface method.

Images