Touchless device interface

Abstract

To provide a retrofit interface device interfacing a target device to provide touchless user input to the target device.SOLUTION: A touchless sensing system comprises one or a plurality of sensors responding to touchless input performed by a human user, thereby generating one or a plurality of corresponding sensor input signals. A retrofit controller is connected so as to receive one or a plurality of sensor input signals from the touchless sensing system, and generates corresponding control signals on the basis of the one or the plurality of sensor input signals. The retrofit controller can be connected to a target device. A control signal is provided as input to the existing control system of the target device so as to bypass the touch-based input of the target device. As a result, the control system of the target device operates the target device on the basis of the control signal.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present technology relates to touchless interaction with machines and the like. Certain embodiments provide a retrofit touchless interface and / or input for providing a touchless function in place of a contact-based human-machine interface (HMI) and / or contact-based input for such target devices, as well as methods of installing and / or operating them, for use in conjunction with existing target devices (target systems, target equipment). Other embodiments provide custom and / or modular touchless interfaces and / or inputs for facilitating human interaction with machines or other target devices, as well as methods of installing and / or operating them.

Background Art

[0002] Touch-based interfaces such as touchscreens, keypads, light switches, elevator buttons, etc. are ubiquitous. These interfaces can be manufactured inexpensively and are primitively intuitive to use. Touchless types of human-machine interfaces (HMIs) are currently limited mainly to industry-specific niche applications such as gaming, entertainment, automotive panels, home automation, and clinical sterile interactions.

[0003] This paradigm is changing in view of the COVID-19 pandemic and the increased awareness of avoiding common means of infection spread, such as high-traffic touch surfaces that are generally exposed to a large number of individuals. There is a growing need for touchless (touch-free) and / or contactless solutions for human-machine interface (HMI) systems. As a specific example, without limiting the generality of the common needs, there is a specific need for touchless and / or contactless solutions for HMI systems used in hospitals, clinics, surgical environments, and / or the like. However, current touchless human-machine interface (HMI) systems are inferior to touch-based systems in many applications. Current touch-based human-machine interface (HMI) systems often not only have tactile feedback, but can also be faster, easier to use, and / or more intuitive than touchless solutions.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Accordingly, there is a need for an improved touchless human-machine interface (HMI) that provides an intuitive and practical user experience, tactile feedback, and / or the like. There is a desire to provide touchless interaction for the most common touch-based human-machine interfaces (HMIs) and input interfaces that exist in today's society (e.g., retrofitting the touch-based human-machine interface (HMI) and input interface of a target device with a touchless solution). **Means for Solving the Problem**

[0006] Aspects of the present invention provide a method and apparatus for providing touchless user input to a target device by retrofitting an existing target device equipped with a touch-based user input device. The retrofitting method and apparatus according to some embodiments include a capacitance sensor for touchless sensing (capacitive sensor, capacitive type sensor), and a touch-actuated component (TAC) that presses, slides, rotates, or otherwise interacts with a touch-based input of the target device (e.g., buttons, latches, door handles, and / or the like). The retrofitting method and apparatus may also include user feedback that can be embodied, by way of non-limiting examples, using sonic wave transmission (SWT) for touchless tactile feedback (THF) (i.e., touch), using a display screen (e.g., LCD or LED) for visual feedback, using a sound cue for auditory feedback, and / or using similar methods. The user feedback may desirably provide an intuitive user experience that is easy to learn and quickly usable. The timing of use of the retrofitting method and apparatus is desirably on the same (or equivalent) time scale as the target touch-based system, or not much longer than the target touch-based system.

[0007] Another aspect of the present invention provides a retrofit interface device for interfacing with a control system of a target device. This retrofit interface device (this device) comprises a touchless sensing system for detecting a touchless input corresponding to a gesture made by a human user and generating one or more corresponding input signals. The device also includes a retrofit controller (controller, interface controller) connected to receive the input signal from the touchless sensing system and to generate a corresponding control signal and a corresponding display signal based on the input signal. The retrofit controller is connected to provide the display signal to the control system of the target device and / or to the display of the target device. The display signal can cause the display to display a corresponding visual indicator based on the display signal. The retrofit controller is connected to provide the control signal to the control system of the target device, whereby the control system of the target device operates the target device based on the control signal.

[0008] Further aspects and exemplary embodiments are shown in the accompanying drawings and / or described in the following description. The accompanying drawings illustrate non-limiting exemplary embodiments of the present invention.

Brief Description of the Drawings

[0009]

Figure 1

Figure 1A

Figure 1B

Figure 1C

Figure 2

Figure 3A

Figure 3B

Figure 3C

Figure 4A

Figure 4B

Figure 5A

Figure 5B

Figure 6

Figure 6A

Figure 7A

Figure 7B

Figure 8

Figure 8A

Figure 8B

Figure 9

DETAILED DESCRIPTION OF THE INVENTION

[0010] In the following description, specific details are set forth in order to provide a more thorough understanding of the present invention. However, the present invention may be practiced without these specific details. In other instances, well-known elements have not been shown or described in detail to avoid unnecessarily obscuring the present invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

[0011] Aspects of the present invention provide a method and apparatus for providing touchless user input to an existing target device having a target touch-based user input device (e.g., a touch-based human machine interface (HMI)) by retrofitting the existing target device.

[0012] By retrofitting an existing target device having a target touch-based interface (or other target touch-based user input), a method and apparatus are provided for providing a touchless user input function to the target device. Such target input is typically integrated into a given device (the target device), so in some embodiments, the retrofitted interface device is arranged relative to the existing (target) touch-based input, such as adjacent to it or in some other way, such that when a user provides input to the target device by performing a touchless gesture (e.g., hover, point / tap, wave, etc.), the retrofitted interface device can interact with at least a portion of the target touch-based input (e.g., buttons, knobs, handles, sliders, etc.). In some embodiments, the retrofitted interface device is connected to communicate directly with the control system of the target device by bypassing the existing target touch-based input of the target device. In some embodiments, the retrofitted interface device is configured to emulate the existing target touch screen interface of the target device.

[0013] Generally, a retrofit interface device according to a particular embodiment can be retrofitted to any suitable target device to provide a touchless user input function to the target device. Exemplary target devices and their target touch-based inputs include, but are not limited to, pedestrian control devices for street light signals, lighting switches and panels, key pads of intercom systems, payment (POS) terminals, ATMs (individual and in-bank units), elevator panels, parking meters / terminals, keypad locks, touch screen kiosks (e.g., airport check-in kiosks, retail self-checkout kiosks, advertising kiosks, hospital check-in kiosks, and / or the like), computers, faucets, water dispensers, lunchroom equipment, vending machines, gaming machines, game controllers, casino slots and other gambling equipment, door handles, latches, knobs, industrial equipment, medical equipment, etc.

[0014] FIG. 1 schematically shows a retrofit interface device 10 according to an exemplary embodiment of the present invention. The retrofit interface device 10 is configured to be retrofitted to a target device 50 having a touch-based input 50A to provide the target device 50 with touchless input functionality that can be used by a user (in addition to or instead of the touch-based input 50A) to provide an input to the target device 50. In the illustrated embodiment of FIG. 1, the retrofit interface device 10 interfaces with the target control system 51 of the target device 50 by using a touchless control signal 16 and, optionally, interfaces with the target display 22 of the target device 50 by using a touchless display signal 18. The retrofit interface device 10 includes a touchless sensing (non-contact sensing) system 12 for detecting touchless input (e.g., an input corresponding to a gesture made by a human user) and a retrofit controller (interface controller) 20 connected to receive a touchless input signal 14 generated by the touchless sensing system 12. In some embodiments, the touchless input signal 14 may be generated in response to the touchless sensing system 12 detecting a touchless input from the user or otherwise determining that the user has made a touchless input. That is, the touchless sensing system 12 may comprise its own controller or internal logic circuit (not shown) that interprets signals from the sensors of the touchless sensing system 12 and provides a touchless input signal 14 indicative of a particular user input to the retrofit controller 20. In some embodiments, such as in the illustrated embodiment of FIG. 1, the touchless input signal 14 may be received by the retrofit controller 20, and the retrofit controller 20 may determine whether the touchless input signal 14 corresponds to a particular touchless user input. Appropriate signal conditioning circuitry (not shown), such as amplifiers, filters, multiplexers MUX, and / or the like, known to those skilled in the art may be provided between the touchless sensors of the touchless sensing system 12 and the retrofit controller 20.

[0015] The retrofit controller 20 is connected to receive (receive, accept) the touchless input signal 14 from the touchless sensing system 12. The retrofit controller 20 may be configured (e.g., suitably programmed) to generate a touchless control signal 16 and / or a touchless display signal 18 for the target device 50 based on the touchless input signal 14 received from the touchless sensing system 12. The retrofit controller 20 may be operably connected to the target control system 51 of the target device 50 by providing the touchless control signal 16 to the target control system 51, whereby the target control system 51 operates the target device 50 based on the touchless control signal 16. For example, the retrofit controller 20 may be connected to the target control system 51 by delivering the touchless control signal 16 to the target control system 51, whereby the target control system 51 operates the target device 50 based on the touchless user input detected by the touchless sensing system 12, such as the movement or estimated position (location) of a part of the user's hand (e.g., a finger). The target control system 51 of the target device 50 may use the touchless control signal 16 from the retrofit controller 20 of the retrofit interface device 10 in addition to or instead of the touch-based control signal 19 from the touch-based interface (50A) of the target device 50 to control the operation of the target device 50.

[0016] The retrofit interface device 10 may optionally include a retrofit display 60. The retrofit display 60 may be part of the retrofit feedback mechanism 55 (discussed further below). The retrofit controller 20 can cause the retrofit display 60 to display a corresponding image, video, visual indicator, and / or the like by providing a video signal (61) to the retrofit display 60. In this way, the retrofit display 60 may provide some feedback to the user regarding the state of the retrofit interface device 10 (e.g., whether the retrofit interface device 10 has detected a touchless input) and / or the state of the target device 50. In some embodiments, the target device 50 may comprise its own display or some other form of target output device (22) by which feedback regarding the state of the target device 50 can be provided to the user. For example, the target device 50, which is an elevator control panel (control panel), may include a target output device (22) that is a flat screen display (e.g., to indicate the next floor at which the elevator car will stop), or may include a target output device (22) that simply illuminates the corresponding floor button (or part of the button) that has been pressed. For the sake of brevity, the target output device (22) of the target device 50 may be referred to herein as the target display 22, and such references should be understood to include any suitable form of the target output device (22) unless the context indicates otherwise.

[0017] The target control system 51 of the target device 50 can control the target display 22 of the target device 50 via an appropriate target display signal 21. In some embodiments, the retrofit controller 20 of the retrofit interface device 10 can be optionally connected to the target display 22 of the target device 50 additionally or alternatively to provide a touchless display signal (output signal) 18 to the target display 22, whereby the target display 22 can display a corresponding image, video, visual indicator, and / or the like based on the touchless display signal 18. In some embodiments, the retrofit controller 20 of the retrofit interface device 10 may provide a touchless control signal 16 to the target control system 51 of the target device 50, whereby the target control system 51 can cause the target display 22 to display a corresponding image, video, visual indicator, and / or the like (e.g., via the target display signal 21).

[0018] The touchless sensing system 12 (non-contact sensing system) 12 includes one or more sensors for detecting touchless user input (non-contact user input). In some embodiments, the touchless sensing system 12 of the retrofit interface device 10 and / or other retrofit interface methods and devices described herein includes one or more capacitance sensors 30 that are sensitive to an electric field (electric field, electrostatic field) in the vicinity of the sensor (i.e., having a capacitance that changes in response to the electric field). For example, gestures made by a user (e.g., taps, finger movements, etc.) can cause corresponding changes in the electric field adjacent to the capacitance sensor 30, thereby potentially causing a detectable change in the capacitance of the capacitance sensor 30. For simplicity, a single capacitance sensor 30 is shown in FIG. 1, but it will be understood that the touchless sensing system 12 may be configured with a plurality of appropriately arranged capacitance sensors 30. Advantageously, the capacitance sensor 30 can facilitate accurate short-range detection of touchless user interaction events (e.g., hand gestures, finger gestures, and / or the like).

[0019] In some embodiments, the capacitance sensor 30 is fabricated on a printed circuit board (PCB) (not shown) or otherwise supported. The printed circuit board PCB may be configured to include a multi-layer printed circuit board PCB. In some such embodiments, one surface or layer of the printed circuit board PCB (known as the receiving surface or receiving electrode) may be exposed to the object to be sensed (e.g., the user's finger or hand). In some such embodiments, this printed circuit board PCB receiving surface or receiving layer is not directly exposed, but rather is covered by a protective layer of non-conductive material. Each capacitance sensor 30 may be configured to include a pair of electrodes (a transmitting electrode and a receiving electrode) between which an electric field is established. In some embodiments, a plurality of capacitance sensors 30 may be provided by a single transmitting electrode and a plurality of receiving electrodes (or vice versa). The transmitting electrode may be located relatively farther from the user than the receiving electrode (e.g., in a layer that is relatively farther from the user than the first layer of the printed circuit board PCB). A layer of non-conductive material may be disposed between the transmitting electrode(s) and the receiving electrode(s) to electrically insulate the transmitting electrode from the receiving electrode.

[0020] In operation, an object (e.g., a part of a human body (a body part)) that is desired to be sensed by the touchless sensing system 12 causes interference in the electric field between the transmitting electrode and the receiving electrode of the capacitance sensor 30 when the object is located at or proximate to the capacitance sensor 30, thereby changing the capacitance of the capacitance sensor 30. Each capacitance sensor 30 causes the capacitance sensor 30 to output a signal m (e.g., a touchless input signal 14) that depends on the capacitance between its receiving electrode and transmitting electrode, and may thus be configured to include an appropriate circuit (not explicitly shown) that depends on the location of a part of the user's body (e.g., a hand or a finger). Thereby, by using the capacitance sensors 30 of the touchless sensing system 12 (e.g., by an appropriately configured controller such as the retrofit controller 20 of the retrofit interface device 10, or an internal controller (not shown) of the capacitance sensor 30 or the touchless sensing system 12, etc.), the movement of a part of the user's body (e.g., a hand or a finger) can be detected and tracked. With an array of preferably arranged capacitance sensors 30 and corresponding output signals, a machine learning algorithm can be trained to infer location characteristics regarding a part of the user's body. It will be understood that the output signals from the sensors (including the capacitance sensors 30) can be interpreted by any suitable combination of one or more controllers that may include the retrofit controller 20 of the retrofit interface device 10, or an internal controller (not shown) of the capacitance sensor 30 or the touchless sensing system 12, and / or the like.

[0021] For some human machine interface (HMI) applications, the capacitance sensor 30 can provide the following non-limiting advantages over other types of sensors (e.g., 3D camera sensors) to facilitate touchless input detection: the detection range of the capacitance sensor 30 is concentrated in a range of about several centimeters; the firmware-level algorithm is adjustable to detect a single point of interest (e.g., the center of mass of a conductive object such as the user's finger); the influence of ambient factors (e.g., light, background movement, etc.) on the function is limited; the capacitance sensor 30 does not raise privacy concerns similar to those of camera-based sensors. Such advantages enable the capacitance sensor 30 to provide more consistent and / or more accurate estimations about the types of user gestures that would typically be used to interact touchlessly with the retrofit interface device 10 (e.g., by a suitably configured controller such as the retrofit controller 20 of the retrofit interface device 10, or an internal controller (not shown) of the capacitance sensor 30 or the touchless sensing system 12, etc.) when used with the retrofit interface device 10.

[0022] The layout, size, and / or shape of the receiving and / or transmitting electrodes of the capacitance sensor 30 can be customized (e.g., on a printed circuit board PCB). In some embodiments, the receiving electrodes of the capacitance sensor 30 are spaced apart, and the signal of each receiving electrode is based on the proximity (closeness, proximity) of a part of the human body (e.g., a finger). The capacitance sensor 30 and / or the touchless sensing system 12 may include an internal controller suitably configured (e.g., programmed) with signal processing algorithms that utilize these signals (from each receiving electrode) to determine the position (e.g., three-dimensional x, y, z coordinates; suitable subsets or representations of these coordinates; and the like) of the conductive part (e.g., fingertip) of the user's body. In some embodiments, the retrofit controller 20 is suitably configured (e.g., programmed) with these signal processing algorithms to determine the position of the conductive part of the user's body. The shape and / or size and / or position of each receiving electrode of the capacitance sensor 30 can be designed, for example, based on whether the touchless sensing system 12 is wired or battery-powered, the desired detection range, the desired detection accuracy, and / or mechanical constraints.

[0023] In some embodiments, the layer of the printed circuit board PCB that supports the transmitting electrode(s) has a copper expansion layer and / or a copper hatch layer (e.g., a "crisscross" copper trace) having dimensions (e.g., x- and y-dimensions in the plane of the printed circuit board PCB) that are slightly larger than the corresponding x- and y-dimensions of the receiving electrodes.

[0024] In some embodiments, the receiving and transmitting electrodes of the capacitance sensor 30 are arranged (e.g., paired) in a concentric configuration within a rectangular cell, and a schematic diagram thereof is shown in FIG. 2. In the embodiment of FIG. 2, the capacitance sensor 30 includes an array of individual capacitance sensor cells 30A, and each of the individual capacitance sensor cells 30A is configured to include a receiving electrode 36 and a transmitting electrode 34. For an individual capacitance sensor cell 30A, the receiving electrode 36 surrounds the transmitting electrode 34, or, in an alternative arrangement, the transmitting electrode 34 surrounds the receiving electrode 36. In such embodiments, each capacitance sensor cell 30A (i.e., an individual capacitance sensor cell 30A) outputs a unique capacitive detection signal (e.g., a signal from each receiving electrode 36) by being connected to a controller (e.g., an internal controller of the capacitance sensor 30 or a touchless sensing system 12 (not shown), a retrofit controller 20 of the retrofit interface device (system) 10, and / or the like) via a suitable signal conditioning circuit (e.g., an amplifier, a filter, a digital-to-analog converter, a multiplexer, and / or the like). In such an embodiment, the voltage applied to each capacitance sensor cell 30A (e.g., to the transmitting electrode) may be configured to control the electric field output corresponding to the capacitance sensor cell 30A. By independently controlling the electric fields of the individual capacitance sensor cells 30A in this way, the shape and boundary of the overall sensor electric field of the capacitance sensor 30 can be accurately controlled. For example, by individually scanning each receiving electrode 36 of each capacitance sensor cell 30A of the capacitance sensor 30, a series of accurate measurement values can be obtained. Advantageously, such a modular design (involving independent control of each capacitance sensor cell 30A) provides good scalability due to its modular cell design (i.e., such a design can be expanded to a large size (e.g., by increasing the number of capacitance sensor cells 30A), and can be shaped while maintaining an acceptable detection / sensitivity volume (e.g., by the appropriate positioning of the capacitance sensor cells 30A)).

[0025] In some embodiments, the capacitance sensor 30 is arranged and / or oriented (aligned) to detect one or more position characteristics and / or motion characteristics (e.g., velocity, acceleration, and / or the direction associated with such velocity and / or acceleration) of a part of the user's body or one or more other conductive objects, typically at the center or mass or centroid of a part of the body or other conductive object. It will be understood that appropriate digital or analog signal processing (e.g., taking a derivative) may be used to obtain motion characteristics (movement characteristics, motion characteristics) from position characteristics. Thus, the position characteristics and / or movement characteristics of the centroid of a part of the body or other conductive object may be determined by the capacitance sensor 30 in cooperation with a suitably configured controller and / or signal processing hardware such as the retrofit controller 20 of the retrofit interface device 10, or an internal controller (not shown) of the capacitance sensor 30 or the touchless sensing system 12, and / or the like. For the sake of brevity, in the remainder of the present disclosure, reference is made to position characteristics, but it is understood that such position characteristics may include movement characteristics that may be derived from or independently detected from the position characteristics, unless the context indicates otherwise. For the sake of simplicity, the present disclosure may refer to the touchless sensing system 12 and / or its sensors (e.g., the capacitance sensor 30 or any other sensor) that perform certain operations (e.g., determination of characteristics, detection of characteristics, sensing of characteristics, generation of signals and / or information, and / or the like). Unless the context indicates otherwise, it should be understood that such sensors may perform such operations in cooperation with one or more suitably configured controllers and / or signal processing hardware such as the retrofit controller 20 of the retrofit interface device 10, or an internal controller (not shown) of the sensor or the touchless sensing system 12, and / or the like.

[0026] The position characteristic may correspond to a gesture. For example, the position characteristic may be configured to include the position of a part of the user's body (e.g., the centroid (center of gravity) of a finger, hand, fist, wrist, forearm, etc.) when the user makes a gesture using a part of the body. Unless the context indicates otherwise, the term "gesture" (as used herein) refers to a position (location, position), movement (motion, movement), and / or configuration made by a part of the user's body, and thus should not be construed as being limited to hand gestures. As a non-limiting example, the capacitance sensor 30 may be configured to detect hand gestures, finger gestures, the angle of movement of a part of the body relative to the sensor 30 (e.g., approach), the position of a part of the body, and / or the like. Such position characteristics may be used (e.g., by the retrofit controller 20) to determine a specific touchless input to the target device 50. For example, if a touch-based interface (50A) includes a series of input buttons (such as an elevator control panel), such position characteristics may be used to determine a touchless input representing the selection (e.g., pressing) of one such button.

[0027] In some embodiments, the retrofit controller 20 is configured to generate a control signal (e.g., the touchless control signal 16, the actuator control signal 39 (further described below), or the emulator control signal (emulator signal) 43 (further described below)) that affects the operation of the target device 50 based on the detected circular motion by detecting the position characteristics of a part of the user's hand that includes a circular motion performed based on the touchless input signal 14. In some embodiments, the retrofit controller 20 is configured to generate a control signal (e.g., the touchless control signal 16, the actuator control signal 39 (described further below), or the emulator control signal 43 (described further below)) that affects the operation of the target device 50 based on the detected lack of movement by detecting the position characteristics of a part of the user's hand that lacks movement (within a threshold) over a threshold period based on the touchless input signal 14. In some embodiments, the retrofit controller 20 is configured to generate a control signal (e.g., the touchless control signal 16, the actuator control signal 39 (described further below), or the emulator control signal 43 (described further below)) that affects the operation of the target device 50 based on the proximity of a part of the user's hand to a part of the retrofit display 60 or the target display 22 (e.g., for an appropriate period) by detecting the position characteristics that include the proximity of a part of the user's hand to a part of the retrofit display 60 or the target display 22 based on the touchless input signal 14. In some embodiments, the retrofit controller 20 is not only configured to detect position characteristics that include the proximity of a part of the user's hand to a part of some other generally planar surface, such as a panel (e.g., for an appropriate period) that indicates icons, virtual buttons, etc., based on the touchless input signal 14, and, based on the proximity of a part of the user's hand to a part of the other planar surface, generate a control signal (e.g., the touchless control signal 16, the actuator control signal 39 (further described below), or the emulator control signal 43 (further described below)) that affects the operation of the target device 50.

[0028] The touchless sensing system 12 may optionally include one or more additional sensors 33 (e.g., a second set of one or more capacitance sensors 35, one or more optical sensors (40) described in more detail elsewhere in this document, etc.) that are arranged and / or oriented (directed) to detect one or more secondary characteristics. Such secondary characteristics may be different from (i.e., additional or alternative to) the position characteristics detected by the capacitance sensor 30, but this is not essential. Such secondary characteristics may include secondary position characteristics (and / or secondary motion characteristics) of a part of the user's body. Such a part of the body, while not essential, may be different from the part of the body associated with the primary position characteristics. The secondary position characteristics may correspond to a secondary gesture of a part of the body. Such additional sensors 33 may generate a secondary position sensor signal(s) when detecting the secondary position characteristics of the user's body.

[0029] Examples of secondary characteristics may include, but are not limited to, the volume, shape, position, and / or angle of movement (e.g., approach) of a part of the user's body that performs a gesture that can be used to provide touchless input to the retrofit interface device 10 to control the target device 50 in sequence.

[0030] The additional sensors 33 may be arranged adjacent to the first capacitance sensor (30). The first capacitance sensor (30) and the additional sensors 33 may have different detection ranges or other different detection characteristics from each other. For example, the additional sensors 33 may have a detection range that is wider (e.g., in the x - y plane corresponding to the printed circuit board PCB on which the capacitance sensor 30 is implemented) and / or farther (e.g., in the z direction orthogonal to the x - y plane) than the detection range of the first capacitance sensor (30). In some embodiments, a significant portion (e.g., 75%, 90%, or 100%) of the detection range of the first capacitance sensor (30) is arranged within the detection range of the additional sensors 33. In other embodiments, the detection range of the first capacitance sensor (30) and the detection range of the additional sensors 33 are non - overlapping.

[0031] In some embodiments, the touchless input signal 14 (from the touchless sensing system 12 to the retrofit controller 20) comprises both a position sensor signal generated by a first capacitance sensor (30) and a secondary sensor signal generated by an additional sensor 33. In some embodiments, the retrofit controller 20 receives the touchless input signal 14 from the first capacitance sensor (30) and the additional sensor 33, and the retrofit controller 20 is configured to determine location characteristics and / or secondary location characteristics. The retrofit controller 20 may be configured to generate a touchless control signal 16 and / or a touchless display signal 18 based on either or both of the position sensor signal from the capacitance sensor 30 and the secondary sensor signal from the additional sensor 33.

[0032] In some embodiments, the retrofit controller 20 is configured to determine the location characteristics of a part of the user's body based on the touchless input signal 14 from both the capacitance sensor 30 and the additional sensor 33. For example, the retrofit controller 20 may be configured to process (e.g., scale, filter, weight, other modification, interpretation, calibration, and / or the like) the information from the capacitance sensor 30 based on the information from the additional sensor 33. In some embodiments, the retrofit controller 20 is configured to determine the presence of a user located in the vicinity of the target device 50 based on the secondary sensor signal (from the additional sensor 33). The retrofit controller 20 may be configured to calibrate the position sensor signal (touchless input signal 14) from the capacitance sensor 30 based on the determination of the presence or absence of a user in the vicinity of the target device 50. The retrofit controller 20 may be configured in other ways to calibrate the position sensor signal (touchless input signal 14) from the capacitance sensor 30 based on the location characteristics detected by the additional sensor 33 and / or based on the output signal generated by such an additional sensor 33.

[0033] The retrofit interface device 10 and / or other retrofit interface methods and devices described herein may, according to some embodiments, comprise a voice detection technology (such as Alexa (registered trademark), Google (registered trademark), and / or the like) with a microphone (e.g., as an additional sensor 33), and (by learning appropriate machine learning or artificial intelligence software - may be programmed in the retrofit controller 20 and / or another controller) an appropriate voice detection algorithm. Such voice detection technology may be used in addition to or instead of the touchless capacitance sensor 30. Such voice detection technology may generally be included as part of the touchless sensing system 12 (e.g., as an additional sensor 33) and / or as part of the retrofit interface device 10. Such voice detection may be used, for example, to input buzzer numbers, names, license plate numbers, elevator floors, door lock codes, etc.

[0034] In some embodiments, the retrofit interface device 10 and / or other retrofit interface methods and devices described herein may comprise an optical sensing system 40. The optical sensing system 40 may be embodied as part of the touchless sensing system 12 (e.g., as an additional sensor 33), as shown in FIG. 1, but this is not essential, and the optical sensing system 40 may be a stand-alone component of the retrofit interface device 10. The optical sensing system 40 may be configured to include one or more optical sensors (e.g., a thermal camera, an infrared camera, an RBG camera, a time-of-flight camera, a stereoscopic camera, a structured light camera, a 3D camera, etc.). Such optical sensors may be sensitive (sensitive) to changes in electromagnetic radiation reflected from a part of the user's body (e.g., changes caused by gestures made by the user).

[0035] Advantageously, the optical sensing system 40 can be configured to calibrate and thereby enable the retrofit controller 20 to obtain more accurate position characteristics of a part of the user's body (e.g., a finger) compared to the case of using only the capacitance sensor 30. The optical sensing system 40 can be used to detect any of various characteristics of a part of the body (e.g., size and shape, left hand / right hand, fist position / forearm position, hand angle, etc.) to calibrate or otherwise compensate for information from the capacitance sensor 30 (e.g., by the retrofit controller 20). As a non-limiting example, the optical sensor (40) can be used to ascertain the following. That the user is particularly tall (and thus may tilt a finger downward when interacting (conversing, interacting) with the capacitance sensor 30), that the user's forearm is relatively close to the capacitance sensor 30 (and thus may affect the amplitude of the signal received from a particular capacitance sensor 30, or the estimation of the center of gravity of an object within the sensing volume may be biased towards the user's forearm rather than the user's fingertip and thus may need to be compensated), and that the user is approaching or interacting with the sensing volume of the capacitance sensor 30 with the left or right hand (whereby the retrofit controller 20 may use different left hand / right hand machine learning inference engines to predict the position characteristics of the user's finger), and / or the like.

[0036] In some embodiments, the retrofit controller 20 (and / or additional controllers (not shown) associated with the touchless sensing system 12) is configured with a machine learning data training model (e.g., a neural network model) that has learned to determine position characteristics of a part of a user's body by using a position sensor signal (e.g., a signal generated by the capacitance sensor 30) and a secondary sensor signal (e.g., a signal generated by the optical sensing system 40, and / or a second capacitance sensor 35, and / or some other additional sensors 33). Such position sensor signals and secondary sensor signals may be part of the touchless input signal 14. The retrofit controller 20 may be configured to generate a touchless control signal 16 and / or a touchless display signal 18 based on the position characteristics determined from the position sensor signal and / or the secondary sensor signal. For example, the retrofit controller 20 may be configured to select the touchless control signal 16 from among a plurality of potential control signals based on the position characteristics determined from such position sensor signal and / or secondary sensor signal.

[0037] In some embodiments, the optical sensing system 40 is configured to detect or otherwise confirm the presence (or absence) of a user in the vicinity of the retrofit interface device (system) 10 and / or the target device 50. In such embodiments, after detecting the presence of the user, the capacitance sensor 30 may be configured to record a measurement of the noise level. The retrofit controller 20 may be able to perform system calibration (e.g., of the capacitance sensor 30 and / or generally other sensors of the touchless sensing system 12) based on the measured background noise.

[0038] The optical sensing system 40 may optionally include one or more lasers for emitting laser radiation (e.g., infrared laser light, near-infrared laser, or other visible light), and one or more detectors (e.g., photodetectors) for detecting the emitted laser radiation. For example, the optical sensing system 40 may be configured to include a detector arranged to receive (receive) the laser radiation reflected from a part of the user's body (e.g., the part of the body making a gesture). The optical sensing system 40 may also include a suitable optical system arranged to receive the emitted (e.g., reflected) laser radiation. In some embodiments, one or more lasers of the optical sensing system 40 may be adapted to be used as the primary sensor of the retrofit interface device 10 - for example, the optical sensing system 40 may be used in place of the capacitance sensor 30 described herein.

[0039] In some embodiments, the optical sensing system 40 is configured to generate a two-dimensional plane of laser radiation, sometimes referred to herein as a detection plane (detection surface). This two-dimensional detection plane may have a normal vector that is substantially parallel (e.g., within 10° or 15°) to the normal vector of the plane of the retrofit display 60 (of the retrofit interface device 10) or the target display 22 (or target device 50), a plane that is in contact with the outer surface of the retrofit display 60 or the target display 22 (contact plane), and / or a plane that is in contact with some other surface of the target device 50 and / or the retrofit interface device 10 (e.g., the plane on which icons, virtual buttons, etc. are displayed). This two-dimensional detection plane may be arranged adjacent to (but spaced from) the plane of the retrofit display 60 or the target display 22, the contact plane of the retrofit display 60 or the target display 22, and / or the contact plane of the surface of the target device 50 and / or the retrofit interface device 10. In some embodiments, this two-dimensional detection plane may be arranged adjacent to (but spaced from) some other generally planar surface (e.g., a panel adorned with an image such as an icon). The position on this two-dimensional detection plane may be characterized by a suitable pair of coordinates (e.g., a Cartesian set of orthogonal x and y coordinates). The retrofit controller 20 may be configured to estimate the x and y coordinates of a part of the user's body based on a secondary sensor signal (e.g., a signal generated by the optical sensing system 40) indicating the position where a part of the body intersects the two-dimensional detection plane.

[0040] The optical sensing system 40 may be disposed adjacent to (or at some other suitable position relative to) the capacitance sensor 30. The optical sensing system 40 may be configured to detect an intersection of a part of a user's body (e.g., a finger) at a configurable distance (i.e., in the z-direction orthogonal to the x-y directions) from the capacitance sensor 30, the retrofit display 60, the target display 22, and / or any other generally planar surface (e.g., at x and y-positions on a detection plane). The optical sensing system 40 may be configured to sense the lateral x and y coordinates of a user's finger such that the finger intersects the sensing plane as the finger approaches the capacitance sensor 30 and / or the retrofit display 60, the target display 22, and / or some other generally planar surface, and the capacitance sensor 30 may be able to sense the orthogonal z coordinate of the finger as it approaches the sensing plane (e.g., before and / or after the finger intersects the sensing plane of the optical sensor (40)). The capacitance sensor 30 may also be able to detect the lateral (x-y) coordinates of the finger (or other body part), although the lateral coordinates measured by the capacitance sensor 30 may be relatively coarse compared to the lateral coordinates detected by the optical sensing system 40. Such x and y coordinates detected by the optical sensing system 40 may be associated with the selection of a particular input (e.g., a virtual button) that may be displayed at particular x and y coordinates on the retrofit display 60, the target display 22, and / or some other general plane. Detection of the z coordinate may help the touchless sensing system 12 anticipate the user's next selection. By way of non-limiting example, the z coordinate may be used by the retrofit controller 20 as part of a "virtual click" detection algorithm (e.g., to ascertain whether the user is attempting to touchlessly press or click a virtual button, or whether a part of the user's body is simply lingering within the sensing volume of the sensing system 12). Such detection of user interaction with virtual buttons is discussed further below.As another example, the z coordinate may be used to provide feedback to the user (e.g., via a suitable post-feedback mechanism 55) to indicate that a part of the user's body has been detected. The capacitance sensor 30 and the ability of the capacitance sensor 30 to detect the x, y, and z coordinates may also be used to determine the angle of approach to the detection plane. This angle of approach may be used together with more accurate lateral (x - y) information from the optical sensing system 40 as part of a "virtual click" detection algorithm. Such detection of user interaction with the virtual button is discussed further below.

[0041] The optical sensing system 40 may comprise a number of optional features that can be used to control the spatial position and / or orientation of the two-dimensional detection plane of the optical sensing system 40. For example, the optical sensing system 40 may comprise one or more laser emitters that may be disposed beside the plane of the target display 22 or the touch-based input 50A or the target device 50 and / or another suitable surface of the retrofit interface device 10 (or a plane in the tangential direction thereof), and may be oriented to direct radiation parallel to the normal of such a plane. In such an implementation, the optical sensing system 40 may optionally include one or more mirrors (optimized for reflection of emitted radiation) that are oriented to reflect the laser radiation in a direction at an angle of approximately 90 degrees with respect to the radiation direction, at an angle of approximately 45 degrees with respect to the emitted laser radiation direction, to form a detection plane parallel to the plane of the target display 22 or the touch-based input 50A (or a plane in the tangential direction), or a plane of some other suitable surface of the target device 50 and / or the retrofit interface device 10. The optical sensing system 40 may comprise different combinations of mirror shapes / sizes, and / or different placement angles, and / or different combinations of other optical elements (such as lenses, waveguides, and / or the like), to provide a desired detection plane at a desired position and / or orientation. The optical sensing system 40 may be configured to comprise a commercially available laser-based optical sensor such as Neonode® or zForce sensor®.

[0042] Figures 3A-3C illustrate a touchless sensing system 12 including one or more capacitance sensors 30 and an optical sensing system 40 including a plurality (e.g., two) of optical sensors 42A, 42B (collectively, optical sensors 42). The optical sensing system 40 can be used in conjunction with the retrofit interface device 10 (FIG. 1) and / or other retrofit interface methods and devices described herein, according to a particular exemplary embodiment. In the exemplary embodiment of FIGS. 3A-3C, the optical sensors 42A, 42B are optically oriented to direct radiation toward (or intersect) the sensing region 32 of the capacitance sensor 30. In the exemplary embodiment shown in FIGS. 3A-3C, each of the optical sensors 42 includes a radiation source for emitting radiation and a radiation receiver (e.g., a photodetector, a receiving diode, etc.) for receiving radiation reflected from any object in the path of the emitted radiation. Unless the context indicates otherwise, the term "optically oriented" (as used herein) is to be construed to mean that the optical sensor 42 (or other sensor of the optical sensing system 40) described herein may include any number (e.g., 0, 1, 2, 3, etc.) of suitable optical elements (e.g., lenses, mirrors, waveguides, etc.) arranged to shape, align, and / or orient radiation from the radiation source in the optically oriented direction.

[0043] As shown using the Cartesian axes shown in FIG. 3A, the radiation sensor (optical sensor 42) may be oriented or otherwise configured together to detect the x-position and y-position of an object (e.g., a finger) relative to the coordinate system of the capacitance sensor 30. For example, in the illustrated embodiment shown in FIG. 3A, the optical sensor 42B can detect not only the x'-position and y'-position (see the x', y', z' axes in FIG. 3A), but also the optical sensor 42A can detect the x''-position and y''-position (see the x'', y'', z'' axes in FIG. 3A). By using either or both of these detected positions (x', y') and (x'', y''), appropriate geometric calculations may be used (see the X, Y, Z-axes in FIGS. 3A and 3B) to determine the x-position and y-position in the coordinate frame of the capacitance sensor 30 (e.g., by the retrofit controller 20 and / or by a controller associated with the optical sensor 42 or the touchless sensing system 12). That is, the optical sensing system 40 may be configured to determine the lateral (x-y) position of the object relative to the capacitance sensor 30 based on the lateral (x'-y') and / or (x''-y'') positions detected by the optical sensor 42.

[0044] As described above, while the optical sensing system 40 (e.g., the optical sensor 42) can accurately determine the x-position and y-position of an object (e.g., a finger) relative to the capacitance sensor 30, the capacitance sensor 30 can accurately detect the distance of the object relative to the capacitance sensor 30 (i.e., the z-position of the object in the coordinate frame of the capacitance sensor 30). Thus, the optical sensing system 40 and the capacitance sensor 30 can complement each other. When an object such as a human finger enters the detection area 5 of the optical sensing system 40 defined by the optical sensor 42 (see FIG. 3B) (e.g., intersects one or more touchless detection planes 6 of the touchless sensing system 12 shown in FIG. 3B or otherwise enters the detection area 5 of the optical sensor), the optical sensing system 40 can be configured to determine the x-position and y-position of the object relative to the capacitance sensor 30 based on information from the radiation emitting sensor (optical sensor 42).

[0045] Since the sensing area (detection area 5) of the optical sensor 42 may be different from the sensing area 32 of the capacitance sensor 30, in some cases, the x-position and y-position of the object are first determined and tracked or temporarily stored until the object enters the sensing area 32 of the capacitance sensor 30 and / or until the object intersects the touchless detection plane 6 of the touchless sensing system 12. In some embodiments, after the x-position and y-position of the object are determined by the optical sensing system 40, the z-position of the object is detected by the capacitance sensor 30. In other embodiments, the z-position of the object is detected by the capacitance sensor 30 before the x-position and y-position of the object are determined by the optical sensing system 40. In some embodiments, the touchless sensing system 12 is configured to register the x-position and y-position (detected by the optical sensing system 40) of the object after the z-position of the object reaches a configurable threshold distance from the capacitance sensor 30 and / or from some other reference surface such as the surface of the target display 22 (or a plane in contact with the target display 22). For example, the touchless sensing system 12 may be configured to register the x- and y-positions when the object intersects the touchless detection plane 6. This threshold z-distance (at which the x- and y-positions are registered) can be evaluated by the capacitance sensor 30. In some embodiments, the retrofit controller 20 or some other controller associated with the touchless sensing system 12 can be configured to make determinations (e.g., detection of a specific gesture or detection of a gesture based on the registered x- and y-positions).

[0046] In some embodiments, the actuation plane (actuation surface) (6) is defined at a fixed position relative to the capacitance sensor 30 and / or at a fixed position from some other reference plane, such as the surface of the target display 22 or a plane in contact with the target display 22 - (e.g., at a distance corresponding to the position where the radiation 5 emitted by the optical sensors 42A, 42B intersects). In other embodiments, the position of the actuation plane (6) can be adjusted by appropriate programming (or user settings) of the retrofit controller 20 or some other controller associated with the touchless sensing system 12.

[0047] FIG. 3C schematically shows an exemplary arrangement of a capacitance sensor 30 and an optical sensing system 40 according to a particular embodiment. In the example shown in FIG. 3C, the touchless sensing system 12 includes a plurality (e.g., two) of optical sensors 42A, 42B (collectively, optical sensors 42) that are optically oriented to emit radiation from positions that may be in the same plane as the capacitance sensor 30. In some embodiments, the optical sensors 42 and the capacitance sensor 30 are formed as part of an integrated module (e.g., as part of a thin slate having the shape of a tablet device). In some embodiments, the optical sensors 42 and the capacitance sensor 30 are encapsulated in a medium 44, such as a resin, to fix the relative positions of the optical sensors 42 and the capacitance sensor 30. The medium 44 may be transparent at the wavelength of the optical sensors 42. In some embodiments, the medium 44 can shape, position, and / or direct (orient) such radiation by providing optical elements in the path of the radiation emitted (and / or received) by the optical sensors 42. The medium 44 may be configured to include glass or quartz. In some embodiments, the capacitance sensor 30 is formed using a transparent conductive material such as indium tin oxide (ITO) and is located within a glass medium (44), and the optical sensors 42 are oriented at a desired angle behind the glass. Advantageously, the glass medium (44) can be used to form a flat (non-protruding) design that can be flush-mounted to another surface (e.g., a wall or a horizontal surface) by enclosing the capacitance sensor 30 and the optical sensing system 40. This flush mounting is in contrast to prior art optical designs that protrude away from the surface (the wall or horizontal surface on which it is mounted (mounted in or on)). This flat design can prevent, for example, tampering with a retrofit interface device.

[0048] Referring to FIGS. 3A to 3C, in some embodiments, a second capacitance sensor 35 (not shown in FIGS. 3A to 3C) is arranged adjacent to the first capacitance sensor (30) so as to be oriented to sense the user's hand interacting with the first capacitance sensor (30). Next, the second capacitance sensor 35 may detect a second distance estimate (coordinate) that provides another degree of information regarding the object (hand / finger) relative to the first capacitance sensor (30). This information, along with the known orientation of the second sensor, may be used to determine the orientation of the hand / finger, which can provide useful additional information for identifying a particular input (e.g., a particular virtual button) that the user is attempting to interact with.

[0049] Additional optical sensors (not shown in FIGS. 3A - 3C) can be deployed by stacking on top of the existing optical sensors 42A, 42B and can provide more coverage and accuracy of the detected object. The additional optical sensors can be used to determine the orientation of the object and provide additional depth information for further analysis. One can imagine that optical sensors at different angles are triggered at different points of finger movement towards a virtual button or other touchless input. This can not only determine the "tap vector" or trajectory, but by using this information, it can be determined which virtual button the user is trying to select. For example, the finger may be placed over the virtual button corresponding to "2", but since it is facing right, the retrofit controller 20 can be configured (by appropriate software) to determine that the user intends to click "3" if the angle of the tap vector is sufficient. In addition to or as an alternative to providing more optical sensors (e.g., in addition to optical sensors 42A, 42B), other sensors and techniques can be used to determine the "tap - vector". Such additional or alternative techniques can include, but are not limited to, determining the tap - vector based on first and second three - dimensional position estimations based on information from a capacitance sensor (e.g., when detecting the intersection of first and second threshold distances (z - coordinate distances) away from the plane of the virtual button). It should be noted in this regard that the capacitance sensor can detect lateral (x, y) coordinates, but these lateral coordinates may be less accurate than the coordinates detected by the optical sensor (40). Such additional or alternative techniques can include, but are not limited to, a first three - dimensional detection based on information from the capacitance sensor 30 (e.g., when detecting the intersection of a first threshold (z - coordinate) away from the plane of the virtual button), and a second three - dimensional detection based on lateral information (x, y) coordinates based on information from the optical sensor (40) and the z - coordinate determined by the capacitance sensor 30, and determining the tap - vector based on these.In any of these tap vector estimation techniques, an appropriate time threshold may be used as another criterion related to tap determination (discrimination).

[0050] An optical sensor (40) (such as sensors 42A, 42B) can detect a plurality of points at once. The intention of the user operating at either a plurality of points or a single point can be determined within a threshold period frame of an object entering the detection area of the optical sensor. Depending on how many points are detected within the threshold period frame, the optical sensor (40) can enter a single-point or multi-point operation mode. As described above, multi-point accuracy can be well achieved by additional optical sensor(s) (if any) (i.e., in addition to sensors 42A, 42B shown in FIGS. 3A to 3C). Multi-point detection can be used for two-finger gestures such as pinch-to-zoom and pinch-to-pan.

[0051] Further improvement in the range and accuracy of the optical sensor (40) (such as optical sensors 42A, 42B) can be achieved by vibrating the detection plane of the optical sensor between an angle range (by using, for example, a rotary actuator). By such oscillation of the detection plane, in addition to the z-direction information obtained from the capacitance sensor 30, the optical sensor itself can have depth (depth, z-direction) information. Such a vibrating detection plane also enables obtaining lateral x-y positions at different object depths (i.e., different z coordinates) as opposed to a fixed depth.

[0052] In some embodiments, the retrofit controller 20 is configured to estimate a human user's line-of-sight vector and / or the distance between the human user and the retrofit display 60 based on the touchless input signal 14 from the touchless sensing system 12. The retrofit controller 20 may be configured to estimate the line-of-sight vector based on an estimated value of the position of a part of the user's hand (e.g., a fist, wrist, forearm, etc.) and / or an estimated value of the approach angle of a part of the user's hand. The line-of-sight vector may also, in some cases, be estimated based on the position of the user's head / torso. Information for performing such line-of-sight estimation may be provided by the touchless sensing system 12.

[0053] The user's line of sight and / or position may, in some cases, be detected by using additional line-of-sight sensors (e.g., capacitance sensors, optical sensors, cameras, and / or others (not shown)) provided as part of the touchless sensing system 12. For example, a suitable optical sensor may be used to detect the user's head pose, line of sight, and handedness (i.e., left or right hand). In some embodiments, the capacitance sensor 30 may be used to detect the position and / or approach angle of a part of the user's hand (e.g., by using a machine learning algorithm). In some embodiments, the line-of-sight sensor includes a height or altitude sensor (e.g., a laser-based sensor) configured to determine the height of the retrofit interface device 10 (or a particular part thereof). In such embodiments, the user's line of sight may be estimated based on the height of the retrofit interface device 10 and the average human height. This height information may be used in conjunction with the other detected values described above to estimate the user's line of sight.

[0054] Returning to FIG. 1, the retrofit interface device 10 (e.g., the retrofit controller 20) may be connected to directly communicate with the target control system 51 of the target device 50 by using the touchless control signal 16. In this way, the retrofit interface device 10 may replace or complement (i.e., provide an alternative or additional) the input function of the existing touch-based input 50A and the touch-based control signal (input signal) 19 of the target device 50. In some embodiments, the retrofit interface device 10 comprises wiring having one or more suitable communication protocols (e.g., USB, SPI, I2C, CAN, RS-232, serial, etc.). The wiring of the retrofit interface device 10 may be provided in an easy plug package for easy integration with the target device 50, and the target device 50 may comprise wiring having one or more corresponding communication protocols. For example, if the existing touch-based input 50A of the target device 50 is a USB mouse or a similar USB input device, the retrofit interface device 10 can provide a touchless input function to the target device 50 by being directly plugged into the USB jack of the target device 50. The connection of the retrofit interface device 10 to the target device 50 can additionally or alternatively be achieved by coupling to an existing electrical node (not shown) of the target device 50. For example, such an electrical node can comprise the same node to which the touch-based input 50A is connected to the target control system 51. In some embodiments, the connection of the touch-based input 50A may be disconnected from such an electrical node, whereby the retrofit interface device 10 replaces the function of the touch-based input 50A with a touchless input, but this is not essential.

[0055] The design of the retrofit interface device 10 can take several expandable forms. Some embodiments of the retrofit interface device 10 enable a manufacturer, and OEM third party, or the end user themselves, to cover a larger touch-based interface by connecting a single "block" unit to a larger unit. The single unit is applicable to a target device 50 (having only a single button) that is a street light signal, while 12 units combined in a 3×4 grid can be used if the target device 50 has an intercom system with a keypad. This modular design would mean that each module block could comprise a capacitive sensor, a method for visual or tactile feedback, and a touch-actuated component TAC, although some such implementations may share some of these components between module blocks.

[0056] Figure 1A is a schematic depiction of a retrofit interface device 10A according to another exemplary embodiment of the present invention. Similar to the retrofit interface device 10 described above, the retrofit interface device 10A provides touchless input to the target device 50 (which includes a touch-based input 50A) by being retrofitted to the target device 50. Instead of (or optionally in addition to) interfacing with the target device 50 using the touchless control signal 16 (as in the case of the retrofit interface device 10 - see Figure 1), the retrofit interface device 10A includes one or more actuators 37 that are positioned and movable to physically interact with the touch-based input 50A of the target device 50. The retrofit interface device 10A can also optionally communicate with the target display 22 of the target device using the touchless display signal 18. The actuator 37 may be referred to herein in the singular or plural, and it is understood that the retrofit interface device 10A may be configured to include one or more actuators 37. The actuator 37 may be connected to the retrofit controller 20 of the retrofit interface device 10A to receive the actuator control signal 39 from the retrofit controller 20. The actuator control signal 39 may be used by the retrofit controller 20 to control the movement (motion) of the actuator 37. The retrofit interface device 10A may include a suitable actuator drive circuit (not explicitly shown), such as a power source, amplifier, and / or the like, that can be used to drive the actuator 37 in response to (react to) the actuator control signal 39. The actuator 37 may move to physically interact with the touch-based input 50A of the target device 50 in response to the actuator control signal 39, thereby providing touchless (from the user's perspective) input to the target device 50.

[0057] The actuator 37 of the attachable interface device 10A physically interacts with and / or applies force to the touch-based input 50A of the target device 50 by pushing, pulling, sliding, rotating, or other means of the touch-based input 50A of the target device 50. The touch-based input 50A of the target device 50 may come in many forms, and as a result, the actuator 37 may include many different types of actuators 37 desirable for interacting with a particular touch-based input 50A. By way of non-limiting example, the actuator 37 may be used to push a keypad button, turn a dial or knob, pull or push, turn a door handle, slide a latch or slider input, touch a touch sensor, and / or the like. The actuator 37 may be suitably positioned relative to the target device 50 so as to interact with the touch-based input 50A of the target device 50 in a suitable manner.

[0058] In some embodiments, while the touch-based input 50A of the target device 50 includes a touch screen interface, the actuator 37 may be used to touch the touch screen interface (e.g., by using a conductive stylus, chip, and / or the like). In some such embodiments, the actuator 37 is arranged, moved into position, and / or sized to interact with a portion of the touch screen where an input may be provided (e.g., the portion where a touch button, keypad, or keyboard typically appears) so that the visibility of other portions of the touch screen interface is not obstructed by the actuator 37.

[0059] In some embodiments, the actuator 37 may be configured to include an electromechanical actuator driven by electric power, such as a motor, solenoid, linear actuator, hydraulic actuator, piezoelectric actuator, and / or the like. Some embodiments of the retrofit interface device 10A may be configured to include an array of actuators 37 for retrofitting to a target device 50 that includes a selection of touch-based inputs 50A. One exemplary embodiment of the actuator 37 is an array of discrete solenoids or motors arranged in a grid-like pattern and configured to press corresponding buttons of the touch-based input 50A, such as keypad buttons, keyboard buttons, touch screen virtual buttons, and / or the like. In some such embodiments, there may be a one-to-one correspondence between the actuator 37 and individual inputs of the touch-based input 50A (e.g., buttons, virtual buttons, touch screen positions, and / or the like), but this is not essential. In some embodiments, one actuator 37 may be used to interact with two or more individual inputs of the touch-based input 50A (e.g., buttons, virtual buttons, touch screen positions, and / or the like) by way of placement, movement to a position, sizing, and / or other means.

[0060] In some embodiments, the actuator 37 is supported for movement relative to the touch-based input 50A. For example, the actuator 37 may be supported on a movable gantry or on a framework having a movable head so that the actuator 37 can be moved in the vicinity of the touch-based input 50A (or a portion thereof) to facilitate interaction with the touch-based input 50A. For example, the actuator 37 may be supported on a "gantry" or on a framework having a movable head such that a grid of solenoids is arranged adjacent to a keypad button so that the keypad button can be pressed. In some embodiments, a suitable system with the use of magnetic fields and / or electric fields can be used to arrange one or more suitable actuators 37 for interaction with individual inputs of the touch-based input 50A.

[0061] The retrofit interface device 10A may include a solenoid actuator (37). The solenoid actuator (37) includes one or more solenoids that utilize a change in current to generate a magnetic field within each coil, thereby creating linear motion. When activated, a metal rod located inside the solenoid is either pushed outside the coil or pulled into the coil. In some embodiments, the retrofit interface device 10A may be configured with a matrix of discrete (e.g., independently controllable) solenoid actuators (37). As a non-limiting example, a grid arrangement of solenoid actuators (37) disposed adjacent to buttons (or virtual buttons) of the touch-based input 50A of the target device 50 would be able to press a given button. The solenoid actuator (37) can have a compact mechanical design compared to the motor-based actuator 37.

[0062] The retrofit interface device 10A may additionally or alternatively include a motor-based actuator 37. For example, the motor-based actuator 37 may be configured to include one or more servo motors, DC motors, or stepper motors for pushing, pulling, linearly actuating, or rotating the touch-based input 50A of the target device 50. The motor-based actuator 37 may be provided with or equipped with an appropriate mechanism for manipulating the direction, power, speed, and / or other characteristics of the force generated by each motor. The motor-based actuator 37 configured or equipped with such a mechanism may be utilized, for example, to generate a linear force to press a button of the touch-based input 50A or to rotate a set of gears (e.g., a gear train) to generate sufficient power and / or range to turn a door handle of the touch-based input 50A. The motor-based actuator 37 may be more power-efficient when compared to the solenoid-based actuator 37.

[0063] As mentioned above, the actuator 37 is not limited to solenoids and / or motors. Other suitable actuators 37 can include, by way of non-limiting example, linear actuators, hydraulic / pneumatic actuators, piezoelectric actuators, thermal actuators, spring-based actuators, magnetic actuators, electrostatic actuators, and / or the like. FIG. 9 is a schematic exploded perspective view of the retrofit interface device 10A of FIG. 1A including a plurality of linear actuators (37) according to a particular embodiment.

[0064] In other respects, the retrofit interface device 10A may be similar to the retrofit interface device 10 described herein. References to the features of the retrofit interface device 10 should be understood to be applicable to the retrofit interface device 10A unless the context specifically indicates otherwise.

[0065] Figure 1B schematically shows a retrofit interface device 10B according to another exemplary embodiment. Similar to the retrofit interface devices 10, 10A described herein, the retrofit interface device 10B provides touchless input to the target device 50 (which includes a touch-based input 50A) by being retrofitted to the target device 50. In the case of the retrofit interface device 10B of the embodiment of FIG. 1B, the touch-based input 50A of the target device 50 is configured to include a target touch screen (input) 50B. Instead of (or optionally in addition to) interfacing with the target device 50 using the touchless control signal 16 (as was the case with the retrofit interface device 10 - see FIG. 1) or using the actuator 37 that moves (as was the case with the retrofit interface device 10A - see FIG. 1A), the retrofit interface device 10B is configured to include one or more touch emulators 41 that are controllable to electrically interact with the target touch screen (touch panel) 50B of the target device 50 by emulating (mimicking) the contact of the user's body. The retrofit interface device 10B can also optionally communicate with the target display 22 of the target device using the touchless display signal 18. In some embodiments, the retrofit interface device 10B may optionally include an actuator (such as the actuator 37 of the retrofit interface device 10A) for interacting with the touch-based input 50A.

[0066] The touch emulator 41 may be referred to herein in the singular or plural, and it is understood that the retrofit interface device 10B may be configured to include one or more touch emulators 41. The touch emulator 41 may receive (receive) an emulator control signal 43 from the retrofit controller 20 by being connected to the retrofit controller 20 of the retrofit interface device 10B. The emulator control signal 43 may be used by the retrofit controller 20 to control the electrical characteristics of the touch emulator 41. The touch emulator 41 may emulate a human touch by electrically interacting with the touch-based input 50A (target touch screen 50B) of the target device 50 in response to the emulator control signal 43, thereby providing a touchless (from the user's perspective) input to the target device 50.

[0067] FIG. 4A schematically shows a typical capacitive target touch screen 50B of a typical target device 50. The target touch screen 50B is typically coated with a layer of a transparent conductive material (e.g., indium tin oxide (ITO)). When the target touch screen 50B is contacted by a conductive element (such as a user's finger), the local electrical characteristics (specifically, local charge, local capacitance, and / or local electric field) change at the location of the touch, so this touch event is observed by the target touch screen 50B and / or by the target control system 51 of the target device 50 (see FIG. 1B).

[0068] Figure 4B schematically shows a touch emulator 41 according to a particular exemplary embodiment. The touch emulator 41 of the embodiment of FIG. 4B can comprise one or more elements (e.g., cells) of a transparent conductive film 45 that can in turn comprise indium tin oxide (ITO), conductive carbon nanotubes, graphene, polymeric materials, thin metal (e.g., silver) nanowires, and / or the like. The transparent conductive film 45 may be disposed adjacent to (e.g., within a suitable interaction proximity) one or more regions of the target touch screen 50B. For example, the transparent conductive film 45 may directly overlap (and may be in contact with) the target touch screen 50B. The transparent conductive film 45 can be controllably and selectively connected to ground (i.e., to ground of the target touch screen 50B or to a ground that is sufficiently close to the ground potential of the target touch screen 50B) so as to vary local charge, local capacitance, and / or local electric field, thereby emulating a “touch” in that region of the target touch screen 50B. For example, in the exemplary illustrated embodiment of FIG. 4B, the transparent conductive film 45 is divided into a plurality of cells 52, and each cell 52 may be in direct physical contact with a corresponding region of the target touch screen 50B (e.g., via a conductive adhesive or applied pressure). The number of cells 52 shown in FIG. 4B, and the number of corresponding regions of the target touch screen 50B, are for illustration purposes, and it will be understood that the number and / or layout of cells 52 may vary in different embodiments.

[0069] Each cell 52 may be controllably and selectively connectable to ground via a conductor (electrical conductor) 53 (e.g., a wire, a printed circuit board PCB trace, and / or the like) and an appropriate switch (shown as switches 54A - 54L (collectively switch 54) in FIG. 4B). Each cell 52 of the transparent conductive film 45 may be electrically insulated from other cells 52. Each conductor 53 may be electrically insulated from other conductors 53. The cells 52 may be laid out in an appropriate grid arrangement or other appropriate arrangement (which may depend on the positions of the virtual buttons on the target touch screen 50B). The number, size, and / or arrangement of the cells 52 may determine the touch emulation resolution of the touch emulator 41 and the retrofit interface device 10B. In some embodiments, the conductor 53 is transparent (e.g., made of one or more transparent conductive materials) or made of thin (narrow) strands of a conductive material (e.g., metal nanowires) that appear transparent to the user. The conductor 53 may be electrically connected to the cell 52 with a transparent conductive adhesive or solder. The conductor 53 may be arranged to minimize the amount of wiring (wire) located between the target touch screen 50B and the user.

[0070] Switch 54 (e.g., a MOSFET transistor, a switching relay, and / or the like) is electrically connected between each cell 52 and electrical ground. The retrofit controller 20 of the retrofit interface device 10B may control the switch 54 by using an emulator control signal 43 (see FIG. 1B). Specifically, the retrofit controller 20 may be configured to switch a particular switch (electrical switch) 54 between an ON configuration in which the switch 54 conducts to connect the corresponding cell 52 to ground and an OFF configuration in which the switch 54 is non-conductive. When the switch 54 is conducting in the ON configuration, the target touch screen 50B of the target device 50 may interpret this (ON configuration) as a touch event at the position of the corresponding cell 52. When the touchless sensing system 12 detects a touchless input from the user (e.g., a gesture of a finger tapping), the retrofit controller 20 may switch one or more switches (electrical switches) 54 to the ON position by using the emulator control signal 43. The retrofit controller 20 may be configured to control the switch (electrical switch) 54 in real time (i.e., with little or no time delay between the touchless tap detected by the touchless sensing system 12 and the touch-emulated tap).

[0071] In some embodiments, the retrofit interface device 10B is configured to establish a spatial correspondence between the touchless sensing system 12 and the cells 52 of the touch emulator 41. Advantageously, the spatial correspondence may enable the user to activate the cells 52 of the touch emulator 41 by entering a touchless input (e.g., a finger tapping gesture) directly adjacent to the cells 52 (e.g., a touchless tap may be detected by the touchless sensing system 12, and the retrofit controller 20 may use the emulator control signal 43 to turn on the switch 54 corresponding to the cell 52 adjacent to the position of the user's touchless tap). Preferably, the switch (electrical switch) 54 has a low floating / parasitic capacitance to establish a strong connection to electrical ground when the switch is turned on. The retrofit controller 20 may be adjacent to or disposed behind the target touch screen 50B. In some embodiments, the retrofit controller 20 may be integrated with the touchless sensing system 12.

[0072] The retrofit interface device 10B may be configured not only to detect a wide variety of touchless inputs (via the touchless sensing system 12), but also to operate the target touch screen 50B in various ways. For example, since the touchless input gesture detected by the touchless sensing system 12 may be transient (e.g., in the case of a tap or double-tap gesture), the retrofit interface device 10B may be configured to temporarily electrically ground the cell 52 and simulate the corresponding gesture on the target touch screen 50B. As another example, since the touchless input detected by the touchless sensing system 12 may be held for a certain period of time, the retrofit interface device 10B may be configured to electrically ground the cell 52 for a corresponding period to simulate a longer "touch and hold" on the target touch screen 50B. As another example, since the touchless input detected by the touchless sensing system 12 may include hover, drag, swipe, pinch, etc., the retrofit interface device 10B may be configured to electrically ground one or more of the cells 52 simultaneously, over a period of time, or sequentially to emulate such movement across the conductive surface of the target touch screen 50B.

[0073] In some embodiments, the transparent conductive film 45 of the touch emulator 41 and the capacitance sensor 30 of the touchless sensing system 12 are provided on the same printed circuit board PCB, but this is not essential. In some such embodiments, the transparent conductive film 45 and the capacitance sensor 30 may be electrically insulated from each other. For example, the printed circuit board PCB (e.g., a transparent printed circuit board PCB) may include a transmitting electrode and a receiving electrode of the capacitance sensor 30 located in two of the plurality of layers of the printed circuit board PCB that are distal from the target touch screen 50B of the target device 50. The cells 52 of the transparent conductive film 45 can be located in one of the plurality of layers of the printed circuit board PCB so as to be close to (e.g., closest to) the target touch screen 50B for contact therewith.

[0074] In some embodiments, the retrofit interface device 10B and / or the touch emulator 41 may comprise a coating or other layer material to provide waterproof or water-resistant properties. In some embodiments, the retrofit interface device 10B and / or the touch emulator 41 can be constructed to withstand various ranges of temperature and humidity (i.e., to match the target touch screen 50B of the target device 50 to which it is retrofitted).

[0075] Advantageously, it is not essential for the retrofit interface device 10B to be directly connected to the target control system 51 of the target device 50 (such a connection is optional), and it can be easily integrated into an existing target touch screen 50B (i.e., a common type of touch-based interface in a public space). The retrofit interface device 10B can be configured such that a user can interact with the existing target touch screen 50B of the target device 50 in a touchless manner (interact, communicate).

[0076] In other respects, the retrofit interface device 10B may be similar to the retrofit interface device 10 described herein. Unless the context specifically indicates otherwise, references to the features of the retrofit interface device 10 are to be understood as being applicable to the retrofit interface device 10B as well.

[0077] FIG. 1C schematically shows an embodiment of an additional aspect of the invention relating to attaching a retrofit interface device 10C to a target machine device 150. The target machine device 150 does not include a controller unique to the target machine device 150, but includes a touch-based input 50A (which may be described as a self-actuating object 150A). In the touch-based input 50A (self-actuating object 150A), the user's interaction with the self-actuating object 150A (in a touch-based context) affects the operation of the corresponding target mechanism 150B of the target machine device 150. Non-limiting examples of the target machine device 150 include a door handle / knob (where the self-actuating object 150A is the handle / knob mechanism while the target mechanism 150B is the tongue / bolt), a mechanical lock mechanism (where the self-actuating object 150A is the lock handle while the target mechanism 150B is the lock shaft), a light switch (where the self-actuating object 150A is the light switch lever while the target mechanism 150B is the switch contact), and / or the like. The retrofit interface device 10C may be configured to include a unique actuator 37 for interacting with each self-actuating object 150A. In other respects, the retrofit interface device 10C may be similar to the retrofit interface device 10A (FIG. 1A) described herein, and unless the context indicates otherwise, the features of the retrofit interface device 10C are to be considered similar to the features of the retrofit interface device 10A described herein.

[0078] In some embodiments, the retrofit interface device 10C and the target machine device 150 may be manufactured or otherwise provided as a single device. In the raw case, the self-actuating object 150A is manufactured to operate like a normal touch-based device, while being equipped with the retrofit interface device 10C to enable the self-actuating object 150A to operate touchlessly. In some such embodiments, the retrofit interface device 10C may include a retrofit display 60 for interaction or instructions with the user. For example, the target machine device 150 may be configured to include a light switch that is pressed or flicked by a hover tap or swipe without generating a physical touch of the light switch. As another example, the target machine device may be configured to include a door handle that can be rotated by hovering an FOB over a suitable virtual button or the like.

[0079] Any of the retrofit interface devices described herein (e.g., retrofit interface devices 10, 10A, 10B, 10C) can facilitate interaction with a user's smartphone or handheld computing device (portable computing device) 47 (e.g., communication between the retrofit controller 20 and the handheld computing device 47 via the mobile device signal 49). By way of non-limiting example, the retrofit interface device 10 includes a QR code (registered trademark), NFC / RFID tag, WIFI / Bluetooth (registered trademark) connection, and / or the like, so that the user can not only wirelessly provide inputs (e.g., button selections, alphanumeric characters, swipe gestures, and / or the like) to the retrofit controller 20 via the mobile device signal 49, but also utilize the keyboard / keypad input of the handheld computing device 47 to provide inputs corresponding to the target device 50. Thereby, the user can interact with the retrofit interface device 10 using their handheld computing device 47 and then, by interacting with the target device 50, the user is enabled to communicate (e.g., provide inputs) with the target device 50 by merely physically contacting their handheld computing device 47. In some embodiments, the handheld computing device 47 may be used by the user in addition to or instead of the touchless sensing system 12 to provide an input signal (mobile device signal 49) to the retrofit controller 20.

[0080] The retrofit interface device 10 may be configured (e.g., by appropriate programming of the retrofit controller 20) to use a software application (e.g., a smartphone app) that facilitates quick connection to the retrofit interface device 10 and quick automatic disconnection from the retrofit interface device 10. The retrofit interface device 10 may be configured (e.g., by appropriate programming of the retrofit controller 20) with a software application that facilitates quick connection to a handheld computing device (handheld device) 47 and / or quick disconnection from the handheld computing device 47.

[0081] In some embodiments, the connection between the handheld computing device (apparatus) 47 and the retrofit interface device 10 is limited to a particular user's handheld computing device 47 or to one of a particular handheld computing device at a time, to prevent the target device 50 from receiving (receiving) user input from multiple handheld computing devices 47 or from receiving input from multiple handheld computing devices at once. Such embodiments may be suitable for secure applications (e.g., for entering PIN codes in POS terminals and / or the like) and / or personal use applications (e.g., parking meters and / or the like). One exemplary implementation of such embodiments uses a timer configured to disconnect each handheld computing device (user device) 47 from the retrofit interface device 10 after a threshold period (e.g., after an inactive threshold period), and / or uses an event detection algorithm that disconnects each handheld computing device (user device) 47 from the retrofit interface device 10 so that a new user can connect after the detection or completion of an event. In another additional or alternative example of such embodiments, the retrofit interface device 10 is configured to require the user device (handheld computing device 47) to maintain a QR code (registered trademark) or NFC tag within the field of view of the camera or NFC sensor of the handheld computing device 47 to communicate with the retrofit interface device 10. Another additional or alternative implementation of such embodiments includes providing the retrofit interface device 10 with suitable communication hardware and / or software (e.g., WIFI and / or Bluetooth (registered trademark) antennas) configured to estimate the proximity of the handheld computing device 47, such that the connection between the handheld computing device 47 and the retrofit interface device 10 can be disabled as soon as the proximity of the handheld computing device 47 reaches a threshold distance away from the retrofit interface device 10.

[0082] The retrofit interface device 10 can be provided with various ways to receive communication of input selection from the handheld computing device 47. Exemplary ways include, but are not limited to, encoded acoustic signals, LED light signals, viewing alphanumeric text or QR codes (registered trademarks) displayed on the smartphone screen and / or similar (e.g., paper) by using a camera system, NFC or RFID.

[0083] Any of the retrofit interface devices described herein (e.g., retrofit interface devices 10, 10A, 10B, 10C) may include one or more retrofit feedback mechanisms 55 for providing feedback to a user who interacts with the target device 50 via the retrofit interface device 10. Such a retrofit feedback mechanism 55 may be controlled by the retrofit controller 20 by using an appropriate feedback control signal 57. As discussed elsewhere in this specification, the retrofit display 60 of the retrofit interface device 10 may constitute a part of the retrofit feedback mechanism 55. The retrofit interface device 10 according to a particular embodiment may include one or more of the following retrofit feedback mechanisms 55.

[0084] The retrofit interface device 10 may include a retrofit feedback mechanism 55 that includes an ultrasonic and / or acoustic wave transmission (collectively referred to herein as sonic wave transmission (SWT)) feedback mechanism. The sonic wave transmission SWT feedback may be used to create a touchless tactile feedback by simulating a button click sensation (e.g., the sensation typically perceived when a finger strikes a key on the surface of a conventional keyboard or the left / right button of a touchpad mouse, or other appropriate sensation) that can be felt by human skin, thereby indicating to the user that an input has been made (e.g., a selection has been made, a button has been pressed, and / or the like).

[0085] A typical off-the-shelf printed circuit board (PCB) equipped with a capacitance sensor 30 does not support acoustic wave transmission (SWT). An acoustic transducer located behind such a PCB may not be sensed by fingers present on the opposite side of the PCB. A retrofit interface device 10 (and / or other retrofit interface devices 10A, 10B, 10C described herein) can be provided with a design configuration that allows a capacitance sensor and acoustic wave transmission (SWT) to coexist in a single form factor. Tactile feedback can be achieved by arranging a touchless sensing system 12 (e.g., one or more capacitance sensors 30) and an acoustic transducer 55A (e.g., on or otherwise on a printed circuit board PCB) such that the tactile feedback does not interfere with (or unduly interfere with) the sensing operation of the sensors of the touchless sensing system 12 while allowing acoustic energy from the acoustic wave transmission (SWT) transducer (acoustic transducer 55A) to reach the sensing area (sensing zone) of the touchless sensing system 12.

[0086] For example, the retrofit interface device 10 and / or the retrofit interface device 10A may have an arrangement of circular, elliptical, or rectangular acoustic transmission SWT transducers disposed around or partially around the printed circuit board PCB of the capacitance sensor - perhaps directed towards the center of the sensing zone on the user-facing side (the side facing the user) of the printed circuit board PCB of the capacitance sensor - a "perimeter" (perimeter) arrangement. This arrangement of the acoustic transmission SWT transducers around or partially around the printed circuit board PCB of the capacitance sensor provides a free path for the acoustic waves that strike a human finger to feel a button click sensation. The acoustic transmission SWT transducer may be configured to simulate the sensation of a button click on the user's finger and / or the like by directing acoustic energy towards the position of the user's finger. In some embodiments, the finger position detected by the touchless sensing system 12 is processed by a suitably configured controller (e.g., the retrofit controller 20) to enable the acoustic wave to be directed towards the detected position of the finger by controlling the acoustic transducer (via the feedback control signal 57).

[0087] Figures 7A and 7B schematically show an acoustic transmission type retrofit feedback mechanism 55 that can be used with any of the retrofit interface devices 10, 10A, 10B, 10C described herein according to a particular embodiment. The acoustic transmission type retrofit feedback mechanism 55 includes an acoustic transducer (acoustic transmission SWT transducer) 55A and a touchless sensing system (touchless sensing mechanism) 12. The touchless sensing system 12 includes a capacitance sensor(s) 30. In the illustrated embodiment of FIGS. 7A and 7B, the acoustic transmission SWT type retrofit feedback mechanism 55 has a layout of acoustic transducers 55A, which may be positioned on a surface (which may be formed to be concave or flat). In the illustrated embodiment of FIG. 7A, the first support surface 38A that supports the acoustic transducer 55A is generally flat, and the acoustic transducer 55A is supported on the first support surface 38A and is positioned behind (i.e., farther from the user) the touchless sensing system 12 (e.g., behind one or more capacitance sensors(s) 30). In the embodiment of FIG. 7A, the capacitance sensor(s) 30 or a part of their electrodes (or generally the touchless sensing system 12) are perforated at an opening 31 through which the acoustic transmission SWT transducer (acoustic transducer 55A) can direct sound waves into the sensing zone of the capacitance sensor 30 (or generally the touchless sensing system 12). In some embodiments, a small number of openings such as a single large opening 31 may be provided. In some embodiments, a matrix with a plurality of small openings 31 may be provided. In the illustrated embodiment of FIG. 7B, the second support surface 38B that supports the acoustic transducer 55A not only has a recess, but also the acoustic transducer 55A is positioned around the touchless sensing system 12 (e.g., around one or more capacitance sensors(s) 30), so that the acoustic transducer 55A is directed to generate sound waves at the user-facing surface (the side facing the user) of the touchless sensing system 12 (e.g., generally in the sensing area of the capacitance sensor(s) 30 and / or the touchless sensing system 12).In some embodiments, the configuration of FIG. 7B can be reversed. That is, since the capacitance sensors 30 (or a set of their electrodes) can be arranged in a ring shape, the acoustic transducer 55A can be arranged in the central opening of the ring of the capacitance sensors 30. In some embodiments, aspects of the feedback systems (retrofit feedback mechanisms 55) shown in FIGS. 7A and 7B can be combined. For example, in the embodiment of FIG. 7A, the first support surface 38A that supports the acoustic transducer can be made to have a concave shape. In some embodiments, the acoustic transducer 55A can be made (at least mainly) from a non-conductive material.

[0088] The retrofit interface device 10 may include a retrofit feedback mechanism 55 that includes a visual feedback device. In some embodiments, the visual feedback device may be implemented via the target display 22 of the target device 50 (and / or indirectly via the touchless control signal 16 and the target display signal 21) by using the touchless display signal 18. In some embodiments, the visual feedback device of the retrofit feedback mechanism 55 of the retrofit interface device 10 may be configured to include a retrofit display 60 of the retrofit feedback mechanism 55 itself that is controlled by a retrofit display signal (display signal) 61. In some embodiments, while the target device 50 does not have a target display 22, the retrofit feedback mechanism 55 of the retrofit interface device 10 includes the only display or visual indicator(s). In some embodiments, the retrofit feedback mechanism 55 provides visual feedback via a display screen such as an LCD screen, an e-ink screen, or any other suitable display screen. The retrofit display 60 may be incorporated into an external panel (for example, a panel disposed adjacent to a printed circuit board PCB that houses the touchless sensing system 12). The retrofit interface device 10 may provide an interactive touchscreen-like user experience by the touchless sensing system 12 by using the retrofit display 60 and / or the target display 22. For the sake of brevity, the remainder of this description regarding visual feedback refers to the retrofit display 60 that is part of the retrofit interface device 10 controlled by the retrofit display signal 61. Such visual feedback may additionally or alternatively be provided by the target display 22 that is part of the target device 50 controlled directly by the touchless display signal 18 or indirectly by the touchless control signal 16 and the target display signal 21 without loss of generality.

[0089] FIG. 8 is a schematic exploded view of a touchless sensing system 12 that can be used with a retrofit feedback mechanism 55 of any of the retrofit interface devices 10, 10A, 10B, 10C described herein according to a particular embodiment. The touchless sensing system 12 includes one or more capacitance sensors 30 arranged in a “ring” shape around a retrofit display 60, and optionally one or more optical sensors (40), and / or additional sensors 33 (not explicitly shown in FIG. 8). For clarity, the retrofit controller 20, although not shown in FIG. 8, may be housed together with the touchless sensing system 12 and / or the retrofit display 60. As shown in FIG. 8, since the capacitance sensors 30 of the touchless sensing system 12 may be arranged adjacent to the retrofit display 60, the printed circuit board PCB that supports the capacitance sensors 30 provides a sensing area around and in front of the retrofit display 60 (i.e., the user-facing surface (the side facing the user)) such that the receiving electrodes 36 of the capacitance sensors 30 are arranged outside the boundary of the retrofit display 60 (e.g., part of the perimeter, completely around, or around). That is, the printed circuit board PCB that supports the capacitance sensors 30 may be designed to form a “ring” around or a part of the perimeter of the retrofit display 60 such that the sensing area of the capacitance sensors 30 is located on the user-facing surface of the retrofit display 60. In some embodiments, the printed circuit board PCB that supports the capacitance sensors 30 may be slightly displaced towards the user (relative to the retrofit display 60) to mitigate interference that may be caused by the retrofit display 60 in the electric field of the capacitance sensors 30. The center 46 of the ring of the capacitance sensors 30 may generally correspond in size and shape to the active portion of the retrofit display 60, i.e., the portion of the retrofit display 60 that actually displays the content. FIG. 8A shows a schematic depiction of a capacitance sensor 30 comprising one or more receiving electrodes 36 and one or more transmitting electrodes 34 arranged in a ring around an aperture (46) according to a particular embodiment.In some embodiments, the capacitance sensor 30 may be disposed around (or partially around) an aperture (46) that may be used for purposes other than the aftermarket display 60. For example, the target display 22 or some other surface may be viewable through the aperture (46).

[0090] In some embodiments, the center 46 of the ring of the capacitance sensor 30 is an empty opening. In some embodiments, this center 46 of the ring of the capacitance sensor 30 may be configured with a suitable transparent material (e.g., glass, plexiglass (poly(methyl methacrylate)), and / or the like). In some embodiments, the transparent material of the center 46 of the ring may be configured with a transparent conductive material (e.g., indium - tin - oxide (ITO) or any other suitable transparent material). In some such embodiments, the transparent conductive material may form one of the electrodes (e.g., the transmitting electrode) of the capacitance sensor 30 such that only one of the electrodes (e.g., the receiving electrode) of the capacitance sensor 30 is disposed in the ring around the retrofit display 60. Having an electrode in the central region (46) can help increase the accuracy of the capacitance sensor 30 by providing a more uniform electric field of the capacitance sensor 30 by providing a stronger electric field at the center of the printed circuit board PCB. Providing a strong electric field at the center of the printed circuit board PCB can help the touchless sensing system 12 better detect fingers rather than a larger fist / forearm located further away - thus increasing the accuracy of the touchless sensing system 12. In some such embodiments, both the transmitting and receiving electrodes of the capacitance sensor 30 may be made of a transparent conductive material, in which case the capacitance sensor 30 may be disposed directly on the user - facing surface of the retrofit display 60 - that is, since both the transmitting electrode 34 and the receiving electrode 36 are transparent, the ring - based layout of the capacitance sensor 30 around the opening (46) shown in FIG. 8 may be changed to a layout that covers the "opening" (46). In the illustrated embodiment of FIG. 8, the ring of the capacitance sensor 30 may be partially or completely covered (on the user - facing surface) by a suitable sheet 48 of a transparent material (e.g., glass, plexiglass (poly(methyl methacrylate)), and / or the like). The transparent sheet or panel (48) may be used to form a flat, non - protruding design that fits flush with the wall or other surface to which the retrofit interface device 10 can be attached by enclosing the capacitance sensor 30.Such a flat design may, by reducing tampering with the retrofit interface device 10, meet industry safety requirements in some cases, such as when the target device 50 is, for example, an elevator panel. Such an example of the retrofit interface device 10 applied to a target device equipped with an elevator panel is shown in FIG. 8B.

[0091] Advantageously, such a "ring" design of the capacitance sensor 30 can minimize (at least to an acceptable extent) the interference that can be caused by the retrofit display 60 in the electric field of the capacitance sensor 30. Also, the touchless sensing system 12 (shown in FIGS. 8 and 3A - 3C) advantageously includes a fusion of the capacitance sensor 30 and the optical sensor (40) located at a position away from the retrofit display 60, thereby providing precise lateral (x - y) detection by the optical sensor (40), precise z - coordinate detection and lateral detection trigger by the capacitance sensor 30, and useful user feedback provided by the combination of the touchless sensing system 12 and the retrofit display 60.

[0092] In some embodiments, the retrofit controller 20 is configured to generate a retrofit display signal 61 that causes the retrofit display 60 to display indicia (e.g., a cursor, pointer, and / or the like) on the retrofit display 60. The position of the displayed indicia on the retrofit display 60 may correspond to, or be based on, an estimated position of a part of the user's hand (i.e., the position estimated by the retrofit controller 20 based on the touchless input signal 14 received from the touchless sensing system 12). The position of the displayed indicia on the display screen (retrofit display 60) may additionally or alternatively correspond to, or be based on, the proximity of a part of the user's hand to the display screen (retrofit display 60) (e.g., the proximity estimated by the retrofit controller 20 based on the touchless input signal 14 received from the touchless sensing system 12). For example, in some embodiments, the retrofit controller 20 may cause the retrofit display 60 to display indicia indicating an estimated position of a part of the user's hand within a plane parallel to the plane of the retrofit display 60 (e.g., the x-y plane), within a plane tangential to the surface of the retrofit display 60, or within a plane tangential to some other suitable surface of the target device 50 and / or the retrofit interface device 10, and then cause the retrofit display 60 to display, for example, characteristics of the indicia (e.g., size, luminance, color, and / or animation characteristics (s) of the indicia, and / or the like) indicating the proximity (e.g., in the z direction) of a part of the user's hand to the plane. In some embodiments, the indicia are always displayed on the retrofit display 60 to inform the user of the position of the user's finger.

[0093] In some embodiments, the retrofit controller 20 may generate a retrofit display signal 61 that causes a display screen (retrofit display 60) to change the appearance of the displayed indicia based on an estimated proximity (e.g., in the z-direction) of a portion of the user's hand and / or an estimated position of a portion of the user's hand (e.g., relative to the plane of the retrofit display 60, relative to the plane that touches the surface of the retrofit display 60, or relative to some other suitable surface of the target device 50 and / or the retrofit interface device 10). The change in the appearance of the displayed indicia may comprise at least one of a change in the color of the displayed indicia, the size of the displayed indicia, the brightness (intensity) of the displayed indicia, a change in the shape of the displayed indicia, a change in the color gradient of the displayed indicia, an animation function of the displayed indicia, adding other indicia to the displayed indicia, and / or the like. For example, the displayed indicator (indicia) may be a "ring cursor" and the size of the ring cursor may decrease (i.e., the diameter of the ring cursor may decrease) as the user's hand approaches a threshold distance away from the retrofit display 60.

[0094] In some embodiments, the retrofit display 60 is configured to display one or more virtual inputs (e.g., virtual buttons) (by the retrofit display signal 61 from the retrofit controller 20). In these embodiments, the retrofit display signal 61 can cause the retrofit display 60 to change the appearance of the virtual inputs based on an estimated position of a portion of the user's hand and / or an estimated proximity (estimated proximity, estimated proximateness) of a portion of the user's hand.

[0095] In some embodiments, the retrofit controller 20 is configured to detect a circular motion made by a part of the user's hand based on the touchless input signal 14 from the touchless sensing system 12. In some such embodiments, the retrofit controller 20 may be configured to generate a retrofit display signal 61 that causes the retrofit display 60 to provide some visual feedback (e.g., change the appearance of one or more displayed virtual inputs) based on the detected circular motion.

[0096] In some embodiments, the retrofit controller 20 is configured to detect that a part of the user's hand lacks (is lacking) movement (within a threshold) over a threshold period based on the touchless input signal 14. In these embodiments, the retrofit controller 20 may be configured to generate a retrofit display signal 61 that causes the retrofit display 60 to provide some visual feedback (e.g., change the appearance of one or more displayed virtual inputs) based on the detected lack of movement.

[0097] In some embodiments, the retrofit controller 20 is further configured to generate a retrofit display signal 61 that varies the size of the displayed indicia in a manner correlated to the proximity of a portion of the user's hand to the retrofit display 60, by the retrofit display 60. In some such embodiments, the retrofit controller 20 may be configured to modify the touchless control signal 16 to affect the operation (e.g., make a selection) of the target device 50 when the size of the displayed indicia is the same (within an appropriate threshold) as the size of one of the plurality of displayed virtual inputs, and the position of a portion of the user's hand lacks movement (within an appropriate threshold) during a threshold period. That is, the retrofit controller 20 is enabled to confirm the selection of the displayed virtual input when the user brings a portion of the hand in proximity (within the threshold) to the displayed virtual input and then does not move (during the threshold period), and to provide feedback to the user of the proximity element of this selection based on the size of the indicia being the same as the size of the virtual input when the user brings a portion of the hand sufficiently close.

[0098] The retrofit display 60 can optionally comprise one or more visible light sources, in addition to or as an alternative to a flat screen display. The retrofit controller 20 may be configured to generate a retrofit display signal 61 that illuminates or changes the color of illumination of at least one of the visible light sources by the retrofit display 60, depending on, for example, the estimated position of a portion of the user's hand in a plane (e.g., the x-y plane) corresponding to the plane of the retrofit display 60, the tangent plane of the surface of the retrofit display 60, or the tangent plane of some other suitable surface of the target device 50 and / or the retrofit interface device 10, and / or the estimated proximity of a portion of the user's hand to such a plane.

[0099] In some embodiments, the retrofit controller 20 is configured to generate a retrofit display signal 61 that causes the retrofit display 60 to provide feedback (or otherwise modify the output of the retrofit display 60) based at least in part on an estimated line-of-sight vector of the user. In such embodiments, the retrofit display signal 61 can cause the retrofit display 60 to display an image skewed (biased, distorted) in a direction towards or away from the user based on the estimated line-of-sight vector. Advantageously, causing the retrofit display 60 to display a "distorted" image can help the user avoid inaccurate touchless interactions with the retrofit interface device (retrofit interface system) 10 that may otherwise be caused by the user's field of view and viewing point with respect to the retrofit display 60 and / or the virtual inputs displayed thereon. By skewing (distorting) the position of the virtual input (or the detection zone of the virtual input) based on the estimated line-of-sight vector of the user, the probability that the user will accurately interact with the touchless sensing system 12 (e.g., select a particular virtual input) will increase. In some embodiments, the retrofit controller 20 can generate a retrofit display signal 61 that causes the retrofit display 60 to display a distorted indicator (e.g., a cursor) corresponding to (for the convenience of) the user's viewing point as determined by the estimation of the user's line of sight.

[0100] The retrofit interface device 10 may optionally include additional visual indicators (as part of the retrofit feedback mechanism 55) that are useful for notifying the user of selections, the presence of a finger / hand, etc. For example, the retrofit interface device may include a light (e.g., a light-emitting diode) that turns on to indicate that a finger / hand tap / wave has been successful. The light may be synchronized with the retrofit display 60 to provide visual feedback to the user together. For example, the retrofit display 60 may be configured to display various icons, images, and / or text for various keypad numbers or other possible selections, and the light may be configured to indicate the presence of a finger and / or a successful selection.

[0101] The retrofit interface device 10 may additionally or alternatively include a retrofit feedback mechanism 55 that supports auditory feedback. The auditory feedback mechanism may be configured to provide an audible cue (a sound cue) (e.g., with an appropriate feedback signal 57 from the appropriately programmed retrofit controller 20) indicating a successful touchless selection or other interaction between the user and the retrofit interface device 10 (e.g., the touchless sensing system 12). For example, when a touchless input corresponding to alphanumeric text is input by the user (e.g., when an appropriate finger press gesture is detected by the touchless sensing system 12), an audible cue may be generated to simply indicate a button press or repeat the input (e.g., "9", "m"). The auditory feedback mechanism may include a speaker. The speaker may be used, as a non-limiting example, to notify the user of system states and / or changes in system states of the retrofit interface device 10 and / or the target device 50, such as when the user's presence is detected, when the system is waiting for user input, when the user reaches the "main menu" or "sub-menu X", and / or the like.

[0102] As discussed elsewhere in this specification, the retrofit interface device 10 not only receives (receives, accepts) touchless input from a user (via the touchless sensing system 12), but also uses such touchless input to provide corresponding input to the target device 50. Here, a number of non-limiting exemplary embodiments are described as to how the retrofit interface device 10 may determine a particular touchless input. These examples may be configured to include additional or alternative techniques to those described elsewhere in this specification. In some exemplary embodiments, the capacitance sensor 30, the optical sensing system 40, and / or other sensors forming part of the touchless sensing system 12 may detect movement of a part of the user's body (e.g., a finger). In some embodiments, the retrofit feedback mechanism 55 may provide visual feedback via a display (which may include the retrofit display 60, which is part of the retrofit interface device 10, and / or the target display 22, which is part of the target device 50). Such visual feedback may include displaying a visual indicator corresponding to the position characteristics that are detectable and / or perceivable by the capacitance sensor 30, the optical sensing system (optical system) 40, and / or other sensors forming part of the touchless sensing system 12.

[0103] Figures 5A and 5B show a front view and a side view of a panel 100 of a retrofit (user) interface device 10 (or any one of the other retrofit (user) interface devices 10A, 10B, 10C described herein) that displays a virtual button 104 according to an exemplary embodiment. As schematically shown in FIGS. 5A and 5B, the retrofit interface device 10 may include a panel 100 labeled with an icon, graphic, or alphanumeric text 102 that can be used to represent, for example, a button on a keypad or other selectable virtual button (or input) 104. Such an icon, graphic, or alphanumeric text 102 can be considered a virtual button (or input) 104 that operates touchlessly and / or exists only on the panel 100 (as opposed to being a physical button). Such a panel 100 that displays a virtual button (or input) 104 may include a display (e.g., a retrofit display 60 that is part of the retrofit interface device 10 and / or a display that is part of a retrofit feedback mechanism 55 that includes a target display 22 of a target device 50). Such a panel 100 that displays a virtual button (or input) 104 may additionally or alternatively be configured to include a static panel or surface that displays a static virtual button (or input) 104. In the exemplary touchless input selection embodiments described below, it is assumed without loss of generality that there is a panel 100 labeled with an icon and / or alphanumeric text 102 such that various virtual buttons 104 are visible to the user. FIGS. 5A and 5B also show orthogonal axes x, y, z that are used to describe directions in the following description of exemplary touchless input selection, where x and y are orthogonal lateral directions and the z direction is generally perpendicular to the panel 100.

[0104] For the sake of brevity, the exemplary touchless input selection embodiments described herein refer to virtual buttons 104 shown on a retrofit display 60 of a retrofit interface device 10 that can be controlled by a retrofit controller 20 (via a retrofit display signal 61). The control by the retrofit controller 20 may additionally or alternatively be provided by a target display 22 that is part of a target device 50 whose functions associated with the retrofit display 60 can be controlled by the retrofit controller 20 (via a touchless display signal 18 and / or indirectly via touchless control signal 16 and target display signal 21), or without loss of generality, such virtual buttons are displayed on a static panel or surface. The exemplary touchless input selection embodiments described below provide techniques by which the retrofit interface device 10 can determine (or conclude) that a particular virtual button 104 has been touchlessly selected by a user. As described elsewhere herein, the retrofit interface device 10 can use the determination of the selection of such virtual buttons 104 to effect corresponding operations of the target device 50.

[0105] FIG. 6 is a schematic depiction of a method 200 for determining touchless input selection by using a retrofit interface device 10 (or any other retrofit interface device 10A, 10B, 10C described herein) according to a particular exemplary embodiment. For purposes of explaining the method 200 of FIG. 6, the touchless sensing system 12 is assumed to comprise a combination of an array of capacitance sensors 30 similar to that shown in the embodiments of FIGS. 3A - 3C and an optical sensing system 40, and the selection of an input is assumed to comprise an interaction with virtual buttons 104 as described elsewhere herein. The method 200 can be implemented by the touchless sensing system 12 by combining a retrofit controller 20 and optionally one or more retrofit feedback mechanisms 55.

[0106] Method 200 begins at block 202, where the touchless sensing system 12 waits for a part of the user's body (assumed to be a finger for the purpose of explaining method 200 without loss of generality) to enter its detection range. In some embodiments, block 202 comprises detecting that the user's finger has entered the sensing area 32 of the capacitance sensor 30, which sensing area 32 may be at a z-distance from the plane displaying one or more virtual buttons 104, and the z-coordinate may be perpendicular to the plane displaying the virtual buttons 104. When the touchless sensing system 12 detects the user's finger in its sensing area 32, method 200 proceeds to block 204, where the touchless sensing system 12 (e.g., capacitance sensor 30) tracks the position of the user's finger as it moves in the sensing area 32. In some embodiments, block 204 may comprise tracking only the z-coordinate of the finger. In some embodiments, block 204 may comprise tracking the lateral (x-y) coordinates of the user's finger in addition to the z-coordinate. At block 204A, the retrofit controller 20 optionally provides some kind of feedback by using the retrofit feedback mechanism 55 when the user's finger is detected in the sensing area 32. For example, the retrofit controller 20 may cause the retrofit display 60 (or target display 22) to not only track the lateral (x-y) position of the user's finger, but also display a cursor or the like having some indicator (e.g., size, color, etc.) indicating the z-coordinate of the user's finger.

[0107] In block 206A, method 200 queries whether the user's finger has been pulled out of the sensing area 32. If so, method 200 returns to block 202. If the user's finger is still within the sensing area 32, method 200 proceeds to optional block 206B. Optional block 206 includes a query as to whether the user's finger has crossed a detection plane. The detection plane may be implemented using a threshold z - coordinate. That is, the touchless sensing system 12 detects an intersection with the detection plane (the finger has crossed the detection plane) in block 206B when it confirms that the z - coordinate of the user's finger has changed from above a configurable threshold to below the configurable threshold. If no intersection with the detection plane is detected in block 206B, method 200 returns to block 204. However, if an intersection with the detection plane is detected in block 206B, method 200 proceeds to optional block 208. In block 208, when the user's finger crosses the detection plane, the lateral (x - y) position of the user's finger is determined. This determination in block 208 may be performed by an optical sensor (40) that can enable a more accurate determination of the lateral (x - y) coordinates (compared to the capacitance sensor 30), as described elsewhere in this specification.

[0108] Regardless of whether it arrives via option blocks 206B and 208 or directly from block 206A, method 200 comprises checking in block 210 whether certain selection criteria are met. As described elsewhere in this specification, various techniques may be used to determine whether the selection criteria are met in block 210. Such techniques may generally be based on the tracked position of a part of the user's body (e.g., the user's finger) as determined by the touchless sensing system 12 (e.g., the capacitance sensor 30 and / or the optical sensing system 40). In certain embodiments, block 210 comprises determining that a selection has been made when the user's finger crosses another detection plane (similar to the detection plane described above in connection with blocks 206B and 208). In other specific embodiments, the selection criteria for block 210 may comprise detection of a tap gesture, a double-tap gesture, a hover gesture, and / or the like, some examples of which are described below. If the selection criteria for block 210 are met, method 200 proceeds to block 212, which comprises determining which selection (e.g., which particular virtual button 104) was made. Depending on the nature of the selection criteria satisfied in block 210, the selection determination in block 212 may be made as part of the query in block 210.

[0109] In some embodiments, since each virtual button 104 has a corresponding range of lateral (x-y) coordinates, block 212 is configured to select a particular virtual button 104 corresponding to the tracked x-y position of the user's finger when the selection criteria of block 210 are met, or corresponding to the determined x-y position of the user's finger when crossing the detection plane at block 208. In some embodiments, block 212 may consider the approach angle of the user's finger. Such an approach angle may be determined, for example, by the capacitance sensor 30 when the capacitance sensor 30 tracks the user's finger between blocks 204 and 210. Such an approach angle may be determined between the x-y position of the first detection plane intersection (as determined at block 208) and the x-y position of the second detection plane intersection (as determined at block 210 when the selection criteria are met). If the approach angle (with respect to the z direction) is greater than a threshold angle, block 212 may include a determination that the user was attempting to select an adjacent virtual button 104 (i.e., the virtual button adjacent to the virtual button corresponding to the tracked x-y position of the user's finger when the selection criteria of block 210 are met, or the virtual button adjacent to the virtual button corresponding to the determined x-y position of the user's finger when crossing detection plane 208). In some embodiments, block 212 includes the calculation of a tap vector based on (x,y,z) detection pairs (x1,y1,z1) and (x2,y2,z2). By utilizing these two separate detections, a vector in space can be obtained, and then the virtual button the user was attempting to select can be predicted. As a non-limiting example, such (x,y,z) detection pairs can each include a laser optical sensor for each detection. Detection occurs when the detection plane of sensor A hits (contacts), while another detection occurs when the detection plane of sensor B hits. The z coordinate can be calculated by the angle of the optical sensor or by the capacitance sensor 30. Generally, any combination of information from the capacitance sensor 30 and / or optical sensor(s) (40) described herein can be used to perform one or both of the (x,y,z) detection pairs.In some embodiments, an appropriate time threshold may be used between (x,y,z) detection pairs to identify whether the user has intentionally made a gesture.

[0110] In block 212A, the retrofit controller 20 may optionally cause a feedback mechanism 55 to display some kind of feedback indicating that the virtual button 104 has been selected by the user. For example, the retrofit controller 20 may cause the color of an icon associated with the virtual button to change, display or change an animation, and / or do the like. Method 200 then proceeds to block 214, which comprises an inquiry as to whether the retrofit interface device (system) 10 is waiting for additional input from the user. For example, if the target device 50 is an elevator panel and the user presses the virtual button corresponding to the 8th floor, no further input is required. However, if the target device 50 is an airport kiosk and the user enters the first letter of their surname, the retrofit interface device (system) 10 may request further input. If the inquiry in block 214 is negative, method 200 ends. Otherwise, method 200 returns to either block 202 or block 210 via optional blocks 218 and 220. In optional block 218, the retrofit controller 20 can cause a feedback mechanism to indicate that the retrofit interface device is waiting for additional input. Optional block 220 comprises an inquiry as to whether the user's finger has withdrawn beyond the detection plane evaluated in blocks 206B and 208 (away from the virtual button). If the inquiry in block 220 is negative, method 200 returns to block 210. If the inquiry in block 220 is positive, method 200 returns to block 202.

[0111] Returning to the query of block 210, the selection criteria may not be met, in which case method 200 proceeds to block 216. Block 216 can include an inquiry as to whether a restart (restart, reboot) is desired. If there is no desire to restart, method 200 returns to block 210. If a restart is desired, appropriate restart feedback may be displayed (block 216A), and method 200 may return to block 204. The restart criteria evaluated at block 216 may be configured to include, for example, an appropriate time threshold, a determination that an inverse intersection of the detection planes of blocks 206B, 208 has been detected, and / or the like.

[0112] As discussed elsewhere herein, the touchless sensing system 12 (e.g., the capacitance sensor 30 and / or the optical sensing system 42) is enabled to detect the x, y, and z positions (see FIGS. 3A, 5A, 5B) of a part of the user's body (e.g., the user's finger). An exemplary method of touchless input selection criteria that may be used, for example, at block 210 of method 200 is that the user hovers a finger over a particular virtual button 104 (e.g., sufficiently close and within a threshold z-distance (FIGS. 3A, 5A, 5B)) and at the correct lateral (x-y) position relative to the particular virtual button 104, and the x, y, z positions of the finger do not change by more than a threshold (e.g., 0.5 cm or 1 cm) over a threshold period (e.g., 0.5 seconds or 1 second). In other embodiments, the selection criteria for the virtual button (or icon) 104 that may be used at block 210 is to determine that the user first moves a finger within a threshold distance (e.g., z-distance) from the virtual button 104 (determined by the lateral (x-y) position of the finger) for a threshold period, and then detects a rapid change in the z-position of the finger (e.g., moving the finger towards or away from the virtual button 104) within the threshold period with a threshold change in z-position (e.g., a change in the z-direction of 1 cm or more in less than 0.25 seconds) and / or a rate of change of z-position greater than an appropriate threshold (e.g., a rate of change in the z-direction greater than 4 cm / second).

[0113] The attachable display 60 may display a visual indicator (visual indicia) or animation (e.g., a loading bar / circle) by the attachable feedback mechanism 55 to indicate that a selection has been made (e.g., at block 212A) and that the finger can be moved to a new position (e.g., at block 218), or other types of feedback may be provided (including those described elsewhere in this document). In some embodiments, the attachable feedback mechanism 55 of the attachable interface device 10 used at block 212A is configured with, for example, a sound wave emitter that directs sound waves towards the user's finger to mean a selection (provide feedback corresponding to the selection).

[0114] The exemplary hover selection touchless input selection technique can be used in addition to or instead of any of the other touchless input techniques described herein (e.g., at block 210).

[0115] In block 210 of method 200 for determining the selection of virtual button 104, another exemplary method of touchless input selection criteria that can be used, for example, is that the user not only hovers over the virtual button (or icon) 104 for a threshold period (e.g., while being sufficiently close (within a threshold z - distance) and at a relatively correct lateral (x - y) position), but then also taps their finger further (in the z - direction) towards the virtual button 104. The tapping motion may include not only the user moving their finger further towards the virtual button 104 (e.g., within a close threshold distance in the z - direction), but then moving or releasing their finger away from the virtual button 104. The touchless sensing system 12 (e.g., the capacitance sensor 30 and / or the optical sensing system 40) may detect hovering over a particular virtual button that can be a pre - selection (qualifying) event for a subsequent tap event. Then, when it is detected that the x - y position does not change (by more than a threshold) and the z - direction decreases (by a threshold, for example, compared to the z - value during hover detection), a "tap" is identified.

[0116] This exemplary method may also include tapping and dragging a finger horizontally after an initial hover. As a non-limiting example, an initially identified tap is used to effect a drag (e.g., of an icon), and subsequent x-y movement of the finger may be used to effect the drag until a subsequent tap, an increase in the z coordinate, and / or the like, is used to release the drag.

[0117] Another exemplary method of touchless input selection criteria that may be used in block 210 of method 200 for determining selection of virtual button 104, for example, includes the user double-tapping a virtual button or icon. This selection method potentially enables faster interaction with the post-attached interface device 10 (when compared to the hover selection described above). The double-tap method of selection includes the user tapping a finger towards and away from virtual button 104 (the lateral position of the virtual button) twice within a threshold period to enter a selection. The touchless sensing system 12 (e.g., capacitance sensor 30 and / or optical sensing system 40) not only detects such double-tapping in a similar manner as the detection of the first tap (described above), but may also then detect an increase in the z position following the second tap within the threshold period. The double-tap advantageously enhances the robustness of the click and reduces false detection or accidental clicks of the virtual button as compared to the hover-type selection criteria discussed above, as some users may naturally hover over a particular virtual button when trying to decide which button to select. In some embodiments, the selection criteria for block 210 may include a combination of the double-tap scheme and the hover selection scheme described above. For example, the user may use the hover selection method to pre-select an icon first and then use the double-tap method (or alternatively a single-tap method) to confirm the selection.

[0118] Another exemplary method of a touchless input selection criterion that can be used in block 210 of method 200 for determining the selection of virtual button 104, for example, includes the user pointing a finger towards a virtual button or icon and making a gesture of creating (i.e., drawing) a circle around the virtual button or icon to be selected. In some such embodiments, the touchless sensing system 12 can track the lateral x-y position of the user's finger when the finger enters the proximity of a threshold z-distance to the virtual button. During the tracking, the touchless sensing system 12 may check whether the lateral position of the finger changes by a threshold amount and then whether it returns within the proximity of the same position within a configurable time threshold (e.g., 1 second). If "true", the touchless sensing system 12 analyzes the shape formed while tracking the lateral position. One example and non-limiting method for determining whether the movement is a circle or some other "closed" shape is to check whether the movement sequentially crosses four Cartesian quadrants during the movement (e.g., by detecting four "corner" points that may correspond to the extreme ends of specific coordinates and verifying that these corner points are in four separate quadrants), and to confirm that there is a minimum threshold radius or threshold separation between opposing corner points. It will be understood that other techniques may be used to confirm whether the user's finger is crossing a closed shape. The touchless sensing system 12 may additionally check whether the x-y coordinates of the virtual button are completely or partially enclosed within the detected circle before confirming the selection of a particular virtual button. In some embodiments, the retrofit display 60 is configured to provide feedback indicating the circle drawn around the selected virtual button or icon. Such feedback may be provided, for example, before the YES conclusion that the selection criteria are met in block 210.

[0119] Another example of a touchless input selection method involves the user positioning their finger over a "ring" (circle) indicator (or other suitable "selector" indicator) that can be displayed on the retrofit display 60, and then "dragging" the selector indicator over the virtual button 104 they wish to select. In some such embodiments, the touchless sensing system 12 may first detect a hover gesture proximate to the ring indicator and interpret this action as moving the ring in concert with the finger. Next, the touchless sensing system 12 may track the lateral x-y position of the finger as it moves over the virtual button. Next, the touchless sensing system 12 may detect a hover gesture, a leaving gesture (where the z-distance increases by a fixed threshold within a threshold period), or other suitable gesture, to select the virtual button corresponding to the x-y position of the ring. Next, the block 210 selection criterion may determine the selection of the virtual button 104 by the user holding the selector indicator over the virtual button (or icon) 104 for a threshold period, or by pulling the finger away from the virtual button 104 (e.g., in the z-direction) after the selector indicator has been dragged over the virtual button (or icon) 104. Advantageously, since this ring selection technique requires an intentional user action similar to the double-tap method described above, this ring selection technique can reduce false positives and user errors similar to the double-tap method compared to hover-based selection.

[0120] Another exemplary method of touchless input selection criteria that can be used in block 210 of method 200 for determining the selection of virtual button 104 involves the user waving (waving a hand) or swiping a hand across the surface of touchless sensing system 12. In some such embodiments, the touchless sensing system 12 first ensures that a part of the user's body is sufficiently close (within the z - distance threshold), and then tracks the lateral (x - y) position of a part of the user's body (e.g., finger, hand, forearm) over a threshold period (e.g., 1 second). If the change in the lateral position of a part of the user's body is greater than a settable threshold within the threshold time (threshold period), or if the rate of lateral change during the threshold lateral movement is higher than a settable threshold, a swipe is detected. In some implementations, the directionality of the swipe can be ascertained by the touchless sensing system 12 by detecting that the movement of the swipe is closest to one of the left (negative x) / right (positive x) / up (positive y) / down (negative y) directions. Assume that the hand is always present within the detection range of the sensor. A threshold change - distance is required for a valid swipe to be identified. In some embodiments, the detection of a swipe, in addition to or as an alternative to being used as a technique for selecting a virtual button in block 210 of method 200, may be used as a navigation gesture (e.g., to change between a set of virtual buttons).

[0121] For example, another exemplary method of touchless input selection criteria that can be used in block 210 of method 200 for determining the selection of virtual button 104 is that the user makes a circular motion with their hand to cycle or scroll through selectable items, operates a slider widget, selects a numerical value, or adjusts a setting (e.g., the temperature of a thermostat). In some such embodiments, the touchless sensing system 12 first confirms that a part of the user's body is close enough (within the z-distance threshold), and then tracks the lateral x-y position of the finger / hand to check if a consistent circular motion is being performed. This circular motion can be grasped in a manner similar to the circular selection described above. In some embodiments, by tracking the radius and speed of the motion, the speed of the cycle through selection, etc. can be confirmed. In some embodiments, tracking these rotations can include: an assumption of circular motion; detection and calculation of the instantaneous velocity vector of the motion; and then calculation of the cross product vectors based on the current and previous velocity vectors - the direction of these cross products provides direction information while the magnitude of these cross products provides speed information. In some embodiments, the detection of circular motion, in addition to or as an alternative to being used as a technique for virtual button selection in block 210 of method 200, may be used as a navigation gesture (e.g., to change between a set of virtual buttons).

[0122] Selection of a virtual button or icon may include selection of an interactive virtual icon such as a slider button and / or the like. A virtual slider button may be configured to include a slider icon that can be "dragged" across a range of motion. In one particular implementation of implementing a virtual slider button, interaction with the virtual slider button can include: The touchless sensing system 12 detects a finger that meets the selection criteria over the slider icon (such as hover, hover tap, double tap, etc.), and as long as the z-distance does not increase (by a threshold amount or more), the touchless sensing system 12 tracks the lateral x-y position to affect the slider to drag left / right or up / down on the screen. When it is detected that the z-distance of the finger has increased beyond the threshold, the interaction ends.

[0123] In some embodiments, the retrofit display 60 is configured to display a keyboard (e.g., the virtual buttons 104 correspond to the keys of the keyboard), and the touchless sensing system 12 is configured to detect that the user swipes (but at an interval therefrom) their finger across the surface of the retrofit display 60 between various alphanumeric icons to type a word. The touchless sensing system 12 may detect such a swipe as a series of swipes and may track the end point of the swipe as the key of the keyboard that was pressed. After a non-active threshold period or after detection of some other appropriate gesture (such as moving away in the z-direction and / or the like), the detected end points can form an alphanumeric sequence. The retrofit controller 20 may support a prediction algorithm that can be employed to predict which word is desired to be input based on the proximity and end points of the user's finger movement.

[0124] In some embodiments, the touchless sensing system 12 is configured to detect finger movements made by a user, and the retrofit controller 20 is configured to interpret those movements. For example, the user can use finger movements to draw an entire word or code such as "2" or "d", or "open" or "1234". When the user's finger enters within the threshold z - distance, the touchless sensing system 12 tracks the lateral (x - y) position of the user's finger, and when a gesture of moving away (or other appropriate gesture) is detected, either character recognition (CR) or a similar algorithm is employed to identify which alphanumeric input was drawn. The detection of multiple numbers / characters together can form a word or passcode. In some embodiments, the position of the user's finger and the finger movements (i.e., what the user wrote) are displayed on the display screen (retrofit display 60).

[0125] In some embodiments, the retrofit controller 20 is configured to cause the retrofit display 60 to display a warning message when the estimated proximity of a part of the user's hand (e.g., a finger) is too close to the panel 100. This can prevent a user who is not accustomed to the touchless sensing system 12 from touching the surface of the panel 100 (to prevent the spread of infection from contact - type virus infections). This can also reduce the inaccuracy of the sensor's detection ability that can potentially result when, for example, the user's hand or forearm enters the detection range while the retrofit controller 20 is adjusted for the detection of relatively thin fingers, by preventing the finger from getting too close to the panel 100.

[0126] The sensor of the touchless sensing system 12 (e.g., the capacitance sensor 30) may be configured to determine 3D (x, y, z) coordinates corresponding to the position of a part of the body (e.g., a finger). By using such 3D coordinates, the retrofit controller 20 may be configured to detect and interpret different custom gestures as inputs to the retrofit interface device 10. Examples of custom gestures include, but are not limited to, any shape such as a figure-eight, a triangle, a letter, a circular loop, etc.

[0127] FIG. 6A is a schematic diagram of a method 300 for determining a touchless input selection by using any one of the retrofit interface devices 10, 10A, 10B, 10C described herein according to another exemplary embodiment. The method 300 is a specific embodiment, and the touchless sensing system 12 is configured to include the capacitance sensor 30 and the optical sensors 42A, 42B described above in connection with FIGS. 3A-3C. The flowchart diagram of the method 300 is logically decomposed into two parts in total: a capacitance part 310 executed by the capacitance sensor 30 and an optical part 320 executed by the optical sensor (40). The method 300 starts at block 302, which is similar to block 202 described above, and includes waiting for a part of the user's body (e.g., a finger or a hand) to move into the sensing area 32. For the sake of brevity and without limiting the generality of the description, in the remaining part of the method 300, the user's finger is described in relation to being the relevant part of the body.

[0128] When the user's finger enters the detection area (sensing area 32), in the capacitance portion 310, the central capacitance sensor 30 detects and tracks the z - coordinate of the finger (block 312), and it is also possible to optionally display to the user (block 312A) when the finger approaches or moves away from the plane of the virtual button 104 or the attached display 60 or the target display 22. When the user's finger moves outside the detection area (sensing area 32) (YES branch of block 314), the method 300 returns to block 302. When the user's finger moves through the detection plane (crosses from above the configurable z - coordinate to below the configurable z - coordinate), the method proceeds to block 330 via the YES branch of block 316. In this sense, block 316 may be similar to the determination of block 210 as to whether the selection criterion of method 300 is satisfied in a state where the selection criterion is the intersection (crossing) of the detection plane. Thus, the detection plane of block 316 may be called the activation plane. In other embodiments, block 316 may additionally or alternatively utilize any of the selection criteria of the other blocks 210 described herein.

[0129] On one hand, in the optical part 320, the optical sensors 42A and 42B (see FIG. 3A) that are present beside the plane of the virtual button 104 or the retrofitted display 60 or the target display 22 emit two detection planes of laser light at an angle with respect to the plane of the virtual button 104 or the retrofitted display 60 or the target display 22. When the user's finger crosses one or both of the detection planes of the optical sensors 42A and 42B, and / or is determined by comparing the z - coordinate measured by the capacitance sensor 30 with an appropriate threshold value, the optical sensors 42A and 42B are triggered when the user's finger crosses the detection plane (block 322), and the lateral (x - y) position of the user's finger is determined as described in other positions of this document (blocks 324 and 326). The detection plane of block 322 may be the same as the activation plane of block 316, or may be different from the activation plane of block 316. For example, the activation plane of block 316 may be farther from the user than the intersection of the detection planes of the optical sensors 42A and 42B (see intersection 56 shown in FIG. 3A). In this configuration, optical detection (in blocks 322, 324, and 326) occurs before method 300 reaches block 316. When the user moves the finger away from the detection plane of block 322 (YES branch of block 328), method 300 returns to block 322.

[0130] If the query in block 316 is affirmative (i.e., the user's finger crosses the activation plane), method 300 proceeds to block 330. Block 330 may determine an input (e.g., virtual button) selection based on the x - y position grasped in block 326. In this sense, block 330 may be similar to block 212 in FIG. 6. Feedback regarding the selection in block 330 may be provided by the retrofitted feedback mechanism 55 in block 330A. When the user removes the hand from the sensing area 32 (YES branch of block 332), method 300 returns to block 302. Otherwise, the z - coordinate is displayed (block 334), and the method returns to block 332.

[0131] The specific embodiments described and claimed herein are used to provide a retrofit (touchless) interface device 10 that, by retrofitting a target device 50 having touch-based input 50A, provides touchless user input to the target device 50. This is not essential. Those skilled in the art will understand that some embodiments may be suitably modified to provide custom and / or modular touchless interfaces for directly facilitating human interaction with machines or other devices. That is, the retrofit interface device 10 and the target device 50 may be integrated into newly manufactured devices. In some such newly manufactured devices, some aspects of the retrofit interface device 10 and the target device 50 described herein as separate may be integrated. For example, the retrofit controller 20 of the retrofit interface device 10 and the target control system 51 of the target device 50 may be integrated into a single functional controller. As another example, the retrofit display 60 of the retrofit interface device 10 and the target display 22 of the target device 50 may be integrated into a single display. Some of such newly manufactured devices may include a touchless sensing system 12 similar to that described herein.

[0132] <Glossary of Terms> Unless the context clearly requires otherwise, throughout this specification and the claims - “comprise,” “comprising,” and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, in the sense of “including, but not limited to.”

[0133] - “connected,” “coupled,” or variations thereof, mean an direct or indirect connection or coupling between two or more elements, and the coupling or connection between the elements can be physical, logical, or a combination thereof.

[0134] - As used to describe this specification, the terms "this specification", "above", "below", and similar imported terms refer to the specification as a whole and not to a particular part of the specification.

[0135] - "Or" with respect to a list of two or more items covers all interpretations of this word, namely any of the items in the list, all of the items in the list, and any combination of the items in the list.

[0136] - The singular forms "a", "an", and "the" also have the appropriate plural meanings. As used in this specification and any appended claims (if any), directional terms such as "vertical", "lateral", "up", "down", "front", "rear", "in", "out", "orthogonal", "transverse", "left", "right", "forward", "backward", "top", "bottom", "directly below", "directly above", "beneath", etc. are determined by the particular orientation of the device being described and illustrated. The subject matter described in this specification can assume various alternative orientations. Accordingly, these directional terms are not only not strictly defined but should not be narrowly construed.

[0137] Some components of various embodiments of the present invention (including, by way of non-limiting example, the post-attached controller 20 of the post-attached interface devices 10, 10A, 10B, 10C, the target control system 51 of the target device 50, or any other controller described herein) may be implemented by using specially designed hardware, configurable hardware, a programmable data processor configured by providing software (which may optionally include “firmware”) executable on a data processor, a special-purpose computer or data processor specially programmed, configured, or constructed to perform one or more steps in the methods described herein in detail, and / or any combination of two or more of these. Examples of specially designed hardware include logic circuits, application-specific integrated circuits (“ASICs”), large-scale integrated circuits (“LSIs”), very-large-scale integrated circuits (“VLSIs”), and the like. Examples of configurable hardware include one or more programmable logic devices such as programmable array logic (“PAL”), programmable logic array (“PLA”), field-programmable gate array (“FPGA”). Examples of programmable data processors include microprocessors, digital signal processors (“DSPs”), embedded processors, graphics processors, math coprocessors, general-purpose computers, server computers, cloud computers, mainframe computers, computer workstations, and the like. For example, one or more data processors within the control circuit of a device may implement the methods described herein by executing software instructions in a program memory accessible to the processor.

[0138] The processing may be centralized or distributed. When the processing is distributed, the information with software and / or data may be held centrally or distributed. Such information may be exchanged between different functional units by a communication network such as a local area network (LAN), a wide area network (WAN), or the Internet, a wired or wireless data link, an electromagnetic signal, or other data communication channels.

[0139] For example, although the processing or blocks are presented in a given order, alternative examples may execute a routine having steps or employ a system having blocks in a different order, and some of the processing or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternative or sub-combinations. Each of these processing or blocks may be implemented in various different ways. Also, although the processing or blocks may be shown as being executed serially, these processing or blocks may instead be executed in parallel or at different times.

[0140] Furthermore, although elements may be shown as being executed sequentially, they may instead be executed simultaneously or in a different order. Accordingly, the following claims are intended to be construed as including all such variations that fall within their intended scope.

[0141] Software and other modules may reside on a server, workstation, personal computer, tablet computer, image data encoder, image data decoder, PDA, color grading tool, video projector, audio visual receiver, display (such as a television), digital cinema projector, media player, and other devices suitable for the purposes described in this document. Those skilled in the relevant art will understand that aspects of the present system can be implemented in other communication, data processing, or computer system configurations. These aspects include Internet appliances, handheld devices (including personal digital assistants (PDAs)), wearable computers, all kinds of cellular or mobile phones, multiprocessor systems, microprocessor-based or programmable home appliances (such as video projectors, AV receivers, displays such as televisions, etc.), set-top boxes, color grading tools, network PCs, minicomputers, mainframe computers, and so on.

[0142] In some embodiments, the present invention may be implemented, in part, in software. To be more specific, "software" comprises any instructions that are executed on a processor and can include (but is not limited to) firmware, resident software, microcode, and the like. As is known to those skilled in the art, both the processing hardware and the software may be wholly or partially centralized or distributed (or a combination thereof). For example, software and other modules may be accessible via local memory, via a network, via a browser or other application in a distributed computing context, or via other means suitable for the purposes described above.

[0143] When a component (such as a software module, a processor, an assembly, a device, a circuit, etc.) is mentioned above, unless otherwise indicated, a reference to that component (including a reference to a "means") is to be construed as including equivalents of that component (i.e., functionally equivalent) that have a structure that is not structurally equivalent to the disclosed structure that performs that function in the exemplary embodiments of the present invention.

[0144] Specific examples of systems, methods, and apparatuses are described herein for purposes of illustration. These are merely examples. The techniques provided herein can also be applied to systems other than the exemplary systems described above. Many changes, modifications, additions, omissions, and permutations are enabled within the practice of this invention. The present invention includes variations of the described embodiments and variations obtained by the following methods. That is, the following methods are: replacing features, elements, and / or acts with equivalent features, elements, and / or acts; mixing and matching features, elements, and / or acts from different embodiments; combining features, elements, and / or acts from the embodiments described herein with features, elements, and / or acts of other technologies; and omitting combinations of features, elements, and / or acts of the described embodiments, at least one of which.

[0145] In this specification, various features are described as being present in "some embodiments". Such features are not essential and thus may not be present in all embodiments. Embodiments of the present invention may comprise zero, any one, or two or more, of such features in any combination. This is limited only to the extent that those having such features are precluded from constructing a practical embodiment that combines such incompatible features with others having such features, in the sense that a person skilled in the art would be unable to do so. Thus, the recitation that "some embodiments" have only feature A and that "some embodiments" have only feature B should be construed as expressly indicating that the inventors contemplate embodiments that combine feature A and feature B as well (except where the recitation is otherwise qualified or where feature A and feature B are fundamentally incompatible).

[0146] Accordingly, the following appended claims and the claims introduced hereafter are intended to be construed to include all modifications, permutations, additions, omissions, and subcombinations that are reasonably inferable. The claims should not be limited by the preferred embodiments shown in the examples, but should be given the broadest interpretation consistent with the specification as a whole.

Claims

1. 1. A system for receiving input for controlling a device, the system comprising: a display for displaying video content based on the display signal; one or more capacitive sensors sensitive to electric field disturbances caused by a human body proximate to a user side of the display; a controller connected to receive one or more capacitive sensor signals as one or more sensor input signals from one or more of the capacitive sensors and to generate control signals based on the one or more capacitive sensor signals, the controller being connectable to the device to provide the control signals as inputs to a control system of the device such that the control system of the device operates the device based on the control signals; Equipped with at least one electrode of each of the one or more capacitive sensors is disposed on a user-facing side of the display, and the at least one electrode of each of the one or more capacitive sensors is made from a transparent conductive material such that the human user can view the display through the at least one electrode of each of the one or more capacitive sensors. system.

2. at least a portion of the one or more capacitive sensors are disposed around a periphery of the display and are sensitive to electric field disturbances caused by a part of a human body proximate a user-facing side of the display at a central portion of the periphery; The system of claim 1 .

3. The controller is configured to estimate a position of a portion of a human user's hand based on one or more of the sensor input signals; and The controller: generating a display signal based on the estimated position of a portion of the user's hand to cause an indicia to be displayed on the display, the position of the displayed indicia on the display being based on the estimated position of a portion of the hand; estimating a proximity of a portion of the user's hand to the display; configured to perform one or more of the following: The system of claim 1 .

4. the display signal causes the display to change an appearance of the displayed indicia based on the estimated proximity of a portion of the user's hand; the display signal causes the display to change the appearance of the displayed indicia based on the estimated proximity of a portion of the user's hand, the change in the appearance of the displayed indicia comprising at least one of: a change in color of the displayed indicia, a change in size of the displayed indicia, a change in brightness of the displayed indicia, a change in shape of the displayed indicia, a change in color gradient of the displayed indicia, and the addition of other indicia to the displayed indicia; At least one of the following holds: The system of claim 3.

5. The controller, based on one or more of the sensor input signals, a swipe motion performed with a portion of the user's hand; a circular motion performed by a portion of the user's hand; configured to detect one or more of: The system of claim 3.

6. the display signal causes the display to vary a size of the displayed indicia in a manner correlated with the proximity of the portion of the hand to the display; the controller is configured to vary operation of the device by varying the control signal when a size of the displayed indicia is the same as one of one or more virtual inputs within a suitable threshold and when the position of a hand portion lacks movement equal to or greater than a suitable threshold for a threshold period of time. The system of claim 3.

7. the system further comprises one or more optical sensors disposed around the periphery of the display, the one or more optical sensors being sensitive to changes in electromagnetic radiation reflected from a human body on a user-facing side of the display; The one or more sensor input signals are one or more of the capacitive sensor signals from one or more of the capacitive sensors; one or more optical sensor signals from one or more of said optical sensors; Equipped with the controller is configured to generate the control signal based on both the capacitive sensor signal or signals and the optical sensor signal or signals. The system of claim 1 .

8. the controller is configured to estimate a gaze vector of the human user based on one or more of the sensor input signals. The system of claim 1 .

9. the controller is configured to estimate a position of a portion of the user's hand based on one or more of the sensor input signals; the controller is configured to generate the control signal based at least in part on a combination of the estimated gaze vector and the estimated position of a portion of the user's hand. The system of claim 8.

10. the controller is configured to generate a display signal that causes the display to distort an image displayed by the display based on the estimated line of sight vector. The system of claim 8.

11. The system further includes a layer of non-conductive material disposed over a user-facing side of one or more of the capacitive sensors. The system of claim 1 .

12. 1. A system for receiving input for controlling a device having a display, the system comprising: one or more sensors for responding to touchless input by a human user and generating one or more corresponding sensor input signals; a controller coupled to receive one or more sensor input signals from one or more of the sensors, the controller configured to generate corresponding control signals based on the one or more sensor input signals; Equipped with the controller being connectable to the device and providing the control signal as an input to a control system of the device such that the control system of the device operates the device based on the control signal; the one or more sensors comprising one or more capacitive sensors sensitive to electric field disturbances caused by body parts proximate to the one or more capacitive sensors; at least one electrode of each of the one or more capacitive sensors is disposed on a user-facing side of the display, and at least one electrode of each of the one or more capacitive sensors is made from a transparent conductive material such that a human user can view the display through the at least one electrode of each of the one or more capacitive sensors. system.

13. the system comprises one or more optical sensors disposed around the periphery of the display, the optical sensors being sensitive to changes in electromagnetic radiation reflected from the body of the human user on a side of the display facing the user; the one or more sensor input signals comprise one or more capacitive sensor signals from one or more of the capacitive sensors and one or more optical sensor signals from one or more of the optical sensors; the controller is configured to generate the control signal based on both the capacitive sensor signal or signals and the optical sensor signal or signals. The system of claim 12.

14. the controller is configured to estimate a gaze vector of the human user based on one or more of the sensor input signals. The system of claim 12.

15. the controller is configured to estimate a position of a portion of the user's hand based on one or more of the sensor input signals; the controller is configured to generate the control signal based at least in part on a combination of the estimated gaze vector and the estimated position of a portion of the user's hand; the controller is configured to generate a display signal that causes the display to distort an image displayed by the display based on the estimated line of sight vector. The system of claim 14.

16. the controller is configured to estimate a position of a portion of a hand of the human user based on one or more of the sensor input signals; The controller: generating a display signal based on the estimated position of a portion of the user's hand to cause the display to display an indicia, the position of the displayed indicia on the display being based on the estimated position of a portion of the hand; estimating a proximity of a portion of the user's hand to the display; configured to perform one or more of the following: The system of claim 12.

17. the display signal causes the display to change an appearance of the displayed indicia based on the estimated proximity of a portion of the user's hand; the display signal causes the display to change the appearance of the displayed indicia based on the estimated proximity of a portion of the user's hand, the change in the appearance of the displayed indicia comprising at least one of: a change in color of the displayed indicia, a change in size of the displayed indicia, a change in brightness of the displayed indicia, a change in shape of the displayed indicia, a change in color gradient of the displayed indicia, and the addition of other indicia to the displayed indicia; At least one of the following holds:

17. The system of claim 16.

18. The controller, based on one or more of the sensor input signals, a swipe motion performed with a portion of the user's hand; a circular motion performed by a portion of the user's hand; configured to detect one or more of:

17. The system of claim 16.

19. 1. A method of receiving input to control a device, the method comprising: providing a display for displaying video content based on the display signal; providing one or more capacitive sensors sensitive to electric field disturbances caused by a part of a human body proximate a user side of the display; disposing at least one electrode of each of one or more of said capacitive sensors on a user-facing side of said display, wherein at least one electrode of each of one or more of said capacitive sensors is made from a transparent conductive material such that a human user can view the display through the at least one electrode of each of one or more of said capacitive sensors; receiving, at a controller, one or more capacitive sensor signals from one or more of said capacitive sensors as one or more sensor input signals; generating, by the controller, a control signal based on one or more of the capacitive sensor signals and communicating the control signal as an input to a control system of the device such that the control system of the device operates the device based on the control signal; The method comprises:

20. The method further comprises estimating a position of a portion of a human user's hand based on the one or more sensor input signals; The method further comprises: generating a display signal based on the estimated position of a portion of the user's hand to cause an indicia to be displayed on the display, the position of the displayed indicia on the display being based on the estimated position of a portion of the hand; estimating a proximity of a portion of the user's hand to the display; performing one or more of the following steps:

20. The method of claim 19.

Images