Systems, devices, and methods for contactless typing

Abstract

Systems, devices, methods, and non-transitory computer-readable media for receiving data input through contactless gestures and motions are provided. For example, a data input device includes at least one processor for receiving information from a sensor. The processor can be configured to receive sensor data from the sensor of a user's hand, the user's hand being separated from and not in contact with a displayed keyboard; and to track one or more fingers in the air at a distance from a displayed keyboard image using the received sensor data. The processor can also be configured to correlate the positions of the one or more fingers in the air with images of the multiple keys in the displayed keyboard, and select a key from the keyboard image based on the correlated positions of the one or more fingers in the air and detection of a predefined gesture performed by the user.

Description

[0001] Related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 61 / 889,348, filed October 10, 2013, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present invention relates to contactless data input, such as typing, and more particularly to devices and computer-readable media that use one or more sensors to detect contactless data input. Background Art

[0004] Allowing a user to interact with a device or an application running on the device is useful in many different settings. For example, electronic systems include devices (e.g., keyboards, mice, and joysticks) that enable a user to input and manipulate data and cause the system to perform various other actions. However, increasingly, touch-sensitive input devices, such as keyboards, mice, and joysticks, are being replaced or supplemented by devices that allow for contactless user interaction. For example, a system may include an image sensor that captures an image of the user, including, for example, the user's hands and / or fingers. The device may include a processor configured to receive such an image and initiate actions based on contactless gestures performed by the user's hands and fingers.

[0005] Contactless systems typically detect large movements of the entire hand, but these systems lack sufficient accuracy and precision to enable data input capabilities that compete with the use of physical computer peripherals. For example, these systems cannot detect subtle movements and gestures to adequately simulate typing on a keyboard. Therefore, there is a need to improve contactless typing and data input technology to provide a better and more precise user experience. Summary of the Invention

[0006] In one disclosed embodiment, a non-transitory computer-readable medium is disclosed having instructions stored thereon, the instructions executable by at least one processor to perform contactless data input operations, such as typing. The operations may include displaying a keyboard image including a plurality of keys; receiving sensor data of a user's hand from at least one sensor, the user's hand being spaced apart from the displayed keyboard image and not in contact with the displayed keyboard image; tracking one or more fingers of the user's hand in mid-air at a distance from the displayed keyboard image based on the received image, associating positions of the one or more fingers in mid-air with images of a plurality of keys; and selecting a key from the keyboard image based on the associated positions of the one or more fingers in mid-air and detection of a predefined gesture performed by the user.

[0007] In another disclosed embodiment, a data input device is disclosed. The data input device includes at least one processor configured to: display a keyboard image including a plurality of keys; receive sensor data of a user's hand from at least one sensor, the user's hand being spaced apart from the displayed keyboard and in non-contact with the displayed keyboard image; track one or more fingers of the user's hand in mid-air at a distance from the displayed keyboard image based on the received sensor data; associate mid-air locations of the one or more fingers with images of the plurality of keys; and select a key from the keyboard image based on the associated mid-air locations of the one or more fingers and detection of a predefined gesture performed by the user.

[0008] Additional aspects related to the embodiments will be set forth in part in the description which follows and will be understood from the description, or may be learned by practice of the disclosed embodiments.

[0009] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various embodiments disclosed, in which:

[0011] Figure 1A An exemplary system for implementing the disclosed embodiments is illustrated;

[0012] Figure 1B Another exemplary system for implementing the disclosed embodiments is illustrated;

[0013] Figure 2 Exemplary methods according to some disclosed embodiments are illustrated;

[0014] Figure 3 Illustrated are graphical representations of keyboard and mouse modes according to some disclosed embodiments.

[0015] Figure 4 Illustrate a graphical representation of the button area dimensions.

[0016] Figure 5A and 5B Illustrate a graphical representation of keyboard row selection. DETAILED DESCRIPTION

[0017] The disclosed embodiments relate to systems, devices, and methods for contactless typing. In some embodiments, contactless typing can include entering data into a computing device without physical contact between a user and the computing device or any peripheral devices connected to the computing device. That is, data can be entered into the computing device using information collected by one or more sensors regarding hand and / or finger movements detected in mid-air. Detection and analysis of individual hand and / or finger movements in mid-air can be used to improve the accuracy of contactless data entry, thereby providing an efficient mechanism for entering data.

[0018] To facilitate the collection of sensor information for contactless typing, the display device can display a keyboard image as a reference for the user's hand and finger movements in the air to simulate typing or mouse operations. The processor can receive the collected sensor information through a wired or wireless connection to the sensor and analyze the collected sensor information. Based on the analysis, the processor can distinguish individual fingers and hands and follow the air movement of each finger and hand to determine the position of the finger and hand at multiple points in time, thereby tracking the hand and fingers in three-dimensional space. The processor can allocate portions of the three-dimensional space to keys in the displayed keyboard and associate the positions of the hands and fingers with the allocated areas, thereby associating individual fingers with keys in the displayed keyboard.

[0019] When predefined conditions are met while a finger remains associated with the keyboard, the processor may select one or more keys for data input. For example, when a finger performs a typing motion (e.g., a generally downward movement of the fingertip), the processor may determine that the user intends to enter data associated with the associated key and select the key for data input.

[0020] As described in more detail below, the processor can modify the spatial regions assigned to keyboard keys and analyze detected hand and finger movements, postures, gestures, and directions to improve the efficiency and accuracy of contactless typing and thereby improve the processor's ability to identify data input and select intended keys.

[0021] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numerals used in the drawings represent the same or similar components.

[0022] Embodiments of the present invention may include a data input system having one or more components, such as a data input device, configured to receive data input via contactless gestures. Other inventive embodiments may include a non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations for receiving data input via contactless gestures.

[0023] Figure 1A 1 is a diagram illustrating an example of a data input system 100 that can implement the disclosed embodiments. The system 100 can detect contactless gestures, postures, and movements from one or more fingers 106 and / or one or both hands 104 of a user 102. Figure 1A As shown, the system 100 may include at least one sensor 110, a CPU 120, and a display 130. These components of the system 100 may be communicatively coupled via one or more wired or wireless communication links.

[0024] Sensor 110 may be configured to collect information about activity in a space proximate to sensor 110. In some embodiments, for example, sensor 110 may include one or more cameras, light sensors, infrared (IR) sensors, ultrasonic sensors, proximity sensors, CMOS image sensors, short-wave infrared (SWIR) image sensors, reflective sensors, a single photoreceptor or a 1-D line sensor capable of scanning an area, a 2-D sensor, a stereo sensor including, for example, multiple 2-D image sensors, a depth sensor, a microphone, a motion sensor such as an accelerometer, a position sensor such as a GPS receiver, or a gaze tracker. Sensor 110 may be associated with a lens to focus light from a specific area on sensor 110.

[0025] The sensor 110 can be close to the display 130 and obtain an image of the three-dimensional visual space. Figure 1A As described, user 102 is located in visual space and is therefore represented in sensor data, such as image data, obtained by sensor 110. For example, the output of sensor 110 may be 2D (two-dimensional) color or infrared (IR) video. Alternatively, sensor 110 may be a depth video system including an image sensor or two 2D stereo image sensors. Sensor 110 may include a combination of some or all of the sensor types mentioned above.

[0026] The image captured by the sensor 110 may be digitized by the sensor 110 and input into the processor 122 of the CPU 120, or may be input into the processor 122 in analog form and digitized by the processor 122. Exemplary proximity sensors may include, among others: capacitive sensors, capacitive displacement sensors, laser rangefinders, sensors using time-of-flight (TOP) technology, infrared sensors, sensors that detect magnetic distortion, or any other sensor capable of generating information indicating the presence of an object proximate to the proximity sensor. In some embodiments, the information generated by the proximity sensor may include the distance of the object from the proximity sensor. The proximity sensor may be a single sensor, or may be a group of sensors. Although Figure 1AAlthough a single sensor 110 is shown in FIG, the system 100 may include multiple types of sensors 110 and / or multiple sensors 110 of the same type. For example, the multiple sensors 110 may be disposed within a single device, such as a data input device that houses all components of the system 100, within a device external to the other components of the system 100, or within various other configurations having at least one external sensor 110 and at least one sensor 110 disposed within another component of the system 100 (e.g., the CPU 120 or the display 130).

[0027] In some embodiments, the CPU 120 may include at least one processor 122 and a memory 124. The CPU 120 may be incorporated into a data input device, including, among others, a mobile phone, smart glasses, a personal computer (PC), an entertainment device, a set-top box, a television, a mobile game console, a tablet computer, an e-reader, a portable game console, a portable computer such as a laptop or ultra-laptop, a home appliance such as a kitchen appliance, a communication device, an air conditioner, a docking station, a gaming console such as a mobile video game device, a digital camera, a watch, an entertainment device, a speaker, a smart home device, a media player or media system, a positioning device, a micro-projector or embedded projector, a medical device such as a medical display device, a vehicle, an in-vehicle / in-flight infotainment system, a navigation system, a wearable device, an augmented reality-enabled device, wearable goggles, a robot, an interactive digital signage, a digital kiosk, a vending machine, an automated teller machine (ATM), or any other device that can receive data from a user or output data to a user. Furthermore, the data input device including the CPU 120 may be handheld (e.g., held by a user's hand) or non-handheld.

[0028] The CPU 120 may be connected to the sensor 110 via one or more wired or wireless communication links and may receive data (such as images) or any data that can be collected by the sensor 110 from the sensor 110. In some embodiments, the CPU 120 may receive data from multiple sensors 110 via one or more wired or wireless communication links. The CPU 120 may also be connected to the display 130 and may send instructions to the display 130 to display one or more images, such as the keyboard image 140. Figure 1A Shown as separate components, in some embodiments, the sensor 110 , CPU 120 , and display 130 may be incorporated into a single data input device, or into two devices having various combinations of sensors 110 , CPU 120 , and display 130 .

[0029] The processor 322 used herein may include, among other things, any circuit or non-electronic biological circuit that can be configured to perform a logical operation on at least one input variable, including, for example, one or more integrated circuits, microchips, microcontrollers, microprocessors (which may be all or part of a central processing unit (CPU)), digital signal processors (DSPs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), or any other circuit known in the art suitable for executing instructions or performing logical operations. The processor 122 may include, for example, one or more general-purpose processors, DSPs (digital signal processors), GPUs (graphics processing units), or any other device configured to run one or more software programs or otherwise execute instructions. Alternatively, the processor 122 may be dedicated hardware, an application-specific integrated circuit (ASIC). As another alternative, the processor 122 may be a combination of dedicated hardware, an application-specific integrated circuit (ASIC), and any one or more of a general-purpose processor, a DSP (digital signal processor), a GPU (graphics processing unit). Although Figure 1A Although one processor is shown in FIG, system 100 may include multiple processors that may provide different processing capabilities (e.g., specialized image processing) and / or may provide parallel processing functions. A single processor may be used to implement multiple functions, or multiple related and / or unrelated functions may be divided among multiple processors.

[0030] In some embodiments, storage 124 may include, for example, non-volatile memory, ROM, EEPROM, EAROM, flash memory devices, magnetic disks, magneto-optical disks, CD-ROM, DVD-ROM, Blu-ray media, non-electronic bio-memory, etc., and may include instructions (such as software or firmware) or other data. Figure 1A The memory 124 connected to the processor 122 is described by way of example. In general, the processor 122 can receive instructions and data stored by the memory 124. Thus, in some embodiments, the processor 122 executes software or firmware to perform functions by operating on input data and generating output. However, the processor 122 can also be, for example, dedicated software or an application-specific integrated circuit (ASIC) that performs processing by operating on input data and generating output. The processor 122 can be any combination of dedicated software, one or more ASICs, one or more general-purpose processors, one or more DSPs, one or more GPUs, or one or more other processors capable of processing digital information. Figure 1AThe memory 124 is described as being part of the CPU 120. However, in alternative embodiments, the memory 124 may be external to the CPU 120. In some embodiments, information captured by the sensor 110 may be stored in the memory 124 to be processed later or may be processed immediately by the processor 122 upon capture without being stored in the memory 124.

[0031] Embodiments may also include a display 130. The display 130 may include, for example, one or more televisions, computer monitors, head-mounted displays, broadcast reference monitors, liquid crystal display (LCD) screens, light emitting diode (LED)-based displays, LED-backlit LCD displays, cathode ray tube (CRT) displays, electroluminescent (ELD) displays, electronic paper / ink displays, plasma display panels, organic light emitting diode (OLED) displays, thin film transistor (TFT) displays, high performance addressed (HPA) displays, surface conduction electron emitter displays, quantum dot displays, interferometric modulator displays, volume scanning displays, carbon nanotube displays, varifocal displays, emissive wavelength displays, laser displays, holographic displays, light field displays, projectors, and surfaces that can project an image (including directly onto the retina of an eye, such as an implanted artificial eye retina of the user 102), or any other electronic device for outputting visual information and / or causing the user 102 to perceive the presence of visual information.

[0032] As described above, the display 130 can receive and execute instructions from the CPU 120 to display one or more images, such as a keyboard image 140. The keyboard image 140 can include graphical illustrations, shapes, or icons of one or more keys that represent different data entry and computer functions. The keys can represent alphanumeric characters, including symbols and emoticons, that are entered into the processor 122 in response to selection of an appropriate key; or represent functions that are performed by the processor 122 in response to selection of an appropriate key. In some embodiments, the keyboard image 140 can include an arrangement in a grid pattern, or in a layout that suits the needs of the user 102. In some embodiments, the keyboard image 140 can include one or more of a QWERTY keyboard, a numeric keypad, function keys, media controls, and any other type of keys required based on the data entry and functional needs of the user 102.

[0033] Figure 1B To illustrate another example of a system, the system may use detected contactless gestures, postures, and motions from one or more fingers 106 and / or one or both hands 104 of a user 102 to implement the disclosed embodiments. Figure 1BThe system 100 is described as being embodied in a wearable device 150. As shown, the device 150 may include one or more sensors 160, a CPU 170, and one or more displays 180. Although the device 150 is shown as a pair of glasses worn on the head of the user 102, the device 150 may take other forms that are configured to be worn by the user 102 (e.g., attached to clothing) and capable of presenting visual content such as a keyboard image 140 to the user 102. For the purposes of the following discussion, the device 150 is described as Figure 1B Furthermore, common components of portable devices (e.g., batteries) are not described in detail here.

[0034] Relative to Figure 1A In the sensor 110, the sensor 160 may include any one or more sensor types and configurations described above. In particular, for Figure 1B In the wearable configuration shown, depending on the type of sensor 160, the sensor 160 can be positioned in the center of the device 150, on the side of the device 150, in one or more corners of the device 150, or anywhere else on the device 150 suitable for placement of the sensor 160. In some embodiments, multiple sensors 160 can be positioned on the device 150.

[0035] Relative to Figure 1A In the CPU 120, the CPU 170 may include components similar to those described above, such as the processor 122 and the memory 124. For example, in Figure 1B In the wearable configuration shown, the CPU 170 may be disposed on or within the eyeglass frame of the device 150. In some embodiments, the CPU 170 may be an external device that communicates with the device 150 via wired or wireless communication.

[0036] The device 150 may include one or more displays 180, such as a display 180 on one or both lenses of glasses of the device 150. Figure 1A In the embodiment of the wearable device compatible display 130, the display 180 may include any suitable display type similar to those described above. In some embodiments, the display 180 may include a projector that projects an image directly onto the pupils of the eyes of the user 102. In other embodiments, the display 180 may display an image on or in front of one or both lenses of the device 150 to create the illusion of augmented reality, so that the user 102 perceives the displayed image as floating in the air and in front of the user 102.

[0037] Figure 2An example process 200 for contactless data input according to some disclosed embodiments is illustrated. In some embodiments, process 200 may use a detected typing action to trigger the selection of a key displayed on a virtual keyboard. The detected typing action may include user 102 moving their finger 106 in mid-air to trigger the selection of a key, while the virtual keyboard is positioned at a distance from user 102's finger 106. That is, the detected typing action involves movement in mid-air that does not contact the surface on which the virtual keyboard is displayed or projected. Process 200 is described herein as being performed by sensor 110, processor 122 of CPU 120, and display 130, but in some embodiments, some steps of process 200 may be performed by a processing device other than processor 122 of CPU 120. Furthermore, in some embodiments, one or more steps of process 200 may be performed using a distributed computer system comprising multiple processors, such as processor 122 performing at least one step of process 200 and another processor in a networked device, such as a mobile phone, performing at least one step of process 200. Furthermore, in some embodiments, one or more steps of process 200 may be performed using a cloud computing system.

[0038] In step 202, the processor 122 may instruct a device, such as the display 130, to display an image of a keyboard. The display 130 may respond by displaying an image of a keyboard, such as keyboard image 140. The displayed image may include a plurality of keys. The keyboard image 140 may be displayed on a tangible display device, such as an LCD screen, projected directly onto the user 102's eyes, displayed in mid-air and separated from a tangible device or surface using holographic technology, or displayed using any other suitable display technology.

[0039] In step 204, processor 122 may receive sensor data collected by sensors 110, including data from one or more sensors 110 of the types described above. In some embodiments, the processor may be configured to perform operations including receiving sensor data from at least one sensor of a user's hand, the user's hand being spaced apart from and not in contact with the displayed keyboard. For example, the sensor data may include an image of user 102's hand 104 and finger 106 spaced apart from and not in contact with keyboard image 140. The image may include one or more analog images captured by sensor 110, digital images captured or determined by sensor 110, a subset of digital or analog images captured by sensor 110, digital information further processed by processor 122, a mathematical representation or transformation of information related to data sensed by sensor 110, information presented as visual information (e.g., frequency data representing an image), or conceptual information (e.g., the presence of an object in the sensor's field of view). The image may also include information indicating the state of the sensors during image capture, and their parameters (e.g., exposure, frame rate, image resolution, color bit rate, depth resolution, and field of view of sensor 110), information from other sensors during image capture (e.g., proximity sensor information, accelerometer information), information describing further processing to further capture the image, lighting conditions during image capture, features extracted from the digital image by sensor 110, or any other information related to sensor data sensed by sensor 110. Furthermore, "image" may include information related to still images, motion images (i.e., video), or any other visual data. In some embodiments, the sensor data received from one or more sensors 110 may include motion data, GPS location coordinates and / or direction vectors, line of sight information, sound data, and any other type of data measurable by different types of sensors 110. In some embodiments, the sensor data may include metrics derived by combining and analyzing data from two or more sensors 110.

[0040] At step 206, the processor 122 may analyze the received sensor data to identify motions, postures, and gestures associated with contactless typing and data entry. In some embodiments, the processor may be further configured to perform operations including tracking one or more fingers of a user's hand in mid-air at a distance from the displayed keyboard image based on the received image. In some embodiments, the processor 122 may analyze the sensor data using one or more known data analysis methods, such as image analysis based on temporally received sensor data (e.g., a series of images) and motion tracking of objects in three-dimensional space. The processor 122 may compare the sensor data with a database of predetermined motions, gestures, positions, and postures stored in memory 124 or the like. Step 260 may include one or more of determining and tracking object motion in the sensor data (step 207), determining the position and / or position change of one or more hands 104 and / or one or more fingers 106 in the sensor data (step 208), or determining the position of the hand (step 209). Steps 207, 208, and 209 may include identifying individual hands 104 and fingers 106 of the user in the received sensor data, and tracking the motion, position, and orientation of the identified objects in the received series of time-ordered sensor data. Figure 1A As shown, the processor 122 may receive image data from a sensor 110 of a user's 102 hand 104 (including five fingers 106 identified by the processor 122 as "A," "B," "C," "D," and "E"). As the sensor 110 captures data, such as image data, the processor 122 may determine at least one of a change in motion, position, and orientation of some or all of the fingers AE of the user's 102 hand 104 in real time, thereby tracking one or more hands 104 and one or more fingers 106 in the air at a distance from the displayed keyboard image. In some embodiments, the changes determined using a sensor such as a 2D camera may include changes in a two-dimensional plane. In some embodiments, the changes determined using a single sensor 110 such as a 3D camera, or a combination of sensors 110 such as a 2D camera and a depth sensor, an image sensor and a proximity sensor, a linear sensor and a "time-of-flight"-based proximity sensor, a stereo camera, or any system including one or more sensors that can extract the position of a user's hand, fingers, or fingertips in 3D space may include changes in three-dimensional space.

[0041] As used herein, “motion” may include one or more of a three-dimensional path in space, velocity, acceleration, angular velocity, motion path, and other known characteristics of changes in the physical position or location of the user's 102 hand 104 and fingers 106 .

[0042] As used herein, "position" may include a location in one or more dimensions in three-dimensional space, such as the X, Y, and Z coordinates of an object relative to the location of sensor 110. Position may also include a location or distance relative to another object detected in the sensor data received from sensor 110, such as the location of an object in a three-dimensional space. Figure 1A In the example shown, the position of finger “B” is relative to the position of finger “A.” In some embodiments, the position may also include the position of one or more hands 104 and / or fingers 106 relative to the body of user 102 , indicating the body position of user 102 .

[0043] As used herein, "orientation" may include the placement of one or more hands 104 or one or more fingers 106, including the position or direction in which the hands 104 and fingers 106 are pointing. In some embodiments, "orientation" may relate to the position or direction of a detected object relative to another detected object, relative to the detection field of a sensor 110, or relative to the detection field of a displayed device or displayed content.

[0044] As used herein, a "posture" may include a configuration of the hand 104 and / or one or more fingers 106 determined at a fixed point in time, as well as a predetermined configuration of the hand 104 and / or one or more fingers 106 positioned relative to each other. In some embodiments, an example hand 104 posture may include the hand being open and the fingers 106 being spread out, such as when arranged to type on a keyboard. Another schematic gesture may include the hand being generally closed and one or two fingers being extended, such as when arranged to operate a computer mouse. Other schematic gestures may include, for example, a hand being generally spread out downward with two or more fingers spread apart, as if placed on a physical keyboard, and a hand being generally downward with two or more fingers being generally bent, as if placed on a physical computer mouse.

[0045] As used herein, "gesture" may include detecting / identifying a predefined motion pattern using sensor data received from sensor 110. "Using sensor data," as used herein, may include analyzing raw sensor data received from sensor 110 and / or analyzing one or more metrics extended from the raw sensor data. In some embodiments, a gesture may include a predefined gesture corresponding to the identified predefined motion pattern. The predefined gesture may involve a motion pattern indicating operation of an activatable object, such as typing a keyboard key, clicking a mouse button, or moving a mouse housing. "Activatable object," as used herein, may include any displayed visual representation that, when selected or operated, causes data to be entered or a function to be performed. In some embodiments, a visual representation may include an image entry or portion of a displayed image, such as a keyboard image 140, a virtual key, a virtual button, a virtual icon, a virtual knob, a virtual switch, and a virtual slider.

[0046] The predefined gestures may be, for example, a sliding motion on an activatable object, performing a pinching motion with two fingers, or pointing to the activatable object, a left-to-right gesture, a right-to-left gesture, an upward gesture, a downward gesture, a push gesture, opening a clenched fist, opening a clenched fist and moving it toward the sensor 8 (also known as an "explosion gesture"), a tapping gesture, a push gesture, a waving gesture, a clapping gesture, a reverse clapping gesture, clenching the hands into a fist, a pinching gesture, a reverse pinching gesture, spreading the fingers on the hand gesture, a reverse spreading the fingers on the hand gesture, pointing to the activatable object, holding the active object at the activatable object for a predefined period of time, clicking on the activatable object, double-clicking, clicking on the right, clicking on the left, clicking on the bottom, clicking on the top, grabbing the object, gesturing to the object from the right, gesturing to the object from the left, passing through the object, pushing the object, clapping on the object, waving on the object, performing an explosion gesture, performing a tapping gesture, performing a clockwise or counterclockwise gesture on an object where the activatable object is grabbed with two fingers, performing a click-drag-release gesture, or sliding an icon (such as a volume bar). The speed of the scroll command may depend on the speed or acceleration of the scroll action.Two or more activation objects may be activated simultaneously using different activation objects, such as different hands or fingers, or using different gestures simultaneously.

[0047] Reference again Figure 2 , in step 210, the processor 122 may use the determined motion, position, and orientation to associate the detected one or more fingers 106 with one or more keys in the displayed keyboard image 140. In some embodiments, the processor may be further configured to perform operations including associating the positions of the one or more fingers in the air with the images of a plurality of keys. Notably, for the processor 122 to associate the fingers 106 with the keys of the keyboard image 140 in step 210, the association between the fingers 106 and the keys of the keyboard image 140 is performed based on sensor data associated with the fingers 106 in the air and does not require any indication that the fingers 106 are in contact with the keyboard image 140. Figure 1A In the example shown, fingers AE are associated with five keys labeled keys AE in keyboard image 140. Note that Figure 1A The key AE in the figure is not a key for inputting alphanumeric characters "A", "B", "C", etc., but rather the processor 122 indicates the key corresponding to the finger marked AE based on the movement, position / location, and / or orientation of the detected finger AE in the air.

[0048] Reference again Figure 2In step 212, the processor 122 may select one or more keys in the keyboard image 140 based on the associated locations of the one or more fingers 106 in the air and when a predefined condition is satisfied while the one or more fingers 106 remain associated with the one or more keys. In some embodiments, the processor may be further configured to perform operations including selecting a key from the keyboard image based on the associated locations of the one or more fingers in the air and detection of a predefined gesture performed by the user, wherein the predefined gesture may correspond to a predefined condition. The predefined condition may be, for example, a predefined time period during which the finger 106 remains associated with a particular key and / or detection of a predefined gesture performed by the finger 106, detection of movement of the finger 106 over a particular key, a change in direction of motion over a particular key, or a change in direction of motion with respect to a particular key, detection of a gesture indicating the beginning of a word, the end of a word, or a space between words that connects to indicate the end of a word, or any other suitable predefined condition. For example, the processor 122 can detect the writing motion of one or more fingers 106 on the keys of the keyboard image 140, and detect a "tap" gesture performed by the thumb, such as a "space" after a word (which also indicates the end of a word). By using the motion features and gestures detected in the received sensor data, the processor 122 can accurately and quickly input contactless typing data without the need for physical contact with the device and / or the touch screen.

[0049] In some embodiments, the processor 122 may select one or more keys in the keyboard image 140 by detecting a motion path of the hand 104 and / or finger 106 in the received sensor data and determining an order of positions of the hand 104 and / or finger 106 on the keyboard image 140 based on the detected motion path. The processor 122 may associate the detected motion path and / or the determined position order with one or more characters based on factors such as an order of corresponding characters associated with the keys on the keyboard image 140, language information such as a character or word database, statistics about commonly used characters and / or words, and / or statistical learning data of the user 102 collected by the processor 122 over time (including frequent motion paths, particularly for the user 102, and typing habits including frequently used characters, words, and typing errors associated with the user 102 stored in the memory 124).

[0050] As another example, in some embodiments, the predefined condition may include detecting placement of the finger 106 at a location in 3D space for at least a predefined time, such as the finger 106 hovering over an associated key for 3 seconds. As another example, in some embodiments, the predefined gesture may include a tapping action (indicating a click or typing action) toward the displayed keyboard image 140.

[0051] Once the predefined conditions are met, the commands associated with the selected keys are executed. In step 214, processor 122 may record the data input corresponding to the selected one or more keys. The data input may include alphanumeric characters such as letters, numbers, or symbols associated with the selected keys, computer functions associated with the selected keys, or a combination of characters and functions. Repeating the steps of process 200 may allow for the input of multiple characters and / or functions using contactless data input, thereby enabling contactless typing.

[0052] Keyboard mode and mouse mode

[0053] In some embodiments, a version of keyboard image 140 may include a virtual mouse, similar in shape and configuration to computer mouse peripherals known to those skilled in the art. However, in a contactless system, the combined use of a virtual mouse and a virtual keyboard presents some challenges. For example, when using a traditional physical keyboard and mouse, the user's hand physically moves between the keyboard and mouse to switch between inputting data via the keyboard and operating the mouse to perform various operations. However, in a contactless environment with a virtual keyboard and mouse, there is no physical keyboard or mouse. To streamline the virtual transition from keyboard to mouse, in some embodiments, processor 122 may recognize gestures to determine that hand 104 is typing at one moment and seamlessly controlling cursor movement at the next. The processor may be further configured to display an image of a virtual mouse having one or more activatable mouse buttons and / or a virtual mouse cursor for selecting and clicking one or more icons or displayed objects at a distance from the user, and to detect at least one first gesture and at least one second gesture, or at least one first set of gestures and at least one second set of gestures. As used herein, the moment when hand 104 is typing is referred to as "keyboard mode," and the moment when hand 104 is operating a virtual mouse is referred to as "mouse mode." The processor may enable keyboard mode to select at least one key from a displayed keyboard upon detecting a first gesture, and enable mouse mode to operate a virtual mouse upon detecting a second gesture. In some embodiments, the processor may enable the user to switch between keyboard mode and mouse mode by changing between the first gesture and the second gesture.

[0054] Figure 3 Figure 1 shows a graphical representation of keyboard mode and mouse mode according to some disclosed embodiments. Figure 3As shown, processor 122 can distinguish between keyboard mode 320 and mouse mode 370 by detecting different gestures, such as first gesture 310 and second gesture 340. For example, when processor 122 analyzes received sensor data and determines that user 102 has formed first gesture 310 by positioning hand 104 with fingers 106 spread apart, processor 122 can enter keyboard mode 320 and cause display 130 to display a keyboard in keyboard mode interface 330. Processor 122 can continuously process received sensor data to select one or more keys from the displayed keyboard for data entry based on detected gestures (e.g., a click or typing gesture). In some embodiments, display 130 can display one or more virtual hand images corresponding to user 102's hand 104 to illustrate the position of user 102's hand 104 relative to keyboard image 140 in keyboard mode interface 330, thereby facilitating more accurate and enjoyable data entry. The displayed virtual hand can move based on the detected movement of user 102's hand 104 and indicate the positioning of individual fingers on specific keys. For example, by placing virtual fingers of the virtual hand over the keys associated with fingers 106, the displayed virtual hand may show fingers 106 positioned as sitting on top of some of the keys in keyboard image 140. In some embodiments, keyboard image 140 may include any other suitable visual representation for showing one or more fingers 106 positioned over one or more keys, such as by highlighting one or more keys associated with the location of one or more fingers 106.

[0055] If the processor 122 detects that the user 102 has changed their gesture from the first gesture 310 to the second gesture 340 (transition 350), for example, by forming their hand 104 into a generally closed hand and / or a hand with one or more fingers 106 extended and pointing generally upward, the processor 122 may transition from the keyboard mode 320 to the mouse mode 370 (transition 360) to instruct the display 130 to transition 380 from the keyboard mode interface 330 to the mouse mode interface 390 to display a virtual mouse image at a distance from the user 102. In some embodiments, the processor 122 may transition from the keyboard mode 320 to the mouse mode (transition 360) when it detects a change in the orientation of one hand 104 relative to the other hand 104, such as when the user 102 moves one hand 104 right / left / up / down while the other hand remains in generally the same position (indicating that the moving hand 104 is operating the mouse while the stationary hand 104 remains on the keyboard).

[0056] While in mouse mode 370, the processor 122 can continuously analyze sensor data in real time to operate the virtual mouse based on the movements and gestures of the hand 104 and finger 106, thereby moving the virtual mouse or virtual cursor, selecting one or more buttons on the virtual mouse, and / or clicking one or more displayed activatable objects, such as one or more displayed icons. Upon detecting that the user 102 transitions back to the first gesture 310, the processor 122 can transition from mouse mode 370 back to keyboard mode 320 (reverse transition 360) and command the display 130 to transition from the mouse mode interface 390 back to the keyboard mode interface 330 (reverse transition 380).

[0057] Button area size

[0058] On a traditional physical keyboard, many keys are roughly the same size, which is practical for physical keyboards but can cause problems for virtual keyboards, especially for contactless data entry. In particular, the uniform size of the keys in a virtual keyboard can lead to an increase in errors in areas of the keyboard where the hand has difficulty reaching or in areas where keys are used more frequently, because sometimes these errors are made by selecting close keys rather than intended keys. In some embodiments, the error rate can be reduced by modifying the space corresponding to some or all keyboard keys relative to other keys. In some embodiments, the processor can be configured to assign spatial areas to alphanumeric keys on the keyboard, and the size of the spatial area assigned to each key can vary according to criteria that include the expected frequency of use of the key and the motion characteristics of the hand 104 and finger 106 of the user 102.

[0059] In some embodiments, the processor 122 may assign spatial regions of different sizes to some of the keys in the keyboard image 140 and modify them. For clarity, the displayed keyboard image 140 does not change appearance based on the modified key region size. Instead, the processor 122 may assign spatial regions of various shapes and sizes around the user's 102 hand 104 and / or finger 106 within the field of view of the sensor 310. For example, in some embodiments, the assigned region may be set within the field of view of the sensor 110 and associated with a location in space, such as where the hand 104 first appears during a typing session or the location of the last key typed.

[0060] In some embodiments, the processor 122 may dynamically modify one or more key area sizes for one or more keys in the keyboard image 140 based on the probability of the next key being selected, even though the key sizes of the keyboard image 140 displayed on the display 130 remain unchanged. The processor 122 may modify the assigned key areas based on one or more factors, including, for example, the likelihood that a key will be selected next, the frequency of selection of a key, the predetermined difficulty of reaching a key, the risk of skipping a key, detection of a finger hovering over a key, or the likelihood that a key will be selected next based on a word completion algorithm.

[0061] In some embodiments, the processor 122 may use historical data on the contactless typing activity of one or more users to assign spatial regions to keys. In other embodiments, the processor 122 may dynamically change the size of key regions based on data on the contactless typing activity of the user 102 collected by the sensor 110. For example, in some embodiments, the processor 122 may assign spatial regions to alphanumeric keys on a keyboard, with the size of the spatial regions varying based on criteria including the expected frequency of use of the keys. As another example, in some embodiments, the processor 122 may dynamically change the spatial regions assigned to particular keys based on a prediction of which particular key will be selected next. As another example, in some embodiments, the processor 122 may dynamically change the spatial regions assigned to particular keys based on the location of the particular keys in the key image, such as along the edge of the keyboard, at the corner of the keyboard, away from the center of the keyboard, or at other locations that are considered relatively difficult to reach compared to keys positioned closer to a typical keyboard hand. As another example, in some embodiments, the processor 122 may assign spatial regions to particular keys or groups of keys based on a predetermined or observed typing error rate. For example, the processor 122 can add one or more zones assigned to one or more keys that may be the subject of frequent typing errors, such as incorrectly selecting the next key after a particular key or skipping the particular key. In some embodiments, the processor 122 can observe one or more error rates for particular keys using manual typing correction frequencies or using a word completion algorithm. In some embodiments, the one or more error rates can be pre-programmed for one or more keys.

[0062] like Figure 4 As shown, the keyboard pattern 140 may include a plurality of keys, such as four schematic keys corresponding to data input of the alphabet, including an "E" key, an "X" key, a "G" key, and an "N" key. Figure 4A comparison is provided between an unmodified version of these keys and a modified version that has been modified to more accurately detect data input. As shown, the dashed boxes around the hand shown represent the portion of the area in the spatial region allocated to each of the four keys. The solid boxes around each of the four keys represent the unmodified areas that directly correspond to the key sizes shown in the keyboard pattern 140, including an E area 402, an X area 406, a G area 410, and an N area 414. The dashed boxes around each of the four keys represent modified areas of various sizes and shapes assigned to each of the four keys by the processor 122 to more accurately detect data input. The modified allocated areas include a modified E area 404, a modified X area 408, a modified G area 412, and a modified N area 416.

[0063] In some embodiments, the processor 122 may allocate a modified area that includes space that deviates from the original key space. For example, the modified E area 404 is offset to the lower left of the E area 402. This modification may be due to, for example, a history of placing a hand on the modified E area 404 with the intention of selecting the E key through the E area 402. Therefore, to compensate for unintentionally skipping over a desired key (discussed in more detail below), the processor 122 may generate a deviated allocation area, such as the modified E area 404. Furthermore, in some embodiments, the deviated allocation area for frequently selected keys may be larger than the original key area, as shown by the modified E area 404 being higher than the E area 402.

[0064] In some embodiments, the processor 122 may identify one or more keys that are less frequently selected, such as Figure 4 . X area 406 represents the unmodified spatial area corresponding to the X key, and in some embodiments, the processor 122 may allocate a smaller modified X area 408 that, when selected, triggers input of the letter "X." Significantly, all dimensions of modified X area 408 are smaller than X area 406, so that, in this embodiment, selection of the X key may require more careful and precise effort from the user 102. The processor 122 may allocate the smaller modified area to one or more less frequently used keys and reallocate portions of the original key area to keys requiring larger and / or offset areas.

[0065] In some embodiments, the processor 122 can dynamically modify one or more assigned key areas based on the likelihood of some keys being selected next. For example, the processor 122 can assign a larger area to keys that are considered likely to be selected next without modifying the surrounding key areas or while modifying the surrounding key areas. However, in some embodiments, the keyboard image 140 can remain unmodified and only the assigned areas that are not visible in the space can be modified. The processor 122 can use one or more methods to determine the likelihood of a particular key being selected next, such as determining that a hand or finger is hovering over a particular key when the finger continues or has stopped moving over the key for a certain period of time.

[0066] In some embodiments, the processor 122 may allocate the expanded area to keys that have been selected more frequently than other keys in history. Figure 4 As shown, modified G-zone 412 is larger in all dimensions than G-zone 410. By enlarging the area allocated to more frequently selected keys, data entry accuracy is improved because more frequently selected keys have a higher probability of being selected even when user 102's finger 106 is located near the key.

[0067] Depending on the shape and arrangement of the displayed keyboard, some keys may be more difficult to reach than others. For example, when the user's 102 hand 104 is located in the center of the displayed keyboard, keys around the edges or corners of the keyboard may be less accessible and require more effort to reach and select. In some embodiments, the processor 122 may compensate for harder-to-reach keys by allocating larger areas to them. Additionally, in some embodiments, the processor 122 may change the shape of the area allocated to hard-to-reach keys. Figure 4 As shown, the N key is located in the lower right corner of the displayed keyboard and is more difficult to reach when the user's 102 hand 104 is near the center of the displayed keyboard than, for example, the displayed E, G, and X keys. The processor 122 can modify the N region 414 assigned to the N key to generate a modified N region 416. As shown, the modified N region 416 is larger in size than the N region 414 and has a modified shape, such as angled or curved edges, to facilitate key selection.

[0068] Confirm Intent Button

[0069] When moving one or more fingers in mid-air, it is often difficult for the user to control the movement of the fingers to precisely stop within the spatial area assigned to the intended key. As a result, the finger 106 of the user 102 may cross the assigned space, which may cause typing errors. To avoid errors that may be caused by crossing over, in some embodiments, the processor 122 may require the user 102 to hover the finger 106 within the assigned area of ​​the intended key, such as hovering above or near the intended key in the displayed keyboard image for a certain period of time, so that the processor 122 can confirm the intended key.

[0070] To detect hovering, the processor 122 may analyze information received from the sensor 110 to detect the speed of the moving finger 106 and correlate changes in speed with a hover gesture. The processor 122 may determine a key associated with hovering. For example, the processor 122 may use the determined position of the finger 106 in space to determine a key associated with the spatial region over which the finger 106 is hovering. In some embodiments, the processor 122 may also employ one or more algorithms to determine possible intended keys (e.g., an algorithm that specifies a threshold for determining whether a key is proximate to the location of a hover gesture).

[0071] Furthermore, in a contactless typing system, due to the delay between sensor data capture and processing, or the human brain processing the displayed image (such as moving across the displayed key image 140 (such as Figure 3 System lag may exist due to a delay in the time it takes for the user to select a key or icon (e.g., a virtual hand image shown and discussed above). During system lag, the user may believe that the finger needs to maintain motion to select the key or icon, when in fact additional motion would cause the finger to pass over the key. This may cause the user to select the wrong key. In some embodiments, the processor 122 can account for system lag by determining the user's likely intention and displaying the intended motion rather than the actual motion.

[0072] In some embodiments, the processor 122 may predict the next key to be selected to avoid over-selection by determining the likely intent of the user 102 using one or more motion characteristics of one or both hands 104 and / or one or more fingers 106 from the received sensor 110 information. For example, the sensor 110 may analyze the image received from the sensor 110 and determine one or more motion characteristics, such as motion vector, motion speed, change in speed, acceleration and deceleration of one or more hands 104 and / or one or more fingers 106 during the motion, change in orientation of one or more hands 104 and / or one or more fingers 106, and motion speed or path of one or more fingertips 106 relative to the tips of other fingers 106 and / or relative to the center of the hand 104.

[0073] In some embodiments, the processor 122 may also determine likely intent using one or more of the following: an estimated amount of time for the brain to process visual feedback, a predetermined or calculated amount of time for the processor 122 to process received sensor 110 data (e.g., a received image), or a history of previously selected keys. For example, when a detected motion vector is above a certain level or threshold, such as when the processor 122 detects movement of one or more fingers 106, the processor 122 may cause the display 130 to provide visual feedback to the user 102, such as highlighting one or more keys on the keyboard image 140 associated with the position of the fingers 106. The processor 122 may predict likely keys based on the motion vectors to identify keys that are likely to be associated in the near future (e.g., within the next few milliseconds). The processor 122 may instruct the display 130 to highlight likely keys before a finger 106 is actually placed over the likely key, to account for system lag inherent in the processing system and any mental lag on the user 102. Mental lag may include, for example, a small delay between the time an image is displayed on the display 130 and the time the user 102 mentally recognizes and interprets the displayed image. When the user 102 views the virtual hand moving across the keyboard image 140, but the user 102 does not immediately register the exact position of the virtual hand, there may be a mental delay, causing the user 102 to overcompensate or possibly pass over the intended key. By selecting and highlighting the potential key a small interval before the hand 104 or finger 106 reaches the actual position of the potential key, the negative effects of system and mental lags can be minimized.

[0074] As another example, the processor 122 may cause the display 130 to display one or more highlighted keys along the direction of the motion vector path, so that the user 102 perceives that the intended key has been "reached" before the intended key is actually reached, so that the user 102 stops moving the finger 106 to minimize the extent of crossing the intended key. For example, the display 130 may highlight two keys along the detected motion path - the "current key" corresponding to the detected position of the finger 106, and the possible next key. The timing of highlighting the intended key early may depend on a number of factors, including system lag and the lag in the time it takes the human brain to process the displayed image (i.e., mental lag), the detected speed, the motion vector, the likelihood of the key being selected, and the "motion behavior" of the user 102, such as the typing habits of the user 102 that change over time, such as common errors, frequently selected keys, and common motion characteristics.

[0075] In some embodiments, the processor 122 can learn the specific "motion behavior" of the user 102 and use information extracted from the user's "motion behavior" to provide visual feedback (e.g., an increment key on a keyboard) to avoid crossing over these intended keys. "Motion behavior" parameters can include, for example, the average distance crossed over a particular key, and the position crossed over the keyboard image 140. The crossing distance can include a spatial area that is increased to the size of the key area of ​​the possible next key, and a spatial area that is increased in the direction of the detected motion vector. For example, if the finger 106 moves to the right toward the possible key, the processor 122 can increase the crossing distance to the right of the area assigned to the possible key.

[0076] In some embodiments, the processor 122 may determine the flyover distance based on, for example, the distance between the last key typed and the likely next key in the expected word or phrase. For example, if the processor 122 determines that the user 102 is likely to type the word "time," the processor 122 may calculate one or more flyover distances between the "t," "i," "m," and "e" keys on the keyboard image 140. If the keyboard image 140 includes a standard QWERTY keyboard, the processor 122 may calculate a first flyover distance between the "t" and "i" keys based on the distance between the "t" and "i" keys (three keys apart) and / or the user 102's historical accuracy in typing "i" after "t." The processor 122 may then calculate a second flyover distance between the "i" and "m" keys based on the distance (two keys apart) and / or the user 102's historical accuracy in typing "m" after "i." Finally, the processor 122 may calculate a third flyover distance between the "m" and "e" keys based on the distance (four keys apart) and / or the user's historical accuracy in typing "e" after "m." In some embodiments, the calculated flyover distance may be directly related to the key distance and historical accuracy. That is, the processor 122 may calculate a smaller flyover distance for two keys that are closer together in the keyboard image 140, and a smaller flyover distance for keys that the user has historically typed accurately.

[0077] Known image and motion data analysis methods can be used to determine motion characteristics and, based on the determined motion characteristics, determine the most likely intended position of the hand 104 or finger 106. Using the determined motion characteristics, the processor 122 can identify a likely key as the key that the user 102 intended to select next and select the likely key even when the processor 122 determines that the user 102 selected another key than the likely key.

[0078] In some embodiments, the processor 122 can predict the next keystroke while the user 102 is typing. The prediction can use at least one of a word completion algorithm and a previously selected keystroke using information associated with at least one location of the user's hand 104 or one or more fingers 106. The processor 122 can use a known word completion algorithm, but in some embodiments, the processor 122 uses the word completion algorithm in conjunction with information received from the sensor 110 to anticipate the next keystroke to increase key prediction accuracy. For example, the processor 122 can predict the next keystroke using at least one of a word completion algorithm and a previously selected keystroke using information associated with at least one location change of the location of the user's hand 104 or one or more fingers 106. As another example, the processor 122 can predict the next keystroke using at least one of a word completion algorithm and a previously selected keystroke using information associated with one or more motion features of the user's 102 moving hand 104.

[0079] In some embodiments, the processor 122 may be configured to utilize a word completion algorithm to determine a possible word / phrase being typed by the user prior to word completion, and to instruct the display 130 to display the possible word. In some embodiments, the possible word / phrase may include a suitable name, a proverb, a slogan, or a title (e.g., a music album or movie title). The processor 122 may monitor information received from the sensor 110 to identify a position of the hand 104 indicating the user 102's acceptance or rejection of the possible word. In some embodiments, the hand position identified to accept / reject the possible word may include a fixed position of one or more hands 104 or fingers 106, or a dynamic gesture of one or more hands 104 or fingers 106. For example, after determining the possible word and instructing the display 130 to display the possible word, the processor 122 may determine that the user has placed their hand 104 in a first position, predefined to indicate acceptance of the displayed possible word. In some embodiments, processor 122 may determine that user 102 accepts the possible word when it recognizes a gesture, such as user 102 moving hand 104 away from user 102's body, or user 102 moving hand 102 or multiple fingers downward in a "tap down" motion. In some embodiments, processor 122 may determine that user 102 rejects the possible word when it recognizes a gesture, such as user 102 moving hand 104 away from user 102's body, or user 102 moving hand 104 or multiple fingers upward in a "tap up" motion. The examples of hand positions and gestures are non-limiting, and other positions or gestures for accepting or rejecting possible words may be pre-programmed or defined by the user during a setup or learning process. Processor 122 may then complete the possible word based on the first recognized position of hand 104 without requiring further typing input.

[0080] For example, after determining a possible word and instructing the display 130 to display the possible word, the processor 122 may determine that the user 102 has placed the hand 104 in a second orientation, which is predefined as indicating rejection of the displayed possible word. The processor 122 may end displaying the possible word and continue monitoring the information received from the sensor 110 for typing input, and / or may determine a second possible word based on a word completion algorithm and display the second possible word.

[0081] Trigger keyboard display

[0082] To initiate interaction with the virtual keyboard, in some embodiments, the processor 122 can command the display 130 to automatically display the keyboard when the user's 102 hands 104 are raised to one of a predefined set of positions, such as one or more hands 104 raised in a typing position, one or more fingers 106 spread out, the hands 104 raised and placed close together, one or more hands 104 raised to the plane of the sensor 110, or one or more hands in a raised position with the hand orientation parallel to the floor. This functionality enables the system to ignore the display of the keyboard and only access it when needed, thereby saving power and increasing the level of interaction between the user 102 and the system 100. In some embodiments, the processor 122 can continuously process information received from the sensor 110 to detect the presence of the user 102 within the field of view of the sensor 110 and to detect the position and movement of one or more hands 104. In some embodiments, when the processor 122 detects one or more hands 104 moving from a lowered position to a raised position, such as raising the hands 104 upward from the sides of the user 102 and above a predetermined height threshold, such as waist or chest level, the processor 122 may command the display 130 to display the keyboard image 140 due to the detection of at least one raised hand. In some embodiments, when the one or more raised hands 104 are lowered, such as when the user 102 lowers the one or more hands 140 from waist or chest to the sides of the user 102, or below the predetermined height threshold, the processor 122 may command the display 130 to end displaying the keyboard image 140. In some embodiments, after detecting the lowered hand, the processor 122 may delay the command to end displaying the keyboard image 140 for a predetermined period of time to confirm the user 102's intent to end display by lowering the one or more hands 104. The sensor 122 may trigger display of the keyboard image 140 based on detecting only one hand, or require detection of two hands to transition from an active display state to an inactive display state, or vice versa.

[0083] In some embodiments, it may be necessary to avoid misinterpreting user 102's intended actions, such as presenting keyboard image 140 when user 102 raises one or more hands 104 without intending to type. In such embodiments, processor 122 may require that one or more predefined conditions be met to display keyboard image 140 and to terminate display of keyboard image 140. For example, in some embodiments, keyboard image 140 may be presented only when typing functionality is available to user 102, only when certain running computer applications support data entry and typing, or only when certain computer applications support typing, such as a searchable TV program listing displayed in a television application. In this example, during video playback (a portion of a television application that does not support typing), processor 122 may not display keyboard image 140 despite user 102 providing the same hand orientation and position. In some embodiments, processor 122 may provide a visual indication of the detection of a gesture in which user 102's hands are positioned and / or oriented to trigger display of keyboard image 140, but may not display keyboard image 140 itself.

[0084] Keyboard row selection

[0085] In some embodiments, when typing on a virtual keyboard, rather than the traditional action of moving a finger forward or backward to tap a key in the up or down row, the user can move their entire hand to select the up or down row more efficiently. In some embodiments, the processor 122 can be configured to select different parts of the displayed keyboard for data input detection. For example, the processor 122 can select a first subset of keys based on a particular motion, gesture, hand position, or a posture detected based on a hand position monitored in the received sensor data. The processor 122 can be configured to select different subsets of keys based on different motions, gestures, hand positions, or postures monitored. In some embodiments, the subset of keys can be a row in the keyboard, or a block of keys.

[0086] Figure 5A An example of selecting a first and second subset of keys based on a detected first motion is shown, where the subset of keys is a first row 506 and a second row 508 on a displayed keyboard 512, and the detected motion is a generally horizontal motion of one or more hands toward the displayed keyboard 512. Figure 5A As shown, when the hand 104 is in a first position, such as H1 512, the first row 506 may be a subset of keys that are not necessarily arranged in a straight line, but may include a first plurality of keys corresponding to the monitored positions of the fingers 106 of the user 102. Similarly, when the hand 104 is in a second position, such as H2 504, the second row 508 may include a second plurality of keys corresponding to the positions of the hand 104 or fingers 106 of the user.

[0087] exist Figure 5A In the illustrated embodiment, the processor 122 may receive sensor data from the sensor 110 regarding the motion of the hand 104 of the user 102 moving from a first position H1 502 to a second position H2 504. The motion may be generally toward or away from the displayed keyboard 512, such as a generally horizontal motion along the Z-axis relative to the displayed keyboard 512. Upon detecting the forward motion, the processor 122 may transition the selection of keys from the first row 506 to the second row 508 (transition 510). After the transition, the processor 122 may cause the display 130 to change the displayed keyboard 512, such as by graphically illustrating the selection of the second row 508. Figure 5A In the example shown, a black circle appears over the key that the processor 122 determines corresponds to the position of the detected finger 106. After executing the transition 510, the displayed black circle may move from the first row 506 to the second row 508. In other embodiments, the key image 512 may display a different indication of the transition 510, or may not display any indication of the transition 510.

[0088] Notably, in some embodiments, the first row 506 and the second row 508 may include one or more commonly used keys. For example, if a particular key in the keyboard image 512 corresponds to the finger 106 in the first position H1 502 and the second position H2 504, then the key may be included in the first row 506 and the second row 508, such as one of the keys shown in the right column of the keyboard image 512.

[0089] In some embodiments, upon detecting movement from the second position H2 504 to the first position H1 502 , such as movement generally rearward away from the displayed keyboard 512 , such as horizontal movement along the Z axis away from the displayed keyboard 512 , the processor 122 may select Down.

[0090] Figure 5B Another example of keyboard row selection using detection of second hand movement is shown. Figure 5B In the illustrated embodiment, the processor 122 may receive sensor data from the sensor 110 regarding movement of the hand 104 of the user 102 from a first position H1 502 to a second position H2 504. The movement may be generally upward or downward, such as vertical movement along a Y-axis parallel to the displayed keyboard 512. Upon detecting the upward movement, the processor 122 may transition the selection of keys from the first row 506 to the second row 508 (transition 510). Following the transition, the processor 122 may cause the display 130 to change the displayed keyboard 512, such as by graphically indicating the selection of the second row 508. Figure 5BIn the illustrated embodiment, black circles appear on keys that the processor 122 determines correspond to the position of the monitored finger 106. After executing the transition 510, the displayed black circles may move from the first row 506 to the second row 508. In other embodiments, the keyboard image 512 may display a different indication of the transition 510, or may not display any indication of the transition 510.

[0091] In some embodiments, the processor 122 may select Down after detecting a generally downward movement (such as a vertical downward movement parallel to the Y-axis of the keyboard image 512 ) from the second position H2 504 to the first position H1 502 .

[0092] As used herein, the terms "for example," "for instance," "for example," and variations thereof describe non-limiting embodiments of the present subject matter. Reference in the specification to "one embodiment," "some embodiments," "other embodiments," or variations thereof means that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the present subject matter. Thus, the appearances of the phrases "one embodiment," "some embodiments," "other embodiments," or variations thereof do not necessarily refer to the same embodiment.

[0093] For clarity, certain features described in this specification in the context of separate embodiments may also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment for brevity may also be provided in multiple separate embodiments or in any suitable subcombinations. Furthermore, while features may be used in a particular combination and even initially claimed, one or more features in a claimed combination may in some cases be removed from that combination, and a claimed combination may lead to subcombinations or variations of subcombinations.

[0094] Operations according to the embodiments described herein may be performed by a computer specially constructed for the required purposes or by a general-purpose computer specially configured for the desired purposes using a computer program stored in a non-transitory computer-readable storage medium.

[0095] While several specific embodiments have been described above, other embodiments are within the scope of the following claims.

Claims

1. A data input device comprising: At least one processor configured to: receiving sensor data of a user's hand from at least one sensor, wherein the user's hand is contactless with the display device; detecting, using the received sensor data, at least one of: a user's hand in air at a distance from the display device or one or more fingers of the user's hand in air at a distance from the display device; Based on the detecting, assigning an area in space within a field of view of a sensor and in air at a distance from the display device, wherein the assigned area is associated with an alphanumeric key; associating the position of the one or more fingers in the air with an assigned area in the space; and selecting an alphanumeric key based on detection of a predefined finger gesture performed by a user's hand in the air and a correlation between the position of one or more fingers in the air and an assigned region in space, Wherein the allocated area in the space varies in size based on a prediction associated with a particular key to be selected.

2. The data input device according to claim 1, wherein The at least one processor is further configured to dynamically change the area in the space allocated to a particular key based on a prediction that the particular key will be selected next.

3. The data input device according to claim 1, wherein The prediction is a prediction associated with a particular key that will be selected next.

4. The data input device according to claim 1, wherein The at least one processor is further configured to display the keyboard image on the display device.

5. The data input device according to claim 1, wherein The at least one processor is further configured to display the keyboard image in mid-air, separate from any physical device or surface.

6. The data input device according to claim 1, wherein The at least one processor is further configured to: Displaying a virtual mouse image at a distance from the user; detecting a first gesture and a second gesture; When the first gesture is detected, enabling a keyboard mode to select at least one key from a displayed keyboard; When a second gesture is detected, enabling a mouse mode to operate the virtual mouse; as well as By changing between the first gesture and the second gesture, the user is enabled to switch between keyboard mode and mouse mode.

7. The data input device according to claim 6, wherein The second gesture is a finger pointing generally upward.

8. The data input device according to claim 6, wherein: The second gesture is a generally closed hand.

9. The data input device according to claim 1, wherein The allocated areas in the space associated with alphanumeric keys vary in size depending on criteria including the expected frequency of key usage.

10. The data input device according to claim 9, wherein The at least one processor is further configured to cause the at least one processor to assign larger areas in the space to more frequently used keys and to assign smaller areas in the space to less frequently used keys.

11. The data input device according to claim 9, wherein It is further configured to cause the at least one processor to allocate a larger area in the space to one or more keys when the user's hand is at rest.

12. The data input device according to claim 1, wherein The at least one processor is further configured to predict a next key to be selected using a motion characteristic of a user's hand.

13. The data input device according to claim 12, wherein The at least one processor is further configured to predict a next key press using a word completion algorithm and at least one of a previous key press selected using information associated with at least one location of a user's hand or one or more fingers of the user's hand.

14. The data input device according to claim 12, wherein The at least one processor is further configured to predict a next key press using a word completion algorithm and at least one of a previous key press, wherein the previous key press is selected using information associated with at least one of a position change, a position of a user's hand, or a position of one or more fingers of a user's hand.

15. The data input device according to claim 12, wherein The at least one processor is further configured to predict a next key press using at least one of a word completion algorithm and a previous key press selected using information associated with motion characteristics of a user's hand.

16. The data input device according to claim 1, wherein The at least one processor is further configured to selectively cause a keyboard image to be displayed, and wherein display of the keyboard image is triggered by detection of at least one lifted hand.

17. The data input device according to claim 16, wherein The at least one processor is further configured to cause display of the keyboard image to end when the raised hand is lowered.

18. The data input device according to claim 1, wherein The at least one processor is further configured to monitor the position of the user's hand and the position of one or more fingers of the user's hand in the received sensor data, and wherein the monitored position of the user's hand is used to determine the selected keyboard row and the monitored position of one or more fingers of the user's hand is used to determine the selected key.

19. The data input device according to claim 18, wherein The selected keyboard row is determined based on a horizontal motion in the monitored position of the user's hand that represents forward or backward movement.

20. The data input device according to claim 18, wherein The selected keyboard row is determined based on a vertical motion in the monitored position of the user's hand representing an upward or downward movement.

21. The data input device according to claim 1, wherein The at least one processor is further configured to determine a possible word that the user is typing before word completion, display the possible word, identify a hand orientation that represents the user's acceptance of the possible word, and accept the possible word in response to the identified hand orientation.

22. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: receiving sensor data of a user's hand from at least one sensor, wherein the user's hand is contactless with the display device; detecting, using the received sensor data, at least one of: a user's hand in air at a distance from the display device or one or more fingers of the user's hand in air at a distance from the display device; Based on the detecting, assigning an area in space within a field of view of a sensor and in air at a distance from the display device, wherein the assigned area is associated with an alphanumeric key; associating the position of the one or more fingers in the air with an assigned area in the space; and selecting an alphanumeric key based on detection of a predefined finger gesture performed by a user's hand in the air and a correlation between the position of one or more fingers in the air and an assigned region in space, Wherein the allocated area in the space varies in size based on a prediction associated with a particular key to be selected.

Images