DEVICE COMPLETING A FRONT DISPLAY AND A TOUCH SCREEN EMULATOR COMPLETING A EYE DETECTION MODULE, A GESTURE DETECTION MODULE AND AN INTEGRATION MODULE, AS WELL AS METHOD FOR OPERATING SUCH A DEVICE

Abstract

Procedure (40), comprising: Receiving a first image from a first front camera (102, 1102) of a device (30, 50, 1000) with a front display (400, 1200); Identifying one or more eyes of a user (10) of the device (30, 50, 1000) based on the first image; Determining a gaze position (12) of one eye or multiple eyes on the front display (400, 1200); Determining a change in gaze position (12); Receiving a second image from a second front camera (101, 1101) of the device (30, 50, 1000); Identifying (42) a point of interest on the front display (400, 1200) based on gaze information, wherein the gaze information includes the gaze position (12) and the change in gaze position (12); Identifying one of the user's hands (10) based on the second image; Identifying one or more fingertips (20) of the user's hand (10) based on the second image; Determining a movement of one or more of the hand and one or more fingertips (20); Identifying (44) a hand action based on gesture information, wherein the gesture information shall include the movement and the movement shall be one or more of a one-dimensional (1D) movement, a two-dimensional (2D) movement and a three-dimensional (3D) movement; and Initiating (46) an action of the device (30, 50, 1000) with respect to the front display (400, 1200) based on the point of interest and the action of the hand, the procedure further includes: Synchronizing gaze information with gesture information; and Capturing one or more pieces of gaze information and gesture information.

Description

BACKGROUND OF THE INVENTION

[0001] Embodiments generally relate to capturing user input. More specifically, embodiments relate to capturing user input by emulating a touchscreen interface. discussion

[0002] A touchscreen interface can allow a user to touch the surface of a device (e.g., a display device) to facilitate user input. Numerous modern consumer electronic devices (e.g., tablets, smartphones, etc.) can utilize touchscreen interfaces. Unfortunately, touchscreen interfaces can typically significantly increase the manufacturing costs of these devices. In fact, in some cases, 30-40% of the cost of a touchscreen device can be attributed to the cost of the touchscreen interface.

[0003] WO 2012 / 082 971 A1 describes a system and method for activating and interacting a user with at least one 3D object displayed on a 3D computer display, through at least the user's gestures, which can be combined with the user's gaze on the 3D computer display. In a first case, the 3D object is a 3D CAD object. In a second case, the 3D object is a radial menu. The user's gaze is captured by a head frame containing at least one endocamera and one exocamera worn by the user. A user's gesture is captured by a camera and recognized from a plurality of gestures. The user's gestures are captured by a sensor and calibrated to the 3D computer display.

[0004] US 6,111,580 A describes an input unit for controlling a device. The input unit first determines whether a user is actively operating the device. If so, the input unit determines whether the user is performing an action corresponding to a command. The user's attention can be indicated by the user observing the device, approaching the device, turning toward the device, or speaking. An input command can be indicated by touch, gestures, speech, or gaze direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The various advantages of embodiments of the present invention will become apparent to one skilled in the art upon reading the following description and the appended claims, as well as upon use of the following drawings, in which: Fig. is an illustration of an exemplary user interaction with a multimodal touch screen emulator according to one embodiment; Fig. is a flowchart of an exemplary method for utilizing a multimodal touch screen emulator according to one embodiment; Fig. is a block diagram of an exemplary multimodal touch screen emulator according to one embodiment; and Fig. is a block diagram of an exemplary system including a multimodal touch screen emulator according to one embodiment. SUMMARY OF THE INVENTION

[0006] The object underlying the invention is achieved by the subject matter of the independent claims. Further advantageous embodiments are specified in the subclaims. DETAILED DESCRIPTION

[0007] Embodiments may generally relate to capturing user input by emulating a touch screen mechanism. Fig. is an illustration of an exemplary interaction of a user 10 with a computer system 30 including a multi-modal touch screen (MMTS) emulator according to one embodiment. In embodiments of the present invention, the MMTS emulator may utilize, among other things, gaze tracking and gesture tracking to emulate a touch screen interface.

[0008] The MMTS emulator may utilize gaze tracking to determine what the attention of the user 10 is directed to. In particular, the MMTS emulator may determine a location of a point of interest 14 (e.g., display screen / device) by identifying one or more eyes of the user 10 and tracking a gaze 12 of the one or more eyes to the point of interest 14.

[0009] In one example, the MMTS emulator may utilize precision tracking to track an iris of user 10. For example, if user 10's attention is focused on a particular icon on the display screen of computer system 30, the MMTS emulator may be configured to track gaze 12 by focusing on the iris and determining the particular icon on which user 10 is focused. The gaze tracking aspect of the MMTS emulator may, for example, be activated upon detecting the presence of a human (e.g., detecting a human head or face) in a captured image, or may occur continuously (e.g., as long as a coupled camera is powered on).

[0010] Second, the MMTS emulator may utilize gesture tracking to determine what the user 10 intends to do. That is, the MMTS emulator may analyze the body movements of the user 10 to detect an input-related gesture and classify the gesture as a particular action of the device requested by the user 10. Thus, in one example, the MMTS emulator may be configured to identify one or more fingertips of a user, such as a fingertip 20, and analyze the movements of the one or more fingertips to identify input-related gestures. In other examples, the MMTS emulator may focus on another body part of the user (e.g., a finger, a hand, an arm, etc.) or on an input device also manipulated by the user (e.g., an input device originating from the user).

[0011] An MMTS emulator, as described, may be configured to identify a number of user gestures. One such gesture recognized by an MMTS emulator may be a single-touch gesture. A single-touch gesture may involve a single body part (e.g., a single fingertip, a single hand, etc.) to input a device action. An example of a single-touch gesture may be a movement of the user's fingertip from a left starting coordinate to a right ending coordinate. This movement may indicate that the user intends to perform a corresponding left-to-right movement within the computer interface (e.g., on a display interface). Similarly, an MMTS emulator may be configured to classify, among other things, right-to-left movement, top-to-bottom movement, bottom-to-top movement, diagonal movements, and so on.Furthermore, the MMTS emulator can be designed to detect and classify, for example (and not limited to), a start movement, a stop movement, an opening movement, a closing movement, an activation movement or a shutdown movement also on the part of the user.

[0012] The MMTS emulator can also be configured to detect a multi-touch gesture. A multi-touch can involve more than one body part for inputting a device action. Thus, for example, in the case of a "zoom in" gesture, the MMTS emulator can be configured to detect that the user is bringing two fingertips together. Alternatively, the MMTS emulator can be configured to identify a "zoom out" gesture by detecting that the user is spreading two fingertips apart. Similar to the gaze tracking aspect, the gesture tracking aspect of the MMTS emulator can operate continuously (i.e., as long as a docked camera is powered on) or be activated upon detecting the presence of a human (e.g., detecting a human head or face) in a captured image.

[0013] Fig. is a flowchart of an exemplary method 40 operating with an MMTS emulator according to one embodiment. The method 40 may be implemented as a set of logical instructions embodied on a machine- or computer-readable storage medium, such as random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), firmware, flash memory, etc., in configurable logic such as programmable logic arrays (PLAs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), in logic hardware with fixed functionality based on circuit technology such as application specific integrated circuits (ASICs), complementary metal oxide semiconductors (CMOS) or transistor-transistor logic (TTL) technology, or any combination thereof.For example, computer program code for performing operations illustrated in method 40 may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++, or the like, as well as conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0014] At processing block 42, an MMTS emulator may track information regarding the user's gaze to identify a point of interest. In this example, the MMTS emulator may be configured to use precision tracking to track the user's iris to an icon presented on a display screen that the user wishes to move (e.g., drag and drop) from left to right.

[0015] The user's gaze may be focused on a first point on the device's display (e.g., with coordinates (G)x1, (G)y1), with the icon initially placed on the display screen. When the user performs a gesture to indicate a desired movement from left to right, the user's gaze moves accordingly from the first point to the second point on the device's display (e.g., with coordinates (G)x2, (G)y2). The MMTS emulator may timestamp this gaze information and input it to a storage device for analysis.

[0016] At processing block 44, the MMTS emulator may continuously track information regarding the user's gestures to identify what the user intends to do. The tracked motion may be one-dimensional (1D) motion, two-dimensional (2D) motion, and three-dimensional (3D) motion, or any combination thereof. Thus, in this example, the user may indicate a desired movement of the icon by moving the fingertip toward the display (e.g., icon selection), performing a corresponding left-to-right movement of the fingertip (e.g., dragging the icon), and moving the fingertip away from the display (e.g., dropping the icon). The MMTS emulator may capture one or more images of the user's hand and perform object recognition and / or detection to identify both the hand and / or fingertip and these corresponding hand actions.The MMTS emulator can time-stamp the hand's actions and input them as gesture information into a storage device for analysis.

[0017] At processing block 46, the MMTS emulator may synchronize the user's gestures and gaze information to initiate actions of the device. Thus, in this example, the MMTS emulator may access the gaze information (i.e., that the user is focused on a particular icon and that the user's focus has moved from the first coordinate (G)x1, (G)y1 to the second coordinate (G)x2, (G)y2)) and synchronize it with the gesture information (i.e., hand actions). The MMTS emulator may synchronize these two pieces of information to determine that the icon should be selected (e.g., due to the detected movement of the fingertip toward the display while gaze is at (G)x1, (G)y1), moved away from a first point on the display screen to a second point on the display screen (e.g.,(due to the detected left-to-right movement of the fingertip) and release (due to the detected movement of the fingertip away from the screen while the gaze is at (G)x2, (G)y2). Accordingly, the MMTS emulator can emulate a touchscreen interface. By synchronizing these two types of information to determine the requested device action, the MMTS emulator can send a control signal to move the icon on the screen accordingly.

[0018] The sequence and numbering of the Fig. The blocks illustrated are not intended to prescribe a sequence of operations to the exclusion of other possibilities. Those skilled in the art will appreciate that the foregoing systems and methods are subject to various modifications, variations, and changes.

[0019] For example, in the embodiment described above, the MMTS emulator may continuously track a user's gesture and gaze information. However, this need not be the case. In another exemplary embodiment, the MMTS emulator may be configured to use the gesture information to "filter" the incoming eye information. This means, for example, that an MMTS emulator may be configured not to respond to incoming eye information until a gesture is detected. Upon detecting a gesture, an MMTS emulator (e.g., via an integration module) may then synchronize the incoming gaze and gesture information, determine the action to initiate, and transmit a control signal to perform the action.

[0020] Likewise, in another exemplary embodiment, the MMTS emulator may utilize gaze information to verify gesture information. Thus, for example, upon detecting and classifying a user gesture, the MMTS emulator may also verify the gesture prior to an action by ensuring that the user's gaze was focused on a relevant portion of the display screen. In another example, the MMTS emulator may verify the gesture prior to an action by verifying that the user's body distance was within a predefined or relevant distance.

[0021] Fig. is a block diagram of an exemplary device 50 according to one embodiment. In this example, the device 50 may include a camera configuration 100, an MMTS emulator 200, a display controller 300, and a display device 400.

[0022] The camera configuration 100 may include any device configured to capture an image and / or video feed of an object or image. Thus, in this example, the camera configuration 100 may include a first camera 101 for gesture tracking and a second camera 102 for gaze tracking. The first camera 101 and the second camera 102 may be arranged and calibrated according to a relevant field of view. For example, the first camera 101 may be arranged for a wide field of view configured to encompass almost the entire user's body and further calibrated to focus on the user's upper body. On the other hand, the second camera 102 may be arranged for a narrow field of view configured to focus on the user's head and further calibrated to focus on the user's eyes.The arrangement and calibration of the camera configuration 100 can be done manually or automatically (ie via a software application).

[0023] As discussed above, the MMTS emulator 200 may be controlled to, among other things, obtain gaze and gesture information from the camera configuration 100, analyze this information to determine a user-requested device action, and initiate the device action. The MMTS emulator 200 may include a camera interface 201, a gaze recognition module 202 (e.g., gaze recognition accelerator), a gesture recognition module 203 (e.g., gesture recognition accelerator), and an integration module 204.

[0024] The camera interface 201 may enable the MMTS emulator 200 to communicate with the camera configuration 100. Specifically, the camera interface 201 may receive video data (including a user's gaze and gesture information) sent from the camera configuration 100 and forward the video data to other components of the MMTS emulator 200 for analysis.

[0025] Gaze detection module 202 may be configured to receive, store, and analyze the user's gaze information to determine where the user's attention is directed. For example, as discussed above, gaze detection module 202 may determine the user's gaze point and map the user's gaze point to a specific location on a display screen.

[0026] Gesture recognition module 203 may be configured to receive, store, and analyze the user's gesture information to determine what the user intends to do. For example, as discussed above, gesture recognition module 203 may recognize a user's gesture by tracking the user's body movements (e.g., fingertip movements) and classify the body movement as a gesture and / or hand action.

[0027] The integration module 204 may receive user gaze information from the gaze recognition module 202 and user gesture information from the gesture recognition module 203 to initiate a device action and emulate a touchscreen interface (as discussed above). Thus, for example, the integration module 204 may synchronize the incoming gesture information and gaze information to determine that the user is requesting the movement of a particular icon from a first point to a second point.

[0028] In these embodiments of the present invention, the MMTS emulator 200 may be at least one of a hardware component, such as a circuit element (e.g., including transistors, resistors, capacitors, inductors, and the like), an integrated circuit, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), digital signal processors (DSP), field-programmable gate array (FPGA), logic gates, a firmware component, a register, semiconductor device, chips, microchips, chipsets, and so on. Furthermore, in other embodiments, the MMTS emulator 200 may also be implemented via a software application. In this embodiment, the MMTS emulator 200 may include a device driver in an input / output module.For example, the integration module 204 may utilize a mouse driver to cause actions of the device according to the aforementioned gaze information and gesture information.

[0029] The display device 400 may be an output device that presents electronic information in visual form. The display controller 300 may be a controller connected to the display device 400. Thus, for example, the display controller 300 may receive a control signal from the integration module 204 and perform a corresponding action of the device presented on the display device 400.

[0030] The sequence and numbering of the Fig. The blocks illustrated are not intended to dictate an order of operations to the exclusion of other possibilities. Those skilled in the art will appreciate that the foregoing systems and methods are subject to various modifications, variations, and changes. For example, in the embodiment described above, the MMTS emulator 200 may be coupled to the display controller 300, which may be coupled to a display device 400. The MMTS emulator 200 may send a control signal to perform a device action on the display device 400. However, this is not necessarily the case. In another example, an MMTS emulator may be coupled to another device through another device controller. For example, an MMTS emulator may be coupled to an input / output (I / O) device controller (e.g., a mouse controller), which may be coupled to an I / O device (e.g., a mouse).

[0031] Fig. 1501 is a block diagram of an exemplary system 1000 including an MMTS emulator 1501 according to one embodiment. Examples of system 1000 may include a laptop, desktop, personal digital assistant (PDA), media player, mobile internet device (MID), any smart device such as a smartphone, smart tablet, smart television, or the like. In this example, system 1000 may be a smartphone. The illustrated system 1000 includes a camera configuration 1100, a display device 1200, an I / O module 11500, a processing component 1400, system memory 1600, and mass storage 1700.

[0032] Similar to camera configuration 100 ( Fig. ), the camera configuration 1100 may include any device configured to capture an image or video feed of an object or image. In this example, the camera configuration 1100 may include a first camera 1101 and a second camera 1102. For example, the first camera 1101 may be used for gesture tracking, while the second camera 1102 may be used for gaze tracking.

[0033] Display device 400 may be an output device that presents electronic information in visual form. For example, a video captured by camera configuration 1100 may be presented on display device 1200 for viewing.

[0034] Processing component 1400 may include at least one computer processor that may be used to execute computer-readable executable instructions. The processor may include an integrated memory controller (IMC) 1401. Processor 1400 may communicate with system memory 1600 via IMC 1401.

[0035] The I / O module 1500 can be used to facilitate inputs (e.g., user inputs) and outputs (e.g., display commands) for the system 1000. The I / O module can include an MMTS emulator 1501, similar to the MMTS emulator 200 ( Fig. ), as already discussed.

[0036] System memory 1600 and mass storage 1700 may be any component or device capable of storing data. In this example, system memory 1600 may be configured to store, among other things, information related to the operation of system 1000. Mass storage 1700 may be used to store, for example, a user's personal data.

[0037] The sequence and numbering of the Fig. The blocks shown are not intended to prescribe a sequence of operations to the exclusion of other possibilities. Those skilled in the art will appreciate that the foregoing systems and methods are subject to various modifications, variations, and changes.

[0038] For example, in the embodiment described above, camera configuration 1100 may include a first camera 1101 and a second camera 1102. However, this need not be the case. An embodiment may include any number of cameras. Thus, for example, in another exemplary embodiment, a camera configuration may include a single camera. This camera may be positioned and calibrated to capture both gaze information and gesture information of a user.

[0039] Embodiments may therefore provide a method comprising receiving a first image from a first front-facing camera of a device having a front-facing display, identifying one or more eyes of a user of the device based on the first image, and determining a gaze position of one or more eyes on the front-facing display. The method may also include determining a change in gaze position, receiving a second image from a second front-facing camera of the device, identifying a point of interest on the front-facing display based on gaze information, wherein the gaze information includes a gaze position and the change in gaze position, and identifying a hand of the user based on the second image.The method may also include identifying one or more fingertips of the user's hand based on the second image and determining a movement of one or more of the hand and the one or more fingertips. Further, the method may include identifying an action of the hand based on gesture information, wherein the gesture information includes the movement, and the movement is one or more of a 1D movement, a 2D movement, and a 3D movement, and initiating an action of the device with respect to the front display based on the point of interest and the hand action.

[0040] In one example, the method may include classifying the movement as an action of the device, wherein the action of the device includes one or more of a single-click operation, a double-click operation, a press operation, and a cursor movement operation.

[0041] In another example, initiating the action of the device includes sending a request to a device driver, wherein the device driver comprises one or more of a mouse driver, a touchpad driver, and a trackball driver.

[0042] In yet another example, the method may include synchronizing the gaze information with the gesture information and detecting one or more of the gaze information and the gesture information.

[0043] Embodiments may also include a computer-readable storage medium comprising a set of instructions that, when executed by a processor, may cause a device to identify a point of interest on a front display of the device based on gaze information from a user of the device. The set of instructions may also cause a device to identify a hand action based on gesture information from the user of the device and initiate an action of the device with respect to the front display based on the point of interest and the hand action.

[0044] In one example, the instructions, when executed, cause the device to receive image data from a front-facing camera configuration of the device and to identify the gaze information and the gesture information based on the image data.

[0045] In another example, the instructions, when executed, cause the device to receive a first image from a first camera of the camera configuration and to receive a second image from a second camera of the camera configuration, wherein the image data includes the first image and the second image, and wherein the gaze information is to be determined based on the first image and the gesture information is to be determined based on the second image.

[0046] In yet another example, the instructions, when executed, cause the device to identify one or more eyes of the user, determine a gaze position of one or more eyes on the front display, and determine a change in gaze position, wherein the gaze information is to include the gaze position and the change in gaze position.

[0047] In yet another example, the instructions, when executed, cause the device to identify a hand of the user, identify one or more fingertips of the user's hand, and determine a movement of one or more of the hand and the one or more fingertips, wherein the gesture information is to include the movement, and the movement is to be one or more of a 1D movement, a 2D movement, and a 3D movement.

[0048] In one example, the instructions, when executed, cause the device to classify the movement as the action of the device, where the action of the device is to include one or more of a single-click operation, a double-click operation, a press operation, and a cursor movement operation.

[0049] In another example, the instructions, when executed, cause the device to send a request to a device driver to initiate the action of the device, where the device driver is to include one or more of a mouse driver, a touchpad driver, and a trackball driver.

[0050] In yet another example, the instructions, when executed, cause the device to synchronize the gaze information with the gesture information and to capture one or more of the gaze information and the gesture information.

[0051] Yet another embodiment may provide a device comprising a gaze detection module to identify a point of interest on a front display of a device based on gaze information of the user, a gesture detection module to identify a hand action based on gesture information of the user of the device, and an integration module to initiate an action of the device with respect to the front display based on the point of interest and the hand action.

[0052] In one example, the device may include a camera interface to receive image data from a front camera configuration of the device, wherein the gaze recognition module is to identify the gaze information based on the image data and the gesture recognition module is to identify the gesture information based on the image data.

[0053] In one example, the camera interface is to receive a first image from a first camera of the camera configuration and a second image from a second camera of the camera configuration, wherein the image data is to include the first image and the second image, the gaze recognition module is to determine the gaze information based on the first image, and the gesture recognition module is to determine the gesture information based on the second image.

[0054] In one example, the gaze detection module is to identify one or more eyes of the user, determine a gaze position of one or more eyes on the front display, and determine a change in gaze position, wherein the gaze information is to include the gaze position and the change in gaze position.

[0055] In another example, the gesture recognition module is to identify a hand of the user, identify one or more fingertips of the user's hand, and determine a movement of one or more of the hand and the one or more fingertips, wherein the gesture information is to include the movement, and the movement is to be one or more of an ID movement, a 2D movement, and a 3D movement.

[0056] In yet another example, the integration module is to classify the movement as an action of the hand, wherein the action of the device is to include one or more of a single-click operation, a double-click operation, a press operation, and a cursor movement operation.

[0057] In yet another example, the integration module is to send a request to a device driver to initiate the action of the device, wherein the device driver is to include one or more of a mouse driver, a touchpad driver, and a trackball driver.

[0058] In one example, the integration module is to synchronize the gaze information with the gesture information and to capture one or more of the gaze information and the gesture information.

[0059] Another embodiment may provide a device comprising a front-end display and a touchscreen emulator. The touchscreen emulator may include a gaze detection module for identifying a point of interest on the front-end display based on gaze information from a user of the device, a gesture detection module for identifying a hand action based on gesture information from a user of the device, and an integration module for initiating an action of the device with respect to the front-end display based on the point of interest and the hand action.

[0060] In one example, the device may include a front-facing camera configuration, wherein the touch screen emulator further includes a camera interface to receive image data from the front-facing camera configuration, the gaze recognition module is to identify the gaze information based on the image data, and the gesture recognition module is to identify the gesture information based on the image data.

[0061] In another example, the camera configuration includes a first camera and a second camera, wherein the camera interface is to receive a first image from the first camera of the camera configuration. The camera interface is to also receive a second image from the second camera of the camera configuration, wherein the image data is to include the first image and the second image, the gaze recognition module is to determine the gaze information based on the first image, and the gesture recognition module is to determine the gesture information based on the second image.

[0062] In one example, the gaze detection module is to identify one or more eyes of the user, determine a gaze position of the one or more eyes on the front display, and determine a change in gaze position, wherein the gaze information is to include the gaze position and the change in gaze position.

[0063] In another example, the gesture recognition module of the device is to identify a hand of the user, identify one or more fingertips of the user's hand, and identify a movement of one or more of the hand and the one or more fingertips, wherein the gesture information is to include the movement, and the movement is to be one or more of a 1D movement, a 2D movement, and a 3D movement.

[0064] In yet another example, the device integration module is to classify the movement as the device action, wherein the device action is to include one or more of a single-click operation, a double-click operation, a press operation, and a cursor movement operation.

[0065] In one example, the device integration module is to send a request to a device driver to initiate the device action, where the device driver is to include one or more of a mouse driver, a touchpad driver, and a trackball driver.

[0066] In yet another example, the device integration module is to synchronize the gaze information with the gesture information and to capture one or more of the gaze information and the gesture information.

[0067] Another embodiment may therefore provide a method that includes identifying a point of interest on a front display of the device based on gaze information from a user of the device and identifying a hand action based on gesture information from the user of the device. The method may also include initiating an action of the device with respect to the front display based on the point of interest and the hand action.

[0068] In one example, the method includes receiving image data from a front camera configuration of the device and identifying the gaze information and the gesture information based on the image data.

[0069] In another example, the method includes receiving a first image from a first camera of the camera configuration and receiving a second image from a second camera of the camera configuration. The image data should include the first image and the second image, the gaze information should be determined based on the first image, and the gesture information should be determined based on the second image.

[0070] In yet another example, the method includes identifying one or more eyes of the user, determining a gaze position of the one or more eyes on the front display, and determining a change in gaze position, wherein the gaze information is to include the gaze position and the change in gaze position.

[0071] In yet another example, the method includes identifying a hand of the user, identifying one or more fingertips of the user's hand, and determining a movement of one or more of the hand and the one or more fingertips. In this example, the gesture information is to include the movement, and the movement is to be one or more of a one-dimensional (1D) movement, a two-dimensional (2D) movement, and a three-dimensional (3D) movement.

[0072] In another example, the method includes classifying the movement as the action of the device, wherein the action of the device is to include one or more of a single-click operation, a double-click operation, a press operation, and a cursor movement operation.

[0073] In one example, the method comprises sending a request to a device driver to initiate the action of the device, wherein the device driver is to include one or more of a mouse driver, a touchpad driver, and a trackball driver.

[0074] In another example, the method comprises synchronizing the gaze information with the gesture information and classifying one or more of the gaze information and the gesture information.

[0075] Various embodiments may be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include processors, microprocessors, circuits, switching elements (e.g., transistors, resistors, capacitors, inductors, and so on), integrated circuits, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), logic gates, registers, semiconductor devices, chips, microchips, chipsets, and so on.Software examples may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (APIs), instruction sets, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an embodiment is implemented using hardware elements and / or software elements may vary according to any number of factors, such as desired computational speed, power, thermal tolerances, processing cycle budget, input data rate, output data rate, memory resources, data bus speeds, and other design or performance constraints.

[0076] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium representing various logic in the processor that, when read by a machine, causes the machine to produce logic to perform the methods described in this patent application. Such representations, known as "IP cores," may be stored on a tangible, machine-readable medium and provided to various customers or manufacturing facilities for input into the manufacturing machines that produce the logic or processor.

[0077] Embodiments of the present invention are applicable to all types of semiconductor integrated circuit (IC) chips. Examples of these IC chips include, but are not limited to, processors, controllers, chipset components, programmable logic arrays (PLAs), memory chips, network chips, and the like. Furthermore, in some drawings, signal traces are represented by lines. Some may be different to indicate multiple individual signal paths, may be numbered to indicate a number of individual signal paths, and / or may have arrows at one or more ends to indicate the primary direction of information flow. However, this should not be construed in a limiting manner. Rather, such additional detail may be used in the context of one or more example embodiments to make a circuit more understandable.All signal lines shown, whether with or without additional information, may comprise one or more signals propagating in multiple directions and may be implemented with any suitable type of signaling scheme, e.g., digital or analog lines realized with differential pairs, fiber optic lines, and / or single-ended lines.

[0078] Examples of sizes / models / values / ranges may have been provided, although the embodiments are not limited to them. As manufacturing techniques (e.g., photolithography) become more sophisticated over time, it is expected that smaller devices could be manufactured. Furthermore, well-known power / ground connections to IC chips and other components may not be shown in the figures for simplicity of illustration and discussion, as well as to avoid obscuring the understanding of certain aspects of the embodiments.Furthermore, arrangements may be presented in block diagram form to avoid obscuring the clarity of embodiments, and also in view of the fact that specific features relating to the implementation of such arrangements of block diagrams are highly dependent on the platform in which the embodiment is to be implemented, i.e., such specific features should be within the knowledge of a person skilled in the art. Where specific details (e.g., circuits) are set forth to describe example embodiments, it should be apparent to those skilled in the art that embodiments may be practiced without, or with modifications to, these specific details. Therefore, the description is to be considered illustrative and not restrictive.

[0079] For example, some embodiments may be implemented using a machine-readable or tangible computer-readable medium or product that can store an instruction or set of instructions that, when executed by a machine, can cause the machine to perform a method and / or operations according to the embodiments. Such a machine may, for example, include any suitable processing platform, computing platform, computing device, processing device, computing system, processing system, computer, processor, or the like, and may be implemented using any suitable combination of hardware and / or software.The machine-readable medium or product may, for example, be any suitable type of memory unit, memory device, memory product, memory medium, data storage device, data storage product, data storage medium and / or data storage unit, for example memory, removable or non-removable media, erasable or non-erasable media, writable or rewritable media, digital or analog media, hard disks, floppy disks, CD-ROMs (Compact Disk Read-Only Memories), CD-Rs (Compact Disks Recordable), CD-RWs (Compact Disks Rewritable), optical disks, magnetic media, magneto-optical media, removable memory cards or disks, various types of DVDs (Digital Versatile Disks), tapes, cassettes or the like.The instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, encrypted code, and the like, implemented using any suitable high-level, low-level, object-oriented, visual, compiled and / or interpreted programming language.

[0080] Unless expressly stated otherwise, it is understood that terms such as "processing," "computing," "calculating," "determining," or the like refer to the action and / or processes of a computer or computing system, or similar electronic data processing device, that manipulates data represented as physical quantities (e.g., electronic) within the registers and / or memories of the computing system and / or converts it into other data similarly represented as physical quantities within the memories, registers, or other such information storage, transmission, or display devices of the computing system. The embodiments are not limited in this context.

[0081] The term "coupled" may be used herein to refer to any type of direct or indirect relationship between the components in question and may apply to electrical, mechanical, fluid, optical, electromagnetic, electromechanical, or other connections. Furthermore, the terms "first" and "second" are used herein only to facilitate discussion and have no particular temporal or chronological significance unless otherwise specified.

[0082] Those skilled in the art will appreciate from the foregoing description that the broad methods of the embodiments of the present invention can be implemented in a variety of forms. Therefore, although the embodiments of this invention have been described in connection with specific examples thereof, the true scope of the embodiments of the invention should not be limited thereto, as other modifications will become apparent to those skilled in the art upon examination of the drawings, specifications, and the following claims.

Claims

[1] Method (40) comprising: Receiving a first image from a first front camera (102, 1102) of a device (30, 50, 1000) having a front display (400, 1200); Identifying one or more eyes of a user (10) of the device (30, 50, 1000) based on the first image; Determining a gaze position (12) of the one or more eyes on the front display (400, 1200); Determining a change in gaze position (12); Receiving a second image from a second front camera (101, 1101) of the device (30, 50, 1000); Identifying (42) a point of interest on the front display (400, 1200) based on gaze information, the gaze information including the gaze position (12) and the change in the gaze position (12); identifying a hand of the user (10) based on the second image; Identifying one or more fingertips (20) of the user's hand (10) based on the second image; Determining a movement of one or more of the hand and the one or more fingertips (20); identifying (44) an action of the hand based on gesture information, wherein the gesture information is to include the movement, and the movement is to be one or more of a one-dimensional, 1D, movement, a two-dimensional, 2D, movement, and a three-dimensional, 3D, movement; and initiating (46) an action of the device (30, 50, 1000) with respect to the front display (400, 1200) based on the point of interest and the action of the hand, the method further comprising: Synchronizing gaze information with gesture information; and Capturing one or more of the gaze information and the gesture information. [2] The method (40) of claim 1, further comprising classifying the movement as an action of the device (30, 50, 1000), wherein the action of the device (30, 50, 1000) includes one or more of a single click operation, a double click operation, a press operation, and a cursor movement operation. [3] The method (40) of claim 1, wherein initiating the action of the device (30, 50, 1000) includes sending a request to a device driver, the device driver comprising one or more of a mouse driver, a touchpad driver, and a trackball driver. [4] A computer-readable storage medium comprising a set of instructions that, when executed by a processor (1400), cause a device (30, 50, 1000): identifies a point of interest (14) on a front display (400, 1200) of the device (30, 50, 1000) based on gaze information of a user (10) of the device (30, 50, 1000); identifies an action of a hand based on gesture information of the user (10) of the device (30, 50, 1000); initiates an action of the device (30, 50, 1000) with respect to the front display (400, 1200) based on the point of interest (14) and the action of the hand, wherein the instructions, when executed, further cause the device (30, 50, 1000) identifies the hand of the user (10); one or more fingertips (20) of the user's hand (10) are identified; determines a movement of one or more of the hand and the one or more fingertips (20), wherein the gesture information is to include the movement and the movement is to be one or more of a one-dimensional, 1D, movement, a two-dimensional, 2D, movement and a three-dimensional, 3D, movement, the gaze information is synchronized with the gesture information; and one or more of the gaze information and the gesture information is captured. [5] The computer-readable storage medium of claim 4, wherein the instructions, when executed, cause the device (30, 50, 1000): receives image data from a front camera configuration (100, 1100) of the device (30, 50, 1000); and identifies gaze information and gesture information based on the image data. [6] The computer-readable storage medium of claim 5, wherein the instructions, when executed, cause the device (30, 50, 1000): receives a first image from a first camera (102, 1102) of the front camera configuration (100, 1100); and receives a second image from a second camera (101, 1101) of the front camera configuration (100, 1100), wherein the image data is to include the first image and the second image, the gaze information is to be determined based on the first image, and the gesture information is to be determined based on the second image. [7] The computer-readable storage medium of claim 4, wherein the instructions, when executed, cause the device (30, 50, 1000): one or more eyes of the user (10) are identified; a gaze position (12) of the one or more eyes on the front display (400, 1200) is determined; and a change in the gaze position (12) is determined, wherein the gaze information should include the gaze position (12) and the change in the gaze position (12). [8] The computer-readable storage medium of claim 4, wherein the instructions, when executed, cause the device (30, 50, 1000) to classify the movement as the action of the device (30, 50, 1000), wherein the action of the device (30, 50, 1000) is to include one or more of a single-click operation, a double-click operation, a press operation, and a cursor movement operation. [9] The computer-readable storage medium of claim 4, wherein the instructions, when executed, cause the device (30, 50, 1000) to send a request to a device driver to initiate the action of the device (30, 50, 1000), wherein the device driver is to include one or more of a mouse driver, a touchpad driver, and a trackball driver. [10] Device (30, 50, 1000) comprising: a gaze detection module (202) for identifying a point of interest (14) on a front display of a device (30, 50, 1000) based on gaze information of a user (10) of the device (30, 50, 1000); a gesture recognition module (203) for identifying an action of a hand based on gesture information of the user (10) of the device (30, 50, 1000); and an integration module (204) for initiating an action of the device (30, 50, 1000) with respect to the front display based on the point of interest (14) and the action of the hand, where the integration module (204): to synchronize the gaze information with the gesture information, and one or more of the gaze information and the gesture information. [11] The device (30, 50, 1000) of claim 10, further comprising a camera interface (201) for receiving image data from a front camera configuration (100, 1100) of the device (30, 50, 1000), wherein the gaze recognition module (202) is to identify the gaze information based on the image data and the gesture recognition module (203) is to identify the gesture information based on the image data. [12] The device (30, 50, 1000) of claim 11, wherein the camera interface (201): to receive a first image from a first camera (102, 1102) of the front camera configuration (100, 1100), and a second image is to be received from a second camera (101, 1101) of the front camera configuration (100, 1100), wherein the image data is to include the first image and the second image, the gaze recognition module (202) is to determine the gaze information based on the first image, and the gesture recognition module (203) is to determine the gesture information based on the second image. [13] The device (30, 50, 1000) of claim 10, wherein the gaze detection module (202): to identify one or more eyes of the user (10), to determine a gaze position (12) of one or more eyes on the front display (400, 1200), and a change in the gaze position (12) is to be determined, wherein the gaze information is to include the gaze position (12) and the change in the gaze position (12). [14] The device (30, 50, 1000) of claim 10, wherein the gesture recognition module (203): to identify the hand of the user (10), to identify one or more fingertips (20) of the user's hand (10), and to determine a movement of one or more of the hand and the one or more fingertips (20), wherein the gesture information is to include the movement, and the movement is to be one or more of a one-dimensional, 1D, movement, a two-dimensional, (2D, movement, and a three-dimensional, 3D, movement. [15] The device (30, 50, 1000) of claim 14, wherein the integration module (204) is to classify the movement as the action of the device (30, 50, 1000), wherein the action of the device (30, 50, 1000) is to include one or more of a single-click operation, a double-click operation, a press operation, and a cursor movement operation. [16] The device (30, 50, 1000) of claim 10, wherein the integration module (204) is to send a request to a device driver to initiate the action of the device (30, 50, 1000), the device driver including one or more of a mouse driver, a touchpad driver, and a trackball driver. [17] Device (30, 50, 1000) comprising: a front display (400, 1200); and a touch screen emulator (200) comprising: a gaze detection module (202) for identifying a point of interest (14) on the front display (400, 1200) based on gaze information of a user (10) of the device (30, 50, 1000), a gesture recognition module (203) for identifying an action of a hand based on gesture information of a user (10) of the device (30, 50, 1000), and an integration module (204) for initiating an action of the device (30, 50, 1000) with respect to the front display (400, 1200) based on the point of interest (14) and the action of the hand, where the integration module (204): to synchronize the gaze information with the gesture information, and one or more of the gaze information and the gesture information. [18] The device (30, 50, 1000) of claim 17, further comprising a front camera configuration (100, 1100), wherein the touch screen emulator (200) further includes a camera interface (201) to receive image data from the front camera configuration (100, 1100), the gaze recognition module (202) is to identify the gaze information based on the image data, and the gesture recognition module (203) is to identify the gesture information based on the image data. [19] The device (30, 50, 1000) of claim 18, wherein the front camera configuration (100, 1100) comprises a first camera (102, 1102) and a second camera (101, 1101), and wherein the camera interface (201) to receive a first image from the first camera (102, 1102) of the front camera configuration (100, 1100), and receive a second image from the second camera of the front camera configuration (100, 1100), wherein the image data includes the first image and the second image, the gaze recognition module (202) determines the gaze information based on the first image, and the gesture recognition module determines the gesture information based on the second image. [20] Device (30, 50, 1000) of claim 17, wherein the gaze detection module (202) to identify one or more eyes of the user (10), to determine a gaze position (12) of one or more eyes on the front display (400, 1200) and a change in the gaze position (12) is to be determined, wherein the gaze information is to include the gaze position (12) and the change in the gaze position (12). [21] Device (30, 50, 1000) of claim 17, wherein the gesture recognition module (203) to identify a hand of the user (10), to identify one or more fingertips (20) of the user’s hand (10) and to identify a movement of one or more of the hand and the one or more fingertips (20), wherein the gesture information is to include the movement, and the movement is to be one or more of a one-dimensional, 1D, movement, a two-dimensional, 2D, movement, and a three-dimensional, 3D, movement. [22] The device (30, 50, 1000) of claim 21, wherein the integration module (204) is to classify the movement as an action of the device (30, 50, 1000), wherein the action of the device (30, 50, 1000) is to include one or more of a single-click operation, a double-click operation, a press operation, and a cursor movement operation. [23] The device (30, 50, 1000) of claim 17, wherein the integration module (204) is to send a request to a device driver to initiate the action of the device (30, 50, 1000), the device driver being one or more of a mouse driver, a touchpad driver, and a trackball driver.

Images