Eye gaze tracking

Abstract

The present disclosure relates to techniques for detecting and tracking eye gaze. An apparatus includes a visible wavelength camera, an infrared (IR) camera, and one or more processors. The one or more processors are configured to generate a three-dimensional (3D) point cloud of a face based on IR data captured from the IR camera, generate a two-dimensional image of the face based on visible wavelength data captured from the visible wavelength camera, and detect a symmetry plane of the face based on the 3D point cloud and the two-dimensional image. The symmetry plane divides the 3D point cloud into two parts. The one or more processors are also configured to reconstruct the 3D point cloud based on the symmetry plane, and track eye gaze of the face based on the reconstructed 3D point cloud.

Description

Technical Field

[0001] The present disclosure relates generally to eye gaze tracking. Background Art

[0002] Bilaterally symmetrical objects are symmetrical about a plane of symmetry. People are bilaterally symmetrical about the sagittal plane. The sagittal plane divides a person into a symmetrical left and right side. A bilaterally symmetrical object can be a whole person or a part of a person, such as a face. In the case of a person, the sagittal plane is a plane of symmetry. Inanimate objects, such as a ball or a drinking glass, can also be bilaterally symmetrical. Bilaterally symmetrical objects do not need to be completely symmetrical about a plane of symmetry. For example, most people have some differences between the left and right sides of their faces. Summary of the invention

[0003] According to one aspect of the present disclosure, a device for eye gaze tracking is provided. The device includes: a visible wavelength camera, an infrared (IR) camera; and one or more processors. The one or more processors are configured to generate a three-dimensional (3D) point cloud of a face based on IR data captured from the IR camera. The one or more processors are configured to generate a two-dimensional image of the face based on the visible wavelength data captured from the visible wavelength camera. The one or more processors are configured to detect a symmetry plane of the face based on the 3D point cloud and the two-dimensional image. The symmetry plane divides the 3D point cloud into two parts. The one or more processors are configured to reconstruct the 3D point cloud based on the symmetry plane. The one or more processors are configured to track the eye gaze of the face based on the reconstructed 3D point cloud.

[0004] Optionally, in any of the foregoing aspects, the one or more processors are further configured to solve a cost function to detect the symmetry plane, the cost function comprising a first parameter providing a first weight for a first symmetrical half of the face and a second parameter providing a second weight for a second symmetrical half of the face.

[0005] Optionally, in any of the aforementioned aspects, the device also includes a display screen, and the one or more processors are further configured to determine a position on the display screen that the person is looking at based on the reconstructed 3D point cloud.

[0006] Optionally, in any of the preceding aspects, the one or more processors are further configured to automatically scroll the content of the display screen in response to a position on the display screen that the person is gazing at that is within a predefined area of ​​the display screen.

[0007] Optionally, in any of the preceding aspects, the one or more processors are further configured to change the appearance of the element on the display screen in response to detecting that the person is looking at the element.

[0008] Optionally, in any of the preceding aspects, the one or more processors are further configured to predict the next location of the person's gaze based on the output of a sequence of long short-term memory (LSTM) units.

[0009] Optionally, in any of the aforementioned aspects, the one or more processors are further configured to integrate image registration and image transformation into each of the LSTM units.

[0010] Optionally, in any of the preceding aspects, the one or more processors are further configured to register the reconstructed 3D point cloud of the face to another 3D point cloud of the face using a long short-term memory (LSTM) unit.

[0011] Optionally, in the aforementioned aspect, the other 3D point cloud is a standard 3D point cloud captured by the person looking at a known position.

[0012] Optionally, in any of the aforementioned aspects, the reconstructed 3D point cloud is a first reconstructed 3D point cloud captured at a first time point, and the other 3D point cloud is a second reconstructed 3D point cloud captured at a second time point before the first time point.

[0013] Optionally, in any of the aforementioned aspects, the one or more processors are further configured to transform the reconstructed 3D point cloud of the face into another 3D point cloud of the face using a long short-term memory (LSTM) unit.

[0014] Optionally, in any of the foregoing aspects, the face comprises a human face, and the one or more processors are further configured to determine corresponding point pairs between a reconstructed 3D point cloud of the face and a standard 3D point cloud of the face using a long short-term memory (LSTM) unit. The standard 3D point cloud is captured by the person looking at a known position. The one or more processors are further configured to measure differences between the corresponding point pairs using the LSTM unit.

[0015] Optionally, in the aforementioned aspect, the one or more processors are further configured to perform facial recognition on the person based on differences between the corresponding point pairs.

[0016] Optionally, in any of the foregoing aspects, the one or more processors are further configured to determine the corresponding point pairs between a first version of the reconstructed 3D point cloud of the face captured at a first point in time and a second version of the reconstructed 3D point cloud of the face captured at a second point in time, including determining the corresponding pairs using a long short-term memory (LSTM) unit. The one or more processors are further configured to measure differences between the corresponding point pairs using the LSTM unit.

[0017] Optionally, in the aforementioned aspect, the human face includes a human face, and the one or more processors are further configured to determine the direction of the person's eye gaze based on the difference between the corresponding point pairs.

[0018] Optionally, in the aforementioned aspect, the one or more processors are also configured to: generate a three-dimensional (3D) point cloud of the bilaterally symmetrical object based on the IR data captured from the IR camera; generate a two-dimensional image of the bilaterally symmetrical object based on the visible wavelength data captured from the visible wavelength camera; detect a symmetry plane of the bilaterally symmetrical object based on the 3D point cloud of the bilaterally symmetrical object and the two-dimensional image of the bilaterally symmetrical object, the symmetry plane dividing the 3D point cloud into two parts; and reconstruct the 3D point cloud of the bilaterally symmetrical object based on the symmetry plane.

[0019] Optionally, in the aforementioned aspect, the one or more processors are further configured to derive color information for voxels in the 3D point cloud based on the two-dimensional image.

[0020] A further aspect includes a method for eye gaze tracking. The method includes generating a three-dimensional (3D) point cloud of a face based on IR data captured from an infrared (IR) camera. The method includes generating a two-dimensional image of the face based on visible wavelength data captured from a visible wavelength camera. The method includes detecting a symmetry plane of the face based on the 3D point cloud and the two-dimensional image, the symmetry plane dividing the 3D point cloud into two parts. The method includes reconstructing the 3D point cloud based on the symmetry plane. The method includes tracking eye gaze of the face based on the reconstructed 3D point cloud.

[0021] Further aspects include an electronic device for detecting eye gaze. The electronic device includes at least one camera and one or more processors. The one or more processors are configured to generate a face image of a person based on data captured from the at least one camera; and input the data into a long short-term memory (LSTM) unit to determine the direction of the person's gaze.

[0022] This summary is provided to introduce in a simplified form some concepts that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in the background technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The aspects of the present disclosure are illustrated by way of example and not limitation in the accompanying figures, in which like references indicate elements.

[0024] Figure 1 is a diagram of an embodiment of an electronic device for eye gaze tracking.

[0025] Figure 2 is a flow chart of one embodiment of a process for performing image processing on a bilaterally symmetrical object.

[0026] Figure 3 is a flow chart of an embodiment of a process for performing image processing on a human face based on reconstructing a 3D point cloud.

[0027] Figure 4 is a flow chart of one embodiment of a process for performing an action based on the location on a display screen that a person is looking at.

[0028] Figure 5 is a flow chart of one embodiment of a process for determining a location on a display screen at which a person is looking.

[0029] Figure 6 A human head and various geometric details are shown at two points in time.

[0030] Figure 7 is a flow chart of an embodiment of determining standard data, which may be used in the process.

[0031] Figure 8 Embodiments of systems configured to perform eye gaze detection are described.

[0032] Fig. 9 is a flowchart of one embodiment of a process for predicting the next eye gaze point by using an LSTM unit.

[0033] Fig.10 One embodiment of an LSTM cell is depicted.

[0034] Fig.11 is a flow chart of one embodiment of a process for an electronic device to provide person-selectable options on a display screen in response to detecting a location on the screen at which the person is looking.

[0035] Fig. 12A A display screen is shown having a first option and a second option.

[0036] Fig. 12B Shows the change Fig. 12A An example of what the first option looks like on a display of .

[0037] Fig.13 is a flow chart of one embodiment of a process for performing an action in response to detecting eye gaze of a person. DETAILED DESCRIPTION

[0038] The present disclosure will be described with reference to the accompanying drawings, Figure 1Generally, it relates to an apparatus and method for image processing of bilaterally symmetrical objects. In some embodiments, a three-dimensional (3D) point cloud is constructed to assist in image processing of bilaterally symmetrical objects. The 3D point cloud can be generated based on IR data captured by an infrared (IR) camera. A two-dimensional (2D) visible wavelength image can also be used to assist in image processing. For example, a 3D point cloud of a face is combined with a 2D visible wavelength image of the face to perform facial recognition or eye tracking.

[0039] One technical challenge is that because the face is not pointed directly at the IR camera, the 3D point cloud may have "missing data". Also, depending on the environment in which the image was taken, the colors on the left and right sides of the face may be very different, primarily because of the light projected on the face. In one embodiment, a "symmetry plane" of the face (or other bilaterally symmetrical object) is detected. The 3D point cloud is then reconstructed based on the symmetry plane. In one embodiment, the missing data in the first portion of the 3D point cloud corresponding to the first half of the bilaterally symmetrical object is filled in based on 3D point cloud data from the second portion of the 3D point cloud corresponding to the second half of the bilaterally symmetrical object. In one embodiment, the symmetry plane is detected by solving (e.g., minimizing) a cost function. In one embodiment, a variational method is used to find the symmetry plane. The techniques disclosed herein for locating the symmetry plane are computationally efficient. In an embodiment, the symmetry plane can be detected without texture information.

[0040] A technical challenge is to effectively determine the eye gaze of a person. In one embodiment, data is input into one or more long short term memory (LSTM) units to detect the direction of eye gaze. In an embodiment, LSTM is a machine learning method that learns long-term dependencies through a sequence of short-term data. Please note that long-term is relative to short-term. For example, long-term may be only a few seconds or even shorter. In an embodiment, the output of the sequence of LSTM units is used to determine the prediction of the next eye gaze point. In an embodiment, this prediction is the next gaze in the sequence of saccadic movements.

[0041] In one embodiment, the LSTM unit is used to perform registration and / or transformation of input data to perform facial recognition and / or eye tracking. In some embodiments, the LSTM unit is used to perform image registration and / or image transformation. As the term "image registration" used herein, image registration is an image processing technique for finding corresponding points in two data sets (e.g., two 3D point clouds). As the term "image transformation" used herein, image transformation is an image processing technique for placing two data sets (e.g., two 3D point clouds) in a coordinate system. In an embodiment, the image transformation measures the difference between the two data sets. Therefore, the term "image registration" defined herein does not require that the two data sets be placed in a coordinate system.

[0042] For example, the LSTM unit can be used to register the above-mentioned reconstructed 3D point cloud to a standard 3D point cloud. In an embodiment, a standard 3D point cloud is generated when a person is looking at a known position (e.g., a point on a display screen). For example, the LSTM unit can be used to determine corresponding points in two 3D point clouds. For another example, the LSTM unit can be used to register the above-mentioned reconstructed 3D point cloud to a standard 3D point cloud. For example, the LSTM unit can be used to measure the difference between the above-mentioned corresponding points in the two 3D point clouds. For another example, the LSTM unit can be used to register the 3D point cloud of the user's face from a time period to the 3D point cloud of the user's face in the previous time period. For another example, the LSTM unit can be used to transform the 3D point cloud of the user's face from a time period into the 3D point cloud of the user's face from the previous time period. Various registrations and / or transformations can be used to perform eye tracking. An embodiment of an LSTM unit that integrates registration and transformation to perform eye tracking is computationally efficient. Therefore, time is saved and battery power is also saved in some cases.

[0043] It should be understood that the embodiments of the present disclosure can be implemented in many different forms, and the scope of the claims should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure thorough and complete, and the embodiments of the present invention are conceived to be fully conveyed to those skilled in the art. In fact, the present disclosure is intended to cover modifications, alternatives and equivalents of all embodiments within the scope and spirit of the present disclosure defined by the appended claims. In addition, in the following detailed description of the embodiments of the present disclosure, a large number of specific details are proposed, the purpose of which is to provide a thorough understanding. However, it is obvious to those of ordinary skill in the art that the embodiments of the present disclosure can be practiced without these specific details.

[0044] Figure 1 is a diagram of an embodiment of an electronic device 100 for eye gaze tracking. The electronic device 100 has an image capture device 104 including an infrared (IR) camera 104A and a visible wavelength camera 104B. The image capture device 104 is capable of capturing an image of a person's head and / or face 120. The electronic device 100 also has a display screen 102, one or more processors 108, and a memory 116. In one embodiment, the bilaterally symmetrical object is a human face. In one embodiment, the electronic device 100 is configured to perform facial recognition. In one embodiment, the electronic device 100 is configured to perform eye tracking. Note that there can be more than one infrared camera and / or more than one visible wavelength camera.

[0045] In one embodiment, electronic device 100 is a wireless communication device. Electronic device 100 may be, for example, a mobile phone, but in other examples may be other devices such as a desktop computer, a laptop computer, a tablet computer, a handheld computing device, an automotive computing device, and / or other computing devices.

[0046] The infrared (IR) camera 104A includes an image sensor 104A1 (e.g., a complementary metal-oxide semiconductor (CMOS) image sensor, a charge coupled device (CCD) sensor) that can be used to capture infrared data. The IR camera 104A also includes an IR emitter 104A2 capable of emitting IR light. The IR emitter 104A2 can be used to project IR light onto a person's face. In one embodiment, infrared data is generally referred to as "near infrared". Near infrared has a wavelength close to visible light. For example, the infrared (IR) camera 104A can be configured to capture electromagnetic radiation having a wavelength from about 700 nanometers (nm) to about 1400nm. The IR data does not need to occupy the entire range of 700 to 1400nm. The infrared (IR) camera 104A can sense some electromagnetic radiation having a wavelength below 700nm or above 1400nm. The infrared data can be used to generate a depth image. In some embodiments, a 3D point cloud is generated based on the infrared data. A variety of techniques can be used, such as structured light and / or time-of-flight (TOF) sensors. 3D point clouds can also be generated or enhanced with images from visible wavelength cameras.

[0047] Visible wavelength camera 104B includes an image sensor (e.g., CMOS image sensor, CCD sensor) that can be used to capture visible wavelength images. In an embodiment, the visible wavelength image is a digital image. Visible wavelength camera 104B captures 2D visible wavelength images. In an embodiment, visible wavelength camera 104B captures 2D color (e.g., RGB) images. "Visible wavelength" is also called "visible spectrum" and refers to the portion of the electromagnetic spectrum that is visible to the human eye. In an embodiment, visible wavelength camera 104B can sense electromagnetic radiation in a band between approximately 380nm and 740nm. Visible wavelength camera 104B does not need to sense electromagnetic radiation spanning the entire range from 380nm to 740nm. In some cases, visible wavelength camera 104B is capable of sensing electromagnetic radiation below 380nm or above 740nm. There may be some overlap between the wavelengths of electromagnetic radiation captured by IR camera 104A and visible wavelength camera 104B. There may be a gap between the wavelengths of electromagnetic radiation captured by IR camera 104A and visible wavelength camera 104B.

[0048] Display screen 102 can be used as a user interface to visually display items (e.g., images, options, instructions, etc.) to a user. Display screen 102 can optionally accept input from a user. In one embodiment, display screen 102 is a touch screen display. In one embodiment, electronic device 100 is configured to determine a location on display screen 102 that a user is looking at.

[0049] In an embodiment, the processor 108 is capable of performing image processing of bilaterally symmetrical objects. Image processing may include, but is not limited to, facial recognition, eye tracking, and prediction of the next eye gaze point. The processor 108 may be used to control various other elements of the electronic device 100, for example, under the control of software and / or firmware stored in the memory 116. There may also be multiple processors 108, such as a central processing unit (CPU) and a graphics processing unit (GPU). The processor 108 can execute computer instructions (stored in a non-temporary computer-readable medium) to enable the processor to perform steps for implementing the embodiments of the present technology described herein. The processor 108 is depicted as having several components (or modules) 108A, 108B, 108C, 108D. In one embodiment, each component is implemented by executing an instruction set (accessed from the memory 116) on the processor 108. The processing unit 108 may include a microprocessor, a microcontroller, a digital signal processor, a field programmable gate array, or an application-specific integrated circuit.

[0050] The symmetry plane detector 108A module is configured to detect the symmetry plane of a bilaterally symmetrical object (e.g., a human face). Figure 1 In the embodiment of the present invention, the symmetry plane detector 108A can detect the symmetry plane 130 of the face 120. The bilaterally symmetrical object can be an inanimate object, such as a ball or a drinking glass. In some embodiments, the symmetry plane detector 108A solves a cost function to detect the symmetry plane. In one embodiment, the symmetry plane detector 108A uses a 3D point cloud and an RGB 2D image of the bilaterally symmetrical object. The technology disclosed herein is very computationally efficient.

[0051] The 3D point cloud constructor 108B is configured to construct and reconstruct a 3D point cloud. The 3D point cloud contains points in any 3D coordinate system. In one embodiment, the coordinate system is a Cartesian (x, y, z) coordinate system; however, another coordinate system may be used. In an embodiment, each point in the 3D point cloud includes an intensity value. In some embodiments, points in the 3D point cloud are assigned color (e.g., RGB) values. The color assignment may be based on a 2D visible wavelength image. In some embodiments, the 3D point cloud includes depth information. For example, the depth information may define the distance between a person (or a part of a person) and a display screen 102 (or a specific location on the display screen 102). Alternatively, the depth information may define the distance between a person (or a part of a person) and an IR camera 104A. In some embodiments, the points in the 3D point cloud include voxels. In one embodiment, a voxel is (x, y, z, i), where "i" represents a monochromatic intensity. In one embodiment, a voxel is (x, y, z, r, g, b), where "r" is red intensity, "g" is green intensity, and "b" is blue intensity. Different color schemes may be used. In some embodiments, the color information of the voxels is derived from the 2D visible wavelength image.

[0052] In one embodiment, the 3D point cloud describes a bilaterally symmetrical object. Initially, due to the limitations of the IR data captured by IR camera 104A, the description of the bilaterally symmetrical object may be incomplete. For example, a person may be looking to the right of IR camera 104A, so that IR camera 104A captures the left side of the person's face well, but does not capture much (if any) information about the right side of the person's face. For example, most of the right side of the person's face may not have any depth information. In one embodiment, the 3D point cloud is reconstructed based on the symmetry plane based on the symmetry plane. In one embodiment, missing data in a first portion of the 3D point cloud corresponding to a first symmetrical half of the bilaterally symmetrical object is filled in based on data from a second portion of the 3D point cloud corresponding to a second symmetrical half of the bilaterally symmetrical object. For example, missing 3D point cloud data on the left side of the face is filled in based on 3D point cloud data on the right side of the person's face.

[0053] The gaze direction detector 108C is configured to detect the direction of a person's gaze. In one embodiment, the gaze direction detector 108C simultaneously inputs the reconstructed 3D point cloud and the 2D color (e.g., RGB) image. Using the 3D point cloud and the 2D color (e.g., RGB) image simultaneously can more accurately detect the gaze direction than using only one of these inputs.

[0054] The gaze predictor 108D is configured to predict the next gaze point of the person. In some embodiments, the gaze direction detector 108C and the gaze predictor 108D are implemented based on a long short-term memory (LSTM) model.

[0055] The memory 116 may be used to store software and / or firmware for controlling the electronic device 100, and to store images captured using the capture device 104, but is not limited thereto. Various different types of memory may be used, including non-volatile and volatile memory. The memory 116 may include a machine-readable medium on which one or more sets of executable instructions (e.g., applications) are stored that embody one or more methods and / or functions described herein. The term "machine-readable medium" as used herein should be understood to include all forms of storage media, whether as a single medium or multiple media in multiple forms; for example, centralized or distributed databases and / or associated caches and servers; one or more storage devices, such as storage drives (including, for example, magnetic and optical drives and storage mechanisms), and one or more instances of storage devices or modules (whether main memory, cache storage internal or external to the processor, or buffers). The term "machine-readable medium" or "computer-readable medium" should be understood to include any tangible, non-transitory medium capable of storing or encoding a sequence of instructions that is executed by a machine and causes the machine to perform any of the methods. The term “non-transitory media” expressly includes all forms of storage drives (optical, magnetic, etc.) and all forms of storage devices (e.g., dynamic random access memory (DRAM), flash (all storage designs), static random access memory (SRAM), magnetic random access memory (MRAM), phase change, etc.), and all other structures designed to store any type of information for later retrieval.

[0056] Figure 2 is a flow chart of one embodiment of a process 200 for performing image processing on a bilaterally symmetrical object. In an embodiment, the process 200 is performed by the electronic device 100. In an embodiment, the bilaterally symmetrical object includes a human face, and the image processing includes eye gaze tracking.

[0057] Step 210 includes generating a 3D point cloud of a bilaterally symmetrical object. In one embodiment, the bilaterally symmetrical object is a human face.

[0058] In one embodiment, the bilaterally symmetrical object is the entire body of a person. In one embodiment, the bilaterally symmetrical object is an inanimate object, such as a ball or a drinking glass. In an embodiment, the 3D point cloud is generated based on IR data from IR camera 104A.

[0059] In one embodiment, the bilaterally symmetrical object includes a human face. In one embodiment, the bilaterally symmetrical object includes a larger portion of a person, up to their entire body. In addition to the face, the bilaterally symmetrical object may also include parts of a person's body.

[0060] Step 212 includes generating a two-dimensional (2D) visible wavelength image of the bilaterally symmetrical object. In an embodiment, the 2D visible wavelength image is generated based on the visible wavelength data from the visible wavelength camera 104B. In an embodiment, the 2D image includes a color (e.g., RGB) image. In an embodiment, the 2D image is a digital image. Steps 210 and 212 may be performed sequentially or in parallel. When performed sequentially, either step may be performed first.

[0061] Step 214 includes detecting a symmetry plane of the bilaterally symmetrical object based on the 3D point cloud and the 2D visible wavelength image. In one embodiment, the symmetry plane is a sagittal plane of a person. In one embodiment, the symmetry plane is a sagittal plane of a face. In one embodiment, the 2D visible wavelength image is used to locate the bilaterally symmetrical object. For example, a face can be extracted from the 2D visible wavelength image. Then, based on the mapping of the 2D visible wavelength image to the 3D point cloud, the bilaterally symmetrical object can be located in the 3D point cloud.

[0062] Step 216 includes reconstructing the 3D point cloud based on the symmetry plane. Step 216 includes, in one embodiment, filling missing data in a first portion of the 3D point cloud corresponding to a first symmetric half of the bilaterally symmetrical object based on 3D point cloud data from a second portion of the 3D point cloud corresponding to a second symmetric half of the bilaterally symmetrical object. Note that in an embodiment, a color (e.g., RGB) value from the 2D visible wavelength image is assigned to each point in the 3D point cloud.

[0063] Step 218 includes performing image processing of a bilaterally symmetrical object based on the reconstructed 3D point cloud. In an embodiment, step 218 includes performing facial recognition. For example, step 218 may include determining whether the person operating the smartphone is the owner of the smartphone. Step 218 may also be used to distinguish between a person who is using the electronic device 100 and a bystander who is just within range of the capture device 104. In an embodiment, step 218 includes performing eye tracking.

[0064] In an embodiment, step 218 includes registering the reconstructed 3D point cloud of the bilaterally symmetrical object to another 3D point cloud of the bilaterally symmetrical object using a long short-term memory (LSTM) unit. Step 218 includes transforming the reconstructed 3D point cloud of the bilaterally symmetrical object into another reconstructed 3D point cloud of the bilaterally symmetrical object using a long short-term memory (LSTM) unit. In an embodiment, step 218 includes determining corresponding point pairs between the reconstructed 3D point cloud of the bilaterally symmetrical object and another 3D point cloud of the bilaterally symmetrical object using a long short-term memory (LSTM) unit, and measuring the difference between the corresponding point pairs using the long short-term memory (LSTM) unit. Further details of image processing using LSTM units are discussed below.

[0065] Figure 3 Flowchart of one embodiment of a process 300 for performing image processing on a face based on a reconstructed 3D point cloud. In one embodiment, the image processing includes eye gaze tracking. In an embodiment, the process 300 is performed by the electronic device 100. The process 300 describes further details of one embodiment of the process 200.

[0066] Step 310 includes mapping the 3D point cloud of the face to a set of 2D visible wavelength images of the face. In one embodiment, the 3D point cloud is a set of 3D data points containing details of the face. In an embodiment, step 310 includes mapping RGB data from the 2D visible wavelength image to points in the 3D point cloud. Thus, in an embodiment, a color (e.g., RGB value) is assigned to each point in the 3D point cloud. Combining the 3D point cloud with color data can better perform image processing of objects on both sides. For example, color data can make gaze detection more accurate.

[0067] Step 312 includes extracting a face from the 2D visible wavelength image. In an embodiment, the 2D visible wavelength image of the face includes an eye image. The eye image may include a corneal reflection image of a bright pupil or a dark pupil, and a color eye image. For a person of ordinary skill, the technology of extracting a face from a 2D visible wavelength image is known.

[0068] Step 314 includes extracting a face from the 3D point cloud. In one embodiment, the extraction is performed based on a mapping of the 2D image to the 3D point cloud. In other words, once the mapping is known, together with the facial extraction of step 314, a face can be extracted from the 3D point cloud.

[0069] Step 316 includes detecting a symmetry plane of the face. The symmetry plane of the face may also be referred to as the sagittal plane of the face. Further details of detecting the symmetry plane are described below. In some embodiments, a cost function is solved to detect the symmetry plane. Solving the cost function is an accurate and effective technique for detecting the symmetry plane.

[0070] Step 318 includes reconstructing the 3D point cloud based on the symmetry plane. In one embodiment, step 318 includes filling missing data in a first portion of the 3D point cloud corresponding to a first half of the face based on 3D point cloud data from a second portion of the 3D point cloud corresponding to a second half of the face.

[0071] Step 320 includes performing image processing of the face based on the reconstructed 3D point cloud. Note that in an embodiment, color (e.g., RGB) values ​​from the 2D visible wavelength image are assigned to the reconstructed 3D point cloud. In an embodiment, step 320 includes performing facial recognition. For example, step 320 may include determining whether the person operating the smart phone is the owner of the smart phone. In an embodiment, step 320 includes performing eye tracking. In an embodiment, step 320 includes performing image processing using an LSTM unit. In some embodiments, the 3D point cloud can be used to detect head movement, and the 2D visible wavelength image can be used for eye detection. In gaze detection, the movement of the head and eyes may be very subtle. The farther the distance between the user and the camera, the more subtle the change. Combining information from the 3D point cloud and the eye image (from the 2D visible wavelength image) can provide more details for gaze detection and obtain better results.

[0072] The following are details of one embodiment of detecting a symmetry plane (e.g., a sagittal plane). Detecting a symmetry plane can be very difficult. One reason for the difficulty is that the 3D point cloud may be missing data for one half due to, for example, a person not looking directly at the IR camera 104A. In one embodiment, the symmetry plane is detected based on a variational function, as described below. Let p = (x, y, z) be a point in the 3D point cloud. A function "g" is used to represent the sagittal plane with g(p) = 0, which separates the left and right sides of a person's face. Equation 1 describes the sagittal plane, Equation 2 describes the left half of a person's face, and Equation 3 describes the right half of a person's face.

[0073] g(x, y, z) = 0 Equation 1

[0074] g(x, y, z)>0 Equation 2

[0075] g(x, y, z)<0 Equation 3

[0076] Furthermore, let p' be symmetric to p with respect to the sagittal plane. That is, the line formed by p and p' is perpendicular to g. Due to symmetry, the distance from p to g should be equal to the distance from p' to g. That is, their difference should be 0. When g is the sagittal plane, the cost function in equation 1 below will be minimized. Note that a face may not be completely symmetrical; therefore, the cost function does not need to be zero. If the face is completely symmetrical, then the cost function can be minimized to zero.

[0077]

[0078] In Equation 1, λ1 and λ2 are parameters related to the left and right sides of a bilaterally symmetrical object (e.g., the left and right sides of a human face). In one embodiment, λ1 is equal to λ2. Using different values ​​for λ1 and λ2 gives more weight to one side of the bilaterally symmetrical object. In Equation 4, K(g) is the Heaviside function, expressed in Equation 5 below.

[0079]

[0080] Note that the number of points on the right side may be different from the number of points on the left side, so there may not be a p' for every p. For simplicity, rewrite Equation 4 as shown in Equations 6, 7, and 8.

[0081]

[0082]

[0083]

[0084] In equations 6, 7, and 8, D, D', and K represent D(p), D(p'), and K(g(x, y, z)) in equation 4, respectively. As described above, in one embodiment, λ1 is equal to λ2. If λ1 is equal to λ2, equation 8 can be simplified as shown in equation 9.

[0085]

[0086] Next, let E equal [D 2 +D′ 2 +2(D+D′) 2 (K 2 -K)], keeping D and D' fixed, derive the relevant Euler-Lagrange equation for g. Then, solve for g and parameterize using step size t, as shown in Equation 10.

[0087]

[0088] g n =g (n-1) +2Δtδ(g(x,y,z))((D+D′) 2 (2K-1))Equation 11

[0089] In Equation 11, Δt is the step size. In addition, in Equation 11, the following substitution may be made, as in Equation 12.

[0090]

[0091] Given that not every point p has a symmetric point p' that can form a straight line perpendicular to f, the definition of p' is relaxed to the nearest ideal point p' within a distance d, where d is a parameter based on the degree of relaxation desired in the definition of p'. In addition, using the points on g(x, y, z), the sagittal plane is obtained by a support vector machine (SVM), and g(x, y, z) = Ax + By + Cz + D, where A, B, C, D are parameters. Therefore, the distance D(p) between p and g can be obtained as shown in Equation 13.

[0092]

[0093] In one embodiment, the initial symmetry plane may be a plane perpendicular to the line between the two eyes, which may be approximately detected in the 2D visible wavelength image. In one embodiment, the step size is set to 1, and the Heaviside function is given by Eq.

[0094]

[0095] Once the symmetry plane (e.g., sagittal plane) is determined, a reconstructed 3D point cloud of the user's face can be made symmetrically based on the sagittal plane. In one embodiment, missing data in a first portion of a 3D point cloud corresponding to a first symmetric half of a bilaterally symmetrical object is filled in based on 3D point cloud data from a second portion of the 3D point cloud corresponding to a second symmetric half of the bilaterally symmetrical object. Embodiments of the techniques for finding symmetry planes are computationally efficient. Therefore, computation time and power can be saved. In addition, if these techniques are employed in a battery-powered device, battery power can be saved.

[0096] Figure 4 4 is a flow chart of an embodiment of a process 400 of performing an action based on a position on a display screen that a person is looking at. The process 400 may be formed by the electronic device 100 or the UE 110, but is not limited thereto.

[0097] Step 410 includes detecting eye gaze of a person based on the reconstructed 3D point cloud and the 2D visible wavelength image of the person's face. In step 410, the 2D visible wavelength image may be used to detect the person's eyes, and the 3D point cloud may be used to determine the person's head position. Step 412 includes determining the location on display screen 102 that the person is looking at. Steps 410 and 412 may be performed in one embodiment of step 218 of process 200 or step 320 of process 300.

[0098] Step 414 includes performing an action based on the location where the person is looking. Example actions include, but are not limited to, highlighting an item on a display screen, turning to the next page in a document presented on the display, scrolling up or down in a document presented on the display.

[0099] Figure 5 is a flow chart of an embodiment of a process 500 for determining a position on a display screen that a person is looking at. The process 500 may be formed by the electronic device 100, the UE 110, but is not limited thereto. Prior to the process 500, so-called "standard data" is collected. The standard data used in the process 500 includes a plurality of 3D point clouds. However, the standard data is not limited to 3D point clouds. In an embodiment, each 3D point cloud is collected together with a specific position on the display screen 102 that the user is looking at. This enables the electronic device 100 to know what the 3D point cloud looks like when the user is looking at certain points on the display screen 102. Figure 7 Further details of one embodiment of collecting normative data are described.

[0100] Step 510 includes accessing standard data including a 3D point cloud collected while the user is looking at different points on the display screen 102 .

[0101] Step 512 includes finding the closest match between the reconstructed 3D point cloud and the standard data. Therefore, step 512 actually determines the gaze point in the standard data that is closest to the person's current gaze point. Step 514 includes calculating the actual (current) gaze point through alignment and transformation. In one embodiment, an LSTM unit is used to perform alignment and / or transformation. Fig. 9 Details of one embodiment of image registration and image transformation are shown.

[0102] Figure 6 A person's head and various geometric details are shown at two points in time. One point in time is when the standard data is collected. The other point in time is when the location on the display screen that the person is looking at is detected based on the standard data (as in process 500). Reference numerals ending with the letter "a" are associated with the time when the standard data is collected. Reference numerals ending with the letter "b" are associated with the time when the location on the display screen that the person is looking at is detected based on the standard data. Note that a human face is an example of a bilaterally symmetrical object.

[0103] Figure 6A head 602 of a person looking at display screen 102 is shown. Display screen 102 is depicted at two points in time. Display screen 102a represents the display screen during the collection of standard data. Display screen 102b represents the display screen when the person's gaze is detected. The person's eye gaze is directed to position 606a on display screen 102a. Line 604al and line 604ar show the direction of the gaze from the person's eye 612a. More specifically, line 604al corresponds to the left eye and line 604ar corresponds to the right eye. Line 614a indicates the orientation of the person's head when looking at point 606a. In one embodiment, line 614a corresponds to the sagittal plane. Line 605a shows the distance between the person and display screen 102a. The standard data includes a 3D point cloud based on IR data captured when the person looks at position 606a at a distance given by line 605a.

[0104] Figure 6 Also shown is a head 602 of a person who is looking at display screen 102b when a position on the display screen that the person is looking at is detected. The person's eye gaze is directed to position 606b on display screen 102b. Lines 604bl and 604br show the direction of the gaze from the person's eye 612b, which is depicted in dotted lines. More specifically, line 604al corresponds to the left eye and line 604ar corresponds to the right eye. Line 614b indicates the orientation of the person's head when looking at point 606b. In one embodiment, line 614b corresponds to the sagittal plane. Line 605b shows the distance between the person and display screen 102b.

[0105] The current data includes a 3D point cloud based on IR data captured when a person is looking at position 606b from a distance given by line 604b. If the lengths of line 604bl and line 604br are different, the average distance can be taken. As described in the discussion of step 512, the standard data is checked to find the 3D point cloud that is closest to the current 3D point cloud. Step 514 involves the registration and transformation of the two 3D point clouds. Performing the registration and transformation will indicate the differences between the two 3D point clouds. Therefore, these differences will indicate the differences in the distance between the person and the display screen (if any). In an embodiment, these differences will also indicate the differences in the angle of the person's head. Based on this information, position 606b on display screen 102b can be determined.

[0106] Figure 7 is a flow chart of an embodiment of determining standard data, which may be used in process 500. Step 710 includes instructing a person to gaze at a point on display screen 102. Step 710 may include highlighting the location of display screen 102 and providing an audio or visual instruction for the gaze location.

[0107] Step 712 includes receiving an indication from the person that the person is looking at a location. The input from the user can be voice, touching a display screen, selecting using an input device (e.g., a mouse), tapping keys on a keyboard, and the like.

[0108] Step 714 includes capturing a 3D point cloud for a location on display screen 102. Step 714 may include IR camera 74A capturing IR data. Step 714 may also include capturing a 2D visible wavelength image.

[0109] Step 716 is to determine whether there are other locations on the display screen 102 where 3D point cloud data can be collected. In one embodiment, data from about five locations are collected. For example, a 3D point cloud can be generated for each of the four corners and the center of the display screen 102. If more data is to be collected, the process 700 returns to step 710. In one embodiment, data from about nine locations are collected. For example, a 3D point cloud can be generated for each of the four corners of the display screen 102, the center, and the midpoint of each of the four edges along the display screen 102. The 3D point cloud collection can be stored in the memory 116. In an embodiment, multiple 3D point cloud collections are stored in a non-volatile memory. However, the 3D point cloud does not need to be stored on the electronic device 70.

[0110] Figure 8 An embodiment of a system 800 configured to perform eye gaze detection is depicted. In one embodiment, the system 800 is used to track eye gaze over time and predict the next eye gaze point. In one embodiment, the system 800 is used to implement the gaze direction detector 108C and the gaze predictor 108D. Thus, the system 800 can be implemented in the electronic device 100. The method 800 can be implemented in the UE 80. In one embodiment, the system 800 is implemented by executing instructions on the processor 108.

[0111] The system 800 includes several long short-term memory (LSTM) units 802. In this embodiment, three LSTM units 802 are depicted. The system 800 may have more or less than three LSTM units 802. Each LSTM unit 802 inputs data (x) for a time point. Data is input for different time points of each LSTM unit 802. In this example, the time periods are referred to as "t+1", "t", and "t-1". The input data may include a reconstructed 3D point cloud. 2D visible wavelength image data may also be input to the LSTM unit 802. In one embodiment, the 2D visible wavelength image data is used to assign color (e.g., RGB) values ​​to the 3D point cloud, in which case the 2D visible wavelength image does not have to be a separate input. Standard 3D point cloud data, such as collected in process 700, may also be input to the LSTM unit 802. Note that although in some embodiments, the system inputs a reconstructed 3D point cloud (e.g., generated in process 200 or 300), the 3D point cloud input to the system 800 does not need to be a reconstructed 3D point cloud. In other words, the 3D point cloud input to system 800 may be generated in any suitable manner.

[0112] LSTM unit 802 can output information to another LSTM unit 802. In one embodiment, LSTM unit 802 outputs the position of the focus of the person's eye gaze at a time point to another LSTM unit 802. In one embodiment, the position is represented by a vector (c t ) description. The LSTM unit 802 receiving this information may modify the position of the focus, or in some cases, not change the position of the focus. Note that additional information may be exchanged between LSTM units 802 as described more fully below.

[0113] In an embodiment, each LSTM unit 802 has an input gate, an output gate, and a forget gate. These three gates can calculate their respective outputs based on the current time step (e.g., t) and the previous time step (e.g., t-1). In some embodiments, various weights can be applied to the gates to affect the contribution of the gate to the final output of the LSTM unit 802. Fig.10 Further details of one embodiment of the LSTM unit 802 are described. Each LSTM unit 802 provides an output h to the integration layer 804. t . The LSTM unit 802 can provide the output ct as the input of another LSTM unit 802.

[0114] The integration layer 804 receives input (h t In one embodiment, the input (h t ) is a vector describing the direction in which the person's eyes are looking. In an embodiment, the input h t and vector ct The combination of may be passed to the next LSTM unit 802. In an embodiment, the integration layer 804 outputs the current location of the person's eye gaze. In an embodiment, the integration layer 804 outputs the location on the display screen 102 that the person is looking at. In an embodiment, the integration layer 804 predicts the next person's eye gaze point. In an embodiment, the integration layer 804 predicts the next gaze point in a series of saccadic movements.

[0115] Fig. 9 is a flow chart of one embodiment of a process 900 for predicting the next eye gaze point by using the LSTM unit 802. The process 900 may be used to Figure 8 In the system 800 of . Step 910 includes inputting data to the LSTM unit 802. The input data may include a reconstructed 3D point cloud for a time point. The input data may include a standard 3D point cloud. The input data may include 2D visual wavelength image data for a time point. In one embodiment, each LSTM unit 802 inputs data for a different time point. In an embodiment, the LSTM units 802 collectively input data for a sequence of time points.

[0116] Step 912 includes tracking eye movement using LSTM cells 802. In one embodiment, each LSTM cell 802 determines a gaze point for a point in time. In one embodiment, step 912 includes registering the reconstructed 3D point cloud to another 3D point cloud using the LSTM cells. In one embodiment, step 912 includes transforming the reconstructed 3D point cloud into another 3D point cloud using the LSTM cells. Further details of how to use LSTM cells to track eye movement are discussed below.

[0117] Step 914 includes predicting the next eye gaze point. Step 914 can be performed by the integration layer 804. The integration layer 804 inputs the vectors from the LSTM unit 802, which describe the direction of the user's eye gaze at different time points. In an embodiment, based on these vectors, the integration layer 804 predicts the next gaze point.

[0118] Fig.10 An embodiment of an LSTM cell 802 is depicted. The LSTM cell 802 has R s Component 1002 is configured to register the 3D point cloud using standard data of the 3D point cloud. s Component 1002 may be configured to find corresponding points in the reconstructed 3D point cloud using the standard data of the 3D point cloud. s Component 1004 is configured to reconstruct the 3D point cloud using a standard data transformation of the 3D point cloud. sComponent 1004 may be configured to measure the difference between the corresponding points. Reconstructing a 3D point cloud may be provided by executing process 200 or process 300 , but is not limited thereto.

[0119] LSTM unit 802 has R T Component 1006 is configured to register the 3D point cloud (for the current time, t) with the 3D point cloud for the previous time (t-1). T Component 1006 can be configured to find corresponding points in two 3D point clouds. LSTM unit 802 has T T Component 1008 is configured to transform the 3D point cloud using the 3D point cloud for the previous time (t-1). T Component 1008 may be configured to measure the difference between the corresponding points in the two 3D point clouds.

[0120] The registration in the embodiment of LSTM unit 802 finds the relationship between two data sets. For example, the registration finds corresponding points in two data sets (e.g., two 3D point clouds). The transformation in the embodiment of LSTM unit 802 quantitatively transforms the data set from one coordinate system to another coordinate system. For example, the transformation measures the difference between two data sets (e.g., two 3D point clouds). There may be some overlap between the registration and the transformation. Therefore, in some embodiments, the registration and the transformation are not performed separately, but can be performed together (e.g., in a single step).

[0121] Input(x t ) refers to the input data to the LSTM unit 802 at time "t". The input data may include a reconstructed 3D point cloud, and 2D visible wavelength data mapped to the 3D point cloud. For example, points in a 3D point cloud may have RGB intensity values ​​assigned to them. In one embodiment, the reconstructed 3D point cloud is generated using process 200. In one embodiment, the reconstructed 3D point cloud is generated using process 300.

[0122] LSTM unit 802 inputs the focus position (c t-1 ), and the focus position (c t ) is output to another LSTM unit 802. Before being output to the next LSTM unit 802, the TS component 1004 and the T T The output of component 1008 modifies the input focus position (c t-1 ).

[0123] In one embodiment, the focus position (c t) includes a unit state vector. In one embodiment, the unit state vector indicates the standard gaze point P at time t s and the current gaze point P t The current gaze point P t Indicates the location on display screen 102 that the user is looking at, which can be obtained by combining the previous gaze point and the movement in eye gaze. In one embodiment, the unit state vector is represented as a 2 by 2 matrix. In one embodiment, each row represents the (x, y) coordinates on display screen 102. Therefore, in one embodiment, a row represents the standard gaze point (P s ), and the other line represents the (x, y) coordinates of the current gaze point P t The (x, y) coordinates of .

[0124] LSTM unit 802 inputs hidden variables (H) for the previous time period from another LSTM unit 802. t-1 ), and the hidden variable (H t ) is output to another LSTM unit 802. Therefore, H t From one LSTM unit to another. In one embodiment, H t = {S, h t}, where S is standard facial data. For example, process 700 can be used to collect standard facial data. In one embodiment, LSTM unit 802 outputs (h t ), which is provided to the integration layer 804. The following describes the generation of h t for further details.

[0125] R s Component 1002 and T s Components 1004 together generate a t )'s "forget gate" result. In one embodiment, R s Component 1002 searches for the 3D point cloud that is closest to the 3D point cloud in the standard data. s Component 1002 is T s Component 1004 provides point pairs, each pair being corresponding points in two 3D point clouds. s Component 1002 executes S and x t Recall that S is the standard face data, x t is the input data. In other words, R s Component 1002 may determine S and x t The corresponding points in .

[0126] In one embodiment, based on the s The corresponding point of component 1002, Ts Component 1004 executes S and x t This determines the transformation of S and x t In one embodiment, T s Component 1004 compares two 3D point clouds (for S and x t ) and determines whether the face is an authorized user. In one embodiment, if this is not an authorized user, the forget gate output is zero. In one embodiment, if this is an authorized user, the forget gate output is one. Based on the corresponding points, T s Component 1004 may also estimate the distance the face has moved (relative to the standard data). s Component 1004 may also estimate the location on the display screen that the person is looking at.

[0127] T s Component 1004 output f t Equation 14 describes R s Component 1002 and T s How component 1004 can generate f t .

[0128] f t =σ f (W f T s (R s (X t ,H t-1 ))) Equation 14

[0129] Equation 14, R s It's S and X t The registration of the standard data S and the current input data X t In equation 14, Ts is the corresponding point pair between S and X. t , which provides a translation between the two data sets. Note that S can be provided as input H t-1 In an embodiment, in Equation 14, σ f is the Heaviside function. In an embodiment, in Equation 14, σ f is the sigmoid function. In Equation 14, W f is the weight matrix.

[0130] R T Components 1006 and T T Component 1008 jointly generates a t ) is the “input gate” result. In one embodiment, R T Component 1006 is T TComponent 1008 provides point pairs, each pair being corresponding points in the two closest 3D point clouds. T Component 1006 executes x t and h t-1 This provides corresponding point pairs for the two data sets. Note that h t-1 Included with S in H t-1 middle.

[0131] In one embodiment, based on the T The corresponding point of component 1006, T T Component 1008 executes X t and h t-1 This indicates that from h t-1 To X t Therefore, in one embodiment, T T Component 1008 can compare corresponding points in two 3D point clouds and determine the movement in the face. Based on the movement in the face, T T Component 1008 can also estimate the location on the display screen that the person is looking at. In other words, based on the movement in the person's face, T T Component 1008 can estimate eye movement. T Component 1008 may also determine eye movement between two points in time, thereby determining movement of the gaze point.

[0132] T T Component 1008 outputs it. Equation 15 describes R T Components 1006 and T T How component 1008 can generate it.

[0133] i t =tanh(W i T T (R T (X t ,H t-1 ))) Equation 15

[0134] In Equation 15, R T Yes X t and h t-1 , which provides corresponding point pairs between the two data sets. Note that h t-1 Included in H with S t-1 However, only h t-1 is active in the registration (i.e., not involved in S). In Equation 15, T T Yes X t and h t-1 A transformation of W that provides the movement between the two data sets. For example, this provides eye movement.i is the weight matrix.

[0135] In an embodiment, the multiplier 1012 is configured to perform an element-wise dot product. The multiplier 1012 performs an input eye vector c t-1 and T s The output f of component 1004 t Adder 1014 combines the output of multiplier 1012 and T T The output i of component 1008 t The output of adder 1014 is the eye vector c for time t t Equation 16 summarizes the generation of the eye vector.

[0136]

[0137] In Equation 17, the symbol represents the element-wise dot product performed by the multiplier 1012. The "+" symbol represents the vector addition by the adder 1014.

[0138] R T Component 1006 outputs a so-called "registration result" to refinement component 1010. t In one embodiment, the registration result is h t-1 and X t Equation 17 describes the corresponding point pairs in R T How component 1006 generates the registration result t .

[0139] o t =W o R T (X t ,H t-1 ) Equation 17

[0140] In Equation 17, R T Yes X t and h t-1 The registration of W provides corresponding point pairs between the two data sets. o is the weight matrix.

[0141] Refinement component 1010 from R T Component 1006 inputs the eye vector c t And the registration result o t , and output h t The refinement component 1010 will remove or adjust small inconsistencies in the data. For example, a person may move her lips slightly, but the movement is very subtle and below the threshold. Although the movement will not change the forget gate f t, but this small amount of movement should not be passed to the integration layer 804 or the next LSTM unit 802.

[0142] Equation 18 shows how the refinement component 1010 can generate h t Function F R Refers to the above-mentioned refinement performed by the refinement component 1010.

[0143] h t =F R (o t ,c t ) Equation 18

[0144] Now discuss again Figure 8 The integration layer 804 in , the integration layer 804 predicts eye gaze based on the input from the LSTM unit 802. In one embodiment, the prediction is based on Eq. 19.

[0145] p n =∑ t W t H t Equation 19

[0146] In Equation 19, p n is the next fixation point, H t W refers to the input from each LSTM cell 802. t is the LSTM output H t The parameters can be learned through training.

[0147] Fig.11 1 is a flow chart of one embodiment of a process 1100 of an electronic device providing a person with options to select on a display screen in response to detecting that the person is looking at a location on the screen. This is an example of performing an action based on the location of the display screen that the person is looking at. Step 1110 includes the electronic device 100 detecting that the person is looking at an option on the display screen 102. Fig. 12A The display screen 102 is shown with a first option 1202 and a second option 1204. The first option 1202 is to go to the next page in the document presented on the display screen 102 (the document is not in Fig. 12A ). The second option 1204 is to go to the previous page in the document presented on the display screen 102. For example, the electronic device 100 determines that the person is looking at the first option 1202. Any technology disclosed herein for detecting a location on a display screen that a person is looking at can be used.

[0148] Step 1112 includes changing the appearance of the option. Fig. 12BOne example of changing the appearance of the first option 1202 is shown. The appearance of the first option 1202 can be changed in any suitable manner, including but not limited to highlighting, brightening, displaying a ring or frame around the option, etc.

[0149] Step 1114 is to determine whether the gaze time is greater than a predetermined time period. The time period can be a user-definable time period. For example, the time period can be one second, two seconds, three seconds, etc. If the person's eye gaze stays on the option within the time period, then in step 1116, an action associated with the option is performed. In this example, the electronic device 100 changes the display screen 102 to present the next page in the document. Other types of actions are possible.

[0150] If the user's gaze does not rest on an option within a predetermined period of time (step 1114 = No), the process ends without taking further action. In other words, the option is not selected in this case.

[0151] Note that steps 1112 and 1116 are examples of performing actions based on the location of the display screen that the person is looking at. Therefore, steps 1112 and 1116 are examples of step 414 (see Figure 4 ) embodiments.

[0152] Many variations of process 1100 are possible. For example, additional user input, such as in the form of a voice command, may be used in step 1114. Thus, electronic device 100 may highlight an option in response to a user gazing at the option, and maintain the highlighting while the user keeps their gaze fixed on the option. However, in an embodiment, the user may use a voice command to select an option.

[0153] Fig.13 1 is a flow chart of one embodiment of a process 1300 for performing an action in response to detecting eye gaze of a person. Process 1300 includes further details of performing an action based on the location of a display screen that a person is looking at. In one embodiment, process 1300 is performed by electronic device 100. Step 1310 includes displaying a document on display screen 102 of the electronic device. Step 1312 includes electronic device 100 detecting the location of the person's eye gaze on display screen 102. Any technique disclosed herein for detecting the location on a display screen that a person is looking at may be used.

[0154] Step 1314 includes determining whether the eye gaze is at the top of the display screen 102. The top can be the top 1 / 3, the top 1 / 4, the top 1 / 5, or some other percentage. If the person is looking at the top, then in step 1316, the electronic device scrolls the document down by X percentage of the display screen 102. In one embodiment, although the X percentage is equal to the size of the top discussed in step 1314, the X percentage can be greater than or less than the size of the top. The term "scroll" in this context has a broad meaning and can include several small movements of the document content, one large movement, etc. After step 1316, process 1300 returns to step 1312 to detect eye gaze. There may be a time gap to allow the person to adjust to the movement of the document on the display screen 102.

[0155] If it is determined in step 1314 that the person's gaze is not at the top of the display screen (step 1314=no), the process 1300 proceeds to step 1318. Step 1318 includes determining whether the eye gaze is at the bottom of the display screen 102. In one embodiment, the bottom has the same size as the top. However, the bottom can have a different size than the top. If the person is looking at the bottom, then in step 1320, the electronic device scrolls the document down by the Y percentage of the display screen 102. In one embodiment, in step 1316, the Y percentage is equal to the X percentage. However, in step 1316, the Y percentage does not need to be equal to the X percentage. In one embodiment, although the Y percentage is equal to the size of the bottom discussed in step 1318, the Y percentage can be greater than or less than the size of the bottom. After step 1320, the process 1300 returns to step 1312 to detect eye gaze. There may be a time gap to allow the person to adjust to the movement of the document on the display screen 102.

[0156] Note that steps 1316 and 1320 are examples of performing actions based on the location of the display screen that the person is looking at. Therefore, steps 1316 and 1320 are examples of step 414 (see Figure 4 ) embodiments.

[0157] Many alternatives to process 1300 are possible. In one embodiment, rather than detecting whether the user's eye gaze is at the top (step 1314) or bottom (step 1318) of display screen 102, electronic device 100 determines whether the user's eye gaze is at the top (step 1314) or bottom (step 1318) of a document being presented on display screen 102.

[0158] The technology described herein can be implemented using hardware, software, or a combination of hardware and software. The software used is stored on one or more of the above-mentioned processor-readable storage devices to program one or more of the processors to perform the functions described herein. The processor-readable storage device may include a computer-readable medium, such as volatile and non-volatile media, removable and non-removable media. As an example and not limitation, a computer-readable medium may include a computer-readable storage medium and a communication medium. The computer-readable storage medium includes any method or technology implementation for storing information such as computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Computer-readable media or media (does not) include propagation, modulation, or transient signals.

[0159] Communication media typically contain computer readable instructions, data structures, program modules, or other data in a propagated, modulated, transient data signal such as a carrier wave or other transport mechanism, and include any information transmission media. The term "modulated data signal" means a signal that has one or more characteristics set or changed so as to encode information in the signal. By way of example and not limitation, communication media include wired media such as a wired network or a direct wired connection, and wireless media such as RF and other wireless media. Combinations of any of the above are also included within the scope of computer readable media.

[0160] In an alternative embodiment, some or all of the software may be replaced by a dedicated hardware logic component. For example, and not limitation, illustrative types of hardware logic components that may be used may include field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), special-purpose computers, etc. In one embodiment, software (stored on a storage device) that implements one or more embodiments is used to program one or more processors. One or more processors may communicate with one or more computer-readable media / storage devices, peripheral devices, and / or communication interfaces.

[0161] It should be understood that this theme can be embodied in many different forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided so that the theme will be thorough and complete, and the disclosure is fully communicated to those skilled in the art. In fact, this theme is intended to cover modifications, alternatives and equivalents of all embodiments within the scope and spirit of this theme defined by the appended claims. In addition, in the following detailed description of this theme, many specific details will be set forth to thoroughly understand this theme. However, for those of ordinary skill in the art, it is apparent that this theme can be practiced without these specific details.

[0162] Various aspects of the present disclosure are described herein with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present disclosure. It should be understood that each frame in the flowchart and / or block diagram and the combination of the frames in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable instruction execution device create a mechanism for implementing the function / action specified in one or more frames of the flowchart and / or block diagram.

[0163] The description of the present disclosure has been presented for the purpose of illustration and description, but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present disclosure. The aspects disclosed herein are selected and described in order to best explain the principles and practical applications of the present disclosure, and to enable others of ordinary skill in the art to understand the disclosure with various variations suitable for a particular intended use.

[0164] For the purposes of this document, each process associated with the disclosed technology can be performed continuously and by one or more computing devices. Each step in the process can be performed by a computing device that is the same or different from the computing device used in other steps, and each step does not necessarily need to be performed by a single computing device.

[0165] Although the subject matter of the present invention is described in language specific to structural features and / or methodological actions, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as example forms of implementing the claims.

Claims

1. A device for eye gaze tracking, the device comprising: Visible wavelength camera; Infrared (IR) cameras; as well as One or more processors are configured to: generating a three-dimensional (3D) point cloud of a human face based on the IR data captured from the IR camera; generating a two-dimensional image of a human face based on visible wavelength data captured from the visible wavelength camera; Minimizing a cost function based on the 3D point cloud and the two-dimensional image to detect a symmetry plane of the face, the symmetry plane dividing the 3D point cloud into two parts, the cost function comprising a first parameter providing a first weight for a first symmetric half of the face and a second parameter providing a second weight for a second symmetric half of the face; Reconstructing the 3D point cloud based on the symmetry plane; and Eye gaze of the face is detected and tracked based on the reconstructed 3D point cloud.

2. The apparatus according to claim 1, wherein in order to reconstruct the 3D point cloud based on the symmetry plane, the one or more processors are further configured to: Based on 3D point cloud data from a second portion of the 3D point cloud corresponding to a second symmetrical half of the face, missing data in a first portion of the 3D point cloud corresponding to a first symmetrical half of the face is filled.

3. The device according to claim 1 or 2, wherein: The device also includes a display screen; and The one or more processors are further configured to determine a location on the display screen at which a person is looking based on the reconstructed 3D point cloud.

4. The apparatus of claim 3, wherein the one or more processors are further configured to: In response to a location on the display screen at which the person is looking that is within a predefined area of ​​the display screen, the content of the display screen is automatically scrolled.

5. The apparatus of claim 4, wherein the one or more processors are further configured to: In response to detecting that the person is looking at the element, the appearance of the element on the display screen is changed.

6. The apparatus of claim 4, wherein the one or more processors are further configured to: Based on the output of a sequence of long short-term memory (LSTM) units, the next location of the person's gaze is predicted.

7. The apparatus of claim 6, wherein the one or more processors are further configured to: Image registration and image transformation are integrated into each of the LSTM units.

8. The apparatus according to any one of claims 1 to 7, wherein the one or more processors are further configured to: The reconstructed 3D point cloud of the face is registered to another 3D point cloud of the face using a long short-term memory (LSTM) unit.

9. The apparatus of claim 8, wherein the other 3D point cloud is a standard 3D point cloud captured by the person gazing at a known position.

10. The apparatus according to claim 8, wherein the reconstructed 3D point cloud is a first reconstructed 3D point cloud captured at a first time point, and the other 3D point cloud is a second reconstructed 3D point cloud captured at a second time point before the first time point.

11. The apparatus according to any one of claims 1 to 10, wherein the one or more processors are further configured to: The reconstructed 3D point cloud of the face is transformed into another 3D point cloud of the face using a long short-term memory (LSTM) unit.

12. The apparatus according to any one of claims 1 to 11, wherein the one or more processors are further configured to: determining corresponding point pairs between a reconstructed 3D point cloud of the person's face and a standard 3D point cloud of the person's face captured by the person gazing at a known location using a long short-term memory (LSTM) unit; and The differences between the corresponding point pairs are measured using the LSTM unit.

13. The apparatus of claim 12, wherein the one or more processors are further configured to: Facial recognition is performed on the person based on the differences between the corresponding point pairs.

14. The apparatus according to claim 12 or 13, wherein the one or more processors are further configured to: determining the corresponding point pairs between a first version of a reconstructed 3D point cloud of the face captured at a first point in time and a second version of the reconstructed 3D point cloud of the face captured at a second point in time, including determining the corresponding pairs using long short-term memory (LSTM) units; and The differences between the corresponding point pairs are measured using the LSTM unit.

15. The device according to claim 14, wherein: The one or more processors are further configured to determine a direction of eye gaze of the person based on differences between the corresponding point pairs.

16. The apparatus according to any one of claims 1 to 15, wherein the one or more processors are further configured to: generating a three-dimensional (3D) point cloud of the bilaterally symmetrical object based on the IR data captured from the IR camera; generating a two-dimensional image of the bilaterally symmetrical object based on visible wavelength data captured from the visible wavelength camera; Detecting a symmetry plane of the bilaterally symmetrical object based on the 3D point cloud of the bilaterally symmetrical object and the two-dimensional image of the bilaterally symmetrical object, wherein the symmetry plane divides the 3D point cloud into two parts; and A 3D point cloud of the bilaterally symmetrical object is reconstructed based on the symmetry plane.

17. The apparatus according to any one of claims 1 to 16, wherein the one or more processors are further configured to: Color information for voxels in the 3D point cloud is derived based on the two-dimensional image.

18. A method for eye gaze tracking, comprising: generating a three-dimensional (3D) point cloud of a human face based on IR data captured from an infrared (IR) camera; generating a two-dimensional image of a human face based on visible wavelength data captured from a visible wavelength camera; Minimizing a cost function based on the 3D point cloud and the two-dimensional image to detect a symmetry plane of the face, the symmetry plane dividing the 3D point cloud into two parts, the cost function comprising a first parameter providing a first weight for a first symmetric half of the face and a second parameter providing a second weight for a second symmetric half of the face; Reconstructing the 3D point cloud based on the symmetry plane; and Eye gaze of the face is tracked based on the reconstructed 3D point cloud.

19. The method of claim 18, wherein reconstructing the 3D point cloud based on the symmetry plane comprises reconstructing a first portion of the 3D point cloud representing a first half of the face based on data from a second portion of the 3D point cloud representing a second half of the face.

20. The method of claim 19, wherein: Detecting and tracking eye gaze of the person's face based on the reconstructed 3D point cloud includes inputting data into a long short-term memory (LSTM) unit and predicting the person's next gaze based on an output of the LSTM unit.

21. The method of claim 18, wherein detecting and tracking the gaze of the eyes of the face based on the reconstructed 3D point cloud comprises: determining corresponding point pairs between a reconstructed 3D point cloud of the face and another 3D point cloud of the face using a long short-term memory (LSTM) unit; as well as The differences between the corresponding point pairs are measured using the LSTM unit.

22. The method of claim 18, wherein detecting and tracking the gaze of the eyes of the face based on the reconstructed 3D point cloud comprises: accessing a set of 3D point clouds, each of which corresponds to the person looking at a particular location on a display screen of an electronic device; selecting a 3D point cloud in the group that is closest to the reconstructed 3D point cloud; as well as Based on the difference between the selected 3D point cloud and the reconstructed 3D point cloud, a position on the display screen corresponding to eye gaze of the person is calculated.

Images