Method, device and equipment for eye movement tracking and storage medium

Abstract

According to the embodiment of the invention, a method and a device for eye movement tracking are provided. The method includes acquiring, by a first camera, a first image of an eye; acquiring a second image of the eye by a second camera; determining one of the first camera and the second camera as a main camera and the other as an auxiliary camera based on the information of the eye components in the first image and / or the second image; the primary camera is controlled to operate at a higher power than the secondary camera. Therefore, through switching of the main camera and the auxiliary camera, power consumption in the eye movement tracking process can be reduced.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for eye tracking. Background Art

[0002] Eye tracking technology can track the gaze direction of human eyes. Eye tracking technology has a wide range of applications, such as in the field of extended reality (XR), including virtual reality (VR), augmented reality (AR), and mixed reality (MR). By using eye tracking technology, various contents can be displayed to users more efficiently. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for eye tracking is provided. The method comprises: acquiring a first image of an eye through a first camera; acquiring a second image of the eye through a second camera; determining one of the first camera and the second camera as a main camera and the other as a secondary camera based on information of an eye component in at least one of the first image and the second image; and controlling the main camera to operate at a higher power than the secondary camera.

[0004] In a second aspect of the present disclosure, a device for eye tracking is provided. The device includes: a first image acquisition module configured to acquire a first image of an eye through a first camera; a second image acquisition module configured to acquire a second image of the eye through a second camera; a switching module configured to determine one of the first camera and the second camera as a main camera and the other as a secondary camera based on information of an eye component in at least one of the first image and the second image; and a control module configured to control the main camera to operate at a higher power than the secondary camera.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the contents described in the summary of the present invention are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 A schematic diagram showing an electronic device with an eye tracking function in which embodiments of the present disclosure can be implemented;

[0011] Figure 3A A flowchart showing a process for eye tracking according to some embodiments of the present disclosure is shown;

[0012] Figure 3B A schematic diagram showing a dynamic switching process of a primary and secondary camera according to some embodiments of the present disclosure is shown;

[0013] Figure 4A A schematic diagram showing a main camera outputting an image in a first mode according to some embodiments of the present disclosure;

[0014] Figure 4B A schematic diagram showing an auxiliary camera outputting an image in a second mode according to some embodiments of the present disclosure;

[0015] Figure 5 A block diagram showing an apparatus for eye tracking according to some embodiments of the present disclosure; and

[0016] Figure 6 A block diagram of a device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0019] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0020] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0021] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0022] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0024] As briefly mentioned above, XR has been widely studied and applied. XR integrates virtual content and real scenes by combining hardware devices and a variety of technical means to provide users with a unique sensory experience. Eye tracking technology can be applied to the XR field. For example, an electronic device with eye tracking function can track the gaze point of the human eye. However, in the process of tracking the gaze point using an electronic device with eye tracking function, the amount of data processing may be large, resulting in high power consumption.

[0025] An embodiment of the present disclosure proposes a scheme for eye tracking. According to various embodiments of the present disclosure, a first camera and a second camera installed on an electronic device with an eye tracking function respectively capture a first image and a second image of the eye. Furthermore, one of the first camera and the second camera is determined as a main camera and the other as an auxiliary camera through information on eye components in the first image and / or the second image (for example, the position and / or size of the pupil in the first image and the second image), and the main camera is controlled to operate at a higher power than the auxiliary camera. In an embodiment of the present disclosure, the main camera and the auxiliary camera can be dynamically switched according to the captured image. In this way, a camera that is more suitable for eye tracking can be operated at a higher power, thereby reducing power consumption without affecting eye tracking.

[0026] The embodiments of the present disclosure are described below in conjunction with the accompanying drawings. Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the example environment 100, a user 150 wears an electronic device 120 with an eye tracking function. The electronic device 120 can collect images of the user's 150 eyes 130-1 and eyes 130-2 (collectively referred to as eyes 130).

[0027] The controller 110 processes the image collected by the electronic device 120 and outputs the gaze point information. In some embodiments, the controller 110 may be a separate device capable of communicating with the electronic device 120, such as a server, a computing node, etc. for image or data processing.

[0028] In some embodiments, the controller 110 may be integrated with the electronic device 120 or at least coupled with the electronic device 120. For example, the controller 110 may be a module of the electronic device 120 or the controller 110 may be implemented by a processing unit in the electronic device 120.

[0029] It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and does not imply any limitation on the scope of the present disclosure.

[0030] Figure 2 A schematic diagram of an electronic device with an eye tracking function in which the embodiments of the present disclosure can be implemented is shown. Figure 2, the electronic device may include a lens 210. The lens 210 is located in front of the eye 130. The electronic device may be, for example, a helmet, glasses, or other appropriate wearable or portable device for VR, AR, XR. In some scenarios, such as when the electronic device is applied to a wearable display device, the electronic device may include lenses 210 arranged in pairs, one for the left eye and the other for the right eye. It should be understood that in some other scenarios, such as when analyzing user behavior based on eye tracking, the lens 210 may also be provided in a single piece.

[0031] In an embodiment of the present disclosure, the lens 210 may be implemented as a folded optical path lens or lens group, which may also be referred to as a pancake lens in the context of the present disclosure. It should be understood that any element having the same or similar structure and optical properties that can refract the optical path may also be used in the electronic device 120 of the present disclosure. The lens 210 may be a lens system that folds the optical path using a semi-transparent and semi-reflective polarizing film, where the light is folded back and forth between the lens, the phase delay plate, and the reflective polarizer multiple times, and finally emitted from the reflective polarizer into the human eye. The lens 210 may also be other types of lenses, and the present disclosure does not make specific limitations in this regard.

[0032] A plurality of light sources 230 - 1 , 230 - 2 , and 230 - 3 are arranged around the lens 210 , which are also collectively or individually referred to as light sources 230 . The light sources 230 are configured to illuminate the user's eyes, for example, illuminate an area adjacent to the pupil 280 to generate the glint 240 .

[0033] In some embodiments, Figure 2 As shown, the plurality of light sources 230 are arranged around the edge of the lens 210. In other embodiments, the plurality of light sources 120 are arranged in the middle portion of the lens 210. Alternatively or additionally, the plurality of light sources 120 may be embedded in the lens 210. Figure 2 The number and layout of the light sources 230 shown are only examples. The number and layout of the light sources 230 can also be any suitable number and layout, and the present disclosure does not limit the number and layout of the light sources 230.

[0034] In an embodiment of the present disclosure, the light source 230 may be implemented as a light emitting diode (LED). Alternatively or additionally, the plurality of light sources 230 may be implemented as vertical cavity surface emitting lasers (VCSEL). It should be understood that the scope of the present disclosure is not limited thereto. Other light emitting elements with suitable sizes, structures and features may be constructed as a plurality of light sources in the electronic device 120 of the present disclosure. Alternatively or additionally, since the optical properties such as the field of view of the plurality of light sources 230 (e.g., light sources 230-1, 230-2, 230-3) may be different, an optical control component may be added in front of the light source 230 (e.g., light sources 230-1, 230-2, 230-3) to adjust the optical properties of the light source 230 to be consistent.

[0035] The electronic device 120 may further include a first camera 250 and a second camera 270 configured to capture images of the user's eyes (e.g., eyes 130). For example, the first camera 250 and the second camera 270 may be eye tracking (ET) cameras. The first camera 250 and the second camera 270 may be located in any suitable location.

[0036] In some embodiments, the first camera 250 and the second camera 270 may be disposed adjacent to the end of the lens 210 along the horizontal axis H of the lens 210. When the user wears the electronic device 120, the direction of the horizontal axis H is substantially consistent with the direction of the line connecting the two eyes of the user. When the lens 210 is a common ellipse-like shape, the horizontal axis H may generally be the long axis of the ellipse.

[0037] As an example, the first camera 250 and the second camera 270 are arranged at both the proximal and distal ends of each lens 210 along the horizontal axis H of the lens 210. The proximal camera will be positioned adjacent to the bridge of the user's nose, for example, between the bridge of the nose and the inner corner of the user's eye. Since the acquisition angle for eye images at the nose and the temporal portion is relatively wide, in such an embodiment, by arranging the camera at such a position, a higher quality image of the eye can be better captured, thereby improving the accuracy of eye position determination and eye tracking. And another camera can be adjacent to the temporal portion (i.e., the temple) of the user. Another image of the eye is captured from this position. Figure 2 In the example of FIG. 1 , assuming that the eye 130 is the right eye of the user, the first camera 250 is positioned adjacent to the bridge of the nose of the user, and the second camera 270 is positioned adjacent to the temple of the user.

[0038] Of course, this arrangement is not necessary. In some other embodiments, the paired cameras can also be arranged along the longitudinal axis L of the lens 210 adjacent to the end of the lens 210. For example, the longitudinal axis L is substantially perpendicular to the transverse axis H.

[0039] It should be understood that the first camera 250 and the second camera 270 of the present disclosure are not limited to Figure 2 Without departing from the principle of the present disclosure, any suitable modification or adjustment may be made to the positions and / or numbers of the first camera and the second camera.

[0040] Considering that the user's gaze point (e.g., a point in the gaze direction of the eye 130) changes dynamically, it is necessary to obtain the real-time position of the gaze point. The controller 110 can determine the user's gaze point information, such as the gaze point position, based on the position of the glint 240 and the position of the pupil 280.

[0041] It should be understood that the above reference Figure 2 The electronic devices described are exemplary only and are not intended to limit the scope of the present disclosure.

[0042] Figure 3A 1 shows a flow chart of a process 300 for eye tracking according to some embodiments of the present disclosure. The process 300 may be implemented at the controller 110. For example only, the following reference Figure 1 and Figure 2 Process 300 is described.

[0043] At block 310, a first image of the eye is acquired by the first camera 250. At block 320, a second image of the eye is acquired by the second camera 270. Figure 2 In the example of FIG. 1 , the light source 230 illuminates the eye 130, and after being reflected by the cornea of ​​the eye 130, the eye 130 is imaged in the first camera 250 and the second camera 270. Accordingly, the first camera 250 and the second camera 270 respectively output the first image and the second image of the user's eye. For example, the first image of the right eye can be collected by the first camera 250, and the second image of the right eye can be collected by the second camera 270.

[0044] In block 330, based on the information of the eye component in the first image and / or the second image, one of the first camera 250 and the second camera 270 is determined as the main camera and the other as the auxiliary camera. The eye component may include any suitable part of the eye, such as the pupil, the iris, a combination of the pupil and the iris, etc. For example, based on the first image and the second image of the user's eye, the controller 110 may determine the information of the pupil, and further determine the main and auxiliary cameras based on the information of the pupil.

[0045] In some embodiments, the information of eye components described herein may include the position and / or size of eye components in the image. Accordingly, the controller 110 may determine one of the first camera 250 and the second camera 270 as the main camera and the other as the auxiliary camera based on the position and / or size of the pupil in the first image and the second image.

[0046] Since the first camera 250 and the second camera 270 are arranged differently along the lens, the positions and sizes of the eye components in the images captured by the first camera 250 and the second camera 270 may be different. Considering the dynamic changes in the gaze direction of the human eye, the positions and sizes of the eye components in the images captured by the first camera 250 and the second camera 270 may also be different at different times and scenes.

[0047] For ease of description, in the following embodiments, the switching of the primary and secondary cameras is mainly described by taking the pupil as an example of an eye component, but it should be understood that this can be similarly applied to other eye components. In some embodiments, if it is determined that the position of the pupil in the first image is closer to the predetermined position than the position in the second image, the controller 110 determines that the first camera 250 is the primary camera and the second camera 270 is the secondary camera. For example, the predetermined position may be the center of the image. In this case, the position of the pupil in the first image is located in the center of the first image, while the position of the pupil in the second image is away from the center of the second image, for example, close to the edge of the image. In such an embodiment, the position of the pupil in the image may be represented by the position of the center of the pupil in the image.

[0048] In some embodiments, if it is determined that the area occupied by the pupil in the first image is larger than the area occupied by the pupil in the second image, the first camera can be determined to be the main camera. For example, the area occupied by the pupil in the first image is larger than the predetermined area range, while the area occupied by the pupil in the second image is smaller than the predetermined area range.

[0049] In some embodiments, if the pupil is not detected in the image captured by a certain camera or the pupil is close to the edge of the image, it can be determined that the camera is a secondary camera and the other camera is a primary camera. For example, during dynamic switching, the image output by the current secondary camera is used to monitor the eye position. If the pupil is not detected in the image output by the current secondary camera or the pupil is close to the edge of the image, the pupil may be located in the image captured by the current primary camera. In this case, the controller 110 can determine to continue to maintain the current primary and secondary camera configuration.

[0050] In some embodiments, if the pupil in an image captured by a certain camera is incomplete, but the pupil in an image captured by another camera is complete, the camera may be determined as a secondary camera, and the other camera may be determined as a primary camera.

[0051] In the initial state of the electronic device 120 being started, the controller 110 does not obtain the images captured by the two cameras, and thus cannot determine the primary and secondary cameras based on the images. When the electronic device 120 is started, the user is likely to look directly at the display screen. In view of this, in some embodiments, in the initial state, the primary camera may be set by default to a camera located near the nose, and the secondary camera may be set by default to a camera located near the temple. As the direction of the human eye gaze changes, the controller 110 determines the primary camera and the secondary camera based on the first image captured by the first camera and the second image captured by the second camera, so that dynamic switching of the primary camera and the secondary camera is achieved between the first camera and the second camera.

[0052] Continue to refer Figure 3A , in box 340, the controller 110 controls the main camera to operate with a higher power than the auxiliary camera. For example, after determining the main camera and the auxiliary camera, the controller 110 can send configuration information to the first camera 250 and the second camera 270. The first camera 250 and the second camera 270 each operate according to the configuration information. During use, the user's eyeballs change dynamically, so the main and auxiliary of the two cameras also change dynamically. The controller 110 can control the dynamic switching of the main camera and the auxiliary camera. After each switch determines the main camera and the auxiliary camera, the main camera is controlled to operate with a higher power than the auxiliary camera. This can reduce power consumption.

[0053] The operating modes of the main and auxiliary cameras can be distinguished in various suitable ways. In some embodiments, the main camera outputs an image in a first mode, in which the size of the image output by the main camera is changed. In other words, the size of the output image changes compared to the image initially captured by the main camera. Specifically, the controller 110 may send configuration information about the first mode (also referred to as the cropping mode) to the main camera. The configuration information may indicate a region of interest (ROI) in the image. According to the configuration information, the main camera may crop the captured original image based on the ROI and output the cropped image. The cropped image contains at least the ROI, and the size of the cropped image is changed relative to the image size before cropping. Reference Figure 4A Describe an example. Figure 4A The image 410 before cropping is presented, and the image 410 before cropping includes an array of pixels in rows. It should be understood that Figure 4A The number and arrangement of pixels shown in are exemplary only and are not intended to limit the scope of the present disclosure. In embodiments of the present disclosure, there may be any suitable number of pixels arranged in any suitable pattern.

[0054] Figure 4AIn the example of , ROI 450 is shown. In some embodiments, ROI 450 can be predetermined, for example, located in the central area of ​​the image. In some embodiments, the image area covered by ROI 450 can be dynamically changed. For example, the controller 110 can determine the coverage of ROI 450 in the image based on various suitable factors, such factors can, for example, include the position and / or size of the pupil, the position and / or size of the iris. As an example, the area occupied by the pupil and the iris in the image can be used as the ROI. Information about the ROI can be included in the configuration information sent by the controller 110 to the main camera. Accordingly, the main camera crops the captured original image based on the configuration information, thereby outputting the cropped image 420. In this example, the image 420 only includes ROI 450.

[0055] In some embodiments, the position of the ROI in the image may change dynamically with each switch between the primary camera and the secondary camera. In the example of FIG. 4 , the position of the ROI 450 may also be located at a different position of the image 410 .

[0056] In some embodiments, the ROI may also be determined by the image of the auxiliary camera and the image of the main camera. To this end, the controller 110 may determine the region of interest in the image captured by the main camera based on the calibration information. The calibration information indicates the conversion relationship between the image coordinates of the auxiliary camera and the image coordinates of the main camera. For example, in the case where the position and / or size of the pupil is also displayed in the image of the auxiliary camera, the controller 110 may convert the position and / or size of the pupil in the image of the auxiliary camera into the position and / or size of the pupil in the image of the main camera based on the calibration information. Further based on the position and / or size of the converted pupil, the position of the ROI in the image captured by the main camera is determined.

[0057] In this way, in the first mode, the size of the image output by the main camera is reduced, and power consumption can be reduced.

[0058] The operating mode of the main camera is described above. In contrast, the auxiliary camera outputs an image in a second mode. In the second mode, the number of pixels of the image output by the auxiliary camera is reduced. In other words, the image output by the auxiliary camera has a reduced number of pixels compared to the original image captured, thereby reducing power consumption.

[0059] Specifically, the controller 110 may send configuration information about the second mode (also referred to as the merge mode) to the auxiliary camera. The configuration information may indicate how to merge adjacent pixels in the image, thereby reducing the number of pixels of the output image. According to the configuration information, the auxiliary camera may merge a certain number of adjacent pixels in the original image and output the merged image.

[0060] refer to Figure 4B Describe an example. Figure 4B An image 430 is presented, which is an example of a raw image captured by the secondary camera. Image 430 includes an array of pixels arranged in rows. It should be understood that Figure 4B The number of pixels and their arrangement shown in are exemplary only and are not intended to limit the scope of the present disclosure. In embodiments of the present disclosure, there may be any suitable number of pixels arranged in any suitable pattern. In the second mode, the auxiliary camera processes the captured image 430 into an output image 435.

[0061] As an example, the secondary camera combines multiple adjacent pixels into a single pixel output. Figure 4B 4 shows that the auxiliary camera combines four adjacent pixels 470 into one pixel 490 for output. For example, the auxiliary camera can use the average pixel value of the four adjacent pixels as the combined pixel value to perform pixel combination. Similarly, the auxiliary camera combines other adjacent pixels in the image 430 to output a low-pixel image 435. It should be understood that Figure 4B The output method of reducing pixels shown is only an example, and the present disclosure is not limited in this regard. For example, six, eight, etc. adjacent pixels can also be combined into one pixel for output.

[0062] In this way, the number of pixels of the image output by the auxiliary camera is reduced, thereby effectively reducing power consumption. In addition, although the pixels of the image output by the auxiliary camera are reduced, the original field of view is maintained. Therefore, the image output by the auxiliary camera can be used to monitor the eye position in order to switch between the main and auxiliary cameras.

[0063] refer to Figure 4A and Figure 4B The cropping mode of the primary camera and the merging mode of the secondary camera are described. It should be understood that in such an embodiment, the first image and the second image described above may be cropped or merged images output by the camera. However, it should be understood that the cropping mode and the merging mode are only examples, and in the embodiments of the present disclosure, the operating modes of the primary and secondary cameras may be distinguished in any suitable manner. Alternatively, in some embodiments, the primary and secondary cameras may both operate in the merging mode, but more adjacent pixels are merged in the secondary camera than in the primary camera.

[0064] Reference below Figure 3B 390 is described to describe the dynamic switching process 390 of the first camera and the second camera as the main camera and the auxiliary camera respectively. In block 3310, the main camera and the auxiliary camera are determined. For example, the controller 110 determines the main camera and the auxiliary camera as described with reference to block 330.

[0065] After determining the main camera and the auxiliary camera, in block 3330, configuration information of the main and auxiliary cameras is generated. The configuration information may be sent to the first camera and the second camera. The configuration information may indicate whether the first camera is the main camera or the auxiliary camera, and whether the second camera is the main camera or the auxiliary camera. For example, the configuration information sent to the main camera may be the configuration information about the first mode mentioned above, and the configuration information sent to the auxiliary camera may be the configuration information about the second mode mentioned above.

[0066] At block 3350, determine whether the first camera is a primary camera. The first camera 250 may determine whether it is a primary camera based on the received configuration information. For example, if the received configuration information indicates a first mode, the first camera may be determined to be a primary camera, and if the received configuration indicates a second mode, the first camera may be determined to be a secondary camera. If the first camera is a primary camera, the process 390 proceeds to block 3371, and the first camera outputs an image in the first mode as a primary camera. If the first camera is a secondary camera, the process 390 proceeds to block 3373, and the first camera outputs an image in the second mode as a secondary camera.

[0067] Similarly, in box 3370, it is determined whether the second camera is the main camera. The second camera 270 can determine whether it is the main camera based on the received configuration information. For example, if the received configuration information indicates the first mode, the second camera can be determined to be the main camera, and if the received configuration indicates the second mode, the second camera can be determined to be the auxiliary camera. If the second camera is the main camera, the process 390 proceeds to box 3372, and the second camera 270 outputs images in the first mode as the main camera. If the second camera is the auxiliary camera, the process 390 proceeds to box 3374, and the second camera outputs images in the second mode as the auxiliary camera. It should be understood that boxes 3371 and 3372 do not occur at the same time, and boxes 3373 and 3374 do not occur at the same time.

[0068] Process 390 returns to block 3310, where the primary and secondary cameras at the next moment are determined based on the image output in the first mode and the image output in the second mode. That is, the controller 110 uses the image output in the first mode and the image output in the second mode for determining the primary and secondary cameras next time.

[0069] The above describes the switching of the primary and secondary cameras. Now, the processing of the images output by the primary and secondary cameras will be described.

[0070] In some embodiments, the image output by the primary camera and the image output by the secondary camera can be processed separately. For example, an eye tracking algorithm can be applied to the image output by the primary camera to determine the gaze point information. The image output by the secondary camera can be used to monitor eye movement in order to switch between the primary and secondary cameras.

[0071] In some embodiments, the image output by the main camera and the image output by the auxiliary camera can be used in combination. The controller 110 combines the image output by the main camera and the image output by the auxiliary camera, and uses the combined image to track the eyes to obtain gaze point information.

[0072] The controller 110 may perform feature extraction on the image output by the main camera (e.g., the cropped image) and the image output by the auxiliary camera (e.g., the merged image). Further, the controller 110 calculates the gaze angle and the position of the pupil (e.g., the center position of the pupil) based on the extracted features. Then, the controller 110 determines the gaze point information, such as the gaze point position, based on the gaze angle and the position of the pupil. In the above process of determining the gaze point information, any suitable ET algorithm may be used, and the scope of the present disclosure is not limited in this respect.

[0073] In an embodiment of the present disclosure, the controller 110 or a portion thereof (e.g., a module for controlling the primary and secondary cameras) may be integrated into the first camera and / or the second camera, or at least may be coupled to the first camera and / or the second camera to obtain the first image and the second image captured by the first camera and the second camera. For example, the controller 110 or a portion thereof (e.g., a module for controlling the primary and secondary cameras) may be implemented as a custom processing chip integrated with the camera sensor. In this way, the data transmission delay from the first camera and / or the second camera to the controller 110 may be saved. Alternatively, the controller 110 may be implemented as a separate computing unit or on the same computing unit as the eye tracking algorithm.

[0074] In the embodiments of the present disclosure, by dynamically switching between the main camera and the auxiliary camera, power consumption can be effectively reduced, while also ensuring the accuracy of gaze angle tracking and the accuracy of pupil center position.

[0075] In addition, according to the embodiments of the present disclosure, the user calibration process can also be simplified. Specifically, since each user's eyes have different characteristics, after the user puts on the electronic device 120, in order to make the user view the display screen more clearly, the user's line of sight needs to be calibrated. In such a dual-camera electronic device, only a small number (for example, one) of calibration points need to be displayed on the display screen. By obtaining the gaze point position when the user views the calibration point, the difference between the user's optical axis and visual axis can be determined, and then calibrated according to the difference between the user's optical axis and visual axis. Thus, a better and simpler calibration process can be achieved.

[0076] Figure 5: A schematic structural block diagram of an apparatus 500 for eye tracking according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in an electronic device 120, such as in a controller 110. Each module / component in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0077] As shown in the figure, the device 500 includes a first image acquisition module 510, which is configured to acquire a first image of the eye through a first camera. The device 500 also includes a second image acquisition module 520, which acquires a second image of the eye through a second camera. The device 500 also includes a switching module 530, which is configured to determine one of the first camera and the second camera as a main camera and the other as a secondary camera based on information of the eye component in at least one of the first image and the second image. The device 500 also includes a control module 540, which is configured to control the main camera to operate at a higher power than the secondary camera.

[0078] In some embodiments, the control module 540 is also configured to determine that the main camera outputs an image in a first mode, in which the size of the image output by the main camera is changed; and to determine that the auxiliary camera outputs an image in a second mode, in which the number of pixels of the image output by the auxiliary camera is reduced.

[0079] In some embodiments, the switching module 530 is further configured to determine the primary camera based on the location and / or size of the eye component in the first image and the second image.

[0080] In some embodiments, the switching module 530 is also configured to: determine that the first camera is the main camera in response to determining that the position of the eye component in the first image is closer to a predetermined position than the position in the second image; and / or determine that the first camera is the main camera in response to determining that the area occupied by the eye component in the first image is larger than the area occupied by the eye component in the second image.

[0081] In some embodiments, the image output by the main camera is obtained by cropping the image captured by the main camera based on the area of ​​interest.

[0082] In some embodiments, the apparatus 500 further comprises a calibration module configured to determine a region of interest in an image captured by the primary camera based on calibration information indicating a conversion relationship between image coordinates of the secondary camera and image coordinates of the primary camera.

[0083] In some embodiments, the device 500 also includes a gaze point information output module, which is configured to combine the image output by the main camera and the image output by the auxiliary camera; and use the combined image to track the eyes to obtain gaze point information.

[0084] Figure 6 A block diagram illustrating a computing device 600 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that Figure 6 The computing device 600 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 6 The computing device 600 shown may be used to implement Figure 1 Controller 110.

[0085] like Figure 6 As shown, computing device 600 is in the form of a general-purpose computing device. Components of computing device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to increase the parallel processing capabilities of computing device 600.

[0086] The computing device 600 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the computing device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the computing device 600.

[0087] The computing device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 6 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0088] The communication unit 640 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the computing device 600 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0089] Input device 650 may be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 may be one or more output devices, such as a display, speaker, printer, etc. Computing device 600 may also communicate with one or more external devices (not shown) through communication unit 640 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with computing device 600, or communicate with any device that allows computing device 600 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0090] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0091] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0092] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0093] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0094] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0095] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for eye tracking, comprising: Acquire a first image of the eye by a first camera; Acquiring a second image of the eye by a second camera; Based on information of eye components in at least one of the first image and the second image, determining one of the first camera and the second camera as a main camera and the other as a secondary camera; as well as The main camera is controlled to operate at a higher power than the auxiliary camera.

2. The method of claim 1, wherein controlling the primary camera to operate at a higher power than the secondary camera comprises: determining that the main camera outputs an image in a first mode, in which a size of the image output by the main camera is changed; The auxiliary camera is determined to output an image in a second mode in which the number of pixels of the image output by the auxiliary camera is reduced.

3. The method of claim 1 , wherein determining one of the first camera and the second camera as a primary camera comprises: The main camera is determined based on the position and / or size of the eye component in the first image and the second image.

4. The method according to claim 3, wherein determining the main camera based on the position and / or size of the eye component in the first image and the second image comprises: In response to determining that the position of the eye component in the first image is closer to a predetermined position than in the second image, determining the first camera to be the primary camera; and / or In response to determining that an area occupied by the eye component in the first image is larger than an area occupied by the pupil in the second image, the first camera is determined to be the main camera. 5 . The method according to claim 2 , wherein the image output by the main camera is obtained by cropping the image captured by the main camera based on a region of interest.

6. The method according to claim 5, further comprising: The region of interest is determined in the image captured by the primary camera based on calibration information, wherein the calibration information indicates a conversion relationship between image coordinates of the secondary camera and image coordinates of the primary camera.

7. The method according to claim 1, further comprising: combining the image output by the main camera and the image output by the auxiliary camera; as well as The eyes are tracked using the combined image to obtain gaze point information.

8. An apparatus for eye tracking, comprising: A first image acquisition module is configured to acquire a first image of the eye through a first camera; A second image acquisition module is configured to acquire a second image of the eye through a second camera; a switching module configured to determine one of the first camera and the second camera as a main camera and the other as a secondary camera based on information of eye components in at least one of the first image and the second image; as well as The control module is configured to control the main camera to operate at a higher power than the auxiliary camera.

9. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 7 when executed by the at least one processing unit.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 7.

Images