Method and system for realizing eye gaze tracking positioning, and computer device

Abstract

The application provides a method and system for realizing eyeball visual line tracking positioning, and a computer device, and relates to the technical field of image processing. The method comprises the following steps: enabling a first camera to detect whether a user gazes at a display device; when it is detected that the user gazes at the display device, enabling a second camera and a third camera to respectively take images of the left and right eyeballs of the user, generating a first eyeball image and a second eyeball image, and constructing an eyeball model; controlling the display device to display a current image, and controlling the second camera and the third camera to respectively take reflection images of the left and right eyeballs of the user to obtain a first current eyeball image and a second current eyeball image; constructing a visual line mapping relationship of the eyeball model according to the first current eyeball image and the second current eyeball image; and obtaining the visual line coordinates of the user according to the visual line mapping relationship, so as to position the eyeball visual line on the display device. The technical scheme of the application does not require the user to wear additional equipment, reduces the dependence on specific environments, and improves the user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a method and system for realizing eye gaze tracking positioning and a computer device. BACKGROUND

[0002] Existing eye gaze tracking technology usually relies on special equipment, such as an eye tracker or an infrared reflection device, to realize gaze positioning by capturing eye reflection light or wearing sensors. However, the limitations of the prior art mainly lie in device dependence and insufficient environmental adaptability. Specifically, the special equipment performs well in a laboratory environment, but usually requires users to wear additional hardware or operate under specific lighting conditions to ensure tracking accuracy, which limits the user experience. SUMMARY

[0003] Therefore, it is necessary to provide a method and system for realizing eye gaze tracking positioning and a computer device.

[0004] In a first aspect, an embodiment of the present application provides a method for realizing eye gaze tracking positioning, which comprises: enabling a first camera to detect whether a user gazes at a display device; when it is detected that the user gazes at the display device, simultaneously enabling a second camera and a third camera to respectively capture left and right eyeballs of the user, to generate a first eyeball image and a second eyeball image, and to construct corresponding first and second eyeball models; controlling the display device to display a current image, and controlling the second and third cameras to respectively capture reflection images of the left and right eyeballs of the user, to obtain a first current eyeball image and a second current eyeball image containing the current image; constructing a gaze mapping relationship of the corresponding eyeball models according to the first and second current eyeball images, to map the relationship between a current eyeball image of the user and a gaze coordinate; and obtaining the gaze coordinate of the user according to the gaze mapping relationship, to position the eye gaze of the user on the display device.

[0005] In a second aspect, the embodiments of the present application provide a system for tracking and positioning eye gaze, which comprises a display device, an image acquisition device and a host device. The display device is configured to display images. The image acquisition device comprises a first camera, a second camera and a third camera. The host device is communicatively connected to the display device and the image acquisition device, and is configured to: enable the first camera to detect whether a user gazes at the display device; when detecting that the user gazes at the display device, enable the second camera and the third camera to capture left and right eyeballs of the user respectively, generate first and second eyeball images, and construct corresponding first and second eyeball models; control the display device to display a current image, and control the second and third cameras to capture reflection images of the left and right eyeballs of the user respectively, so as to obtain first and second current eyeball images containing the current image; construct a gaze mapping relationship of the corresponding eyeball models according to the first and second current eyeball images, so as to map a relationship between a current eyeball image of the user and a gaze coordinate; and obtain the gaze coordinate of the user according to the gaze mapping relationship, and position the eye gaze of the user on the display device.

[0006] In a third aspect, the embodiments of the present application provide a computer device, which comprises a memory and a processor. The memory is configured to store a computer program. The processor is configured to execute the computer program to implement the method for tracking and positioning eye gaze.

[0007] The method and system for tracking and positioning eye gaze, and the computer device, detect a gazing state of a user and construct eyeball models by using multiple cameras, dynamically capture eyeball reflection images and construct a gaze mapping relationship, without the need for the user to wear additional devices, thereby reducing dependence on specific environments and improving user experience. BRIEF DESCRIPTION OF DRAWINGS

[0008] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from the structures shown in the drawings without creative labor.

[0009] Figure 1 The first flowchart of the method for tracking and positioning eye gaze provided by the embodiments of the present application.

[0010] Figure 2 The flowchart of step S102 provided by the embodiments of the present application.

[0011] Figure 3 The flow chart of step S103 provided for the embodiment of the present application.

[0012] Figure 4 The flow chart of step S1031 provided for the embodiment of the present application.

[0013] Figure 5 The flow chart of step S104 provided for the embodiment of the present application.

[0014] Figure 6 The second flow chart of the method for realizing eye gaze tracking positioning provided for the embodiment of the present application.

[0015] Figure 7 The structural schematic diagram of the system for realizing eye gaze tracking positioning provided for the embodiment of the present application.

[0016] Figure 8 The internal structural schematic diagram of the computer device for applying the method for realizing eye gaze tracking positioning provided for the embodiment of the present application.

[0017] Figure 9 The schematic diagram of constructing the gaze mapping relationship provided for the embodiment of the present application.

[0018] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0020] The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, in other words, the described embodiments are implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof can also include other contents, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0021] It should be noted that the terms "first", "second", etc. in the present application are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second" can be explicitly or implicitly included one or more of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the protection scope required by the present application.

[0022] The existing eye gaze tracking technology usually relies on special equipment, such as eye tracker or infrared reflection device, to realize gaze positioning by capturing eye reflection light or wearing sensor. But usually need the user to wear additional hardware or run under certain light conditions to ensure tracking accuracy, which limits the user experience. The present application proposes a method for realizing eye gaze tracking positioning, and realizes tracking by using multiple cameras to detect the user's gaze state and construct eye model and corresponding gaze mapping relationship, without the need for the user to wear additional equipment, thereby improving the user experience.

[0023] Please refer to Figure 1 , which is the first flowchart of the method for realizing eye gaze tracking positioning provided by the embodiment of the present application. The specific steps S101-S105 of realizing eye gaze tracking positioning.

[0024] Step S101, enable the first camera to detect whether the user gazes at the display device.

[0025] In step S101, the field of view range of the first camera 21 covers the area where the user can observe the display content of the display device. For example, the display device is a display screen of a computer device placed on the table, then the first camera 21 can be set above the display screen, and the field of view range of the first camera 21 covers the user operating the computer device in front of the table, so as to improve the capture and tracking effect of the first camera 21 on the user's eye gaze.

[0026] Step S102, when detecting that the user gazes at the display device, simultaneously enable the second camera and the third camera to shoot the left and right eyes of the user respectively, generate the first eye image and the second eye image, and construct the corresponding first eye model and the second eye model.

[0027] In step S102, the second camera 22 is used to capture one of the user's left and right eyeballs, and the third camera 23 is used to capture the other one of the user's left and right eyeballs. The first eyeball model corresponds to the first eyeball image, and the second eyeball model corresponds to the second eyeball image. The second camera 22 and the third camera 23 are used to capture the left and right eyeballs of the user only when it is detected that the user gazes at the display device, that is, the cameras are started immediately when the user gazes at the display device, so that the eyeball state of the user can be quickly captured and the eyeball model can be constructed accordingly, thereby ensuring real-time tracking effect in a dynamic scene. Meanwhile, the left and right eyeballs of the user are captured at the same time, so that when the independent eyeball models are generated, the geometric features and reflection information of the eyeballs at the current moment can be more comprehensively captured, thereby improving the accuracy of the gaze positioning.

[0028] Further, the second camera 22 and the third camera 23 can also be arranged above the display screen. In an embodiment, the first camera 21, the second camera 22 and the third camera 23 are arranged side by side above the display screen, and the first camera 21 is located between the second camera 22 and the third camera 23. In another embodiment, the first camera 21, the second camera 22 and the third camera 23 are integrated into a camera module, and the field of view range of the camera module covers the area in which the user can observe the display content of the display device.

[0029] Please refer to Figure 2 which is a flowchart of step S102 provided by an embodiment of the present application. The construction of the corresponding first eyeball model and second eyeball model includes steps S1021-S1023.

[0030] In step S1021, eyeball data is extracted from the first eyeball image and the second eyeball image, respectively.

[0031] In step S1021, the eyeball data is obtained by combining the visual difference generated when different eyeballs are observed and different eyeball images. The eyeball data includes eyeball size, curvature and center point position. In the present application, the eyeball data includes first eyeball data corresponding to the first eyeball image and second eyeball data corresponding to the second eyeball image. The first eyeball data and the second eyeball data are respectively three-dimensional depth data of a plurality of key points for representing different parts of the eyeball. The plurality of key points can be, for example, pupil center, corneal reflection point, canthus, etc. The three-dimensional depth data is obtained by three-dimensional point cloud.

[0032] Further, the first eyeball image and the second eyeball image in the present application can also be used to judge the distance between the eyeball and the display device, so as to accurately reflect different ways of the user's gaze display state in combination with the three-dimensional depth data.

[0033] In step S1022, the corresponding first eyeball model and second eyeball model are generated based on the extracted eyeball data.

[0034] In step S1022, the corresponding eyeball model is generated mainly by the three-dimensional point cloud of the eyeball part. Specifically, taking the first eyeball model as an example, first, the three-dimensional depth data of the eye corner part, the highest position of the eyeball, the upper part of the eyeball and the lower part of the eyeball are extracted from the corresponding first eyeball data by the three-dimensional point cloud; then, the eyeball size and the eyeball surface are constructed according to the three-dimensional depth data extracted above; then, the center point of the eyeball is determined according to the three-dimensional depth data of the highest position of the eyeball, combined with the pupil position in the first eye image; finally, the eyeball curvature is obtained according to the eyeball surface, combined with the eyeball size and the eyeball center point obtained to obtain the first eyeball model. Correspondingly, the second eyeball model can be extracted according to the above generation method of the first eyeball model from the corresponding eyeball data. Then the three-dimensional depth data required is obtained, and the second eyeball model is obtained.

[0035] In step S1023, the first eyeball image and the second eyeball image are respectively mapped to the corresponding eyeball model, so that the first eyeball image and the second eyeball image are respectively matched to the corresponding eyeball data.

[0036] In step S1023, the first eyeball image is matched to the first eyeball data, and the second eyeball image is matched to the second eyeball data. In the present application, by respectively mapping the first eyeball image and the second eyeball image to the corresponding eyeball model, the corresponding eyeball data can be more accurately positioned to different key points of different eyeballs, which is convenient for analyzing different states of the eyeball when the user gazes at things, such as the eyeball size, the eyeball curvature, etc., improving the accuracy of subsequent acquisition of the eyeball line of sight, and further improving the accuracy of eyeball line of sight tracking.

[0037] In step S103, the display device is controlled to display the current image, and the second camera and the third camera are controlled to respectively capture the reflection images of the left and right eyeballs of the user to obtain the first current eyeball image and the second current eyeball image containing the current image.

[0038] In step S103, the current image is one of a plurality of preset images displayed on the display device. The plurality of preset images can be the same or different images. The display device has a first state of displaying an image and a second state of not displaying an image, and the display device is initially in the second state. More specifically, the display device in the second state can not display any image. The plurality of preset images are displayed in different positions of the display device in turn. When the display device is in the first state and displays a current image, the display device also displays prompt information prompting the user to gaze at the current image, so as to obtain a plurality of eye gaze lines of the user through different preset images, and facilitate analysis of different states of the eye of the user when gazing at an object. In the present application, after the first eye image and the second eye image are acquired by the second camera 22 and the third camera 23 respectively and the corresponding eye models are constructed, the images of the left and right eyes reflecting the current image are photographed again by the second camera 22 and the third camera 23 respectively, so as to take each position of the current image as a reference point of the gaze line mapping, and help the eye model to more accurately calculate the coordinates of the user's gaze line.

[0039] Referring to Figure 3 which is a flowchart of step S103 provided by an embodiment of the present application. The display device is controlled to display a current image, and the second camera 22 and the third camera 23 are controlled to photograph the reflection images of the left and right eyes of the user respectively, so as to acquire the first current eye image and the second current eye image containing the current image, including steps S1031-S1032.

[0040] In step S1031, each of the plurality of preset images is set as a current image in turn.

[0041] In step S1032, the second camera and the third camera are controlled to photograph the reflection images of the left and right eyes of the user respectively to acquire the corresponding first current eye image and the second current eye image each time a current image is displayed, until the first current eye image and the second current eye image corresponding to all preset images are acquired.

[0042] Referring to Figure 4 which is a flowchart of step S1031 provided by an embodiment of the present application. Setting each of the plurality of preset images as a current image in turn includes steps S10311-S10312.

[0043] In step S10311, when it is detected that the user gazes at the display device, the display device is controlled to switch to the first state and display a current image, and the reference position data of the current image is acquired.

[0044] In step S10311, the reference position data of the current image corresponds to the coordinates of the positions of the preset images displayed on the display device in turn.

[0045] Step S10312: When the first current eye image and the second current eye image corresponding to all preset images are not obtained, each of the remaining preset images is set as the current image in sequence until the reference position data of all preset images and the corresponding first current eye image and the second current eye image are obtained.

[0046] Step S104: Based on the first current eye image and the second current eye image, construct the gaze mapping relationship of the corresponding eye model to map the relationship between the user's current eye image and gaze coordinates.

[0047] In step S104, the gaze mapping relationship includes a first gaze mapping relationship corresponding to the first eyeball model and a second gaze mapping relationship corresponding to the second eyeball model. This application obtains multiple current eyeball images when the user is looking at multiple different positions on the display device based on the different reflections of the current image by different eyeballs, in order to comprehensively analyze the different states of the eyeballs when the user is looking at something, facilitating the subsequent acquisition of the user's gaze position on the display device by constructing an eyeball model with gaze mapping relationships.

[0048] Please refer to Figure 5 This is a flowchart of step S104 provided in the embodiments of this application. Constructing the gaze mapping relationship of the corresponding eye model based on the first current eye image and the second current eye image includes steps S1041-S1044.

[0049] Step S1041: Obtain the position data of each first current eye image based on the first current eye image and the first eye model corresponding to each current image.

[0050] Step S1042: Based on each location data and the corresponding reference location data, construct a first line-of-sight mapping relationship and obtain a first correction value.

[0051] Step S1043: Obtain the position data of each second current eye image based on the second current eye image and the second eye model corresponding to each current image.

[0052] Step S1044: Based on each position data and the corresponding reference position data, construct a second line-of-sight mapping relationship and obtain a second correction value.

[0053] The following section will explain in detail how to construct the line-of-sight mapping relationship.

[0054] like Figure 9As shown, let the width of the display device be X and the height be Y. Then, the reference position data of the four corners A, B, C, and D of the display device are (0, 0), (X, 0), (0, Y), and (X, Y), respectively. Assume that multiple preset images are the current images Pa, Pb, Pc, and Pd displayed at the four corners of the display device, and each is a square. First, let F1 and F2 be the distances from the left and right eyeballs to the center points of the two cameras on the sides of the display device. A square-shaped current image, denoted as Pa, is displayed at coordinates (0, 0) in the upper left corner A. Then, the first and second current eye images are captured by the second camera 22 and the third camera 23, determining the position of Pa in the reflection of the left and right eyeballs. Next, based on the corresponding eyeball model, the central axes of the two eyeballs intersect at Pa, denoted as Zal and Zar respectively. Then, a straight line Za is drawn connecting the center points of the two eyeballs to the center point of Pa. Next, a new current image Pb is displayed in the upper right corner of the display device, and the steps for obtaining the central axes of the two eyeballs are executed to obtain the new central axes Zbl and Zbr. A straight line Zb is drawn connecting the center points of the two eyeballs to the center point of Pb, thus obtaining the first angle difference Vxab and the second angle difference Vyab between Za and Zb. When the position data of corner B is obtained as (Xb, Yb), then the first correction value = Xb / Vxab, Yb = 0.

[0055] Similarly, at angle C, a new current image Pc is displayed, and the previously obtained midlines of the two eyeballs are used to obtain the new midlines Zcl and Zcr of the two eyeballs below angle C. Then, a straight line Zc is drawn to obtain the third angle difference Vxac and the fourth angle difference Vyac. When the position data of angle C is obtained as (Xc, Yc), then the second correction value = Yb / Vyab, Xb = 0.

[0056] Similarly, at angle D, a new current image Pd is displayed, and the previously obtained midlines of the two eyeballs are used to obtain the new midlines Zdl and Zdr of the two eyeballs at angle D. Then, a straight line Zd is drawn to obtain the fifth angle difference Vxad and the sixth angle difference Vyad. When the position data of angle D is obtained as (Xd, Yd), combined with the first correction value = Xb / Vxab and the second correction value = Yb / Vyab, the position data of angle D can be expressed as Xd = Vxad * first correction value and Yd = Vyad * second correction value.

[0057] Furthermore, when it is necessary to output the position data of other points on the display device that the user is looking at, the central axis of the eyeball can be obtained through 3D modeling, and the first angle Vx and the second angle Vy formed with Za can be obtained. When the position data of other points is denoted as (X, Y), then X = Vx * first correction value, Y = Vy * second correction value, and the position data of other points can be obtained to obtain the coordinates of the user's gaze.

[0058] Step S105: Obtain the user's gaze coordinates based on the gaze mapping relationship, so as to locate the user's eye gaze on the display device.

[0059] In step S105, the gaze coordinates include eye position information and gaze direction information. Eye gaze can be used for non-contact interaction between the user and the display device. For example, changes in the user's gaze coordinates can be used to obtain different information the user is looking at on the display device. This allows the user to control the display of specific information by maintaining or changing their gaze coordinates when they are particularly focused on it, thereby achieving interaction.

[0060] Please refer to Figure 6 This is a second flowchart of the method for implementing eye-tracking positioning provided in the embodiments of this application. The method for implementing eye-tracking positioning further includes steps S201-S204.

[0061] Step S201: Control the first camera to detect whether there is a change in the user's facial posture.

[0062] In step S201, the facial pose change includes facial movement and eye movement.

[0063] In step S202, when a change in the user's facial pose is detected, the second and third cameras are simultaneously controlled to re-capture the user's left and right eyeballs to update the first and second eyeball images.

[0064] Step S203: Update the corresponding eye model based on the updated first eye image and second eye image.

[0065] In step S203, during the eye tracking process, the user's eye state may undergo various changes in a short period of time, such as slight movements of the eye position and opening and closing of the eyelids. Since the first and second eye images are mapped to the corresponding eye models, by mapping the updated first and second eye images to the updated eye models, the eye models can dynamically adapt to changes in the eyes, thereby maintaining accurate tracking of the eye state.

[0066] Step S204: Based on the current image, construct the gaze mapping relationship of the updated eyeball model.

[0067] Please refer to Figure 7This is a schematic diagram of the structure of a system for implementing eye-tracking positioning provided in an embodiment of this application. This application also provides a system 1000 for implementing eye-tracking positioning. System 1000 includes a display device 1, an image acquisition device 2, and a main control device 3. The display device 1 is used to display images. The image acquisition device 2 includes a first camera 21, a second camera 22, and a third camera 23. The main control device 3 is communicatively connected to the display device 1 and the image acquisition device 2, and is used to perform the following operations: activating the first camera to detect whether the user is looking at the display device; when the user is detected looking at the display device, simultaneously activating the second and third cameras to capture images of the user's left and right eyeballs respectively, generating a first eyeball image and a second eyeball image, and constructing corresponding first and second eyeball models; controlling the display device to display the current image, and controlling the second and third cameras to capture reflection images of the user's left and right eyeballs respectively, to obtain a first current eyeball image and a second current eyeball image containing the current image; constructing a gaze mapping relationship for the corresponding eyeball model based on the first and second current eyeball images, to map the relationship between the user's current eyeball image and gaze coordinates; and obtaining the user's gaze coordinates based on the gaze mapping relationship to locate the user's eye gaze on the display device.

[0068] Please refer to Figure 8 This is a schematic diagram of the internal structure of a computer device for implementing an eye-tracking positioning method according to an embodiment of this application.

[0069] like Figure 8 As shown, the computer device 100 includes a memory 901 and a processor 902. The processor 902 is used to execute computer program instructions stored in the memory 901 to implement a method for eye-tracking positioning.

[0070] The memory 901 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 901 can be an internal storage unit of a computer device, such as a hard disk. In other embodiments, the memory 901 can be an external storage device of a computer device, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., configured in the computer device. Furthermore, the memory 901 can include both internal and external storage units of the computer device. The memory 901 can be used not only to store application software and various types of data installed on the computer device, such as code for methods of eye-tracking positioning, but also to temporarily store data that has been output or will be output.

[0071] Furthermore, the computer device 100 also includes a bus 903. The bus 903 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0072] Furthermore, the computer device 100 may also include a display component 904. The display component 904 may be an LED display, a liquid crystal display, a touch-screen liquid crystal display, or an organic light-emitting diode (OLED) touchscreen, etc. The display component 904 may also be appropriately referred to as a display device or display unit, used to display information processed in the computer device 100 and to display a visual user interface.

[0073] Furthermore, the computer device 100 may also include a communication component 905. The communication component 905 may optionally include a wired communication component and / or a wireless communication component (such as a Wi-Fi communication component, a Bluetooth communication component, etc.), which is typically used to establish a communication connection between the computer device 100 and other computer devices.

[0074] Figure 8 Only a partial computer device 100 with a method for implementing eye-tracking positioning is shown; those skilled in the art will understand that...Figure 8 The structure shown does not constitute a limitation on the computer device 100 and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0075] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, in the form of a computer program product.

[0076] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to embodiments of the present invention is generated. The computer device may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0077] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0078] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0079] The unit described as a separate component may or may not be physically separate. The component shown as a unit may or may not be a physical unit; that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0080] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist independently, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0081] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard disks, read-only storage media (ROM), random access storage media (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0082] In the above embodiments, by using multiple cameras to detect the user's gaze state and construct an eye model, and by dynamically capturing eye reflection images and constructing a gaze mapping relationship, the user does not need to wear additional devices, reducing dependence on specific environments and improving the user experience.

[0083] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0084] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0085] The above-listed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for implementing eye gaze tracking positioning, characterized by, The method comprises: enabling a first camera to detect whether a user gazes at a display device, a field of view range of the first camera covering an area in which the user can observe display content of the display device; when detecting that the user gazes at the display device, simultaneously enabling a second camera and a third camera to respectively capture left and right eyeballs of the user, to generate a first eyeball image and a second eyeball image, and to construct a corresponding first eyeball model and a second eyeball model; controlling the display device to display a current image, and controlling the second camera and the third camera to respectively capture reflection images of the left and right eyeballs of the user, to obtain a first current eyeball image and a second current eyeball image containing the current image; the current image being one of a plurality of preset images displayed on the display device, the plurality of preset images being the same or different images; according to the first current eyeball image and the second current eyeball image, constructing a line-of-sight mapping relationship of the corresponding first eyeball model and the second eyeball model, to map a relationship between a current eyeball image of the user and a line-of-sight coordinate; according to the line-of-sight mapping relationship, obtaining a line-of-sight coordinate of the user, to position an eyeball line-of-sight of the user on the display device; wherein the method further comprises: controlling the first camera to detect whether there is a change in a facial posture of the user, the change in the facial posture including facial movement and eyeball activity; when detecting the change in the facial posture of the user, simultaneously controlling the second camera and the third camera to re-capture the left and right eyeballs of the user, to update the first eyeball image and the second eyeball image; based on the updated first eyeball image and the second eyeball image, updating the corresponding eyeball models; according to the current image, constructing a line-of-sight mapping relationship of the updated eyeball models.

2. The method of claim 1, wherein, constructing the corresponding first eyeball model and the second eyeball model comprises: respectively extracting eyeball data from the first eyeball image and the second eyeball image; based on the extracted eyeball data, generating the corresponding first eyeball model and the second eyeball model.

3. The method of claim 2, wherein, the eyeball data includes eyeball size, curvature and center point position; constructing the corresponding first eyeball model and the second eyeball model further comprises: mapping the first eyeball image and the second eyeball image to the corresponding eyeball models respectively, so that the first eyeball image and the second eyeball image respectively match the corresponding eyeball data.

4. The method of claim 3, wherein, controlling the display device to display a current image, and controlling the second camera and the third camera to respectively capture reflection images of the left and right eyeballs of the user, to obtain a first current eyeball image and a second current eyeball image containing the current image, comprises: sequentially setting each of the plurality of preset images as the current image; controlling the second camera and the third camera to respectively capture reflection images of the left and right eyeballs of the user to obtain the corresponding first current eyeball image and the second current eyeball image for each current image displayed, until the first current eyeball image and the second current eyeball image corresponding to all preset images are obtained.

5. The method of claim 4, wherein, The display device has a first state of displaying an image and a second state of not displaying the image, and the display device is initially in the second state; sequentially setting each of the plurality of preset images as the current image, comprising: when detecting that the user gazes at the display device, controlling the display device to switch to the first state and display the current image, and acquiring reference position data of the current image; when the first current eye image and the second current eye image corresponding to all preset images are not acquired, sequentially setting each of the remaining preset images as the current image until the reference position data of all preset images and the corresponding first current eye image and second current eye image are acquired.

6. The method of claim 5, wherein, The line-of-sight mapping relationship includes a first line-of-sight mapping relationship corresponding to the first eye model and a second line-of-sight mapping relationship corresponding to the second eye model; and the line-of-sight mapping relationship of the corresponding eye model is constructed according to the first current eye image and the second current eye image, comprising: acquiring position data of each first current eye image according to the first current eye image corresponding to each current image and the first eye model; constructing the first line-of-sight mapping relationship and acquiring a first correction value according to each position data and the corresponding reference position data; acquiring position data of each second current eye image according to the second current eye image corresponding to each current image and the second eye model; constructing the second line-of-sight mapping relationship and acquiring a second correction value according to each position data and the corresponding reference position data.

7. The method of claim 1, wherein, The line-of-sight coordinates include eye position information and line-of-sight direction information.

8. A system for implementing eye gaze tracking positioning, the system comprising: The system comprises: a display device for displaying images; an image acquisition device comprising a first camera, a second camera and a third camera, a field of view range of the first camera covering an area in which a user can observe display content of the display device; a master control device communicatively connected to the display device and the image acquisition device, configured to perform the following operations: enabling the first camera to detect whether the user gazes at the display device; when detecting that the user gazes at the display device, simultaneously enabling the second camera and the third camera to respectively capture left and right eye images of the user, generate first eye images and second eye images, and construct corresponding first eye models and second eye models; controlling the display device to display a current image, and controlling the second camera and the third camera to respectively capture reflection images of the left and right eyes of the user to acquire first current eye images and second current eye images containing the current image; the current image is one of a plurality of preset images displayed on the display device, and the plurality of preset images are the same or different images; constructing line-of-sight mapping relationships of the corresponding first eye model and second eye model according to the first current eye image and the second current eye image, to map the relationship between the current eye image of the user and the line-of-sight coordinates; acquiring the line-of-sight coordinates of the user according to the line-of-sight mapping relationship, to position the line-of-sight of the eye of the user on the display device. The host device is further configured to perform the following operations: controlling the first camera to detect whether there is a face posture change of the user, the face posture change including face movement and eyeball movement; controlling the second camera and the third camera to re-shoot the left and right eyeballs of the user to update the first eyeball image and the second eyeball image when detecting the face posture change of the user; updating the corresponding eyeball model based on the updated first eyeball image and the second eyeball image; constructing a line-of-sight mapping relationship of the updated eyeball model according to the current image.

9. A computer device, comprising: The computer device comprises: a memory for storing a computer program; and a processor for executing the computer program to implement the method for realizing eyeball line-of-sight tracking positioning according to any one of claims 1-7.

Images