Method and system for realizing eyeball sight tracking and positioning, and computer equipment

Abstract

The invention provides a method and system for achieving eyeball sight tracking and positioning and computer equipment, and relates to the technical field of image processing, and the method comprises the steps: starting a first camera to detect whether a user watches a display device or not; when it is detected that the user watches the display device, a second camera and a third camera are started at the same time to shoot left and right eyeballs of the user respectively, a first eyeball image and a second eyeball image are generated, and an eyeball model is constructed; controlling a display device to display a current image, and controlling a second camera and a third camera to respectively shoot reflection images of left and right eyeballs of the user to obtain a first current eyeball image and a second current eyeball image; according to the first current eyeball image and the second current eyeball image, constructing a sight line mapping relation of an eyeball model; according to the sight line mapping relation, sight line coordinates of the user are obtained, and eyeball sight lines are positioned on the display device. According to the technical scheme, a user does not need to wear additional equipment, the dependence on a specific environment is reduced, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular, to a method and system for realizing eye gaze tracking and positioning, as well as a computer device. Background Art

[0002] Existing eye gaze tracking technologies usually rely on dedicated devices, such as eye trackers or infrared reflection devices, to achieve gaze positioning by capturing reflected light from the eyes or wearing sensors. However, the limitations of existing technologies are mainly reflected in device dependence and insufficient environmental adaptability. Specifically, dedicated devices perform well in a laboratory environment, but usually require users to wear additional hardware or operate under specific lighting conditions to ensure tracking accuracy, which limits the user experience. Summary of the Invention

[0003] In view of this, it is necessary to propose a method and system for realizing eye gaze tracking and positioning, as well as a computer device.

[0004] In a first aspect, an embodiment of the present application provides a method for realizing eye gaze tracking and positioning. The method includes: enabling a first camera to detect whether a user is looking at a display device; when it is detected that the user is looking at the display device, simultaneously enabling a second camera and a third camera to respectively capture the left and right eyes of the user, generating a first eye image and a second eye image, and constructing corresponding first and second eye models; controlling the display device to display a current image, and controlling the second camera and the third camera to respectively capture the reflected images of the left and right eyes of the user to obtain a first current eye image and a second current eye image containing the current image; constructing a gaze mapping relationship of the corresponding eye model according to the first current eye image and the second current eye image to map the relationship between the current eye image of the user and the gaze coordinates; and obtaining the gaze coordinates of the user according to the gaze mapping relationship to position the user's eye gaze on the display device.

[0005] Second aspect, an embodiment of the present application provides a system for implementing eye gaze tracking and positioning. The system includes a display device, an image acquisition device, and a main control device. The display device is used to display images. The image acquisition device includes a first camera, a second camera, and a third camera. The main control device is communicatively connected to the display device and the image acquisition device respectively, and is configured to perform the following operations: enable the first camera to detect whether the user is looking at the display device; when it is detected that the user is looking at the display device, simultaneously enable the second camera and the third camera to respectively capture the left and right eyes of the user, generate a first eye image and a second eye image, and construct corresponding first and second eye models; control the display device to display the current image, and control the second camera and the third camera to respectively capture the reflection images of the left and right eyes of the user, so as to obtain a first current eye image and a second current eye image including the current image; construct a gaze mapping relationship of the corresponding eye model according to the first current eye image and the second current eye image, so as to map the relationship between the user's current eye image and the gaze coordinates; obtain the gaze coordinates of the user according to the gaze mapping relationship, so as to locate the user's eye gaze on the display device.

[0006] Third aspect, an embodiment of the present application provides a computer device, which includes a memory and a processor. The memory is used to store a computer program. The processor is configured to execute the computer program to implement the above method for implementing eye gaze tracking and positioning.

[0007] The above method and system for implementing eye gaze tracking and positioning, and the computer device, by using multiple cameras to detect the user's gaze state and construct an eye model, and by dynamically capturing the eye reflection images and constructing a gaze mapping relationship, do not require the user to wear additional devices, reduce the dependence on a specific environment, and improve the user experience. Description of the Drawings

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the structures shown in these drawings.

[0009] Figure 1 It is the first flowchart of the method for implementing eye gaze tracking and positioning provided by the embodiment of the present application.

[0010] Figure 2 It is the flowchart of step S102 provided by the embodiment of the present application.

[0011] Figure 3 This is the flowchart of step S103 provided by the embodiment of the present application.

[0012] Figure 4 This is the flowchart of step S1031 provided by the embodiment of the present application.

[0013] Figure 5 This is the flowchart of step S104 provided by the embodiment of the present application.

[0014] Figure 6 This is the second flowchart of the method for realizing eye gaze tracking and positioning provided by the embodiment of the present application.

[0015] Figure 7 This is the schematic structural diagram of the system for realizing eye gaze tracking and positioning provided by the embodiment of the present application.

[0016] Figure 8 This is the schematic internal structure diagram of the computer device for applying the method for realizing eye gaze tracking and positioning provided by the embodiment of the present application.

[0017] Figure 9 This is the schematic diagram for constructing the gaze mapping relationship provided by the embodiment of the present application.

[0018] The realization of the purpose of the present application, functional features and advantages will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0019] In order to make the purpose, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0020] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar planning objects and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances. In other words, the described embodiments are implemented in an order other than the content illustrated or described here. In addition, the terms "include" and "have" and any variations thereof may further include other content. For example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0021] It should be noted that in this application, the descriptions involving "first", "second", etc. are only for descriptive purposes and should not be construed as indicating or implying their relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. Additionally, the technical solutions between various embodiments may be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0022] Existing eye gaze tracking technologies usually rely on dedicated devices, such as eye trackers or infrared reflection devices, to achieve gaze positioning by capturing the reflected light of the eyes or wearing sensors. However, it usually requires users to wear additional hardware or operate under specific lighting conditions to ensure tracking accuracy, which limits the user experience. This application proposes a method for achieving eye gaze tracking and positioning, and realizes the tracking by using multiple cameras to detect the user's gaze state and construct an eye model and the corresponding gaze mapping relationship, without the need for users to wear additional devices, thereby improving the user experience.

[0023] Please refer to Figure 1 , which is the first flowchart of the method for achieving eye gaze tracking and positioning provided by the embodiment of this application. The specific implementation of eye gaze tracking and positioning includes steps S101 - S105.

[0024] Step S101, enable the first camera to detect whether the user is gazing at the display device.

[0025] In step S101, the field of view of the first camera 21 covers the area where the user can observe the display content of the display device. For example, if the display device is the display screen of a computer device placed on a table, then the first camera 21 can be set above the display screen, and the field of view of the first camera 21 covers the user operating the computer device in front of the table to improve the capture and tracking effect of the first camera 21 on the user's eye gaze.

[0026] Step S102, when it is detected that the user is gazing at the display device, simultaneously enable the second camera and the third camera to respectively capture the left and right eyes of the user, generate a first eye image and a second eye image, and construct the corresponding first eye model and second eye model.

[0027] In step S102, the second camera 22 is used to photograph one of the user's left and right eyeballs, and the third camera 23 is used to photograph the other of the user's left and right eyeballs correspondingly. The first eyeball model corresponds to the first eyeball image, and the second eyeball model corresponds to the second eyeball image. In this application, the second camera 22 and the third camera 23 are enabled to photograph the user's left and right eyeballs only when it is detected that the user is gazing at the display device, that is, the cameras are started immediately when the user gazes at the display device, which can quickly capture the user's eyeball state and correspondingly construct an eyeball model to ensure the real-time tracking effect in a dynamic scene. At the same time, this application also photographs the user's left and right eyeballs simultaneously to ensure that when generating independent eyeball models, the geometric features and reflection information of the eyeballs at the current moment can be captured more comprehensively, thereby improving the accuracy of gaze positioning.

[0028] Furthermore, the second camera 22 and the third camera 23 can also be arranged above the display screen. In one embodiment, the first camera 21, the second camera 22, and the third camera 23 are arranged side by side above the display screen, and the first camera 21 is located between the second camera 22 and the third camera 23. In another embodiment, the first camera 21, the second camera 22, and the third camera 23 are integrated into a camera module, and the field of view of the camera module covers the area where the user can observe the display content of the display device.

[0029] Please refer to Figure 2 , which is the flowchart of step S102 provided by the embodiment of this application. Constructing the corresponding first eyeball model and second eyeball model includes steps S1021 - S1023.

[0030] Step S1021, extract eyeball data from the first eyeball image and the second eyeball image respectively.

[0031] In step S1021, the eyeball data is obtained by combining the visual differences generated when observing with different eyeballs and combining different eyeball images. The eyeball data includes the size, curvature, and center point position of the eyeball. In this application, the eyeball data includes the first eyeball data corresponding to the first eyeball image and the second eyeball data corresponding to the second eyeball image. The first eyeball data and the second eyeball data are respectively three-dimensional depth data of several key points used to characterize the characteristics of different parts of the eyeball. Among them, the several key points can be, for example, the pupil center, the corneal reflection point, the eye corner, etc. The three-dimensional depth data is obtained through three-dimensional point cloud.

[0032] Furthermore, in this application, the first eyeball image and the second eyeball image can also be used to judge the distance between the eyeball and the display device, so as to accurately reflect different ways of the user's gaze state when combined with the three-dimensional depth data.

[0033] Step S1022, generate the corresponding first eyeball model and second eyeball model based on the extracted eyeball data.

[0034] In step S1022, the corresponding eyeball model is generated mainly by using the three-dimensional point cloud of the eyeball part. Specifically, taking the first eyeball model as an example, first, the three-dimensional depth data of the canthus part, the highest position of the eyeball, the upper part of the eyeball, and the lower part of the eyeball are respectively extracted from the corresponding first eyeball data through the three-dimensional point cloud; then, the eyeball size and the eyeball curved surface are constructed according to the above-extracted three-dimensional depth data; next, according to the three-dimensional depth data of the highest position of the eyeball and in combination with the pupil position in the first eye image, the center point of the eyeball is determined; finally, the eyeball curvature is obtained according to the eyeball curved surface, and the first eyeball model is obtained by modeling in combination with the obtained eyeball size and the eyeball center point. Correspondingly, the second eyeball model can extract the required three-dimensional depth data from the corresponding eyeball data according to the generation method of the first eyeball model, and then obtain the second eyeball model.

[0035] Step S1023: Map the first eyeball image and the second eyeball image to the corresponding eyeball models respectively, so that the first eyeball image and the second eyeball image respectively match the corresponding eyeball data.

[0036] In step S1023, the first eyeball image corresponds to and matches the first eyeball data, and the second eyeball image corresponds to and matches the second eyeball data. In this application, by mapping the first eyeball image and the second eyeball image to the corresponding eyeball models respectively, the corresponding eyeball data can be more accurately located at different key points of different eyeballs, which is convenient for analyzing different states of the eyeballs when the user is looking at things, such as the eyeball size, the eyeball curvature, etc., improving the accuracy of obtaining the eyeball line-of-sight direction subsequently, and further improving the accuracy of eyeball line-of-sight tracking.

[0037] Step S103: Control the display device to display the current image, and control the second camera and the third camera to respectively capture the reflection images of the user's left and right eyeballs, so as to obtain the first current eyeball image and the second current eyeball image including the current image.

[0038] In step S103, the current image is one of multiple preset images displayed on the display device. The multiple preset images can be the same or different images. The display device has a first state of displaying an image and a second state of not displaying an image, and the display device is initially in the second state. More specifically, the display device in the second state can be not displaying any image. The multiple preset images are sequentially displayed at different positions on the display device. Whenever the display device is in the first state and displays a current image, the display device also displays a prompt message to prompt the user to gaze at the current image, thereby obtaining various eye gazes of the user through different preset images, facilitating the analysis of different states of the user's eyes when gazing at things. In this application, after the first eye image and the second eye image are respectively obtained through the second camera 22 and the third camera 23 and the corresponding eye models are constructed, the second camera 22 and the third camera 23 are respectively used to capture images of the left and right eyes reflecting the current image again, so as to use each position where the current image is located as a reference point for gaze mapping to help the eye model more accurately calculate the coordinates of the user's gaze.

[0039] Please refer to Figure 3 , which is the flowchart of step S103 provided by the embodiment of this application. Controlling the display device to display the current image and controlling the second camera 22 and the third camera 23 to respectively capture the reflection images of the user's left and right eyes to obtain the first current eye image and the second current eye image including the current image includes steps S1031 - S1032.

[0040] Step S1031, sequentially set each of the multiple preset images as the current image.

[0041] Step S1032, every time a current image is displayed, control the second camera and the third camera to respectively capture the reflection images of the user's left and right eyes to obtain the corresponding first current eye image and the second current eye image until the first current eye image and the second current eye image corresponding to all preset images are obtained.

[0042] Please refer to Figure 4 , which is the flowchart of step S1031 provided by the embodiment of this application. Sequentially setting each of the multiple preset images as the current image includes steps S10311 - S10312.

[0043] Step S10311, when it is detected that the user is gazing at the display device, control the display device to switch to the first state and display the current image, and at the same time obtain the reference position data of the current image.

[0044] In step S10311, the reference position data of the current image corresponds to the coordinates of the positions where the preset images are sequentially displayed on the display device.

[0045] Step S10312: When the first current eye image and the second current eye image corresponding to all the preset images are not obtained, each of the remaining preset images is sequentially set as the current image until the reference position data of all the preset images, and the corresponding first current eye image and second current eye image are obtained.

[0046] Step S104: According to the first current eye image and the second current eye image, construct the line-of-sight mapping relationship of the corresponding eye model to map the relationship between the user's current eye image and the line-of-sight coordinates.

[0047] In step S104, the line-of-sight mapping relationship includes a first line-of-sight mapping relationship corresponding to the first eye model and a second line-of-sight mapping relationship corresponding to the second eye model. In this application, multiple current eye images when the user gazes at multiple different positions of the display device are obtained according to the situation that different eyes reflect different current images, so as to comprehensively analyze different states of the eyes when the user gazes at things, which is convenient for subsequently obtaining the line-of-sight position of the user gazing at the display device by constructing an eye model with a line-of-sight mapping relationship.

[0048] Please refer to Figure 5 , which is the flowchart of step S104 provided by the embodiment of this application. Constructing the line-of-sight mapping relationship of the corresponding eye model according to the first current eye image and the second current eye image includes steps S1041 - S1044.

[0049] Step S1041: According to the first current eye image corresponding to each current image and the first eye model, obtain the position data of each first current eye image.

[0050] Step S1042: According to each position data and the corresponding reference position data, construct the first line-of-sight mapping relationship and obtain the first correction value.

[0051] Step S1043: According to the second current eye image corresponding to each current image and the second eye model, obtain the position data of each second current eye image.

[0052] Step S1044: According to each position data and the corresponding reference position data, construct the second line-of-sight mapping relationship and obtain the second correction value.

[0053] Next, how to construct the line-of-sight mapping relationship will be specifically described.

[0054] Such as Figure 9As shown in the figure, if the width of the display device is denoted as X and the height as Y, the reference position data of the four corners A, B, C, and D of the display device are respectively (0, 0), (X, 0), (0, Y), and (X, Y). Assume that multiple preset images are the current images Pa, Pb, Pc, and Pd respectively displayed at the four corners of the display device, and they are all squares. First, denote the distances from the left and right eyeballs to the center points of the two cameras on the upper and lower sides of the display device as F1 and F2. Display a square current image at the coordinates (0, 0) of the upper left corner A, denoted as Pa; then, obtain the first current eyeball image and the second current eyeball image by shooting through the second camera 22 and the third camera 23, and determine the positions where Pa is reflected on the left and right eyeballs; then, according to the corresponding eyeball model, the central axes of the two eyeballs will intersect at Pa, and the central axes of the two eyeballs are respectively denoted as Zal and Zar; then, draw a straight line Za connecting the positions of the center points of the two eyeballs to the center point of Pa; then, display a new current image Pb at the upper right corner of the display device, and execute the previous steps to obtain the central axes Zbl and Zbr of the new two eyeballs, and draw a straight line Zb connecting the positions of the center points of the two eyeballs to the center point of Pb, thereby obtaining the first angle difference Vxab and the second angle difference Vyab between Za and Zb. When the position data of a corner B is obtained as (Xb, Yb), then, the first correction value = Xb / Vxab, Yb = 0.

[0055] Similarly, display a new current image Pc at the corner C, and execute the previous steps to obtain the central axes Zcl and Zcr of the new two eyeballs at the corner C, and then draw a straight line Zc to obtain the third angle difference Vxac and the fourth angle difference Vyac. When the position data of a corner C is obtained as (Xc, Yc), then, the second correction value = Yb / Vyab, Xb = 0.

[0056] Similarly, display a new current image Pd at the corner D, and execute the previous steps to obtain the central axes Zdl and Zdr of the new two eyeballs at the corner D, and then draw a straight line Zd to obtain the fifth angle difference Vxad and the sixth angle difference Vyad. When the position data of a corner D is obtained as (Xd, Yd), combining the first correction value = Xb / Vxab and the second correction value = Yb / Vyab, the expression form of the position data of the corner D can be obtained as Xd = Vxad * the first correction value, Yd = Vyad * the second correction value.

[0057] Furthermore, when it is necessary to output the position data of the user's gaze at other points on the display device, the central axis of the eyeball can be obtained through three-dimensional modeling, and the first angle Vx and the second angle Vy formed with Za. When the position data of other points is denoted as (X, Y), then X = Vx * the first correction value, Y = Vy * the second correction value, and the position data of other points can be obtained to acquire the user's line-of-sight coordinates.

[0058] Step S105: Obtain the user's gaze coordinates according to the gaze mapping relationship to locate the user's eye gaze on the display device.

[0059] In step S105, the gaze coordinates include eye position information and gaze direction information. The eye gaze can be used for contactless interaction between the user and the display device. For example, different information on the display device can be obtained through changes in the user's gaze coordinates. Furthermore, when the user pays special attention to a specific piece of information, the user can control the display of the specific information by maintaining or changing the gaze coordinates, thereby realizing the interaction.

[0060] Please refer to Figure 6 , which is the second flowchart of the method for realizing eye gaze tracking and positioning provided by the embodiment of the present application. The method for realizing eye gaze tracking and positioning further includes steps S201 - S204.

[0061] Step S201: Control the first camera to detect whether there is a change in the user's face posture.

[0062] In step S201, the face posture changes include face movement and eye movement.

[0063] Step S202: When a change in the user's face posture is detected, simultaneously control the second camera and the third camera to re - photograph the user's left and right eyes to update the first eye image and the second eye image.

[0064] Step S203: Update the corresponding eye model based on the updated first eye image and second eye image.

[0065] In step S203, during the tracking process of the eye gaze, the user's eye state may change in various ways in a short period of time, such as slight movement of the eye position, opening and closing of the eyelids, etc. Since the first eye image and the second eye image are mapped to the corresponding eye model, by mapping the updated first eye image and second eye image to the updated eye model, the eye model can dynamically adapt to the changes of the eyes, thereby maintaining accurate tracking of the eye state.

[0066] Step S204: Construct the gaze mapping relationship of the updated eye model according to the current image.

[0067] Please refer to Figure 7, which is a schematic structural diagram of a system for implementing eye gaze tracking and positioning provided by an embodiment of the present application. The present application also provides a system 1000 for implementing eye gaze tracking and positioning. The system 1000 includes a display device 1, an image acquisition device 2, and a main control device 3. The display device 1 is used to display images. The image acquisition device 2 includes a first camera 21, a second camera 22, and a third camera 23. The main control device 3 is communicatively connected to the display device 1 and the image acquisition device 2 respectively, and is used to perform the following operations: enable the first camera to detect whether the user is gazing at the display device; when it is detected that the user is gazing at the display device, simultaneously enable the second camera and the third camera to respectively capture the left and right eyes of the user, generate a first eye image and a second eye image, and construct corresponding first and second eye models; control the display device to display the current image, and control the second camera and the third camera to respectively capture the reflection images of the left and right eyes of the user to obtain a first current eye image and a second current eye image containing the current image; construct a gaze mapping relationship of the corresponding eye model according to the first current eye image and the second current eye image to map the relationship between the current eye image of the user and the gaze coordinates; and obtain the gaze coordinates of the user according to the gaze mapping relationship to position the eye gaze of the user on the display device.

[0068] Please refer to Figure 8 , which is a schematic internal structure diagram of a computer device for applying the method for implementing eye gaze tracking and positioning provided by an embodiment of the present application.

[0069] As Figure 8 shown, the computer device 100 includes a memory 901 and a processor 902. Among them, the processor 902 is used to run the computer program instructions in the memory 901 to implement the method for implementing eye gaze tracking and positioning.

[0070] The memory 901 includes at least one type of readable storage medium, which includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 901 can be an internal storage unit of a computer device in some embodiments, such as the hard disk of a computer device. The memory 901 can also be an external storage device of an external computer device in other embodiments, such as a plug-in hard disk configured in a computer device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 901 can also include both an internal storage unit of a computer device and an external storage device. The memory 901 can be used not only to store application software installed in the computer device and various types of data, such as the code for implementing the method of eye gaze tracking and positioning, but also to temporarily store data that has been output or will be output.

[0071] Further, the computer device 100 further includes a bus 903. The bus 903 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 8 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0072] Further, the computer device 100 can also include a display component 904. The display component 904 can be an LED display, a liquid crystal display, a touch liquid crystal display, and an Organic Light-Emitting Diode (OLED) toucher, etc. Among them, the display component 904 can also be appropriately referred to as a display device or a display unit, which is used to display the information processed in the computer device 100 and to display a visual user interface.

[0073] Further, the computer device 100 can also include a communication component 905. The communication component 905 can optionally include a wired communication component and / or a wireless communication component (such as a Wi-Fi communication component, a Bluetooth communication component, etc.), which is usually used to establish a communication connection between the computer device 100 and other computer devices.

[0074] Figure 8 Only the computer device 100 with some components and implementing the method of eye gaze tracking and positioning is shown. Those skilled in the art can understand that,Figure 8 The structure shown does not constitute a limitation on the computer device 100, and may include fewer or more components than shown, or combine certain components, or have a different component arrangement.

[0075] In the above embodiments, it can be implemented in whole or in part by software, hardware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0076] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part. The computer device may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)).

[0077] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0078] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0079] The unit described as a separate component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0080] In addition, each functional unit in various embodiments of the present application may be integrated into one processing unit, may exist separately for each unit, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0081] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, removable hard disks, read-only storage media (ROM, Read-Only Memory), random access storage media (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0082] In the above embodiment, by using multiple cameras to detect the user's gaze state and construct an eye model, and by dynamically capturing the eye reflection image and constructing a line-of-sight mapping relationship, there is no need for the user to wear additional devices, reducing the dependence on a specific environment and improving the user experience.

[0083] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these changes and modifications.

[0084] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially in the direction of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this document, there is no strict order restriction for the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0085] The above are only preferred embodiments of the present application, and of course, the scope of rights of the present application cannot be limited thereby. Therefore, equivalent changes made according to the claims of the present application still fall within the scope covered by the present application.

Claims

1. A method for realizing eye tracking and positioning, characterized in that: The method comprises: Enable the first camera to detect whether the user is looking at the display device; When it is detected that the user is looking at the display device, the second camera and the third camera are simultaneously enabled to respectively photograph the left and right eyeballs of the user, generate a first eyeball image and a second eyeball image, and construct corresponding first eyeball models and second eyeball models; Controlling the display device to display the current image, and controlling the second camera and the third camera to respectively capture the reflected images of the left and right eyeballs of the user, so as to obtain a first current eyeball image and a second current eyeball image containing the current image; According to the first current eyeball image and the second current eyeball image, constructing a sightline mapping relationship of the corresponding eyeball model to map the relationship between the current eyeball image of the user and the sightline coordinates; According to the sight line mapping relationship, the sight line coordinates of the user are acquired to locate the eye sight line of the user on the display device.

2. The method according to claim 1, characterized in that Constructing the corresponding first eyeball model and second eyeball model, including: extracting eyeball data from the first eyeball image and the second eyeball image respectively; A corresponding first eyeball model and a second eyeball model are generated based on the extracted eyeball data.

3. The method according to claim 2, characterized in that The eyeball data includes the size, curvature and center position of the eyeball; constructing the corresponding first eyeball model and second eyeball model also includes: The first eyeball image and the second eyeball image are respectively mapped to corresponding eyeball models, so that the first eyeball image and the second eyeball image match corresponding eyeball data respectively.

4. The method according to claim 3, characterized in that The current image is one of a plurality of preset images displayed on the display device; controlling the display device to display the current image, and controlling the second camera and the third camera to respectively capture the reflected images of the left and right eyeballs of the user to obtain a first current eyeball image and a second current eyeball image containing the current image, comprising: sequentially setting each of the plurality of preset images as the current image; Each time a current image is displayed, the second camera and the third camera are controlled to respectively capture the reflected images of the left and right eyeballs of the user to obtain the corresponding first current eyeball image and the second current eyeball image, until the first current eyeball image and the second current eyeball image corresponding to all preset images are obtained.

5. The method according to claim 4, characterized in that The display device has a first state in which an image is displayed and a second state in which an image is not displayed, and the display device is initially in the second state; Sequentially setting each of the plurality of preset images as the current image includes: When it is detected that the user is looking at the display device, controlling the display device to switch to the first state and display the current image, and acquiring reference position data of the current image; When the first current eyeball image and the second current eyeball image corresponding to all preset images are not obtained, each of the remaining preset images is set as the current image in turn until the reference position data of all preset images and the corresponding first current eyeball image and the second current eyeball image are obtained.

6. The method according to claim 5, characterized in that The sight line mapping relationship includes a first sight line mapping relationship corresponding to the first eyeball model and a second sight line mapping relationship corresponding to the second eyeball model; constructing the sight line mapping relationship of the corresponding eyeball model according to the first current eyeball image and the second current eyeball image, including: Acquire position data of each first current eyeball image according to the first current eyeball image corresponding to each current image and the first eyeball model; According to each position data and the corresponding reference position data, construct the first sight line mapping relationship and obtain a first correction value; Acquire position data of each second current eyeball image according to the second current eyeball image corresponding to each current image and the second eyeball model; According to each position data and the corresponding reference position data, the second sight line mapping relationship is constructed and the second correction value is obtained.

7. The method according to claim 1, characterized in that The method further comprises: Controlling the first camera to detect whether there is a change in the user's facial posture, where the facial posture change includes facial movement and eye movement; When a change in the user's facial posture is detected, the second camera and the third camera are simultaneously controlled to re-shoot the left and right eyeballs of the user to update the first eyeball image and the second eyeball image; Based on the updated first eyeball image and the second eyeball image, updating the corresponding eyeball model; According to the current image, a sightline mapping relationship of the updated eyeball model is constructed.

8. The method according to claim 1, characterized in that The sight line coordinates include eyeball position information and sight line direction information.

9. A system for realizing eye tracking and positioning, characterized in that: The system comprises: A display device for displaying images; An image acquisition device, comprising a first camera, a second camera and a third camera; The main control device is communicatively connected to the display device and the image acquisition device, and is used to perform the following operations: enabling the first camera to detect whether the user is looking at the display device; When it is detected that the user is looking at the display device, the second camera and the third camera are simultaneously enabled to respectively photograph the left and right eyeballs of the user, generate a first eyeball image and a second eyeball image, and construct corresponding first eyeball models and second eyeball models; Controlling the display device to display the current image, and controlling the second camera and the third camera to respectively capture the reflected images of the left and right eyeballs of the user, so as to obtain a first current eyeball image and a second current eyeball image containing the current image; According to the first current eyeball image and the second current eyeball image, constructing a sightline mapping relationship of the corresponding eyeball model to map the relationship between the current eyeball image of the user and the sightline coordinates; According to the sight line mapping relationship, the sight line coordinates of the user are acquired to locate the eye sight line of the user on the display device.

10. A computer device, characterized in that: The computer device comprises: a memory for storing a computer program; and A processor, used to execute the computer program to implement the method for eye tracking and positioning as described in any one of claims 1-8.

Images