Under-screen eye movement tracking method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of under-screen eye movement tracking, and discloses an under-screen eye movement tracking method and device, computer equipment and a storage medium, the under-screen eye movement tracking method is applied to under-screen eye movement equipment, the under-screen eye movement equipment comprises camera equipment and eye movement equipment, the camera equipment is fixed at the center position of a screen of the eye movement equipment, and the camera equipment is fixed at the center position of the screen of the eye movement equipment. Shielding deviation and the like caused by the fact that the camera equipment is arranged below the eyes are avoided; the camera device and the human eyes are located on the inner side and the outer side of the screen of the eye movement device respectively, and the camera device cannot affect the actual use feeling of the user; on the basis, a first light spot array corresponding to eyeballs of the user is obtained, the first light spot array is formed on the eyeballs of the user by invisible light emitted by camera equipment, and when it is recognized that the user wears the lens, after the first light spot array is corrected, a second light spot array is obtained; the second light spot array obtained at the moment can accurately reflect the rotation of the eyeball, and the accuracy of eye movement tracking is improved.

Description

Technical Field

[0001] The present invention relates to the field of under-screen eye tracking technology, and in particular to an under-screen eye tracking method, device, computer equipment and storage medium. Background Art

[0002] With the development of near-eye display technology and the improvement of glasses display technologies such as AR and VR, users' requirements for eye interaction have increased significantly. Eye movement interaction is expected to replace gesture recognition and become the most frequently used interaction method. The current eye tracking solutions have inaccurate recognition of eye angular momentum, resulting in low recognition accuracy, which in turn causes a poor user experience. Summary of the invention

[0003] In view of this, the present invention provides an under-screen eye tracking method, apparatus, computer device and storage medium to solve the problem of inaccuracy caused by eyelash occlusion and other problems in the prior art eye tracking.

[0004] In a first aspect, the present invention provides an under-screen eye tracking method, the method comprising: being applied to an under-screen eye tracking device, the under-screen eye tracking device comprising a camera device and an eye movement device, the camera device being fixed at the center of a screen of the eye movement device, and the camera device and a human eye being respectively located at inner and outer sides of the screen of the eye movement device; the method comprising:

[0005] Acquire a first light spot array corresponding to the user's eyeball, where the first light spot array is formed on the user's eyeball by invisible light emitted by the camera device, and the first light spot array changes with the rotation of the eyeball;

[0006] When it is recognized that the user is wearing lenses, the first light spot array is corrected according to the relative position relationship between the camera device and the screen to obtain the second light spot array.

[0007] Beneficial effect: the under-screen eye tracking method is applied to an under-screen eye movement device, wherein the under-screen eye movement device comprises a camera device and an eye movement device, the camera device is fixed at the center of the screen of the eye movement device, avoiding the occlusion deviation caused by the camera device being set under the eye; and the camera device and the human eye are respectively located on the inner and outer sides of the screen of the eye movement device, the camera device does not affect the actual use experience of the user, so the device improves the accuracy of eye movement recognition; on this basis, a first light spot array corresponding to the user's eyeball is obtained, wherein the first light spot array is formed by invisible light emitted by the camera device at the user's eye The first light spot array is formed on the eyeball, and the invisible light will not affect the user's experience of using the eye movement device, thereby improving the user experience; the first light spot array will change with the rotation of the eyeball, so the rotation angle of the eyeball can be determined based on the acquired first light spot array, thereby improving the accuracy of eye movement tracking; when it is recognized that the user is wearing a lens, the first light spot array directly acquired at this time has errors, and the first light spot array needs to be corrected to obtain the second light spot array. The second light spot array obtained at this time can accurately reflect the rotation of the eyeball, thereby improving the accuracy of eye movement tracking.

[0008] In an optional embodiment, the invisible light is infrared light.

[0009] Beneficial effect, no impact on user use, improving user experience.

[0010] In an optional implementation, the invisible light is light emitted by 8-10 infrared LED lamps.

[0011] Beneficial effect, improving the accuracy of eye tracking.

[0012] In an optional implementation, the first light spot array is acquired through pixel gaps at preset positions of the screen.

[0013] The beneficial effect is that it is helpful for the camera device to obtain the first light spot array through the screen, thereby improving the accuracy of eye tracking.

[0014] In an optional embodiment, the circuit routing of the screen is arranged outside the preset position.

[0015] The beneficial effect is to avoid the influence of circuit routing on the acquisition of the first light spot array, thereby improving the accuracy of eye tracking.

[0016] In an optional implementation, the screen has a pixel size of 400×400.

[0017] Beneficial effect: Under this pixel, it can be ensured that the first light spot array can pass through the screen, so that the camera device can capture the first light spot array, complete eye tracking with the best user experience, and ensure the accuracy of eye tracking.

[0018] In a second aspect, the present invention provides an under-screen eye tracking device, which is applied to an under-screen eye tracking device, wherein the under-screen eye tracking device comprises a camera device and an eye movement device, wherein the camera device is fixed at the center of the screen of the eye movement device, and the camera device and the human eye are respectively located at the inner and outer sides of the screen of the eye movement device; the device comprises:

[0019] A light spot array acquisition module is used to acquire a first light spot array corresponding to the user's eyeball, where the first light spot array is a first light spot array formed on the user's eyeball by invisible light emitted by the camera device, and the first light spot array changes with the rotation of the eyeball;

[0020] The correction module is used to correct the first light spot array according to the relative position relationship between the camera device and the screen to obtain the second light spot array when it is recognized that the user is wearing a lens.

[0021] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the under-screen eye tracking method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0022] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the under-screen eye tracking method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 is a schematic diagram of an eye movement recognition method in the prior art according to an embodiment of the present invention;

[0025] Figure 2 is a schematic diagram of an eye movement recognition method in the prior art applicable to an embodiment of the present invention;

[0026] Figure 3 is a flow chart of an under-screen eye tracking method according to an embodiment of the present invention;

[0027] Figure 4 is a schematic diagram of an under-screen eye tracking method according to an embodiment of the present invention;

[0028] Figure 5a and Figure 5b is a schematic diagram of an under-screen eye tracking method according to an embodiment of the present invention;

[0029] Figure 6a and Figure 6b is a schematic diagram of an under-screen eye tracking method according to an embodiment of the present invention;

[0030] Figure 7a and Figure 7b Schematic diagram of an under-screen eye tracking method according to an embodiment of the present invention

[0031] Figure 8 is a schematic diagram of an under-screen eye tracking method according to an embodiment of the present invention;

[0032] Fig. 9 is a schematic diagram of an under-screen eye tracking method according to an embodiment of the present invention;

[0033] Fig.10 is a schematic diagram of an under-screen eye tracking method according to an embodiment of the present invention;

[0034] Fig.11 is a structural block diagram of an under-screen eye tracking device according to an embodiment of the present invention;

[0035] Fig.12 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0037] At present, the common solution for eye movement recognition is to use a camera under the eye to collect the light spot image on the eyeball from the invisible light surrounding the eye, and confirm the parameters of the eye movement through the changes in the shape of the light spot.

[0038] This solution is not very friendly to collecting the angular momentum of the eyeball. Figure 1 and Figure 2As shown, placing an IR camera under the eyes will still be blocked by non-effective reference objects such as lower eyelashes. In addition, although the image of the infrared spot collected by the camera under the eyes can be used to calculate the angular momentum of the eyeball through the database and algorithm compensation, the compensated solution still has the disadvantage of recognition bias. Furthermore, for users who wear glasses, there are many variables that affect the infrared spot passing through the lens with degrees and entering the camera under the glasses, such as displacement deviations of different degrees, including possible frame occlusion, etc. In short, the infrared spot collection solution under the eyes will be restricted by many factors, which greatly affects the accuracy and experience of eye movement interaction.

[0039] According to an embodiment of the present invention, an embodiment of an under-screen eye tracking method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] In this embodiment, a method for under-screen eye tracking is provided. Figure 3 : is a flow chart of an under-screen eye tracking method according to an embodiment of the present invention. The under-screen eye tracking method in the embodiment of the present invention is applied to an under-screen eye tracking device, wherein the under-screen eye tracking device includes a camera device and an eye tracking device, the camera device is fixed at the center of the screen of the eye tracking device, and the camera device and the human eye are respectively located on the inner and outer sides of the screen of the eye tracking device, and the process of the under-screen eye tracking method includes the following steps:

[0041] Step S301, obtaining a first light spot array corresponding to the user's eyeball.

[0042] The first light spot array is a first light spot array formed on the user's eyeball by invisible light emitted by the camera device, and the first light spot array changes with the rotation of the eyeball.

[0043] Exemplarily, the first light spot array corresponding to the user's eyeball is the light spot formed on the user's eyeball by the invisible light emitted by the camera device. The light spot array changes accordingly as the user's eyeball moves. Therefore, the light spot array collected and the combination of light spots will correspond to different arrays due to the different angles of the eyeball. Eye movements can be accurately captured through algorithm calculation, and corresponding eye control operations can be implemented, such as the control of VR and AR during use.

[0044] Setting the camera device at the center of the screen can effectively avoid the problem of being blocked by obstacles such as eyelashes when it is set at other positions, so that the first light spot array obtained is more accurate.

[0045] In a preferred embodiment, the invisible light is infrared light, and is emitted by 8-10 infrared LED lamps. The human eye cannot recognize infrared light, which will not affect the actual use of the eye tracking device, thereby improving the user experience.

[0046] In a preferred embodiment, when the camera device is set at the center point of the screen and the user's eyeballs are located on both sides of the screen, the light spot array formed by the camera device and the eyeballs is separated by the screen. Therefore, how to obtain the light spot array from behind the screen is a difficult problem to be solved. In an embodiment of the present invention, the first light spot array is obtained through the pixel gap of the screen at a preset position.

[0047] For example, when acquiring the light spot array, it is required that the light spot array can pass through the screen so as to be acquired by the camera behind the screen. Therefore, the light spot array can be acquired from the pixel gap at a preset position of the screen, where the preset position is the screen area within the FOV area of ​​the camera. Specifically, Figure 4 The figure shows the screen position within the FOV area of ​​the camera device. To facilitate the light spot array to pass through the screen, the pixels in the corresponding area can be proportionally deleted so that the light spot array can pass through the screen through the pixel gap. The corresponding Figure 5a and Figure 5b .like Figure 6a and Figure 6b The pixel size in the FOV area of ​​the camera device is reduced accordingly, and the pixel gap can also be increased to increase the amount of light entering the camera device. The camera device uses a 400×400 pixel recognition spot array, and the screen is to ensure that the spot array within the FOV projection range of the camera device can enter the camera device through the OLED screen.

[0048] If the screen can be transparent to the light spot array, it is required that the screen cannot be blocked by other opaque materials except pixels. Therefore, when designing and evaluating eye tracking equipment, the circuit routing of the corresponding screen is set outside the preset position. Figure 7a and Figure 7b The figure shows the schematic diagram of the screen circuit routing design. The circuit routing needs to be placed behind the pixels to reduce the obstruction of light to the camera equipment. Figure 8 Shown is the FOV area involved above.

[0049] Step S302: when it is recognized that the user is wearing a lens, the first light spot array is corrected according to the relative position relationship between the camera device and the screen to obtain a second light spot array.

[0050] For example, when performing eye tracking, when it is recognized that the user is wearing a lens, the lens refracts the light spot array, such as Fig. 9The figure shows the relative position relationship between the lens (optical lens), screen (OLED), camera (IR Camera) and eyeball. Fig.10 Shown is the refraction path of the light spot array through the optical lens.

[0051] Therefore, it is necessary to correct the first light spot array passing through the optical lens to obtain an accurate second light spot array, wherein the specific correction method can be implemented in the relevant technology and will not be repeated here.

[0052] The under-screen eye tracking method provided in this embodiment is applied to an under-screen eye movement device, wherein the under-screen eye movement device includes a camera device and an eye movement device, the camera device is fixed at the center of the screen of the eye movement device, avoiding the occlusion deviation caused by the camera device being set under the eye; and the camera device and the human eye are respectively located on the inner and outer sides of the screen of the eye movement device, and the camera device does not affect the actual use experience of the user, so this device improves the accuracy of eye movement recognition; on this basis, a first light spot array corresponding to the user's eyeball is obtained, wherein the first light spot array is formed by invisible light emitted by the camera device on the user The first light spot array formed on the eyeball, the invisible light will not affect the user's experience of using the eye movement device, thereby improving the user experience; the first light spot array will change with the rotation of the eyeball, so the rotation angle of the eyeball can be determined based on the acquired first light spot array, thereby improving the accuracy of eye movement tracking; when it is recognized that the user is wearing a lens, the first light spot array directly acquired at this time has errors, and the first light spot array needs to be corrected to obtain the second light spot array. The second light spot array obtained at this time can accurately reflect the rotation of the eyeball, thereby improving the accuracy of eye movement tracking.

[0053] In this embodiment, an under-screen eye tracking device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0054] This embodiment provides an under-screen eye tracking device, such as Fig.11 As shown, it is applied to an under-screen eye tracking device, which includes a camera device and an eye tracking device. The camera device is fixed at the center of the screen of the eye tracking device, and the camera device and the human eye are respectively located on the inner and outer sides of the screen of the eye tracking device; the device includes:

[0055] The light spot array acquisition module 1101 is used to acquire a first light spot array corresponding to the user's eyeball, where the first light spot array is a first light spot array formed on the user's eyeball by invisible light emitted by the camera device, and the first light spot array changes with the rotation of the eyeball;

[0056] The correction module 1102 is used to correct the first light spot array according to the relative position relationship between the camera device and the screen to obtain a second light spot array when it is recognized that the user is wearing a lens.

[0057] In some optional embodiments, the invisible light is infrared light.

[0058] In some optional embodiments, the invisible light is light emitted by 8-10 infrared LED lamps.

[0059] In some optional implementations, the first light spot array is acquired through pixel gaps at preset positions of the screen.

[0060] In some optional embodiments, the circuit routing of the screen is arranged outside the preset position.

[0061] In some optional implementations, the screen has a pixel size of 400×400.

[0062] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0063] The under-screen eye tracking device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0064] The embodiment of the present invention also provides a computer device having the above Fig.11 The under-screen eye tracking device shown.

[0065] See also Fig.12 , Fig.12 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.12As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.12 A processor 10 is taken as an example.

[0066] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0067] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0068] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0069] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0070] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0071] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0072] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for under-screen eye tracking, characterized in that: Applied to an under-screen eye tracking device, the under-screen eye tracking device comprises a camera device and an eye tracking device, the camera device is fixed at the center of the screen of the eye tracking device, and the camera device and the human eye are respectively located at the inner and outer sides of the screen of the eye tracking device; the method comprises: Acquire a first light spot array corresponding to the eyeball of the user, where the first light spot array is a first light spot array formed on the eyeball of the user by the invisible light emitted by the camera device, and the first light spot array changes with the rotation of the eyeball; When it is recognized that the user is wearing a lens, the first light spot array is corrected according to the relative position relationship between the camera device and the screen to obtain a second light spot array.

2. The method according to claim 1, characterized in that The invisible light is infrared light.

3. The method according to claim 1 or 2, characterized in that: The invisible light is light emitted by 8-10 infrared LED lamps.

4. The method according to claim 3, characterized in that The first light spot array is acquired through the pixel gaps at preset positions of the screen.

5. The method according to claim 4, characterized in that The circuit wiring of the screen is arranged outside the preset position.

6. The method according to claim 5, characterized in that The screen has a pixel size of 400×400.

7. An under-screen eye tracking device, characterized in that: Applicable to an under-screen eye tracking device, the under-screen eye tracking device comprises a camera device and an eye movement device, the camera device is fixed at the center of the screen of the eye movement device, and the camera device and the human eye are respectively located at the inner and outer sides of the screen of the eye movement device; the device comprises: A light spot array acquisition module is used to acquire a first light spot array corresponding to the user's eyeball, where the first light spot array is a first light spot array formed on the user's eyeball by the invisible light emitted by the camera device, and the first light spot array changes with the rotation of the eyeball; A correction module is used to correct the first light spot array according to the relative position relationship between the camera device and the screen to obtain a second light spot array when it is recognized that the user is wearing a lens.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the under-screen eye tracking method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the under-screen eye tracking method according to any one of claims 1 to 6.

Images