Eyeball tracking method and device, storage medium and program product

Abstract

The embodiment of the invention provides an eyeball tracking method and device, a storage medium and a program product. The method comprises the steps that a target image is displayed in a non-visual area outside a user view field; the method comprises the following steps: acquiring an eye image of a user, analyzing the eye image, and determining a first coordinate of a corresponding feature point of a feature point of the target image in the eye image in a camera coordinate system of a camera, and obtaining a second coordinate of the feature point in a display coordinate system of the display screen, and determining the sight direction of the eyeballs of the user according to the first coordinate and the second coordinate. According to the method provided by the embodiment of the invention, hardware such as an infrared light source and the like does not need to be specially arranged, the hardware cost is reduced, and the hardware structure of the head-mounted equipment is simplified.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of virtual reality technology, and in particular to an eye tracking method, device, storage medium, and program product. Background Art

[0002] Virtual reality (VR) glasses are head-mounted display devices that allow users to experience the most realistic feelings in the virtual reality world. The authenticity of the simulated environment is difficult to distinguish from the real world, giving people an immersive feeling. Eye tracking technology is a key technology of VR technology. Eye tracking technology can calculate which area of ​​the display screen the user's gaze stays on.

[0003] In the related art, an infrared light emitting diode (IR LED) is set on the edge of the lens of VR glasses to emit infrared light, and then an infrared camera is used to capture an infrared image of the eyeball, and then the line of sight of the eyeball is calculated based on the infrared image.

[0004] However, the inventors have discovered that the related art has at least the following technical problems: In the above method, an IRLED needs to be specially provided as an infrared light source, and the hardware cost is relatively high. Summary of the invention

[0005] The embodiments of the present disclosure provide an eye tracking method, device, storage medium and program product to reduce hardware costs.

[0006] In a first aspect, an embodiment of the present disclosure provides an eye tracking method, which is applied to a head-mounted display device, and the method includes:

[0007] Displaying a target image in a non-visible area outside the user's field of view;

[0008] Acquire an eye image of a user; the eye image includes a reflection image of the target image in the user's eyeball;

[0009] Parsing the eye image to determine first coordinates of corresponding feature points of feature points of the target image in the eye image in a camera coordinate system of the camera;

[0010] Acquire a second coordinate of the feature point in a display coordinate system of the display screen;

[0011] The sight direction of the user's eyeball is determined according to the first coordinate and the second coordinate.

[0012] In a second aspect, an embodiment of the present disclosure provides an eye tracking device, comprising:

[0013] A display module, used for displaying a target image in a non-visible area outside the user's field of view;

[0014] An image acquisition module, used to acquire an eye image of a user; the eye image includes a reflection image of the target image in the user's eyeball;

[0015] A determination module, configured to analyze the eye image and determine first coordinates of corresponding feature points of feature points of the target image in the eye image in a camera coordinate system of the camera;

[0016] A coordinate acquisition module, used to acquire a second coordinate of the feature point in a display coordinate system of the display screen;

[0017] The determination module is further used to determine the sight direction of the user's eyeball according to the first coordinate and the second coordinate.

[0018] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a display screen, a camera, a processor, and a memory;

[0019] The display screen and the camera are both connected to the processor;

[0020] The display screen is used to display the target image in a non-visible area outside the user market;

[0021] The camera is used to photograph the user's eyeball to obtain an eye image; the eye image includes a reflection image of the target image in the user's eyeball;

[0022] The memory stores computer-executable instructions;

[0023] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the eye tracking method as described in the first aspect and various possible designs of the first aspect.

[0024] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the eye tracking method described in the first aspect and various possible designs of the first aspect is implemented.

[0025] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the eye tracking method as described in the first aspect and various possible designs of the first aspect.

[0026] The eye tracking method, device, storage medium and program product provided in this embodiment display a target image in a non-visible area outside the user's field of view; obtain an eye image of the user, the eye image includes a reflection image of the target image in the user's eye, analyze the eye image, determine the first coordinates of the corresponding feature points of the target image in the eye image in the camera coordinate system of the camera, obtain the second coordinates of the feature points in the display coordinate system of the display screen, and determine the sight direction of the user's eye according to the first coordinate and the second coordinate. The method provided in this embodiment uses pixel points in the non-visible area of ​​the display screen to display the target image, and the light corresponding to the target image is irradiated on the iris area of ​​the human eye to form a reflection pattern or light spot, and the eye image is obtained by shooting with a camera, and then the sight direction of the user's eye is determined by the coordinates of the corresponding feature points of the target image in the eye image and the coordinates of the feature points on the display screen, without the need to specially set up hardware such as infrared light sources, thereby reducing hardware costs and simplifying the hardware structure of the head-mounted device. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 A schematic diagram of an application scenario of the eye tracking method provided by an embodiment of the present disclosure;

[0029] Figure 2 A schematic diagram of the eye tracking method provided by an embodiment of the present disclosure;

[0030] Figure 3a A schematic diagram of the structure of a display screen provided in an embodiment of the present disclosure Figure 1 ;

[0031] Figure 3b A schematic diagram of the structure of a display screen provided in an embodiment of the present disclosure Figure 2 ;

[0032] Figure 4 A structural block diagram of an eye tracking device provided in an embodiment of the present disclosure;

[0033] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0035] Virtual reality (VR) glasses are head-mounted display devices that allow users to experience the most realistic feelings in the virtual reality world. The authenticity of the simulated environment is difficult to distinguish from the real world, giving people an immersive feeling. Eye tracking technology is a key technology of VR technology. Through eye tracking technology, it can accurately calculate which area of ​​the display screen the user's line of sight is staying, and then it can perform operations such as screen scrolling and web browsing, realizing the "control" of everything with the eyes.

[0036] In the related art, an infrared radiation light emitting diode (IR LED) is set at the edge of the lens of VR glasses to emit infrared light, and then an infrared camera is used to capture the infrared image of the eyeball, and then the line of sight of the eyeball is calculated based on the infrared image. However, the above method requires a special IR LED as an infrared light source, which has a high hardware cost.

[0037] In order to solve the above technical problems, the inventors of the present disclosure have found that a display screen can be used as a light source. In addition, in order not to affect the normal display function of the visible area of ​​the display screen, the non-visible area outside the visible area of ​​the display screen can be used as a light source to display the target image in the non-visible area outside the user's field of view; and by shooting the user's eyeball with a camera, an eye image including the imaging of the target image in the user's eyeball can be obtained, and then the line of sight direction of the user's eyeball can be determined based on the eye image. Based on this, the embodiments of the present disclosure provide an eye tracking method.

[0038] refer to Figure 1 , Figure 1Schematic diagram of application scenarios of the eye tracking method provided in the embodiment of the present disclosure. The head-mounted display device includes a display screen 101, a lens 102 and a camera 103. Among them, the lens 102 is arranged in front of the display screen 101, so that the user can view the content displayed on the display screen 101 through the lens 102. The camera 103 is arranged below the lens 102 at a preset angle to shoot the user's eyeball. The display screen 101 can be an organic light-emitting diode (OLED) display screen 101, a liquid crystal display 101 (LCD) or an active-matrix organic light-emitting diode (AMOLED). The camera 103 can be a black and white camera or a color camera. The lens 102 can be a Fresnel lens 102 or a Pancake lens 102.

[0039] In the specific implementation process, the target image is displayed in the non-visible area of ​​the display screen 101, and the user's eyes view the display content displayed in the visible area of ​​the display screen 101 through the lens 102. At this time, the target image displayed in the non-visible area of ​​the display screen 101 is also mapped to the user's eyeball, and the user's eyeball is photographed by the camera 103 to obtain an eye image, which includes the imaging of the target image displayed in the non-visible area in the user's eyeball. The eye image is analyzed to determine the first coordinates of the corresponding feature points of the feature points of the target image in the eye image in the camera coordinate system of the camera 103, and obtain the second coordinates of the feature points in the display coordinate system of the display screen 101. According to the first coordinate and the second coordinate, the line of sight direction of the user's eyeball is determined. The eye tracking method provided in the embodiment of the present application uses pixel points in the non-visible area of ​​the display screen 101 to display the target image. The light corresponding to the target image is irradiated on the iris area of ​​the human eye to form a reflection pattern or light spot. The eye image is obtained by photographing the camera 103. Then, the line of sight of the user's eye is determined by the coordinates of the corresponding feature points of the feature points of the target image in the eye image and the coordinates of the feature points on the display screen 101. There is no need to specially spend money to set up hardware such as infrared light sources, which reduces hardware costs and simplifies the hardware structure of the head-mounted device.

[0040] refer to Figure 2 , Figure 2 A flow chart of an eye tracking method provided in an embodiment of the present disclosure. The method of this embodiment can be applied in a head-mounted display device, and the method includes:

[0041] 201. Display a target image in a non-visible area outside the user's field of view.

[0042] Specifically, not all pixels of the display screen are used for normal display. In order to adapt to the viewing habits of the human eye, the middle area of ​​the display screen (e.g., the middle circular area) is usually set as a visible area for normal display, and the area where the remaining pixels are located around the visible area is set as a non-visible area. In the present disclosure, the target image can be displayed by applying corresponding drive signals to the corresponding pixels in the non-visible area. In addition, the selection of the position of the pixel points used to display the target image in the non-visible area must ensure that the target image can be mapped into the human eye.

[0043] In one embodiment of the present disclosure, the target image may be any image containing feature points. In order to facilitate accurate recognition, the target image may be a specially customized image, for example, one or more color blocks with specific colors or shapes. When there are multiple color blocks, the relative position relationship of the multiple color blocks may be designed to have specific features.

[0044] In one embodiment of the present disclosure, a target image is displayed in a non-visible area outside the user's field of view; this may include: selecting multiple sub-areas in the non-visible area of ​​the display screen; the multiple sub-areas are distributed around the visible area of ​​the display screen; and corresponding parts of the target image are displayed through the multiple sub-areas respectively.

[0045] Specifically, the target image may include a plurality of discrete sub-images, and the plurality of sub-images correspond one-to-one to a plurality of sub-areas of the non-visible area of ​​the display screen, and different sub-areas display different sub-images.

[0046] For example, Figure 3a As shown, the display screen 101 includes a visible area 1012 and a non-visible area 1011, and four sub-areas (10111 to 10114) can be selected in the non-visible area 1011 of the display screen 101 to respectively display the corresponding parts of the target image. The four sub-areas (10111 to 10114) are distributed around the visible area. The four sub-areas can be arranged in a manner with specific characteristics. For example, the center points of the four sub-areas (10111 to 10114) are connected to form a square. This can increase the recognition of the target image. It is beneficial for the human eye to see the target image displayed in the non-visible area 1011, and the eye image obtained by shooting is analyzed to obtain the position coordinates of the corresponding feature points of the feature points of the target image.

[0047] 202. Acquire an eye image of a user; the eye image includes a reflection image of the target image in an eyeball of the user.

[0048] Specifically, after the target image is displayed through the non-visible area of ​​the display screen, the content displayed in the non-visible area cannot be refracted to the human eye through the lens system and seen by the human eye, while the light of the displayed content can be refracted through the lens and illuminate the iris area of ​​the human eye, forming a reflection pattern or light spot, which is used to solve the direction of the eyeball.

[0049] 203. Analyze the eye image to determine first coordinates of corresponding feature points of feature points of the target image in the eye image in a camera coordinate system of the camera.

[0050] Specifically, after obtaining the eye image including the non-visible area of ​​the display screen, the eye image can be analyzed to obtain the first coordinates of the corresponding feature points of the target image in the eye image in the camera coordinate system. The feature points of the target image may include color information, grayscale information, shape information, texture features, corner point information, etc. Of course, if the camera is a black and white camera, the color information can be ignored.

[0051] In one embodiment of the present disclosure, when a camera obtains an eye image by photographing through a lens, considering the influence of the distortion effect of the lens on the accuracy of the first coordinates of the corresponding feature points of the feature points of the target image in the eye image, the eye image can be firstly subjected to distortion correction before the eye image is parsed. Specifically, parsing the eye image and determining the first coordinates of the corresponding feature points of the feature points in the eye image in the camera coordinate system of the camera may include: based on the distortion parameters of the lens, performing distortion correction on the eye image to obtain a corrected eye image; parsing the corrected eye image to determine the first coordinates of the corresponding feature points of the feature points in the eye image in the camera coordinate system of the camera.

[0052] Specifically, after the eye image is captured, it can be corrected based on the distortion parameters of the lens, thereby eliminating the influence of the distortion of the lens on the accuracy of the first coordinates of the corresponding feature points obtained by analysis, thereby ensuring the accuracy of the line of sight direction of the eyeball.

[0053] In one embodiment of the present disclosure, parsing the eye image and determining the first coordinates of the corresponding feature points in the eye image of the feature points of the target image in the camera coordinate system of the camera may include: acquiring feature information of the feature points of the target image; determining the target area corresponding to the non-visible area in the eye image, and determining feature information of multiple pixel points in the target area; determining the pixel points whose feature information among the multiple pixel points matches the feature information of the feature point as the corresponding feature points of the feature point in the eye image, and determining the first coordinates of the corresponding feature points in the camera coordinate system of the camera.

[0054] Specifically, the feature points of the target image may be all pixels of the target image or part of the pixels. For example, if the target image is a rectangular red block, the feature points of the target image may be the pixels corresponding to the entire rectangular block or the pixels corresponding to the four vertices of the rectangle, and the corresponding feature information may be grayscale information, color information, shape information, corner point information, etc.

[0055] In one embodiment of the present disclosure, the feature information of the feature points of the target image may include at least one of the following: grayscale information, color information, shape information, and corner point information. Of course, if the camera is a black and white camera, the color information may be ignored.

[0056] In one embodiment of the present disclosure, there is a gap between the pixel area in the non-visible area used to display the target image and the visible area; determining the target area corresponding to the non-visible area in the eye image may include: performing pixel point scanning on the eye image, and determining the boundary line between the non-visible area and the visible area based on the scanning result; and determining the target area corresponding to the non-visible area according to the boundary line.

[0057] Specifically, Figure 3b As shown, the non-visible area 1011 and the visible area 1012 of the display screen 101 are connected. Each pixel of the visible area 1012 is used to display content normally for users to watch, so each pixel in the visible area 1012 has a gray value, and if the target image is displayed through the sub-area (10111 to 10114) in the non-visible area 1011, and the sub-area (10111 to 10114) is spaced from the visible area 1012, since the visible area 1012 is surrounded by the non-visible area 1011, the interval can be an annular interval (shown by the dotted line), then the pixel points located at the annular interval do not display the image, do not have a gray value, or the gray value is 0. Therefore, the eye image can be scanned by pixels. Based on the change of gray value, the annular interval composed of pixels with a gray value of 0 can be found, and then the area outside the annular interval is determined as the target area. The corresponding feature points of the target image are searched for the target area. The ability to quickly lock the target area and search for a smaller target area can improve the efficiency and accuracy of the calculation, and quickly and efficiently determine the first coordinates of the corresponding feature points.

[0058] 204. Obtain a second coordinate of the feature point in a display coordinate system of the display screen.

[0059] Specifically, in one possible implementation, the target image may be a preset fixed image, and the target image may be displayed at a preset position. The second coordinates of the feature points may be obtained by obtaining the coordinates corresponding to the preset position. In another possible implementation, the target image may be a random image, and the feature points of the target image may be extracted by parsing the target image, and the second coordinates of the feature points in the display coordinate system of the display screen may be determined.

[0060] 205. Determine the sight direction of the user's eyeball according to the first coordinate and the second coordinate.

[0061] Specifically, the sight direction of the eyeball may include the position and orientation of the eyeball. The position of the display screen and the position of the camera are relatively fixed. When the eyeball orientation is fixed to the preset sight direction, the conversion relationship between the display coordinate system of the display screen and the camera coordinate system of the camera can be calibrated. If the eyeball orientation is still the preset sight direction, that is, the light of the target image has passed through the same reflection path, then the second coordinate of the feature point of the target image in the display coordinate system and the first coordinate of the corresponding feature point of the feature point of the target image in the eye image in the camera coordinate system satisfy the conversion relationship between the above two coordinate systems. If the first coordinate and the second coordinate do not satisfy the above conversion relationship, that is, there is a gap between the camera coordinate obtained by the second coordinate based on the conversion relationship and the first coordinate, then it indicates that the position or orientation of the eyeball has changed, and the reflection path of the light has changed.

[0062] In one embodiment of the present disclosure, determining the line of sight direction of the user's eyeball based on the first coordinate and the second coordinate may include: determining the camera coordinates corresponding to the second coordinate based on the conversion relationship between the camera coordinate system and the display coordinate system; determining the corneal reflection point based on the camera coordinates and the first coordinate; and determining the line of sight direction of the user's eyeball based on the corneal reflection point.

[0063] Specifically, the conversion relationship between the camera coordinate system of the camera and the display coordinate system of the display screen can be pre-calibrated. Then, based on the conversion relationship, the camera coordinates corresponding to the second coordinates of the feature point can be calculated. Then, the current corneal reflection point can be determined based on the distance between the camera coordinates and the first coordinates of the corresponding feature point and the corneal reflection point during calibration. Then, the line of sight of the user's eyeball can be determined based on the current corneal reflection point.

[0064] From the above description, it can be seen that this embodiment displays the target image by using pixel points in the non-visible area of ​​the display screen. The light corresponding to the target image is irradiated on the iris area of ​​the human eye to form a reflection pattern or light spot, and the eye image is obtained by photographing the camera. Then, the line of sight of the user's eyeball is determined by the coordinates of the corresponding feature points of the feature points of the target image in the eye image and the coordinates of the feature points on the display screen. There is no need to specially spend money to set up hardware such as infrared light sources, which reduces hardware costs and simplifies the hardware structure of the head-mounted device.

[0065] Corresponding to the eye tracking method of the above embodiment, Figure 4 This is a block diagram of the structure of the eye tracking device provided by the embodiment of the present disclosure. For the convenience of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 4 The device 40 includes: a display module 401, an image acquisition module 402, a determination module 403 and a coordinate acquisition module 404.

[0066] The display module 401 is used to display the target image in a non-visible area outside the user's field of view;

[0067] The image acquisition module 402 is used to acquire the user's eye image; the eye image includes the reflection image of the target image in the user's eyeball.

[0068] The determination module 403 is used to analyze the eye image and determine the first coordinates of the corresponding feature points of the feature points of the target image in the eye image in the camera coordinate system of the camera.

[0069] The coordinate acquisition module 404 is used to acquire the second coordinate of the feature point in the display coordinate system of the display screen.

[0070] The determination module 403 is further configured to determine the sight direction of the user's eyeball according to the first coordinate and the second coordinate.

[0071] In one embodiment of the present disclosure, the display module 401 is specifically used to select multiple sub-areas in the non-visible area of ​​the display screen; the multiple sub-areas are distributed around the visible area of ​​the display screen; and corresponding parts of the target image are displayed respectively through the multiple sub-areas.

[0072] In one embodiment of the present disclosure, the determination module 403 is specifically used to: perform distortion correction on the eye image based on the distortion parameters of the lens to obtain a corrected eye image; analyze the corrected eye image to determine the first coordinates of the corresponding feature points of the feature points in the eye image in the camera coordinate system of the camera.

[0073] In one embodiment of the present disclosure, the determination module 403 is specifically used to: obtain feature information of feature points of the target image; determine the target area corresponding to the non-visible area in the eye image, and determine feature information of multiple pixel points in the target area; determine the pixel points whose feature information among the multiple pixel points matches the feature information of the feature point as corresponding feature points of the feature point in the eye image, and determine the first coordinates of the corresponding feature point in the camera coordinate system of the camera.

[0074] In one embodiment of the present disclosure, the feature information of the feature points of the target image includes at least one of the following: grayscale information, color information, shape information, and corner point information.

[0075] In one embodiment of the present disclosure, the determination module 403 is specifically used to: perform pixel scanning on the eye image, and determine a boundary between a non-visible area and a visible area based on the scanning result; and determine a target area corresponding to the non-visible area according to the boundary.

[0076] In one embodiment of the present disclosure, the determination module 403 is specifically used to: determine the camera coordinates corresponding to the second coordinates based on the conversion relationship between the camera coordinate system and the display coordinate system; determine the corneal reflection point based on the camera coordinates and the first coordinates; and determine the line of sight direction of the user's eyeball based on the corneal reflection point.

[0077] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0078] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0079] refer to Figure 5 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure, and the electronic device 500 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as head-mounted display devices, mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0080] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0081] Typically, the following devices may be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0082] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0083] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0084] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0085] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0086] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0087] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0088] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0089] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0090] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0091] In a first aspect, according to one or more embodiments of the present disclosure, an eye tracking method is provided, comprising:

[0092] Displaying a target image in a non-visible area outside the user's field of view;

[0093] Acquire an eye image of a user; the eye image includes a reflection image of the target image in the user's eyeball;

[0094] Parsing the eye image to determine first coordinates of corresponding feature points of feature points of the target image in the eye image in a camera coordinate system of the camera;

[0095] Acquire a second coordinate of the feature point in a display coordinate system of the display screen;

[0096] The sight direction of the user's eyeball is determined according to the first coordinate and the second coordinate.

[0097] According to one or more embodiments of the present disclosure, displaying a target image in a non-visible area outside a user's field of view includes:

[0098] Selecting a plurality of sub-areas in the non-visible area of ​​the display screen; wherein the plurality of sub-areas are distributed around the visible area of ​​the display screen;

[0099] The corresponding parts of the target image are respectively displayed by the plurality of sub-areas.

[0100] According to one or more embodiments of the present disclosure, the head mounted display device includes a camera;

[0101] Accordingly, the step of obtaining the user's eye image includes:

[0102] Obtaining an eye image of the user by photographing through a lens with the camera;

[0103] The step of analyzing the eye image to determine a first coordinate of a corresponding feature point of the feature point in the eye image in a camera coordinate system of the camera includes:

[0104] Based on the distortion parameters of the lens, the eye image is subjected to distortion correction to obtain a corrected eye image;

[0105] The corrected eye image is analyzed to determine the first coordinates of the corresponding feature points of the feature points in the eye image in the camera coordinate system of the camera.

[0106] According to one or more embodiments of the present disclosure, parsing the eye image to determine first coordinates of corresponding feature points of feature points of the target image in the eye image in the camera coordinate system of the camera includes:

[0107] Acquire feature information of feature points of the target image;

[0108] Determine a target area corresponding to the non-visible area in the eye image, and determine feature information of a plurality of pixels in the target area;

[0109] The pixel points whose feature information matches the feature information of the feature point in the plurality of pixel points are determined as corresponding feature points of the feature point in the eye image, and the first coordinates of the corresponding feature point in the camera coordinate system of the camera are determined.

[0110] According to one or more embodiments of the present disclosure, the feature information of the feature points of the target image includes at least one of the following: grayscale information, color information, shape information, and corner point information.

[0111] According to one or more embodiments of the present disclosure, there is a gap between a pixel area in the non-visible area for displaying the target image and the visible area; and determining the target area corresponding to the non-visible area in the eye image includes:

[0112] Scanning pixels of the eye image, and determining a boundary between a non-visible area and a visible area based on the scanning result;

[0113] A target area corresponding to the non-visible area is determined according to the dividing line.

[0114] According to one or more embodiments of the present disclosure, determining the sight direction of the user's eyeball according to the first coordinate and the second coordinate includes:

[0115] Determining a camera coordinate corresponding to the second coordinate according to a conversion relationship between the camera coordinate system and the display coordinate system;

[0116] Determine a corneal reflection point based on the camera coordinates and the first coordinates;

[0117] The sight direction of the user's eyeball is determined according to the corneal reflection point.

[0118] In a second aspect, according to one or more embodiments of the present disclosure, an eye tracking device is provided, comprising:

[0119] Display module 401, used to display the target image in a non-visible area outside the user's field of view;

[0120] The image acquisition module 402 is used to acquire the user's eye image; the eye image includes the reflection image of the target image in the user's eyeball.

[0121] The determination module 403 is used to analyze the eye image and determine the first coordinates of the corresponding feature points of the feature points of the target image in the eye image in the camera coordinate system of the camera.

[0122] The coordinate acquisition module 404 is used to acquire the second coordinate of the feature point in the display coordinate system of the display screen.

[0123] The determination module 403 is further configured to determine the sight direction of the user's eyeball according to the first coordinate and the second coordinate.

[0124] According to one or more embodiments of the present disclosure, the display module 401 is specifically used to select multiple sub-areas in the non-visible area of ​​the display screen; the multiple sub-areas are distributed around the visible area of ​​the display screen; and corresponding parts of the target image are displayed respectively through the multiple sub-areas.

[0125] According to one or more embodiments of the present disclosure, the image acquisition module 402 is specifically used to: obtain an eye image of the user by photographing through a lens by the camera; the determination module 403 is specifically used to: perform distortion correction on the eye image based on the distortion parameters of the lens to obtain a corrected eye image; analyze the corrected eye image to determine the first coordinates of the corresponding feature points of the feature points in the eye image in the camera coordinate system of the camera.

[0126] According to one or more embodiments of the present disclosure, the determination module 403 is specifically used to: obtain feature information of feature points of the target image; determine the target area corresponding to the non-visible area in the eye image, and determine feature information of multiple pixel points in the target area; determine the pixel points whose feature information among the multiple pixel points matches the feature information of the feature point as corresponding feature points of the feature point in the eye image, and determine the first coordinates of the corresponding feature point in the camera coordinate system of the camera.

[0127] According to one or more embodiments of the present disclosure, the feature information of the feature points of the target image includes at least one of the following: grayscale information, color information, shape information, and corner point information.

[0128] According to one or more embodiments of the present disclosure, the determination module 403 is specifically used to: perform pixel scanning on the eye image, and determine a boundary between a non-visible area and a visible area based on the scanning result; and determine a target area corresponding to the non-visible area based on the boundary.

[0129] According to one or more embodiments of the present disclosure, the determination module 403 is specifically used to: determine the camera coordinates corresponding to the second coordinates according to the conversion relationship between the camera coordinate system and the display coordinate system; determine the corneal reflection point based on the camera coordinates and the first coordinates; and determine the line of sight direction of the user's eyeball according to the corneal reflection point.

[0130] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, including: a display screen, a camera, a processor, and a memory;

[0131] The display screen and the camera are both connected to the processor;

[0132] The display screen is used to display the target image in a non-visible area outside the user's field of view;

[0133] The camera is used to photograph the user's eyeball to obtain an eye image; the eye image includes a reflection image of the target image in the user's eyeball;

[0134] The memory stores computer-executable instructions;

[0135] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the eye tracking method as described in the first aspect and various possible designs of the first aspect.

[0136] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the eye tracking method described in the first aspect and various possible designs of the first aspect is implemented.

[0137] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the eye tracking method described in the first aspect and various possible designs of the first aspect is implemented.

[0138] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0139] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0140] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An eye tracking method for a head mounted display device, characterized in that: The method comprises: Displaying a target image in a non-visible area outside the user's field of view; Acquire an eye image of a user; the eye image includes a reflection image of the target image in the user's eyeball; Analyze the eye image to determine first coordinates of corresponding feature points of feature points of the target image in the eye image in a camera coordinate system of a camera; Obtaining a second coordinate of the feature point in a display coordinate system of the display screen; The sight direction of the user's eyeball is determined according to the first coordinate and the second coordinate.

2. The method according to claim 1, characterized in that The method of displaying a target image in a non-visible area outside the user's field of view comprises: Selecting a plurality of sub-areas in the non-visible area of ​​the display screen; wherein the plurality of sub-areas are distributed around the visible area of ​​the display screen; The corresponding parts of the target image are respectively displayed by the plurality of sub-areas.

3. The method according to claim 1, characterized in that The head mounted display device includes a camera; Accordingly, the step of obtaining the user's eye image includes: Obtaining an eye image of the user by photographing through a lens with the camera; The step of analyzing the eye image to determine a first coordinate of a corresponding feature point of the feature point in the eye image in a camera coordinate system of the camera includes: Based on the distortion parameters of the lens, the eye image is subjected to distortion correction to obtain a corrected eye image; The corrected eye image is analyzed to determine the first coordinates of the corresponding feature points of the feature points in the eye image in the camera coordinate system of the camera.

4. The method according to claim 1, characterized in that: The step of parsing the eye image to determine first coordinates of corresponding feature points of feature points of the target image in the eye image in the camera coordinate system of the camera includes: Acquire feature information of feature points of the target image; Determine a target area corresponding to the non-visible area in the eye image, and determine feature information of a plurality of pixel points in the target area; A pixel point whose feature information matches the feature information of the feature point among the plurality of pixel points is determined as a corresponding feature point of the feature point in the eye image, and a first coordinate of the corresponding feature point in the camera coordinate system of the camera is determined.

5. The method according to claim 4, characterized in that There is a gap between the pixel area in the non-visible area for displaying the target image and the visible area; The step of determining the target area corresponding to the non-visible area in the eye image includes: Scanning pixels of the eye image, and determining a boundary between a non-visible area and a visible area based on the scanning result; A target area corresponding to the non-visible area is determined according to the dividing line.

6. The method according to any one of claims 1 to 5, characterized in that: The determining the sight direction of the user's eyeball according to the first coordinate and the second coordinate includes: Determining a camera coordinate corresponding to the second coordinate according to a conversion relationship between the camera coordinate system and the display coordinate system; Determine a corneal reflection point based on the camera coordinates and the first coordinates; The sight direction of the user's eyeball is determined according to the corneal reflection point.

7. An eye tracking device, characterized in that: include: A display module, used for displaying a target image in a non-visible area outside the user's field of view; An image acquisition module, used to acquire an eye image of a user; The eye image includes a reflection image of the target image in the user's eyeball; A determination module, configured to analyze the eye image and determine first coordinates of corresponding feature points of the feature points of the target image in the eye image in a camera coordinate system of a camera; A coordinate acquisition module, used to acquire a second coordinate of the feature point in a display coordinate system of the display screen; The determination module is further used to determine the sight direction of the user's eyeball according to the first coordinate and the second coordinate.

8. An electronic device, characterized in that: include: Display, camera, processor and memory; The display screen and the camera are both connected to the processor; The display screen is used to display the target image in a non-visible area outside the user's field of view; The camera is used to photograph the user's eyeball to obtain an eye image; the eye image includes a reflection image of the target image in the user's eyeball; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the eye tracking method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the eye tracking method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the eye tracking method according to any one of claims 1 to 6 is implemented.

Images