Method for determining binocular gaze points and electronic device

Abstract

The embodiment of the present application discloses a method for determining binocular gaze points and an electronic device, which relates to the field of image processing. The gaze point of another person's eye can be determined based on the gaze point of one person's eye, thereby saving the amount of calculation consumed in the process of detecting the gaze point of another person's eye, and at the same time being able to simplify the hardware structure in the electronic device and reduce hardware costs. The method includes: determining the coordinates of a first gaze point of a first person's eye, the first person's eye is one of the user's eyes, the first gaze point coordinates are the coordinates of the gaze point of the first person's eye on a first display screen, and the first display screen corresponds to the first person's eye; according to the first gaze point coordinates, determining the coordinates of a second gaze point, the second gaze point coordinates are the coordinates of the second person's eye of the user on a second display screen, and the second display screen corresponds to the second person's eye.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of image processing, and in particular to a method for determining binocular gaze points and an electronic device. Background Art

[0002] At present, virtual reality (VR) technology has been widely used. For example, by using virtual reality technology, a three-dimensional virtual world can be generated by computer simulation through VR equipment such as VR helmets worn by users, providing users with simulations of senses such as vision, making users feel as if they are in the scene, and can observe things in the three-dimensional space instantly and without restrictions.

[0003] Taking the VR device as a VR helmet as an example, when the VR helmet is working, the scene of the corresponding area can be displayed to the user according to the different viewing positions of the user. Therefore, in order to provide accurate image display to the user, the VR helmet needs to be able to quickly determine the user's viewing position. In a specific implementation, the VR helmet can determine the user's viewing position by respectively determining the gaze point or gaze area corresponding to the left and right eyes of the user. Exemplarily, corresponding components can be set for the left and right eyes in the VR helmet, so that the gaze points of the left and right eyes can be obtained respectively. However, since it is necessary to obtain and calculate the gaze point related information of the left and right eyes respectively, the current solution has the problem of excessively high hardware cost and excessive calculation in VR devices such as VR helmets. Summary of the invention

[0004] The embodiments of the present application provide a method for determining binocular gaze points and an electronic device, which can determine the gaze point of one person's eye based on the gaze point of another person's eye, thereby saving the amount of calculation consumed in the process of detecting the gaze point of the other person's eye, and at the same time can streamline the hardware structure in the electronic device and reduce hardware costs.

[0005] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0006] In a first aspect, a method for determining binocular gaze points is provided, the method comprising: determining first gaze point coordinates of a first eye, the first eye being one of the user's eyes, the first gaze point coordinates being the coordinates of the gaze point of the first eye on a first display screen, and the first display screen corresponds to the first eye; determining second gaze point coordinates based on the first gaze point coordinates, the second gaze point coordinates being the coordinates of the user's second eye on a second display screen, and the second display screen corresponds to the second eye.

[0007] Based on this scheme, a scheme is provided in which only the gaze point of one eye is needed to determine the gaze point of both eyes. In this example, the coordinates of the second gaze point can be calculated by obtaining the coordinates of the first gaze point. Among them, the coordinates of the first gaze point can be the coordinates in the local coordinate system based on the first display screen. The coordinates of the second gaze point can be the coordinates in the local coordinate system based on the second display screen. Of course, the coordinates of the second gaze point can also be the coordinates in the local coordinate system based on the first display screen. Alternatively, the coordinates of the first gaze point and / or the coordinates of the second gaze point can be based on other two-dimensional or three-dimensional coordinate systems. It should be understood that in the prior art, it is necessary to detect the gaze points of the first eye and the second eye respectively, thereby requiring components for detecting the gaze points to be respectively set near the eyes of the user. After obtaining the corresponding information, the calculation of the gaze point coordinates of the eyes can be performed in parallel or serially, and finally the gaze point coordinates corresponding to the eyes are obtained. Compared with this scheme, the scheme provided in this example only needs to set a component for detecting the gaze point near one eye. Alternatively, when the component is set near both eyes, only one needs to be started to work. After determining the gaze point of the first person's eye, the gaze point coordinates corresponding to the other person's eye can be calculated and obtained according to any possible implementation method provided in this application. This can avoid a large number of repetitive calculations when calculating the gaze point coordinates of both eyes in parallel or serially, and can also compress the amount of calculation. This improves the calculation efficiency and reduces the calculation pressure of the electronic device. Since this solution only requires one detection module to be implemented, it can effectively reduce the hardware cost of the electronic device.

[0008] In a possible design, the position observed by the first human eye is the same or similar to the position observed by the second human eye; according to the coordinates of the first gaze point, the coordinates of the second gaze point are determined, including: according to the coordinates of the first gaze point, the three-dimensional coordinates of the position observed by the first human eye in the virtual three-dimensional space are determined, and according to the three-dimensional coordinates, the coordinates of the second gaze point are determined. Based on this scheme, an example of a scheme for determining the coordinates of the gaze point of another human eye according to the coordinates of the gaze point of the first human eye is provided. It should be understood that when the human eye is observing, the observation points of both eyes are the same or similar, so that objects at corresponding positions can be clearly observed. In this example, the electronic device can determine the coordinates of the gaze point of the second human eye on the second display screen according to the position observed by the gaze point of the first human eye. For example, according to the coordinates of the first gaze point on the first display screen, the specific position of the object observed by the user in the virtual three-dimensional scene can be determined. Since the position corresponding to the second gaze point has a corresponding relationship with the specific position in the virtual three-dimensional scene, the coordinates of the second gaze point on the second display screen can be inferred accordingly. In this way, the coordinates of the gaze point can be determined without separately collecting and calculating the position of the second gaze point.

[0009] In a possible design, the three-dimensional coordinates are determined according to the first gaze point coordinates, including: determining the three-dimensional coordinates according to the first gaze point coordinates and the first space mapping matrix; determining the second gaze point coordinates according to the three-dimensional coordinates and the second space mapping matrix; wherein the first space mapping matrix is ​​used to indicate the mapping relationship between the first gaze point coordinates and the three-dimensional coordinates, and the second space mapping matrix is ​​used to indicate the mapping relationship between the second gaze point coordinates and the three-dimensional coordinates. Based on this scheme, a specific scheme for determining the three-dimensional coordinates according to the first gaze point coordinates and determining the second gaze point coordinates according to the three-dimensional coordinates is provided. It should be noted that the first space mapping matrix and the second space mapping matrix are the corresponding relationships already existing in the existing image rendering process. Therefore, during the execution of this scheme, it is not necessary to re-acquire the above two corresponding relationships, but directly obtain the corresponding relationship from the rendering corresponding computing component (such as ISP or CPU or GPU, etc.) for calculating and obtaining the second gaze point coordinates. This can achieve the effect of data reuse, thereby improving computing efficiency.

[0010] In one possible design, the coordinates of the second gaze point are determined according to the coordinates of the first gaze point, including: determining a search area for the second gaze point according to the coordinates of the first gaze point, the search area being an area on the second display screen, and the coordinates of the second gaze point being in the search area; determining the coordinates of the second gaze point in the search area, wherein the image features corresponding to the coordinates of the second gaze point are the same or similar to the image features corresponding to the coordinates of the first gaze point. Based on this scheme, another example of a scheme for determining the coordinates of the second gaze point according to the coordinates of the first gaze point is provided. In this example, based on the positional relationship between the first gaze point and the second gaze point, and the image feature relationship between the first gaze point and the second gaze point, the specific coordinate position of the second gaze point can be determined.

[0011] In a possible design, according to the coordinates of the first gaze point, the search area of ​​the second gaze point is determined, including: according to the coordinates of the first gaze point and the first space mapping matrix, the three-dimensional coordinates of the position observed by the first human eye in the virtual three-dimensional space are determined, and the first space mapping matrix is ​​used to indicate the mapping relationship between the coordinates of the first gaze point and the spatial coordinates where the three-dimensional coordinates are located; according to the positional relationship between the three-dimensional coordinates and the second human eye, the intersection of the line connecting the second human eye and the corresponding position of the three-dimensional coordinates and the display screen corresponding to the second human eye is determined; near the intersection, according to the coordinates of the first gaze point, the coordinates of the second gaze point are determined, wherein the image features corresponding to the coordinates of the second gaze point are the same or similar to the image features corresponding to the coordinates of the first gaze point. Based on this scheme, a specific scheme for determining the coordinates of the second gaze point according to the positional relationship and image feature relationship between the first gaze point and the second gaze point is provided. Obviously, the first gaze point and the observation position in the virtual three-dimensional space, and the first human eye are on the same straight line. Similarly, the second gaze point and the observation position in the virtual three-dimensional space, and the second human eye are also on the same straight line. In this example, the coordinates of the observation position in the three-dimensional space can be determined according to the first gaze point and the first space mapping matrix. Combined with the coordinate position of the second eye in space, the line between the second eye and the observation position (that is, the line of sight) can be obtained. Since the second gaze point is a point on the second display screen, in theory, the intersection of the line of sight and the second display screen can be the second gaze point. However, in the actual real-time process, due to factors such as spatial interference and calculation errors, the intersection may not absolutely coincide with the second gaze point. Therefore, in the present application, a position with the same or similar pixel features as the first gaze point corresponding to the intersection can be found near the intersection as the second gaze point. Thus, the position of the second gaze point can be accurately determined. It should be noted that in some other implementations of the present application, the electronic device can be preset with a correspondence between each pixel point on the first display screen and the second display screen within a certain depth of field, and then after obtaining the position of the first gaze point, the position of the second gaze point on the second display screen is directly determined according to the correspondence. Among them, the correspondence between each pixel point on the first display screen and the second display screen can be determined by fusion of the first space mapping matrix and the second space mapping matrix.

[0012] In a possible design, according to the coordinates of the first gaze point, the search area of ​​the second gaze point is determined, including: according to the coordinates of the first gaze point and the position relationship between the first gaze point and the user's eyes, the epipolar plane is determined, the epipolar plane is a plane including the first gaze point and the position of the eyes; the intersection of the epipolar plane and the display screen corresponding to the second eye is determined; the coordinates of the second gaze point are determined on the intersection, wherein the image features corresponding to the coordinates of the second gaze point are the same or similar to the image features corresponding to the coordinates of the first gaze point. Based on this scheme, another scheme for determining the position of the gaze point of the second eye according to the position of the gaze point of the first eye is provided. In this scheme, it is not necessary to calculate and obtain the virtual three-dimensional coordinates in the space, but directly determine the possible position of the second gaze point according to the positional relationship between the first gaze point and the first eye and the second eye. In this example, the epipolar plane including the first gaze point, the first eye and the second eye can be determined based on the connection line between the first gaze point and the first eye and the connection line between the first eye and the second eye obtained in the process of removing the position of the first gaze point. It can be understood that the second gaze point should also be on this epipolar plane. In this example, the intersection of the epipolar plane and the second display screen may be referred to as an epipolar line. On the epipolar line, a point having the same or similar image features as the first gaze point may be searched as the second gaze point. Thus, the coordinates of the second gaze point may be determined according to the coordinates of the first gaze point without obtaining the first spatial mapping matrix.

[0013] In a possible design, before the first gaze point coordinates of the first eye, the dominant eye of the user is determined to be the first eye. Based on this solution, the electronic device can use the dominant eye as the first eye, thereby obtaining more accurate first gaze point coordinates, so the second gaze point coordinates calculated based on this are more accurate. The dominant eye can be determined based on the historical data of the current user, or by detecting changes in the gaze points of both eyes.

[0014] In a possible design, the display screen corresponding to the first human eye is the same as the display screen corresponding to the second human eye, or the display screen corresponding to the first human eye is different from the display screen corresponding to the second human eye. Based on this scheme, a display screen setting scheme is proposed, that is, the first display screen and the second display screen can be the same display screen, which is used to display different images at different positions, so as to show the corresponding images to the left eye and the right eye respectively. Of course, in some other implementations, the display screen can also be other components with display functions, for example, its display function can be realized by a micro projector.

[0015] In a second aspect, a device for determining binocular gaze points is provided, the device comprising: an acquisition unit, used to determine a first gaze point coordinate of a first eye, the first eye is one of the user's eyes, the first gaze point coordinate is the coordinate of the gaze point of the first eye on a first display screen, and the first display screen corresponds to the first eye; a determination unit, used to determine a second gaze point coordinate based on the first gaze point coordinate, the second gaze point coordinate is the coordinate of the second eye of the user on a second display screen, and the second display screen corresponds to the second eye.

[0016] In a possible design, the position observed by the first eye is the same as or similar to the position observed by the second eye. The determination unit is used to determine the three-dimensional coordinates of the position observed by the first eye in the virtual three-dimensional space according to the first gaze point coordinates, and determine the second gaze point coordinates according to the three-dimensional coordinates.

[0017] In one possible design, a determination unit is used to determine three-dimensional coordinates based on first gaze point coordinates and a first spatial mapping matrix; and to determine second gaze point coordinates based on the three-dimensional coordinates and a second spatial mapping matrix; wherein the first spatial mapping matrix is ​​used to indicate a mapping relationship between the first gaze point coordinates and the three-dimensional coordinates, and the second spatial mapping matrix is ​​used to indicate a mapping relationship between the second gaze point coordinates and the three-dimensional coordinates.

[0018] In one possible design, a determination unit is used to determine a search area for a second gaze point based on the coordinates of a first gaze point, where the search area is an area on a second display screen, and the coordinates of the second gaze point are within the search area; in the search area, the coordinates of the second gaze point are determined, wherein image features corresponding to the coordinates of the second gaze point are the same as or similar to image features corresponding to the coordinates of the first gaze point.

[0019] In a possible design, a determination unit is used to determine the three-dimensional coordinates of the position observed by the first human eye in the virtual three-dimensional space based on the first gaze point coordinates and a first space mapping matrix, and the first space mapping matrix is ​​used to indicate the mapping relationship between the first gaze point coordinates and the spatial coordinates where the three-dimensional coordinates are located; based on the positional relationship between the three-dimensional coordinates and the second human eye, determine the intersection of a line connecting the position corresponding to the second human eye and the three-dimensional coordinates and the display screen corresponding to the second human eye; near the intersection, determine the second gaze point coordinates based on the first gaze point coordinates, wherein the image features corresponding to the second gaze point coordinates are the same or similar to the image features corresponding to the first gaze point coordinates.

[0020] In one possible design, a determination unit is used to determine an epipolar plane based on the coordinates of a first gaze point and a positional relationship between the first gaze point and the user's eyes, where the epipolar plane is a plane including the first gaze point and the positions of the eyes; determine an intersection line between the epipolar plane and a display screen corresponding to a second eye; and determine the coordinates of a second gaze point on the intersection line, wherein image features corresponding to the coordinates of the second gaze point are the same as or similar to image features corresponding to the coordinates of the first gaze point.

[0021] In a possible design, the display screen corresponding to the first human eye is the same as the display screen corresponding to the second human eye, or the display screen corresponding to the first human eye is different from the display screen corresponding to the second human eye.

[0022] In a third aspect, an electronic device is provided, wherein the electronic device is provided with at least one display screen. The electronic device includes one or more processors and one or more memories; the one or more memories are coupled to the one or more processors, and the one or more memories store computer instructions; when the one or more processors execute the computer instructions, the electronic device executes the method for determining binocular gaze points as described in the first aspect and any one of various possible designs.

[0023] In a fourth aspect, a chip system is provided, which includes an interface circuit and a processor; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive signals from a memory and send signals to the processor, and the signals include computer instructions stored in the memory; when the processor executes the computer instructions, the chip system executes the method for determining the binocular gaze point as described in the first aspect above and any one of various possible designs.

[0024] In a fifth aspect, a computer-readable storage medium is provided, the computer-readable storage medium comprising computer instructions, and when the computer instructions are executed, a method for determining a binocular gaze point as described in the first aspect and any one of various possible designs is executed.

[0025] In a sixth aspect, a computer program product is provided, which includes instructions. When the computer program product runs on a computer, the computer can execute the method for determining the binocular gaze point as described in the first aspect and any one of various possible designs according to the instructions.

[0026] It should be understood that the technical solutions provided in the second, third, fourth, fifth and sixth aspects above and their technical features can all correspond to the method for determining the binocular gaze point provided in the first aspect and its possible design, and therefore the beneficial effects that can be achieved are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is an exemplary illustration of a binocular observation process;

[0028] Figure 2 A schematic diagram of the composition of a display device provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of the composition of a detection module provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of a flow chart of a method for determining binocular gaze points provided in an embodiment of the present application;

[0031] Figure 5A A schematic diagram of a method for determining a first gaze point provided in an embodiment of the present application;

[0032] Figure 5B A schematic diagram of another method for determining a first gaze point provided in an embodiment of the present application;

[0033] Figure 6 A schematic diagram of another method for determining a first gaze point provided in an embodiment of the present application;

[0034] Figure 7 A schematic diagram of a method for determining a second gaze point provided in an embodiment of the present application;

[0035] Figure 8 A flowchart of another method for determining binocular gaze points provided in an embodiment of the present application;

[0036] Fig. 9 A schematic diagram of another method for determining a second gaze point provided in an embodiment of the present application;

[0037] Fig.10 A schematic diagram of the composition of a device for determining binocular gaze points provided in an embodiment of the present application;

[0038] Fig.11 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application;

[0039] Fig.12 A schematic diagram of the composition of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] When observing an object in a three-dimensional space, the human eye can obtain the projection image of the three-dimensional object in the sight plane corresponding to the left eye's sight line through the left eye, and can also obtain the projection image of the three-dimensional object in the sight plane corresponding to the right eye's sight line through the right eye. The above two projection images are two-dimensional images. People can obtain stereoscopic vision by synthesizing the above two two-dimensional images. For example, please refer to Figure 1In (a), when a person uses both eyes to view an object with three-dimensional features, the sight lines of the left eye and the right eye will focus on the observed object through different paths. For example, the sight line of the left eye is as follows Figure 1 The line of sight 1 shown in (a) of the right eye is as follows Figure 1 The line of sight shown in (a) 2. Combined Figure 1 In (b), the human eye can obtain the projection image of the object on the line of sight plane 1 corresponding to the line of sight 1 through the left eye. The human eye can also obtain the projection image of the object on the line of sight plane 2 corresponding to the line of sight 2 through the right eye. Obviously, the projection images of the same object on different line of sight planes are different. People can analyze the difference between the two obtained projection images, thereby restoring the three-dimensional features of the object in their brains, and then obtaining stereoscopic vision. In this application, the stereoscopic vision can also be referred to as three-dimensional stereoscopic perception.

[0041] It should be noted that the three-dimensional stereoscopic experience can be generated by parallax, the observer's observation experience, and the mutual occlusion relationship between the observed objects. In the current display scheme, the two-dimensional image is displayed to the audience through the display screen, and the three-dimensional stereoscopic experience that can be brought about is obvious enough, or even no three-dimensional stereoscopic experience. However, through VR technology, users can be provided with a near-real experience immersed in a virtual three-dimensional scene. As a possible implementation method, in this application, a head-mounted display (head-mounted display, HMD) is used as a carrier to provide services to users using VR technology as an example. It should be noted that since the appearance of the HMD is close to a helmet, in this application, the HMD can also be referred to as a VR helmet or VR glasses. Generally speaking, a VR helmet can perform high-definition rendering of the corresponding pixels near the area of ​​human eye attention (i.e., the area near the gaze point) on the image displayed to the human eye, without the need for high-definition rendering of other areas. This can reduce the computational pressure of the VR helmet during the rendering process. It can be understood that in order to accurately perform high-definition rendering of the area near the gaze point, the VR helmet needs to be able to accurately obtain the gaze point of the human eye at different times.

[0042] In the prior art, corresponding detection modules can be set near the left eye and right eye of the user, respectively. For example, a detection module L can be set near the left eye on the VR helmet to detect the gaze point of the left eye. A detection module R can also be set near the right eye on the VR helmet to detect the gaze point of the right eye. In this way, the gaze points of the left and right eyes of the user at different times are obtained, so that high-definition rendering can be performed on the corresponding areas, which can provide the user with a clear three-dimensional stereoscopic experience while saving computing power.

[0043] Generally speaking, the detection module may include a light source for emitting detection laser, a photodetector, and a mechanical component for adjusting the position of the detection module. In order to obtain the gaze points of the left eye and the right eye, at least two sets of the detection modules are required on the VR helmet. This also leads to an increase in the hardware investment of the VR helmet.

[0044] In addition, in the process of the VR helmet detecting and obtaining the gaze points of the left eye and the right eye, although the gaze points of the left eye and the right eye are different, the calculation process is roughly similar. Since the VR helmet needs to calculate and obtain the gaze points of the left eye and the right eye respectively, the above calculation process needs to be repeated at least twice. It should be noted that, in general, the carrier used in VR technology (such as the above-mentioned VR helmet) needs to meet the requirements of portability and mobility. Therefore, the volume of its processing unit cannot be too large, which also limits the computing power of the VR helmet. Repeating the above calculation process at least twice will cause a very large computing burden on the VR helmet.

[0045] It should be noted that VR technology itself has at least the following three characteristics compared to general display technology: higher refresh rate and lower latency, larger field of view (FoV), and higher resolution. In order to meet these three requirements, the operation of the VR helmet is already close to saturation. Combined with the above description, when the operation pressure of this VR helmet is already high, performing the above-mentioned at least two repetitions of the gaze point operation process will undoubtedly cause a greater burden on the VR helmet.

[0046] In order to solve the above problems, the method for determining the position of the gaze point provided in the embodiment of the present application can obtain the gaze point of one eye by inferring the gaze point of another eye, without repeating the gaze point calculation, so as to obtain the gaze point of both eyes, thereby significantly reducing the requirements for computing power in the process of implementing VR technology. In some implementations of the present application, the implementation of the solution can be achieved by relying on only one set of detection modules, thereby significantly saving the hardware cost of the equipment implementing VR technology.

[0047] It should be noted that, generally speaking, VR helmets need to display different two-dimensional images to the left eye and the right eye at the same time. At the same time, when the VR avatar displays the two-dimensional image of the left eye or the right eye, it only needs to render the image near the corresponding gaze point in a refined manner. In other words, according to the gaze point of the left eye or the right eye, the VR helmet needs to be able to determine the position on the corresponding two-dimensional image. Therefore, in the following description of this application, the coordinates of the gaze point of the left eye and / or the right eye can be calibrated based on the local coordinates of the corresponding two-dimensional image, which is different from the coordinates of the three-dimensional objects in the actual scene that can be calibrated by a global coordinate system such as a geographic coordinate system.

[0048] The following is a detailed description of the method for locating the gaze point provided in the embodiments of the present application in conjunction with the accompanying drawings.

[0049] Please refer to Figure 2 , is a schematic diagram of the composition of a display device 200 provided in an embodiment of the present application. The display device can be used to support the display of VR technology. The location method of the gaze point provided in the embodiment of the present application can be used in the following example: Figure 2 In a specific implementation, the display device 200 may be a VR helmet, or a display component in a VR all-in-one machine, etc.

[0050] like Figure 2 As shown, the display device 200 may include a processor 210, a first display screen 221, a second display screen 222, and a detection module, etc. It should be noted that in some implementations of the present application, the functions of the first display screen 221 and the second display screen 222 may also be implemented by a single display screen. For example, an image corresponding to the left eye is displayed in an area where the corresponding image needs to be displayed to the left eye, and an image corresponding to the right eye is displayed in an area where the corresponding image needs to be displayed to the right eye. Of course, the functions of the first display screen 221 and the second display screen 222 may also be implemented by a larger number of display screens, and the embodiments of the present application do not limit the specific composition of the first display screen 221 and the second display screen 222. The following takes the example that the first display screen 221 and the second display screen 222 are each composed of a single display screen.

[0051] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0052] The processor 210 may include one or more processing units, for example, a processing unit may be a processor (Central Processing Unit, CPU), a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a microcontroller unit (Microcontroller Unit, MCU) and other components with processing functions. In other implementations, the processing unit may also include a field programmable gate array (Field-Programmable Gate Array, FPGA) and other components with computing capabilities.

[0053] The display device 200 can realize the display function through the first display screen 221 and / or the second display screen 222 under the control of the processor 210. Take the processor 210 including a GPU as an example. The GPU can be a microprocessor for image processing, connected to the first display screen 221 and the second display screen 222. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0054] The first display screen 221 and the second display screen 222 are used to display images, videos, etc. The display screen includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc.

[0055] like Figure 2 As shown, the display device 200 may further include a detection module 231. In some implementations, the detection module 231 provided in the display device 200 may be provided near a display screen, for example, the detection module 231 may be provided near the first display screen 221. The detection module 231 may be used to detect and determine the gaze point of the human eye corresponding to the first display screen 221. For example, the detection module 231 may be used to determine the coordinate position of the gaze point of the human eye on the first display screen 221. In other implementations, the display device 200 may further include a detection module 232, and the detection module 232 may be provided near another human eye different from the human eye opposite to the detection module 231. For example, the detection module 232 may be provided near the second display screen 222, and is used to detect the coordinates of the gaze point of the human eye corresponding to the second display screen 222 on the second display screen 222.

[0056] In different implementations of the present application, the detection module 231 and / or the detection module 232 may have different components according to different gaze point detection schemes adopted. For example, the detection module 231 is taken as an example. Figure 3As shown in (a) of FIG. 1 , the detection module 231 may include a laser source 231a for generating detection light, a reflective micromirror 231b for adjusting the incident angle / incident position of the detection light emitted by the laser source 231a relative to the human eye, and a photodetector for detecting an optical signal after the detection light is reflected by the human eye. For example, Figure 3 In the composition example shown in (a) of FIG. 1 , the photodetector may be a photodiode 231c. In some implementations, a scanner may also be provided in the detection module 231. The reflective component in the scanner may be used to implement the function of the reflective micromirror 231b. A micro-electro-mechanical system (MEMS) driver may also be provided in the scanner to adjust the angle of the reflective micromirror 231b. Figure 3 In the composition shown in (a) of FIG. 1 , the detection module 231 can determine the gaze point of the human eye by a photodiode response method. In other implementations, such as Figure 3 As shown in (b) of FIG. 1 , the detection module 231 includes a detection light source 231d and a shooting unit 231e. Figure 3 The detection module 231 can be used to determine the gaze point of the human eye through a model-based video eye diagram scheme. The specific implementation of the photodiode response method and the model-based video eye diagram scheme (hereinafter referred to as the video eye diagram method) will be described in detail in the following method description and will not be repeated here. It should be noted that, if Figure 3 (a) and Figure 3 The components shown in (b) are only one possible example of the detection module 231. In other implementations, the detection module 231 may also include more or fewer components to support the method for determining the gaze point corresponding to a single human eye. The embodiment of the present application does not limit the specific components of the detection module 231.

[0057] In addition, the above example is described by taking the composition of the detection module 231 as an example. Figure 2 When the display device 200 shown also includes a detection module 232, the detection module 232 may also have a similar composition to detect the gaze point of the corresponding human eye. Of course, in the same display device 200, the composition of the detection module 231 and the detection module 232 may be the same or different. The embodiment of the present application does not limit this.

[0058] It should be understood that the solution provided in the embodiment of the present application can determine the corresponding gaze point of a person's eye through a detection module, and infer another gaze point based on the gaze point. That is, when the method of the embodiment of the present application is applied to Figure 2In the display device 200 shown, only one detection module is required to work in the process of determining the gaze point of the user's eyes. That is, when the display device 200 is provided with both the detection module 231 and the detection module 232, only one of the detection modules is required to be used in a detection process. For ease of description, the following description is based on an example in which the display device 200 includes one detection module, namely, the detection module 231.

[0059] The method for determining binocular gaze points provided in the embodiments of the present application can be used in Figure 2 The display device is shown.

[0060] The solution provided in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0061] It should be noted that the solution provided in the embodiment of the present application can be used to determine the gaze point of the human eye. Therefore, it can be applied to various scenarios that require eye tracking, and is not limited to the VR scenario involved in the embodiment of the present application. Among them, eye tracking can also be called line of sight tracking. Its English is eye tracking, or eye pose estimation, or gaze estimation, or gaze tracking, etc. The study of eye tracking originated in the 1930s, and after nearly a hundred years of research, a variety of algorithms have been produced. In addition to the original observation method, the methods currently used in clinical and commercial products can be roughly divided into the following five categories: electro-oculography (EoG), electrode method (electromagnetic), contact lens method (contact lenses), photodiode response method (photodiode correspondence) and video eye diagram method (video-oculography). The first three methods are mostly used in clinical practice and are somewhat invasive to the human eye. Commercial products generally use non-invasive solutions, such as video eye diagram method or photodiode response method. Among them, and because the image processing solution has a long research history, the video eye diagram method is more widely used. The solution provided in the embodiment of the present application, combined with the existing eye tracking solution, can determine the gaze point of the other eye according to the position of the gaze point of either eye of the user. This allows the relevant equipment to save at least half of the computing power, while lowering the hardware requirements of the equipment, thereby achieving the purpose of significantly reducing costs.

[0062] Please refer to Figure 4 , is a flow chart of a method for determining binocular gaze points provided in an embodiment of the present application. Figure 4 As shown, the method may include:

[0063] S401: Determine the position of a first gaze point of a first person's eye on a first display screen.

[0064] The first display screen is a display screen that displays a two-dimensional image for the first person's eye. For example, when the first person's eye is the left eye, the first display screen displays the corresponding two-dimensional image for the left eye. For another example, when the first person's eye is the right eye, the first display screen displays the corresponding two-dimensional image for the right eye.

[0065] As an example, the following is an exemplary description of the method for determining the position of the first gaze point on the first display screen. For ease of description, the following is applied to a VR helmet, in which the following is provided: Figure 2 The detection module 231 shown is used as an example for explanation. It can be understood that when a detection module is set in the VR helmet, the human eye corresponding to the detection module is the first human eye.

[0066] In some embodiments, the detection module has Figure 3 Take the composition shown in (a) as an example. The VR helmet can determine the gaze point of the human eye based on the photodiode response method. It should be understood that Figure 5A As shown in (a) in the figure, the surface of the human eye is covered with a layer of cornea that is not completely in close contact with the eyeball. Therefore, when the angle of the incident light is different, the intensity of the reflected light received at a fixed position is also different. In this example, the position of the human pupil can be determined by monitoring the intensity distribution of the reflected light, and then the gaze point of the human eye can be determined.

[0067] As an example, a VR helmet can obtain the correspondence between the photodiode response and the human eye pupil coordinates, and then determine the human eye pupil coordinates according to the photodiode response when the user is currently viewing the scene (for example, the photodiode can convert the received light signal into an electrical signal, and the intensity of the light signal is different, and the corresponding electrical signal intensity is also different), thereby determining the position of the gaze point. Exemplarily, the correspondence between the photodiode response and the human eye pupil coordinates can be obtained by Figure 5AThe method shown in (b) in FIG. 1 is used to obtain the detection laser. The laser source can emit a detection laser to the reflective micromirror, and the reflective micromirror can reflect the detection laser to the human eye. The detection laser is reflected by the human eye through the refraction of the cornea, and is received by the photodiode after refraction of the cornea again. The photodiode can receive the detection laser, determine the intensity of the detection laser, and generate a corresponding electrical signal response according to the intensity of the received detection laser. By controlling the MEMS driver, the posture of the reflective micromirror can be adjusted (such as adjusting the angle of the reflective micromirror), thereby adjusting the incident position of the detection laser reflected on the human eye on the human eye. It can be understood that when the incident position of the detection laser is different, the intensity of the reflected light after reflection by the human eye is also different. In this example, the intensity of the light signal (such as the reflected light signal) obtained after reflection by the human eye and the cornea can be detected by the photodiode, thereby determining the posture of the reflective micromirror when the intensity of the reflected light signal is the largest at the current position of the human eye. Thus, the corresponding relationship between the pupil of the human eye and the posture of the reflective micromirror at the current position can be obtained. It should be noted that in this example, the adjustment of the position of the reflective micromirror by a MEMS driver is used as an example. In other implementations, other components with similar functions can also be used to adjust the position of the reflective micromirror, and this application does not impose any limitation on this.

[0068] In addition, the position of the pupil is different when the human eye looks at different angles. Different pupil positions can be mapped to a gaze point coordinate. As an example, please refer to Figure 5B When determining the correspondence between the human eye position and the gaze point coordinates, the following can be displayed on the display screen corresponding to the human eye: Figure 5B The example of the gaze point shown is used to guide the human eye to adjust the gaze point to the position of the example of the gaze point. The coordinates of the example of the gaze point are known. Therefore, the coordinates of the gaze point corresponding to the position of the human eye at this time can be calibrated as the coordinates of the example of the gaze point. Similarly, the method can be used to determine the coordinates of the gaze point corresponding to the human eye at different positions.

[0069] When the method is used to determine the gaze point coordinates of the human eye, the current position of the human eye can be determined by adjusting the position of the reflective micromirror, and the gaze point coordinates of the human eye at the current moment can be determined based on the correspondence between the calibrated human eye position and the gaze point coordinates. In some examples of the present application, in combination with the above description, when the human eye is viewing different positions, it corresponds to different pupil coordinates, so the pupil coordinates can be used to distinguish the difference in the gaze point when the human eye is viewing different positions.

[0070] In other embodiments, the detection module has Figure 3 Take the composition shown in (b) as an example. VR helmets can determine the gaze point of human eyes based on eye tracking. Figure 6 , the detection light source can be used as Figure 6The light source shown, when the light emitted by the light source shines on the human eye, the human eye's current human eye response image can be captured by the shooting unit. Then, the current pupil position is determined by analyzing the human eye response image. Take the photodetector as an infrared light emitting diode (IR LED) as an example. The VR helmet can obtain the correspondence between the pupil center coordinates of the human eye and the glint coordinates of the IR LED on the cornea. And according to the vector between the pupil center and the IR LED on the cornea, the pupil coordinates are determined. As an example, the shooting unit can receive the outgoing light obtained after the light of the light source is refracted / reflected by the human eye, thereby drawing a human eye response image corresponding to the current outgoing light. By analyzing the features of the response image, the coordinates of the pupil in the current state can be determined. Subsequently, the gaze point coordinates in the current state can be determined based on the determined pupil coordinates. Among them, in some implementations, the correspondence between the pupil coordinates and the gaze point coordinates in this example can be similar to the above Figure 5B The method shown in is obtained. Considering that the VR helmet may move during the measurement process, when the VR helmet and the pupil move relative to each other during the measurement process, the pupil coordinates cannot be accurately obtained. In some implementations of the present application, a pupil-glint vector can be constructed according to the response spot of the pupil position in the above-mentioned response image, and then the gaze point coordinates are accurately determined according to the pupil-glint vector. It can be seen that in this scheme, the shooting unit is a component that can obtain the response image of the human eye to the incident light. In some implementations, the shooting unit may include a lens for adjusting the field of view and converging light, and an image sensor for photosensitivity. For example, the image sensor may be an image sensor based on a complementary metal oxide semiconductor (CMOS) or an image sensor based on a charge coupled device (CCD). It should be understood that in the above examples, several specific implementations that can realize the functions of the shooting unit are cited, and in other implementations of the present application, the shooting unit can also realize its functions through other components, which will not be repeated here.

[0071] It should be noted that, in the above examples, only two examples of determining the coordinates of the gaze point of the human eye by the detection module are provided. In some other implementations, the coordinates of the gaze point of the first human eye can also be determined by other methods. For example, the iris-sclera edge tracking technology, the calibration scheme based on the cross ratio invariance, etc. can be used. The specific scheme can be flexibly selected according to actual needs, and the embodiments of the present application are not limited to this.

[0072] S402: Determine, according to the first gaze point, a position of an object corresponding to the first gaze point in the three-dimensional scene.

[0073] It is understandable that when a user wears a VR helmet, the VR helmet can show the user the scenery in the virtual three-dimensional scene. In combination with the above description, the display of the scenery in the virtual three-dimensional scene can be achieved by displaying the two-dimensional image of the virtual three-dimensional scene projected on the first display screen on the display screen corresponding to the left eye (such as the first display screen), and at the same time, displaying the two-dimensional image of the virtual three-dimensional scene projected on the second display screen on the display screen corresponding to the right eye (such as the second display screen). In this example, the VR helmet can determine the position of the first gaze point in the virtual three-dimensional scene based on the coordinate position of the first gaze point in the two-dimensional image.

[0074] S403: Determine the position of a second gaze point of a second person's eye on the second display screen according to the position of the first gaze point in the three-dimensional scene.

[0075] In this example, according to the position of the first gaze point, the position of the object observed by the first gaze point in the corresponding virtual three-dimensional scene can be determined. The position of the first gaze point can be the coordinates of the first gaze point in the two-dimensional coordinates on the first display screen corresponding to the first human eye. The coordinates can be local coordinates corresponding to the first display screen. For example, the coordinate center can be the lower left corner of the first display screen.

[0076] It is understandable that, since the VR helmet can provide three-dimensional vision for the human eye, combined with the above description, the two-dimensional image seen by the first human eye on the first display screen can be regarded as the projection image of the object on the first display screen in the corresponding virtual three-dimensional scene. Specifically, in the process of acquiring the projected image, the above-mentioned two-dimensional image can be acquired by methods such as image rendering. Generally speaking, in the process of image rendering, the above-mentioned projection image (such as the above-mentioned two-dimensional image) can be acquired according to the spatial mapping matrix (such as spatial mapping matrix 1) corresponding to the three-dimensional scene and the first display screen. In an embodiment of the present application, the gaze area of ​​the gaze point in the three-dimensional scene can be determined according to the spatial mapping matrix 1 and the position of the first gaze point. For example, please refer to Figure 7, taking the first eye as the left eye and the second eye as the right eye as an example. The user's left eye can see the two-dimensional projection of the object in the corresponding virtual three-dimensional scene (such as the tree in the figure) on the first display screen. According to the above S401, the VR helmet can obtain the position of the left eye's gaze point on the first display screen, such as the coordinates (X11, Y11) of the first gaze point. It can be understood that, Figure 7 As shown, the corresponding position of the left eye at the gaze point is actually the corresponding position of the tree in the three-dimensional scene corresponding to the coordinate (X11, Y11) in the line of sight. Through the above-mentioned spatial mapping matrix 1, the coordinates (X01, Y01, Z01) of the corresponding position in the three-dimensional scene can be determined. Thus, the exact position of the user's gaze point in the three-dimensional scene can be determined. It should be noted that the spatial mapping matrix in this example can be a mapping matrix used to identify the correspondence between the three-dimensional scene and the two-dimensional image.

[0077] It should be understood that if Figure 7 As shown, generally speaking, when a user views an object in a three-dimensional scene, the positions where the left eye and the right eye are looking are approximately the same. That is, when the left eye's gaze point is the coordinates (X01, Y01, Z01) in the three-dimensional scene, the coordinates of the right eye's gaze point in the three-dimensional scene are (X01, Y01, Z01) or close to (X01, Y01, Z01). In this example, the gaze point of the right eye on the second display screen can be determined based on the position in the three-dimensional scene (such as the coordinates (X01, Y01, Z01)).

[0078] Exemplarily, similar to the rendering process of the object in the three-dimensional scene to the projected image on the first display screen, the two-dimensional image displayed to the user's right eye on the second display screen can also be determined by the corresponding spatial mapping matrix (such as spatial mapping matrix 2). For example, according to the coordinates (X01, Y01, Z01) in the three-dimensional scene, the spatial mapping matrix 2 is calculated to obtain the coordinates (X21, Y21) of the position on the second display screen. In this way, the gaze point of the user's right eye, i.e. (X21, Y21), can be determined.

[0079] It can be seen that, based on the above description, the method provided in the embodiment of the present application can infer the position of the gaze point of another human eye through the position of a gaze point. That is to say, even if only one detection module can determine the gaze point of one human eye, the VR helmet can also determine the gaze point of another human eye accordingly. It can be understood that the computing power required to infer the position of another gaze point according to one gaze point is significantly less than the computing power required to re-acquire another gaze point through the detection module. Therefore, it is possible to quickly determine the gaze point of both eyes while significantly reducing the requirements for the computing power of the VR helmet. And since only one detection module is required, the hardware cost can be significantly reduced compared to the solution in the prior art that at least two detection modules need to be set. After obtaining the gaze points of both eyes, the VR helmet can perform other operations accordingly. For example, the VR helmet can perform local rendering optimization for the left eye and the right eye respectively, and reduce the image resolution away from the gaze point area. Of course, after the device obtains the gaze points of both eyes, it can also be used to realize other functions. For example, through the change in the position of the gaze point of the human eye, the corresponding response of the mouse movement, click, drag and other operations on the corresponding interface is executed.

[0080] Please refer to Figure 8 , is a flow chart of another method for determining binocular gaze points provided in an embodiment of the present application. Figure 8 As shown, the program may include:

[0081] S801: Determine the position of a first gaze point of a first person's eye on a first display screen.

[0082] In this example, the execution method of S801 is the same as that of Figure 4 The S401 shown is similar, and its specific process can be used as a reference and will not be repeated here.

[0083] S802: Determine a search area of ​​a second gaze point on a second display screen according to the position of the first gaze point.

[0084] S803: Determine the position of the second gaze point on the second display screen in the search area according to the pixel information of the first gaze point on the first display screen.

[0085] and Figure 4 Different from the method shown, in this example, another solution is provided for determining the second gaze point according to the position of the first gaze point.

[0086] It should be understood that when the user's eyes (such as the left eye and the right eye) observe an object in a three-dimensional environment, they can form a plane with the location of the observed object. In this application, the plane can be referred to as the epipolar plane. The intersection of the epipolar plane and different display screens can be referred to as the epipolar line corresponding to the display screen. In this example, the possible search area of ​​the second gaze point corresponding to the first gaze point on the second display screen can be constrained according to the epipolar line of the second display screen, and the position of the second gaze point can be determined accordingly.

[0087] For example, combined with Fig. 9 , take the image of a virtual tree provided by a VR helmet that the user is watching as an example. The current left eye pupil position P1, the right eye pupil position P2, and the corresponding position P0 of the observed virtual tree can constitute a core surface. The core surface can intersect with the second display screen, and the intersection line is the core line of the second display screen. It can be understood that on this core line, there is a point P4 corresponding to P3. In this example, the core line of the second display screen can be used as the search area for point P4. After determining the search area corresponding to point P4, the VR helmet can display the image corresponding to point P3 on the first display screen according to the image features, and search for points with similar (or identical) image features within the above-mentioned search area (or near the search area), and this point can be used as the position of P4 on the second display screen.

[0088] The epipolar plane can be determined based on the positions of P1, P2 and P0 in the same coordinate system, wherein the position of P0 can be calculated based on the position of P3 on the first display screen and the corresponding space mapping matrix (such as the space mapping matrix 1).

[0089] In some other implementations of the present application, the determination of the core surface may not rely on the calculation of P0. Fig. 9 , the plane formed by P1, P2 and P3 coincides with the epipolar surface. Therefore, the VR helmet can also determine the epipolar surface according to the positions of P1, P2 and P3. Among them, the line between P1 and P3 can be obtained when determining the position of P3. The line between the pupils of two eyes (i.e., P1 and P2) can be determined according to the design parameters of the VR helmet. For example, the optimal wearing position of the user is considered when designing a VR helmet. That is to say, for a VR helmet, during the user's use, the position of the user's eye line relative to the VR helmet is determined. Therefore, for the VR helmet, the straight line where P1 and P2 are located can be determined according to the corresponding design parameters. In this way, according to the line between P1 and P3, and the line between P1 and P2, the plane P1P2P3 can be determined, and this plane is the epipolar surface.

[0090] As an example, the VR helmet can obtain the gaze point of the left eye on the first display screen (i.e., the coordinates of the point P3 in the two-dimensional coordinate system corresponding to the first display screen) and obtain the position of the point P3 in the spatial coordinate system. According to the position of P3 in the spatial coordinate system, combined with the position of the line connecting P1 and P3 in the spatial coordinate system and the position of the line connecting P2 and P1 in the spatial coordinate system, the position of the epipolar surface including P1, P2 and P3 in the spatial coordinate system can be determined. A similar VR helmet can also determine the position of the second display screen in the spatial coordinate system. Thus, the position of the epipolar line corresponding to the second display screen in the spatial coordinate system can be determined. It can be understood that since the epipolar line corresponding to the second display screen is the intersection of the epipolar surface and the plane where the second display screen is located, the epipolar line will fall on the second display screen. In this example, after obtaining the position of the epipolar line in the spatial coordinate system, the VR helmet can convert it into a position in the two-dimensional coordinate system based on the second display screen, which is the position of the epipolar line on the second display screen. Next, the VR helmet can search for the corresponding pixels near the epipolar line on the second display screen based on the image features of the image corresponding to point P3 on the first display screen, and use the points with similar image features to point P3 as the pixels corresponding to point P4. Thus, the VR helmet can use the pixel position corresponding to point P4 as the position of point P4 on the second display screen, and thus obtain the gaze point coordinates of the right eye on the second display screen based on the two-dimensional coordinate system of the second display screen.

[0091] It should be noted that, compared with this application Figure 7 The scheme shown, the Figure 8 and Fig. 9 In the solution shown, the two-dimensional coordinates of P4 on the second display screen can be calculated without the position of P0 in the three-dimensional space. Therefore, in some cases where the position of P0 cannot be calculated, such as when the spatial mapping matrix encapsulated in the rendering process cannot be obtained, the VR helmet can also use the following method: Figure 8 as well as Fig. 9 The scheme shown in FIG. 1 is based on the first fixation point (such as Fig. 9 The position of P3 in the figure is calculated to obtain the second gaze point (such as Fig. 9 The position of P4) is shown.

[0092] It should also be noted that, if the VR helmet can calculate the position of P0, in addition to determining the position of P4 according to the solution described above, the position of P4 can also be determined based on the position of P0 and the position of the user's right pupil P2. Fig. 9Since point P4 is the projection of P0 on the second display screen, the area where point P4 is located on the second display screen can be constrained to be near the intersection of the line connecting P2 and P4 and the second display screen. In this example, the VR helmet can also use the area near the intersection of the line connecting P2 and P4 and the second display screen as the search area for point P4. Then, based on the image features corresponding to point P3 on the first display screen, points that are identical or similar to the above image features are determined within the search area as the position of point P4.

[0093] In this way, the purpose of obtaining the gaze point of one eye can be inferred based on the gaze point of the other eye. Through this solution, it is also possible to determine the gaze points of both eyes when only one detection module is set, thereby significantly reducing the computing power required for obtaining the gaze points of both eyes and the hardware cost.

[0094] It should be noted that in the above examples, the VR helmet is provided with one detection module as an example. In other embodiments of the present application, for an existing VR helmet that has two detection modules, one of the detection modules can be controlled to obtain the gaze point of the corresponding human eye, and the gaze point of the corresponding human eye can be obtained according to the following example: Figure 4 or Figure 8 The scheme shown is a method for estimating the gaze point of another person's eyes and determining the gaze points of both eyes. Therefore, only one detection module is required to work at the same time, so while reducing the pressure on computing power, it can also reduce power consumption and extend the working life of the detection module.

[0095] It should be understood that, as described above, in accordance with Figure 4 or Figure 8 When determining the gaze point of both eyes in the scheme shown, if the device includes only one detection module, the gaze point of the corresponding human eye can be determined according to the setting position of the detection module, and the gaze point of the other human eye can be inferred accordingly. If the device includes two detection modules at the same time, the gaze point of the corresponding human eye can be determined by one detection module randomly or according to a preset strategy (such as using two detection modules at intervals), and the gaze point of the other human eye can be inferred accordingly.

[0096] In some other implementations of the present application, when two detection modules are simultaneously provided in the device, when selecting which detection module to select to determine the gaze point of the first eye, the dominant eye can also be determined and the dominant eye can be used as the first eye, thereby improving the accuracy of the gaze point position of the first eye obtained by detection, and further improving the calibration accuracy of the gaze point coordinates of both eyes.

[0097] Generally speaking, when humans observe objects, they have two eyes, the dominant eye and the auxiliary eye. The dominant eye is also called the fixation eye or the dominant eye. Through a lot of research, it can be found that when people observe objects with both eyes, the fixation point of the dominant eye will be better concentrated in the fixation area. In contrast, the fixation point coordinates of the auxiliary eye will move irregularly around the fixation area. What the dominant eye sees will be accepted by the brain first.

[0098] Since the coordinates of the gaze point of the second person's eye need to be estimated based on the coordinates of the gaze point of the first person's eye, the more stable and accurate the coordinates of the gaze point of the first person's eye are, the more stable and accurate the coordinates of the gaze point of the second person's eye are. Figure 4 S401 as shown or Figure 8 In S801 shown, the VR helmet can first determine the dominant eye of the current user when viewing the scene. As an example, before executing the above S401 and / or S801, the VR helmet can respectively count the gaze coordinate data of the left and right eyes of the current user, analyze the performance differences, and determine the dominant eye. For example, take the example that the VR helmet is provided with detection modules near the left eye and the right eye. The VR helmet can use the detection module to respectively determine the distribution of the gaze points of the left eye and the right eye within a preset time period. If the gaze coordinates of the left eye are more stable than the gaze coordinates of the right eye, then the left eye is considered to be the dominant eye. Since the dominant eye and the auxiliary eye of the same user generally do not change, when the same user uses the VR helmet subsequently, the VR helmet can only start the detection module near the left eye, as shown in the following example. Figure 4 or Figure 8 Similarly, if the right eye is the dominant eye, the VR helmet can only start the detection module near the right eye, as shown in the scheme, to determine the gaze point coordinates of the left eye and the right eye. Figure 4 or Figure 8 The scheme shown determines the gaze point coordinates of the left and right eyes.

[0099] It should be noted that the above-mentioned solution for determining the dominant eye can be applied to a device with two detection modules at the same time, or can be set in a device with only one detection module. Exemplarily, the detection module can be provided with a moving structure, through which the detection module can be controlled to move flexibly near the left eye and the right eye. Therefore, after determining the dominant eye, the detection module can be moved to the detection position of the corresponding dominant eye by controlling the moving structure, so as to detect the gaze point of the dominant eye.

[0100] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of electronic equipment. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0101] The embodiment of the present application can divide the functional modules of the devices involved therein according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0102] Please refer to Fig.10 , which shows a schematic diagram of a binocular gaze point determination device 1000 provided in an embodiment of the present application. Fig.10 As shown, the device includes: an acquisition unit 1001, used to determine the first gaze point coordinates of a first eye, the first eye is one of the user's eyes, the first gaze point coordinates are the coordinates of the gaze point of the first eye on the first display screen, and the first display screen corresponds to the first eye; a determination unit 1002, used to determine the second gaze point coordinates according to the first gaze point coordinates, the second gaze point coordinates are the coordinates of the second eye of the user on the second display screen, and the second display screen corresponds to the second eye.

[0103] In a possible design, the position observed by the first eye is the same as or similar to the position observed by the second eye. The determination unit 1002 is used to determine the three-dimensional coordinates of the position observed by the first eye in the virtual three-dimensional space according to the first gaze point coordinates, and determine the second gaze point coordinates according to the three-dimensional coordinates.

[0104] In one possible design, determination unit 1002 is used to determine the three-dimensional coordinates based on the first gaze point coordinates and the first spatial mapping matrix; and to determine the second gaze point coordinates based on the three-dimensional coordinates and the second spatial mapping matrix; wherein the first spatial mapping matrix is ​​used to indicate the mapping relationship between the first gaze point coordinates and the three-dimensional coordinates, and the second spatial mapping matrix is ​​used to indicate the mapping relationship between the second gaze point coordinates and the three-dimensional coordinates.

[0105] In one possible design, the determination unit 1002 is used to determine a search area for a second gaze point based on the coordinates of the first gaze point, where the search area is an area on the second display screen, and the coordinates of the second gaze point are within the search area; in the search area, the coordinates of the second gaze point are determined, wherein image features corresponding to the coordinates of the second gaze point are the same as or similar to image features corresponding to the coordinates of the first gaze point.

[0106] In one possible design, the determination unit 1002 is used to determine the three-dimensional coordinates of the position observed by the first human eye in the virtual three-dimensional space based on the first gaze point coordinates and the first space mapping matrix, and the first space mapping matrix is ​​used to indicate the mapping relationship between the first gaze point coordinates and the space coordinates where the three-dimensional coordinates are located; based on the positional relationship between the three-dimensional coordinates and the second human eye, determine the intersection of the line connecting the position corresponding to the second human eye and the three-dimensional coordinates and the display screen corresponding to the second human eye; near the intersection, determine the second gaze point coordinates based on the first gaze point coordinates, wherein the image features corresponding to the second gaze point coordinates are the same or similar to the image features corresponding to the first gaze point coordinates.

[0107] In one possible design, the determination unit 1002 is used to determine the epipolar plane based on the coordinates of the first gaze point and the position relationship between the first gaze point and the user's eyes, where the epipolar plane is a plane including the first gaze point and the positions of the eyes; determine the intersection line of the epipolar plane and the display screen corresponding to the second eye; determine the coordinates of the second gaze point on the intersection line, wherein the image features corresponding to the coordinates of the second gaze point are the same as or similar to the image features corresponding to the coordinates of the first gaze point.

[0108] In a possible design, the display screen corresponding to the first human eye is the same as the display screen corresponding to the second human eye, or the display screen corresponding to the first human eye is different from the display screen corresponding to the second human eye.

[0109] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0110] Fig.11 FIG. 1 is a schematic diagram showing the composition of an electronic device 1100. Fig.11 As shown, the electronic device 1100 may include: a processor 1101 and a memory 1102. The memory 1102 is used to store computer-executable instructions. Exemplarily, in some embodiments, when the processor 1101 executes the instructions stored in the memory 1102, the electronic device 1100 may execute any of the binocular gaze point determination methods shown in the above embodiments.

[0111] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0112] Fig.12 A schematic diagram of the composition of a chip system 1200 is shown. The chip system 1200 may include: a processor 1201 and a communication interface 1202, which are used to support related devices to implement the functions involved in the above embodiments. In a possible design, the chip system also includes a memory for storing program instructions and data necessary for the terminal. The chip system may be composed of chips, or may include chips and other discrete devices. It should be noted that in some implementations of the present application, the communication interface 1202 may also be referred to as an interface circuit.

[0113] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0114] The functions or actions or operations or steps in the above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or may include one or more servers, data centers and other data storage devices that can be integrated with the medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0115] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for determining binocular gaze points, characterized in that: The method comprises: Determine a first gaze point coordinate of a first eye, where the first eye is one of the eyes of a user, the first gaze point coordinate is the coordinate of the gaze point of the first eye on a first display screen, and the first display screen corresponds to the first eye; Determine a second gaze point coordinate according to the first gaze point coordinate, where the second gaze point coordinate is a coordinate of a second eye of the user on a second display screen, and the second display screen corresponds to the second eye; The step of determining the second gaze point coordinates according to the first gaze point coordinates includes: Determine the three-dimensional coordinates of the position observed by the first human eye in the virtual three-dimensional space according to the first gaze point coordinates, and determine the second gaze point coordinates according to the three-dimensional coordinates; or Determine a search area for the second gaze point according to the coordinates of the first gaze point, wherein the search area is an area on the second display screen, and the coordinates of the second gaze point are within the search area; In the search area, coordinates of the second gaze point are determined, wherein image features corresponding to the coordinates of the second gaze point are the same as or similar to image features corresponding to the coordinates of the first gaze point.

2. The method according to claim 1, characterized in that: The determining the three-dimensional coordinates according to the first gaze point coordinates includes: Determining the three-dimensional coordinates according to the first gaze point coordinates and a first space mapping matrix; Determine the second gaze point coordinates according to the three-dimensional coordinates and a second space mapping matrix; The first space mapping matrix is ​​used to indicate the mapping relationship between the first gaze point coordinates and the three-dimensional coordinates, and the second space mapping matrix is ​​used to indicate the mapping relationship between the second gaze point coordinates and the three-dimensional coordinates.

3. The method according to claim 1, characterized in that The step of determining a search area for the second gaze point according to the coordinates of the first gaze point includes: Determine the three-dimensional coordinates of the position observed by the first human eye in the virtual three-dimensional space according to the first gaze point coordinates and a first space mapping matrix, wherein the first space mapping matrix is ​​used to indicate a mapping relationship between the first gaze point coordinates and the space coordinates where the three-dimensional coordinates are located; According to the positional relationship between the three-dimensional coordinates and the second eye, determining an intersection point between a line connecting the second eye and a position corresponding to the three-dimensional coordinates and a display screen corresponding to the second eye; Using a first area of ​​a preset size centered at the intersection as the search area; Determining the coordinates of the second gaze point in the search area includes: In the first area, the second gaze point coordinates are determined according to the first gaze point coordinates, wherein image features corresponding to the second gaze point coordinates are the same as or similar to image features corresponding to the first gaze point coordinates.

4. The method according to claim 1, characterized in that: The step of determining a search area for the second gaze point according to the coordinates of the first gaze point includes: Determine an epipolar plane according to the coordinates of the first gaze point and the positional relationship between the first gaze point and the user's eyes, where the epipolar plane is a plane including the first gaze point and the positions of the eyes; Determining an intersection line between the epipolar plane and the display screen corresponding to the second human eye; Using a second area of ​​a preset size centered on the intersection line as the search area; Determining the coordinates of the second gaze point in the search area includes: In the second area, the second gaze point coordinates are determined, wherein image features corresponding to the second gaze point coordinates are the same as or similar to image features corresponding to the first gaze point coordinates.

5. The method according to any one of claims 1 to 4, characterized in that Before the first gaze point coordinates of the first eye, the method further includes: Determine that the dominant eye of the user is the first eye.

6. An electronic device, characterized in that: The electronic device is provided with at least one display screen. The electronic device comprises one or more processors and one or more memories; the one or more memories are coupled to the one or more processors, and the one or more memories store computer instructions; When the one or more processors execute the computer instructions, the electronic device executes the method for determining binocular gaze points as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed, the method for determining the binocular gaze point according to any one of claims 1 to 5 is executed.

8. A chip system, characterized in that: The chip system includes an interface circuit and a processor; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from a memory and send the signal to a processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the chip system executes the method for determining the binocular gaze point as described in any one of claims 1-5.

Images