Human eye sight line detection method, device and electronic equipment

Abstract

The application relates to a human eye sight line detection method, device and electronic equipment, and the method comprises the following steps: acquiring an image containing a target observation point projected by a projector through a second image acquisition device; obtaining a first target observation point three-dimensional coordinate of the target observation point under a first image acquisition device according to a first calibration result and a second calibration result; performing human face detection on a human face image of an observed person collected by the first image acquisition device to obtain an eye point three-dimensional coordinate corresponding to the human face image under the first image acquisition device; and calculating a human eye sight line vector and an included angle of the observed person based on the first target observation point three-dimensional coordinate and the eye point three-dimensional coordinate. Based on the above method, the human eye sight line detection and the precision evaluation of the human eye sight line detection can be realized, the current human eye sight line detection system can be expanded with high cost and inconvenient operation, resources can be saved, and the precision of the human eye sight line detection is effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, device and electronic device for detecting human eye gaze. Background Technology

[0002] Human eye gaze detection is a crucial problem in the field of computer vision, with numerous applications in scenarios such as autonomous driving and smart homes. For example, when determining whether a driver is distracted or whether the central control screen needs to be illuminated for interaction with the vehicle's infotainment system, the driver's gaze can be detected based on their facial image, enabling safety checks and intelligent in-vehicle control.

[0003] Existing methods for calculating human eye gaze involve displaying a specific circle on multiple fixed screens and using a multi-camera system to capture images of the human eye focusing on that circle. The gaze distance is then calculated based on the information from these images. However, this method requires even more screens to expand the gaze area, resulting in high scalability costs. Furthermore, mobile devices are inconvenient for experimental use. Summary of the Invention

[0004] The purpose of this application is to provide a method, device, and electronic device for detecting human eye gaze, so as to realize the detection of human eye gaze and the evaluation of the accuracy of human eye gaze detection, solve the problems of high cost of scalability and inconvenient operation of current human eye gaze detection systems, and effectively improve the accuracy of human eye gaze detection while saving resources.

[0005] In a first aspect, this application provides a method for detecting human eye gaze, the method comprising:

[0006] The second image acquisition device acquires an image containing the target observation point projected by the projector.

[0007] Based on the first calibration result and the second calibration result, the three-dimensional coordinates of the first target observation point under the first image acquisition device are obtained, wherein the first calibration result is the coordinate system transformation relationship between the projector and the second image acquisition device, and the second calibration result is the coordinate system transformation relationship between the first image acquisition device and the projection device;

[0008] Face detection is performed on the face image of the observer acquired by the first image acquisition device to obtain the three-dimensional coordinates of the eye point corresponding to the face image under the first image acquisition device;

[0009] Based on the three-dimensional coordinates of the first target observation point and the three-dimensional coordinates of the eye point, the line-of-sight vector and the included angle of the observer's eyes are calculated.

[0010] The above-mentioned calibration of the projection device and the first image acquisition device and the projection device yields a first calibration result and a second calibration result with high accuracy. The accuracy of the three-dimensional coordinates of the first target observation point obtained from the first calibration result and the second calibration result is high, which makes the subsequent calculation of the human eye line vector and angle based on the three-dimensional coordinates of the first target observation point and the three-dimensional coordinates of the eye point more accurate, thus improving the accuracy of human eye line detection.

[0011] In one possible design, before acquiring an image containing the target observation point projected by the projector through the second image acquisition device, the method further includes: calibrating the projection device to obtain a first calibration result; and calibrating the first image acquisition device and the projection device to obtain a second calibration result.

[0012] By calibrating the projection device and the first image acquisition device and projection device, high-precision first and second calibration results can be obtained, which makes the subsequent detection of human eye line more accurate.

[0013] In one possible design, obtaining the three-dimensional coordinates of the target observation point under the first image acquisition device based on the first calibration result and the second calibration result includes: converting the two-dimensional coordinates of the target observation point under the projector into the first three-dimensional coordinates under the second image acquisition device based on the first calibration result; converting the first three-dimensional coordinates into the second three-dimensional coordinates under the first image acquisition device based on the second calibration result; and using the second three-dimensional coordinates as the three-dimensional coordinates of the first target observation point.

[0014] The accuracy of the three-dimensional coordinates of the first target observation point obtained from the first and second calibration results is high, which makes the accuracy of the human eye line vector and angle calculated subsequently based on the three-dimensional coordinates of the first target observation point and the three-dimensional coordinates of the eye point higher, thus improving the accuracy of human eye line detection.

[0015] In one possible design, after calculating the observer's eye line vector and angle based on the three-dimensional coordinates of the first target observation point and the three-dimensional coordinates of the eye point, the method further includes: determining the three-dimensional coordinates of the observer's eye point with the optical center of the first image acquisition device as the origin and the three-dimensional coordinates of the second target observation point of the target observation point, wherein the three-dimensional coordinates of the second target observation point are the theoretical coordinates of the target observation point; and calculating the error value between the three-dimensional coordinates of the first target observation point and the coordinates of the second target observation point based on the three-dimensional coordinates of the first target observation point, the three-dimensional coordinates of the second target observation point, and the three-dimensional coordinates.

[0016] In one possible design, determining the three-dimensional coordinates of the observer's eye point with the optical center of the first image acquisition device as the origin and the three-dimensional coordinates of the second target observation point of the target observation point includes: performing face detection on the face image of the observer acquired by the first image acquisition device to obtain the two-dimensional coordinates of the face feature points corresponding to the face image; determining the three-dimensional face model corresponding to the face image based on the two-dimensional coordinates of the face feature points; calculating the head posture and the three-dimensional coordinates of the observer based on the two-dimensional coordinates of the face feature points and the three-dimensional face model; and calculating the three-dimensional coordinates of the second target observation point based on the head posture, the three-dimensional coordinates, the eye gaze vector, and the gaze point plane, wherein the gaze point plane is the plane on which the target observation point is located as projected by the projector.

[0017] The above method evaluates the accuracy of human eye line detection by using the three-dimensional coordinates of the first target observation point, the three-dimensional coordinates of the second target observation point, and the three-dimensional coordinates of the third target observation point. The error value between the three-dimensional coordinates of the first target observation point and the three-dimensional coordinates of the second target observation point is calculated. Based on this error value, the accuracy of the human eye line detection system can be determined, which facilitates subsequent adjustments to the human eye detection system.

[0018] Secondly, this application provides a human eye gaze detection device, the device comprising:

[0019] The acquisition module acquires an image containing the target observation point projected by the projector through the second image acquisition device;

[0020] The acquisition module obtains the three-dimensional coordinates of the target observation point under the first image acquisition device based on the first calibration result and the second calibration result. The first calibration result is the coordinate system transformation relationship between the projector and the second image acquisition device, and the second calibration result is the coordinate system transformation relationship between the first image acquisition device and the projection device.

[0021] The detection module performs face detection on the face image of the observer acquired by the first image acquisition device to obtain the three-dimensional coordinates of the eye point corresponding to the face image under the first image acquisition device.

[0022] The calculation module calculates the line-of-sight vector and the included angle of the observer's eyes based on the three-dimensional coordinates of the first target observation point and the three-dimensional coordinates of the eye point.

[0023] In one possible design, the device further includes:

[0024] The first calibration module calibrates the projection device to obtain the first calibration result;

[0025] The second calibration module calibrates the first image acquisition device and the projection device to obtain the second calibration result.

[0026] In one possible design, the acquisition module is specifically used to: convert the two-dimensional coordinates of the target observation point under the projector into a first three-dimensional coordinate under the second image acquisition device according to the first calibration result; convert the first three-dimensional coordinates into a second three-dimensional coordinate under the first image acquisition device according to the second calibration result, and use the second three-dimensional coordinates as the three-dimensional coordinates of the first target observation point.

[0027] In one possible design, the device is further configured to: determine the three-dimensional coordinates of the observer's eye point with the optical center of the first image acquisition device as the origin and the three-dimensional coordinates of a second target observation point of the target observation point, wherein the three-dimensional coordinates of the second target observation point are the theoretical coordinates of the target observation point; and calculate the error value between the three-dimensional coordinates of the first target observation point and the coordinates of the second target observation point based on the three-dimensional coordinates of the first target observation point, the three-dimensional coordinates of the second target observation point, and the three-dimensional coordinates.

[0028] In one possible design, the device is further configured to: perform face detection on the face image of the observer acquired by the first image acquisition device to obtain the two-dimensional coordinates of the face feature points corresponding to the face image; determine the three-dimensional face model corresponding to the face image based on the two-dimensional coordinates of the face feature points; calculate the head pose and the three-dimensional coordinates of the observer based on the two-dimensional coordinates of the face feature points and the three-dimensional face model; and calculate the three-dimensional coordinates of the second target observation point based on the head pose, the three-dimensional coordinates, the eye gaze vector, and the gaze point plane, wherein the gaze point plane is the plane on which the target observation point is projected by the projector.

[0029] Thirdly, this application provides an electronic device, the electronic device comprising:

[0030] Memory, used to store computer programs;

[0031] When the processor executes the computer program stored in the memory, it implements the above-described method steps for detecting human eye gaze.

[0032] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps for detecting human eye gaze.

[0033] For the various aspects of the second to fourth aspects mentioned above, and the technical effects that each aspect may achieve, please refer to the above description of the technical effects that can be achieved for the first aspect or the various possible solutions in the first aspect, which will not be repeated here. Attached Figure Description

[0034] Figure 1 A flowchart of a human eye line detection method provided in this application;

[0035] Figure 2 This is a schematic diagram of a human eye line detection system provided in this application;

[0036] Figure 3 A schematic diagram of a possible projection pattern provided for this application;

[0037] Figure 4 A schematic diagram illustrating a possible human eye line vector and angle provided for this application;

[0038] Figure 5 A schematic diagram of a human eye line detection device provided in this application;

[0039] Figure 6 A schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operating methods in the method embodiments can also be applied to the device embodiments or system embodiments.

[0041] In the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A connected to B can represent: A and B directly connected, or A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0042] To facilitate a better understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the following is a brief explanation of the relevant technical terms:

[0043] A homography matrix (H) describes the mapping relationship between two planes. If feature points all fall on a certain plane in the scene, the camera pose can be calculated using the homography matrix.

[0044] The method provided by exemplary embodiments of this application will now be described with reference to the accompanying drawings.

[0045] See Figure 1 The diagram shown is a flowchart of the human eye gaze detection method provided in this application embodiment. The specific implementation process of this method is as follows:

[0046] Step 101: Acquire an image containing the target observation point projected by the projector using the second image acquisition device;

[0047] Step 102: Based on the first calibration result and the second calibration result, obtain the three-dimensional coordinates of the first target observation point under the first image acquisition device;

[0048] Step 103: Perform face detection on the face image of the observer acquired by the first image acquisition device to obtain the three-dimensional coordinates of the eye point corresponding to the face image under the first image acquisition device;

[0049] Step 104: Based on the three-dimensional coordinates of the first target observation point and the three-dimensional coordinates of the eye point, calculate the line-of-sight vector and the included angle of the observer's eyes.

[0050] In this embodiment, the human eye gaze detection method is applied to a human eye gaze detection system, which is as follows: Figure 2 As shown, it includes: a projection device, a first image acquisition device, and a fixed target plate. The projection device further includes: a projector and a second image acquisition device. The first image acquisition device can be a 3D camera, and the second image acquisition device can be an industrial camera. Compared with ordinary cameras (camcorders), industrial cameras have high image stability, high transmission capacity, and high anti-interference capability.

[0051] It should be noted that the first image acquisition device can be any other acquisition device with the same function as a 3D camera, and is not limited to a 3D camera. Similarly, the second image acquisition device can be any other acquisition device with the same function as an industrial camera, and is not limited to an industrial camera. The embodiments of this application will be described in detail below using a 3D camera as the first image acquisition device and an industrial camera as the second image acquisition device as an example.

[0052] In this embodiment, before acquiring an image of the target observation point projected by the projector using an industrial camera, it is necessary to first calibrate the projection device and both the 3D camera and the projection device. A first calibration result is obtained by calibrating the projection device; a second calibration result is obtained by calibrating the 3D camera and the projection device. The first calibration result represents the coordinate transformation relationship between the projector and the industrial camera, and the second calibration result represents the coordinate transformation relationship between the 3D camera and the projection device.

[0053] Specifically, the first step is to calibrate the projection equipment. The specific workflow for projection equipment calibration is as follows:

[0054] 1) Generate a projected pattern using a projector. The projected pattern can be a checkerboard, circle, square, or other patterns. The coordinates p1(x, y) of each corner point in the projected pattern are fixed coordinates. For example, when the projected pattern is a checkerboard, ... Figure 3 As shown, the corner coordinates of each grid point on the chessboard are fixed coordinates. The corner coordinates p1(x, y) are the pixel coordinates in the projector coordinate system;

[0055] 2) When the relative positions of the projector and industrial camera in the projection device are fixed (the relative position is the position where the industrial camera can simultaneously acquire images of the fixed target board and the projected pattern), the industrial camera acquires images containing both the fixed target board and the projected pattern, and obtains at least four corner coordinates p2(x, y) and four corresponding pixel coordinates C2(u, v) of the fixed target board. The corner coordinates p2(x, y) are the two-dimensional coordinates of the fixed target board in the world coordinate system, and the pixel coordinates C2(u, v) are the pixel coordinates of the fixed target board in the industrial camera coordinate system. Based on the corner coordinates p2(x, y) and the corresponding pixel coordinates C2(u, v), the homography matrix H between the fixed target board and the industrial camera is calculated.

[0056] 3) Calculate the plane equation (i.e., the gaze point plane equation) of the calibration plane where the fixed target plate is located based on the homography matrix H, the coordinates p2(x, y) of each corner point of the fixed target plate, and the intrinsic parameters of the industrial camera.

[0057] It should be noted that the gaze point plane equation is a plane equation in the industrial camera coordinate system. All points on this plane satisfy this gaze point plane equation. If the projector projects a pattern onto this plane, then all points on the projected pattern satisfy this gaze point plane equation.

[0058] 4) Extract the corner coordinates C1(u, v) of the projected pattern in the industrial camera coordinate system. Based on the industrial camera intrinsic parameters, the corner coordinates C1(u, v), and the equation of the gaze point plane, analyze the 3D coordinates C of the projected pattern in the industrial camera coordinate system. wi ;

[0059] 5) Move the projection device and change the projection angle. Use an industrial camera to take multiple images from different angles, including the projected pattern and the fixed target plate. At this time, p1(x,y), C1(u,v), C wi Given that both are known, dual-target calibration is performed to determine the poses (extrinsic parameters R and T) of the projector and the industrial camera, thereby obtaining a more accurate first calibration result, namely the coordinate transformation relationship between the projector and the industrial camera.

[0060] By moving the projection device, images of the projected pattern and the fixed target plate at different angles are obtained. Based on p1(x,y), C1(u,v), and C at different angles... wi Performing dual-target calibration results in a more accurate first calibration result and a more precise coordinate transformation relationship between the projector and the industrial camera.

[0061] After completing the calibration of the projection device, the 3D camera and projection device are calibrated to obtain a second calibration result, which is the coordinate transformation relationship between the 3D camera and the projection device. Using this second calibration result, the 3D coordinates of a point in the industrial camera coordinate system can be converted to the 3D coordinates of the point in the 3D camera coordinate system. In this embodiment, the 3D camera and projection device calibration method can employ Zhang's calibration method, or it can employ higher-precision Gray code or positive grating methods.

[0062] Through the above-mentioned calibration of the projection equipment and the 3D camera and projection equipment, a first calibration result and a second calibration result with high accuracy can be obtained, thereby making the subsequent detection of human eye line more accurate.

[0063] After the calibration is completed, the human eye gaze detection can begin. The specific workflow for human eye gaze detection is as follows:

[0064] 1) Determine the 3D coordinates of the target observation point

[0065] First, adjust the position of the projection device according to the needs, and project the target observation point through the projector. The target observation point can be a circle, square, triangle, etc.

[0066] An image of the target observation point, including the projection of a projector, is acquired using an industrial camera. The 2D coordinates of the target observation point under the projector are extracted. At this point, the 2D coordinates of the projector are known. Based on the 2D coordinates of the target observation point under the projector, the 2D coordinates of the projector, and the first calibration result, the 2D coordinates of the target observation point under the projector are converted into the first 3D coordinates of the target observation point under the industrial camera.

[0067] Based on the second calibration result, the first 3D coordinates of the target observation point under the industrial camera are converted into the second 3D coordinates under the 3D camera coordinate system, and the second 3D coordinates are used as the 3D coordinates gt(x, y, z) of the target observation point.

[0068] 2) Determine the 3D coordinates of the observer's eye point.

[0069] Specifically, a 3D camera is used to capture a face image of an observer looking at a target observation point. Face detection is performed on the captured face image to obtain the 2D coordinates of the face feature points corresponding to the face image. Then, based on the 2D coordinates of the face feature points, the 3D coordinates of the observer's eye point in the 3D camera coordinate system are obtained.

[0070] 3) Calculate the line-of-sight vector and the included angle.

[0071] Specifically, based on the determined 3D coordinates of the target observation point and the 3D coordinates of the observer's eye point, the observer's eye-line vector Vr and the included angle are calculated. (See [reference]). Figure 4 As shown, the human eye gaze vector Vr is the vector of the line connecting the 3D coordinates of the target observation point and the 3D coordinates of the observer's eye point. The included angles are the angle a1 between the human eye gaze vector Vr and the xoy plane (vertical angle, such as 45° upward) and the angle a2 between the human eye gaze vector Vr and the zoy plane (horizontal angle, such as 15° to the right).

[0072] The above methods and systems improve the accuracy of the first and second calibration results, thereby increasing the accuracy of the target observation point 3D coordinates obtained from the first and second calibration results. The accuracy of the human eye line vector and angle calculated from the target observation point 3D coordinates and the eye point 3D coordinates is also higher, thus improving the accuracy of human eye line detection.

[0073] Furthermore, after calculating the observer's line-of-sight vector and angle based on the 3D coordinates of the target observation point and the observer's eye point, the accuracy of the line-of-sight detection can also be calculated based on this system. The specific process for calculating the accuracy of the line-of-sight detection is as follows:

[0074] 1) Determine the three-dimensional coordinates t of the observer's eye point with the optical center of the 3D camera as the origin, and the 3D coordinates gc(x, y, z) of the second target observation point of the target observation point;

[0075] In this embodiment of the application, the second target observation point 3D coordinates of the target observation point are the theoretical coordinates of the target observation point, which can be understood as the three-dimensional coordinates of the intersection between the observer's line of sight and the plane of the pattern projected by the projector.

[0076] Specifically, based on the face detection performed on the acquired face image of the observer mentioned above, the 2D coordinates of the face feature points corresponding to the face image are obtained, and a 3D face model is determined based on the 2D coordinates of the face feature points. Based on the 2D coordinates of the face feature points and the 3D face model, the head pose R of the observer and the three-dimensional coordinates t of the observer's eye point with the optical center of the 3D camera as the origin are determined.

[0077] Based on the previously calculated human eye gaze vector Vr, head pose R, three-dimensional coordinates t, and gaze point plane equation, the second target observation point 3D coordinates gc(x, y, z) are calculated.

[0078] The above method is used to determine the 3D coordinates of the second target observation point, which is the theoretical coordinates of the target observation point. This facilitates subsequent comparison with the 3D coordinates of the first target observation point to obtain the error value between the two.

[0079] 2) Based on the 3D coordinates gt(x, y, z) of the first target observation point, the 3D coordinates gc(x, y, z) of the second target observation point, and the three-dimensional coordinate t, calculate the error value between the 3D coordinates of the first target observation point and the 3D coordinates of the second target observation point.

[0080] In this embodiment of the application, the accuracy evaluation index of human eye line detection is the error between the 3D coordinates of the first target observation point and the 3D coordinates of the second target observation point. This error may include: relative distance error and angle error.

[0081] Specifically, the formula for calculating the relative distance error is as follows:

[0082]

[0083] The formula for calculating angular error is as follows:

[0084]

[0085] In the embodiments of this application, a = |t-gt|, b = |t-gc|, and c = |gt-gc|.

[0086] The above method evaluates the accuracy of human eye line detection by using the 3D coordinates gt of the first target observation point, gc of the second target observation point, and the three-dimensional coordinate t. It calculates the error value between the 3D coordinates of the first target observation point and the 3D coordinates of the second target observation point. Based on this error value, the accuracy of the human eye line detection system can be determined, which facilitates subsequent adjustments to the human eye detection system.

[0087] In summary, the human eye line detection method and system provided in this application have high accuracy. The target observation point is reconstructed using structured light 3D, which can be accurate to the millimeter level. It is simple to use, flexible to expand, and the projection device can be moved as needed to collect different fields of view. The projection pattern can also be flexibly modified according to the density of the target observation point. Furthermore, it integrates the acquisition system and the accuracy evaluation system, saving resources.

[0088] Based on the same inventive concept, this application also provides a human eye gaze detection device for detecting human eye gaze and evaluating the accuracy of human eye gaze detection. This solves the problems of high scalability and inconvenient operation in current human eye gaze detection systems, effectively improving the accuracy of human eye gaze detection and saving resources. See [link to relevant documentation]. Figure 5 The device includes:

[0089] The acquisition module 501 acquires an image containing the target observation point projected by the projector through the second image acquisition device;

[0090] The acquisition module 502 obtains the three-dimensional coordinates of the target observation point under the first image acquisition device based on the first calibration result and the second calibration result. The first calibration result is the coordinate system transformation relationship between the projector and the second image acquisition device, and the second calibration result is the coordinate system transformation relationship between the first image acquisition device and the projection device.

[0091] The detection module 503 performs face detection on the face image of the observer acquired by the first image acquisition device to obtain the three-dimensional coordinates of the eye point corresponding to the face image under the first image acquisition device.

[0092] The calculation module 504 calculates the line-of-sight vector and the included angle of the observer's eyes based on the three-dimensional coordinates of the first target observation point and the three-dimensional coordinates of the eye point.

[0093] In one possible design, the device further includes:

[0094] The first calibration module calibrates the projection device to obtain the first calibration result;

[0095] The second calibration module calibrates the first image acquisition device and the projection device to obtain the second calibration result.

[0096] In one possible design, the acquisition module 502 is specifically used to: convert the two-dimensional coordinates of the target observation point under the projector into a first three-dimensional coordinate under the second image acquisition device according to the first calibration result; convert the first three-dimensional coordinates into a second three-dimensional coordinate under the first image acquisition device according to the second calibration result, and use the second three-dimensional coordinates as the three-dimensional coordinates of the first target observation point.

[0097] In one possible design, the device is further configured to: determine the three-dimensional coordinates of the observer's eye point with the optical center of the first image acquisition device as the origin and the three-dimensional coordinates of a second target observation point of the target observation point, wherein the three-dimensional coordinates of the second target observation point are the theoretical coordinates of the target observation point; and calculate the error value between the three-dimensional coordinates of the first target observation point and the coordinates of the second target observation point based on the three-dimensional coordinates of the first target observation point, the three-dimensional coordinates of the second target observation point, and the three-dimensional coordinates.

[0098] In one possible design, the device is further configured to: perform face detection on the face image of the observer acquired by the first image acquisition device to obtain the two-dimensional coordinates of the face feature points corresponding to the face image; determine the three-dimensional face model corresponding to the face image based on the two-dimensional coordinates of the face feature points; calculate the head pose and the three-dimensional coordinates of the observer based on the two-dimensional coordinates of the face feature points and the three-dimensional face model; and calculate the three-dimensional coordinates of the second target observation point based on the head pose, the three-dimensional coordinates, the eye gaze vector, and the gaze point plane, wherein the gaze point plane is the plane on which the target observation point is projected by the projector.

[0099] Based on the above system, the detection of human eye gaze and the evaluation of the accuracy of human eye gaze detection are realized, which solves the problems of high cost of scalability and inconvenient operation of current human eye gaze detection systems, and effectively improves the accuracy of human eye gaze detection and saves resources.

[0100] Based on the same inventive concept, this application also provides an electronic device that can realize the function of the aforementioned human eye gaze detection device, see reference. Figure 6 The electronic device includes:

[0101] At least one processor 601 and a memory 602 connected to at least one processor 601. In this embodiment, the specific connection medium between the processor 601 and the memory 602 is not limited. Figure 6 The example shown is the connection between processor 601 and memory 602 via bus 600. Bus 600 is... Figure 6 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 600 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 6 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 601 can also be called a controller; there is no restriction on the name.

[0102] In this embodiment, memory 602 stores instructions executable by at least one processor 601. By executing the instructions stored in memory 602, at least one processor 601 can perform the human eye gaze detection method described above. Processor 601 can implement... Figure 5 The functions of each module in the device shown.

[0103] The processor 601 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 602 and calling data stored in memory 602, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0104] In one possible design, processor 601 may include one or more processing units. Processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 601. In some embodiments, processor 601 and memory 602 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0105] The processor 601 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the human eye gaze detection method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0106] Memory 602, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 602 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 602 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 602 may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0107] By designing and programming the processor 601, the code corresponding to the human eye gaze detection method described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute the code during operation. Figure 1 The steps of the human eye gaze detection method in the illustrated embodiment are described below. How to design and program the processor 601 is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0108] Based on the same inventive concept, embodiments of this application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the aforementioned method for detecting human eye gaze.

[0109] In some possible implementations, various aspects of the human eye gaze detection method provided in this application can also be implemented in the form of a program product, which includes program code that, when the program product is run on a device, causes the control device to perform the steps in the human eye gaze detection method according to the various exemplary embodiments of this application described above.

[0110] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0114] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for detecting human eye gaze, characterized in that, The method is applied to a human eye line detection system, the system comprising: a projection device, a first image acquisition device, and a fixed target plate, the projection device comprising a projector and a second image acquisition device, the method comprising: The second image acquisition device acquires an image containing the target observation point projected by the projector. Based on the first calibration result, the two-dimensional coordinates of the target observation point under the projector are converted into the first three-dimensional coordinates under the second image acquisition device, wherein the first calibration result is the coordinate system transformation relationship between the projector and the second image acquisition device; According to the second calibration result, the first three-dimensional coordinates are converted into the second three-dimensional coordinates under the first image acquisition device, and the second three-dimensional coordinates are used as the three-dimensional coordinates of the first target observation point under the first image acquisition device. The second calibration result is the coordinate system transformation relationship between the first image acquisition device and the projection device. Face detection is performed on the face image of the observer acquired by the first image acquisition device to obtain the two-dimensional coordinates of the face feature points corresponding to the face image, and based on the two-dimensional coordinates of the face feature points, the three-dimensional coordinates of the eye point of the observer under the first image acquisition device are obtained. The vector connecting the three-dimensional coordinates of the first target observation point and the three-dimensional coordinates of the observer's eye point is taken as the observer's eye line vector, and the angle between the eye line vector and the xoy plane and zoy plane is taken as the observer's angle.

2. The method as described in claim 1, characterized in that, Before acquiring an image containing the target observation point projected by the projector through the second image acquisition device, the method further includes: The projection device was calibrated to obtain the first calibration result; The first image acquisition device and the projection device are calibrated to obtain the second calibration result.

3. The method as described in claim 1, characterized in that, After calculating the observer's line-of-sight vector and angle based on the three-dimensional coordinates of the first target observation point and the three-dimensional coordinates of the eye point, the method further includes: The three-dimensional coordinates of the observer's eye point with the optical center of the first image acquisition device as the origin and the three-dimensional coordinates of the second target observation point of the target observation point are determined, wherein the three-dimensional coordinates of the second target observation point are the theoretical coordinates of the target observation point; Based on the three-dimensional coordinates of the first target observation point, the three-dimensional coordinates of the second target observation point, and the three-dimensional coordinates, the error value between the three-dimensional coordinates of the first target observation point and the three-dimensional coordinates of the second target observation point is calculated.

4. The method as described in claim 3, characterized in that, Determining the three-dimensional coordinates of the observer's eye point with the optical center of the first image acquisition device as the origin, and the three-dimensional coordinates of the second target observation point of the target observation point, includes: Face detection is performed on the face image of the observer acquired by the first image acquisition device to obtain the two-dimensional coordinates of the face feature points corresponding to the face image; The three-dimensional model of the face corresponding to the face image is determined based on the two-dimensional coordinates of the face feature points; Based on the two-dimensional coordinates of the facial feature points and the three-dimensional model of the face, the head pose and three-dimensional coordinates of the observer are calculated. Based on the head posture, the three-dimensional coordinates, the human eye gaze vector, and the gaze point plane, the three-dimensional coordinates of the second target observation point are calculated, wherein the gaze point plane is the plane on which the target observation point is projected by the projector.

5. A human eye gaze detection device, characterized in that, The device includes: The acquisition module acquires images of the target observation point projected by the projector through a second image acquisition device; The acquisition module converts the two-dimensional coordinates of the target observation point under the projector into the first three-dimensional coordinates under the second image acquisition device according to the first calibration result, wherein the first calibration result is the coordinate system transformation relationship between the projector and the second image acquisition device; According to the second calibration result, the first three-dimensional coordinates are converted into second three-dimensional coordinates under the first image acquisition device, and the second three-dimensional coordinates are used as the three-dimensional coordinates of the first target observation point under the first image acquisition device. The second calibration result is the coordinate system transformation relationship between the first image acquisition device and the projection device. The detection module performs face detection on the face image of the observer acquired by the first image acquisition device, obtains the two-dimensional coordinates of the face feature points corresponding to the face image, and obtains the three-dimensional coordinates of the eye point of the observer under the first image acquisition device based on the two-dimensional coordinates of the face feature points. The calculation module uses the vector of the line connecting the three-dimensional coordinates of the first target observation point and the three-dimensional coordinates of the observer's eye point as the observer's eye line vector, and uses the angle between the eye line vector and the xoy plane and zoy plane as the observer's angle.

6. The apparatus as claimed in claim 5, characterized in that, The device further includes: The first calibration module calibrates the projection device to obtain the first calibration result; The second calibration module calibrates the first image acquisition device and the projection device to obtain the second calibration result.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a computer program stored in the memory, implements the method steps of any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-4.

Images