Method and device for line of sight detection based on monocular camera

Abstract

The present disclosure provides a method and device for line of sight detection based on a monocular camera. The method comprises: extracting facial features of a facial region and an eye region included in the facial region based on an image acquired by a monocular camera, wherein the eye region comprises an eye socket region and an iris region; respectively inferring the line of sight direction of the left eye and the line of sight direction of the right eye based on the eye socket region and the corresponding iris region; and inferring three-dimensional line of sight information based on the line of sight direction of the left eye, the line of sight direction of the right eye and the facial features.

Description

Technical Field

[0001] The present disclosure relates to the field of eye tracking, and in particular, provides a method for line of sight detection based on a monocular camera and a device using the method. Background Art

[0002] Gaze direction detection has been an active research topic for decades. Gaze detection plays an important role in psychology. For example, based on gaze detection, statistics can be used to calculate states that can reflect psychological activities, such as saccades, fixations, and scans. Gaze detection can also be widely used in the field of human-computer interaction and can be used as a new means of human-computer interaction.

[0003] The existing line of sight detection methods are mainly divided into wearable devices and medium- and long-distance multi-camera detection methods. The equipment is complex and expensive, which limits the possibility of wide application in actual application scenarios. Summary of the invention

[0004] The present disclosure provides a method for line of sight detection based on a monocular camera and a device using the method. The embodiments provided by the present disclosure can infer the line of sight direction based on a monocular camera (e.g., an ordinary RGB camera) without the need for special equipment or special cameras. This reduces the cost of the equipment and the difficulty of obtaining the equipment.

[0005] One aspect of the present disclosure provides a method for line of sight detection based on a monocular camera, the method comprising: extracting facial features of a facial area and an eye area included in the facial area based on an image acquired by the monocular camera, wherein the eye area comprises an eye socket area and an iris area; inferring a left eye line of sight direction and a right eye line of sight direction based on the eye socket area and the corresponding iris area, respectively; and inferring three-dimensional line of sight information based on the left eye line of sight direction, the right eye line of sight direction and facial features.

[0006] Optionally, the step of inferring three-dimensional line of sight information based on the left eye line of sight direction, the right eye line of sight direction and facial features may include: inferring the three-dimensional line of sight information based on the left eye line of sight direction, the right eye line of sight direction and facial features using a pre-trained multi-layer perceptron network, wherein the pre-trained multi-layer perceptron network includes: an input layer, configured to use the left eye line of sight direction, the right eye line of sight direction and facial features as input; a first hidden layer, configured to include a first sublayer and a second sublayer, wherein the left eye line of sight direction and facial features in the input layer form a full connection with the first sublayer, and the right eye line of sight direction and facial features in the input layer form a full connection with the second sublayer; a second hidden layer, configured to form a full connection with the first hidden layer; and an output layer, configured to form a full connection with the second hidden layer.

[0007] Optionally, the step of inferring three-dimensional line of sight information based on the left eye line of sight direction, the right eye line of sight direction and facial features may include: filtering the line of sight direction through a monocular line of sight filter, and inferring the three-dimensional line of sight information based on the filtered line of sight direction and facial features, wherein the line of sight direction includes the left eye line of sight direction and the right eye line of sight direction, wherein the monocular line of sight filter may include: an autoregressive integral moving average model, which takes the line of sight direction of the current frame as input and outputs the filtered line of sight direction of the current frame based on model parameters; a multilayer perceptron, which takes the line of sight direction of the current frame and the filtered line of sight direction of the current frame as input, and outputs the optimized model parameters to the autoregressive integral moving average model, wherein the autoregressive integral moving average model filters the line of sight direction of the next frame based on the optimized model parameters.

[0008] Optionally, the steps of respectively inferring the left eye sight direction and the right eye sight direction based on the orbital area and the corresponding iris area may include: dividing each orbital area into four equal quadrants; calculating the integral value of the corresponding iris area in each of the four quadrants; and respectively inferring the left eye sight direction and the right eye sight direction based on the integral values.

[0009] Another aspect of the present disclosure discloses a device for line of sight detection based on a monocular camera, the device comprising: a feature extraction unit, configured to extract facial features of a facial area and an eye area included in the facial area based on an image acquired by the monocular camera, wherein the eye area comprises an eye socket area and an iris area; a monocular line of sight estimation unit, configured to estimate the left eye line of sight direction and the right eye line of sight direction respectively based on the eye socket area and the corresponding iris area; and a three-dimensional line of sight estimation unit, configured to estimate three-dimensional line of sight information based on the left eye line of sight direction, the right eye line of sight direction and facial features.

[0010] Optionally, the three-dimensional line of sight estimation unit is configured to: use a pre-trained multi-layer perceptron network to infer three-dimensional line of sight information based on the left eye line of sight direction, the right eye line of sight direction and facial features, wherein the pre-trained multi-layer perceptron network includes: an input layer, configured to use the left eye line of sight direction, the right eye line of sight direction and facial features as input; a first hidden layer, configured to include a first sublayer and a second sublayer, wherein the left eye line of sight direction and facial features in the input layer are fully connected with the first sublayer, and the right eye line of sight direction and facial features in the input layer are fully connected with the second sublayer; a second hidden layer, configured to be fully connected with the first hidden layer; and an output layer, configured to be fully connected with the second hidden layer.

[0011] Optionally, the three-dimensional line of sight estimation unit is configured to: filter the line of sight direction through a monocular line of sight filter, and estimate the three-dimensional line of sight information based on the filtered line of sight direction and facial features, wherein the line of sight direction includes the line of sight direction of the left eye and the line of sight direction of the right eye, wherein the monocular line of sight filter includes: an autoregressive integral moving average model, which takes the line of sight direction of the current frame as input and outputs the filtered line of sight direction of the current frame based on model parameters; a multi-layer perceptron, which takes the line of sight direction of the current frame and the filtered line of sight direction of the current frame as input, and outputs optimized model parameters to the autoregressive integral moving average model, wherein the autoregressive integral moving average model filters the line of sight direction of the next frame based on the optimized model parameters.

[0012] Optionally, the steps of respectively inferring the left eye sight direction and the right eye sight direction based on the orbital area and the corresponding iris area include: dividing each orbital area into four equal quadrants; calculating the integral value of the corresponding iris area falling in each of the four quadrants; and respectively inferring the left eye sight direction and the right eye sight direction based on the integral values.

[0013] Another aspect of the present disclosure provides a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the line of sight detection method as described above is implemented.

[0014] Another aspect of the present disclosure provides a computer device, characterized in that the computer device includes: a processor; and a memory storing a computer program, and when the computer program is executed by the processor, the line of sight detection method described above is implemented.

[0015] According to one or more aspects of the present disclosure, a method for eye recognition and tracking based on a monocular visible light camera and a device using the method are proposed. Line of sight detection based on ordinary RGB images can be achieved with only one monocular visible light camera. Compared with the existing technology, the present disclosure has better usability in practical application scenarios. The arrangement is simple, no active light source is required, and no need to wear it. In addition, since only one monocular visible light camera is used, the cost of use is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other aspects, features and advantages of the present disclosure will be more clearly understood through the following detailed description in conjunction with the accompanying drawings, in which:

[0017] Figure 1 is a schematic diagram showing a usage scenario of a method for line of sight detection based on a monocular camera according to an embodiment of the present disclosure;

[0018] Figure 2is a flow chart illustrating a method for line of sight detection based on a monocular camera according to an exemplary embodiment;

[0019] Figure 3 is a schematic diagram showing a monocular sight filter according to an exemplary embodiment;

[0020] Figure 4 is a flow chart illustrating inferring three-dimensional line of sight information according to an exemplary embodiment;

[0021] Figure 5 is a schematic diagram showing a multi-layer perceptron network for inferring three-dimensional line of sight information according to an exemplary embodiment;

[0022] Figure 6 is an example showing estimated three-dimensional visual line information according to an exemplary embodiment;

[0023] Figure 7 is a block diagram illustrating an apparatus for monocular camera-based line of sight detection according to an exemplary embodiment; and

[0024] Figure 8 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following specific embodiments are provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be apparent to those of ordinary skill in the art. For example, the order of operations described herein is merely an example and is not limited to the order set forth herein, but changes that will be apparent to those of ordinary skill in the art may be made in addition to the operations that must be performed in a particular order. In addition, in order to improve clarity and brevity, descriptions of features and structures that will be known to those of ordinary skill in the art may be omitted. The features described herein may be implemented in different forms and will not be construed as being limited to the examples described herein. More specifically, the examples described herein have been provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those of ordinary skill in the art.

[0026] Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. Examples of the embodiments are shown in the accompanying drawings, wherein the same reference numerals refer to the same components throughout. The embodiments will be described below with reference to the accompanying drawings in order to explain the present disclosure.

[0027] Figure 1 It is a schematic diagram showing a usage scenario of implementing a method for line of sight detection based on a monocular camera according to an embodiment of the present disclosure.

[0028] Different from the conventional method of performing sight line detection by means of wearable devices or multiple cameras, the sight line detection method according to the embodiment of the present disclosure only needs to be implemented based on the image obtained by a monocular camera. Figure 1 As shown, a person can be in front of the camera in various postures. The present disclosure provides a method for line of sight detection based on a monocular camera and a device using the method. For example, the line of sight detection method of an embodiment of the present disclosure can infer three-dimensional line of sight information based on an image obtained by a monocular camera, thereby inferring the area where the line of sight is focused. For example, when a person's line of sight is focused on an object at a long distance, the inferred three-dimensional line of sight information is angle information. and depth information D1; when a person's line of sight is looking at an object that is relatively close, the inferred three-dimensional line of sight information is angle information And depth information D2. Figure 1 In the example shown, three-dimensional polar coordinates are used to represent the relative relationship between the human eye and the object. However, this is only an example and should not be understood as a limitation of the present disclosure. Any method of representing position correlation may be included in the scope of the present disclosure.

[0029] Compared with the existing technology, the present invention has better usability in practical application scenarios, is simple to arrange, does not require active light source, does not need to be worn, and because only one monocular visible light camera is used, the use cost is reduced.

[0030] Figure 2 is a flow chart illustrating a method of line of sight detection based on a monocular camera according to an exemplary embodiment. Figure 3 is a schematic diagram illustrating a monocular sight filter according to an exemplary embodiment. Figure 4 is a flow chart illustrating a method of estimating three-dimensional visual line information according to an exemplary embodiment. Figure 5 is a schematic diagram illustrating a multi-layer perceptron network for inferring three-dimensional line of sight information according to an exemplary embodiment.

[0031] According to an embodiment of the present disclosure, in step S10, a facial region and an eye region included in the facial region are obtained based on an image acquired by a monocular camera, wherein the eye region includes an eye socket region and an iris region.

[0032] In the embodiment, since the original image has a lot of redundant information for binocular sight line estimation, in order to facilitate the establishment of the subsequent binocular sight line estimation model, it is necessary to extract the effective information therein. Among them, the spatial distribution information in the facial image is more useful for sight line estimation, so the facial area can be processed by extracting features including but not limited to HOG, SIFT, CNN, etc. that can describe the spatial distribution information, thereby obtaining facial features (e.g., coded features) of the facial area.

[0033] In an embodiment, sometimes there are interferences such as eye sockets and eyebrows in the eye region of an image that affect the eye socket detection. The image can be subjected to eye socket detection, so as to correctly extract the eye socket and eliminate such interferences. In an embodiment of performing eye socket detection on an image, first, a color space conversion is performed, for example, the eye image can be converted into an HSV space; secondly, the color gamut can be counted, for example, the most frequently appearing elements (high_h) in the H space can be counted; then, the eye image is scanned, an eye image template is created, and the S value of the elements that are not in the high_h range is set to 0; then, the holes in the template image are searched, and the largest hole information is retained, which is the eye socket region.

[0034] In an embodiment of iris detection on an eye image corresponding to an eye socket region, first, the eye image is binarized; second, small noise points are eliminated from the binarized eye image by dilation and corrosion; then, the image distribution is statistically calculated, and the eye binary image after noise elimination is statistically calculated to obtain the vertices of the Gaussian distribution in the horizontal direction; then, the vertex position is recorded, and the position of the white dots with a certain width next to the vertex position is recorded, which is the iris region.

[0035] In an embodiment, the step of obtaining the eye region may further include the step of optimizing the orbital region and the iris region. For example, according to the orbital region and the iris region obtained by the above method, the geometric dimensions and geometric proportions of the orbital region and the iris region are calculated, and the geometric dimensions of the orbital region and the geometric dimensions of the iris region are adjusted to ensure the accuracy of the position.

[0036] In an embodiment, step S10 may further include an angle correction step. For example, according to the Gaussian distribution vertices calculated in the above steps, the head deflection angle is calculated and corrected. For example, the binocular line can be obtained by connecting the binocular Gaussian distribution vertices of the binocular Gaussian distribution vertices, and the horizontal line can be connected to calculate the deflection angle of the binocular line and the horizontal line, and an affine transformation is performed according to the deflection angle, so as to achieve angle correction of the image.

[0037] In step S20, the left eye sight direction and the right eye sight direction are estimated based on the eye socket area and the corresponding iris area, for example, the left eye sight direction is estimated based on the left eye socket area and the left eye iris area, and the right eye sight direction is estimated based on the right eye socket area and the right eye iris area.

[0038] In an embodiment, step S20 may include: dividing each orbital area in the orbital area into four equal quadrants; calculating the integral value of the corresponding iris area falling in each of the four quadrants; and inferring the left eye sight direction and the right eye sight direction respectively based on the integral value.

[0039] For example, the left eye orbital area or the right eye orbital area is divided into four equal quadrants. The area integrals of the corresponding iris area in each quadrant are calculated, and the integral values ​​of the corresponding quadrants are recorded. The integral of the sum of quadrants 1 and 2 is compared with the integral of the sum of quadrants 3 and 4 to determine the vertical angle of the left eye or the right eye. The integral of the sum of quadrants 1 and 4 is compared with the integral of the sum of quadrants 2 and 3 to determine the horizontal angle of the left eye or the right eye. The vertical angle and the horizontal angle are combined to obtain the direction of the left eye and the right eye.

[0040] In step S30, three-dimensional sight line information is estimated based on the left eye sight line direction, the right eye sight line direction and the facial features.

[0041] Since monocular sight information (e.g., left eye sight direction and right eye sight direction) is easily affected by complex application scenarios and noise, in order to filter out the noise, it is necessary to filter the monocular realization information, and the filtering methods may include but are not limited to: Kalman filtering, particle filter, mathematical statistics analysis, etc.

[0042] In an embodiment, the line of sight direction may be filtered by a monocular line of sight filter, and the three-dimensional line of sight information may be inferred based on the filtered line of sight direction and facial features, wherein the line of sight direction includes a left eye line of sight direction and a right eye line of sight direction.

[0043] Reference Figure 3 , an autoregressive integrated moving average model (MLP-ARIMA) with parameter optimization based on a multi-layer perceptron can be used for noise filtering. The monocular sight line filter MLP-ARIMA may include: an autoregressive integrated moving average model ARIMA, which takes the sight line direction of the current frame as input and outputs the filtered sight line direction of the current frame based on the model parameters; a multi-layer perceptron MLP1, which takes the sight line direction of the current frame and the filtered sight line direction of the current frame as input, and outputs the optimized model parameters to the autoregressive integrated moving average model, wherein the autoregressive integrated moving average model ARIMA filters the sight line direction of the next frame based on the optimized model parameters. In other words, the update of the model parameters is asynchronous.

[0044] In order to obtain a binocular 3D sight line estimation model, a multi-layer perceptron, a convolutional neural network, a long short-term memory network, etc. may be used. Figure 4 , based on the left eye gaze direction, the right eye gaze direction and facial features, a pre-trained multi-layer perceptron network MLP2 is used to infer the three-dimensional gaze information.

[0045] Reference Figure 5The pre-trained multi-layer perceptron network MLP2 includes an input layer IL, a first hidden layer HL1, a second hidden layer HL2, and an output layer OL. The input layer IL is configured to use the left eye sight direction, the right eye sight direction, and the facial features as input; the first hidden layer HL1 is configured to include a first sublayer HL1-1 and a second sublayer HL1-2, wherein the left eye sight direction and the facial features in the input layer IL are fully connected with the first sublayer HL1-1, and the right eye sight direction and the facial features in the input layer IL are fully connected with the second sublayer HL1-2; the second hidden layer HL2 is configured to be fully connected with the first hidden layer HL1; and the output layer OL is configured to be fully connected with the second hidden layer HL2.

[0046] In order to train and test the multi-layer perceptron network, a three-dimensional line of sight estimation data set can be established. Specifically, training samples can be established with predetermined angle separations. In an embodiment, the horizontal direction is 0 degrees as the positive direction, 15 degrees as the separation, 15 degrees, 30 degrees, 45 degrees and 60 degrees in the left and right directions, and the vertical direction is also 0 degrees as the positive direction, 20 degrees as the separation, 20 degrees and 40 degrees in the up and down directions, and the depth is separated by 0.5 meters, and the distances are 1 meter, 1.5 meters, 2 meters, 2.5 meters and 3 meters. In this case, a total of 225 parameter combinations can be obtained. Each parameter combination in the data set has no less than 100 standard samples of mixed different heights, face shapes, and pupil colors. It should be noted that the sample establishment process described here and the number of samples used in the establishment process are only examples.

[0047] Figure 6 2 is an example showing estimated three-dimensional visual line information according to an exemplary embodiment.

[0048] In step S10, based on the image acquired by the monocular camera (for example, Figure 6 A in the figure) to extract the facial region (e.g. Figure 6 ) and the eye region included in the face region (e.g., Figure 6 The eye area includes the orbital area and the iris area.

[0049] For example, based on Figure 6 The facial region shown in part B in FIG. 1 is subjected to HOG feature extraction, and the extracted HOG feature is used as the facial feature. In addition, the head deflection angle can be calculated based on the facial region and corrected. Figure 6 As shown in part B of FIG. 1 , the deflection angle of the head can be calculated to be (-17.2°, -3.2°, 1.0°). The angle is further corrected with reference to the description in step S10.

[0050] In step S20, the left eye sight direction and the right eye sight direction are estimated based on the eye socket area and the corresponding iris area. Figure 6 As shown in part C in the figure, it can be inferred that the left eye's line of sight is in the direction of (-27.8°, -10°) and the right eye's line of sight is in the direction of (-14.1°, -14.7°).

[0051] Then, in step S30, three-dimensional sight line information is estimated based on the left eye sight line direction, the right eye sight line direction and the facial features. Figure 2 The method described in step S30 infers the angle information and depth information of the line of sight, and redundant detailed descriptions will be omitted here.

[0052] Figure 7 is a block diagram illustrating an apparatus for monocular camera-based line of sight detection according to an exemplary embodiment.

[0053] The apparatus 100 for line of sight detection based on a monocular camera includes: a feature extraction unit 110 , a monocular line of sight estimation unit 120 and a three-dimensional line of sight estimation unit 130 .

[0054] According to an exemplary embodiment of the present disclosure, the feature extraction unit 110 is configured to extract features based on an image acquired by a monocular camera to obtain facial features of an eye region and a facial region excluding the eye region, wherein the eye region includes an eye socket region and an iris region. The feature extraction unit 110 may be configured to perform a reference Figure 2 Step S10 of the method is described.

[0055] The monocular sight line estimation unit 120 is configured to estimate the left eye sight line direction and the right eye sight line direction respectively based on the eye socket area and the corresponding iris area. Figure 2 Step S20 of the method is described.

[0056] The 3D sight line estimation unit 130 is configured to estimate 3D sight line information based on the left eye sight line direction, the right eye sight line direction and the facial features. The 3D sight line estimation unit 130 may be configured to perform a reference Figure 2 Step S30 of the method is described.

[0057] Regarding the device in the above embodiment, the specific manner in which each module / unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0058] Figure 8 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0059] Figure 8The electronic device 800 for monocular camera-based line of sight detection shown includes at least one memory 801 and at least one processor 802, wherein the at least one memory 801 stores computer-executable instructions, and when the computer-executable instructions are executed by the at least one processor 802, the at least one processor 802 is prompted to execute the method for monocular camera-based line of sight detection according to an embodiment of the present disclosure.

[0060] Each unit in the electronic device 800 may be configured as software, hardware, firmware, or any combination of the above items to perform a specific function. For example, each unit may correspond to a dedicated integrated circuit, or may correspond to a pure software code, or may correspond to a module that combines software and hardware. In addition, one or more functions implemented by each unit may also be uniformly executed by components in a physical entity device (e.g., a processor, a client, or a server, etc.).

[0061] In addition, refer to Figure 2 The described method of line of sight detection based on a monocular camera can be implemented by a program (or instruction) recorded on a computer-readable storage medium. For example, according to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the method of line of sight detection according to the present disclosure.

[0062] The computer program in the computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. It should be noted that the computer program can also be used to perform additional steps in addition to the above steps or perform more specific processing when performing the above steps. The contents of these additional steps and further processing have been described in reference to Figure 2 It is mentioned in the description of the related method, so it will not be repeated here to avoid repetition.

[0063] It should be noted that the various units in the line of sight detection device according to the exemplary embodiment of the present disclosure can completely rely on the operation of the computer program to realize the corresponding function, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (for example, lib library) to realize the corresponding function.

[0064] on the other hand, Figure 7 The various units shown may also be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segment for performing the corresponding operation may be stored in a computer-readable medium such as a storage medium, so that the processor may perform the corresponding operation by reading and running the corresponding program code or code segment.

[0065] For example, an exemplary embodiment of the present disclosure may also be implemented as a computing device, which includes a storage component and a processor, wherein a set of computer executable instructions is stored in the storage component, and when the set of computer executable instructions is executed by the processor, a method of line of sight detection according to an exemplary embodiment of the present disclosure is performed.

[0066] Specifically, the computing device can be deployed in a server or client, or can be deployed on a node device in a distributed network environment. In addition, the computing device can be a PC, a tablet device, a personal digital assistant, a smart phone, a web application, or other device capable of executing the above instruction set.

[0067] Here, the computing device is not necessarily a single computing device, but may also be any device or circuit collection that can execute the above instructions (or instruction sets) individually or jointly. The computing device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0068] In a computing device, a processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, a processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0069] According to the exemplary embodiment of the present disclosure, some operations described in the method of line of sight detection may be implemented by software, some operations may be implemented by hardware, and furthermore, these operations may be implemented by a combination of software and hardware.

[0070] The processor may execute instructions or codes stored in one of the storage components, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0071] The storage component may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage component may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage component and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the storage component.

[0072] In addition, the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the computing device may be connected to each other via a bus and / or a network.

[0073] The method for line of sight detection based on a monocular camera according to an exemplary embodiment of the present disclosure can be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams can be equally integrated into a single logical device or operated according to non-exact boundaries.

[0074] Therefore, refer to Figure 2 The described method of monocular camera-based line of sight detection may be implemented by a system comprising at least one computing device and at least one storage device storing instructions.

[0075] According to an exemplary embodiment of the present disclosure, at least one computing device is a computing device for executing a method for line of sight detection according to an exemplary embodiment of the present disclosure, and a computer executable instruction set is stored in the storage device. When the computer executable instruction set is executed by the at least one computing device, the execution reference Figure 2 The described gaze detection method.

[0076] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.

Claims

1. A method for line of sight detection based on a monocular camera, characterized in that: The method comprises: Extracting facial features of a facial region and an eye region included in the facial region based on an image acquired by a monocular camera, wherein the eye region includes an eye socket region and an iris region; Inferring the left eye sight direction and the right eye sight direction respectively based on the eye socket area and the corresponding iris area; The left eye sight direction and the right eye sight direction are filtered by a monocular sight filter, and Based on the filtered left eye sight direction, the filtered right eye sight direction and the facial features, a pre-trained multi-layer perceptron network is used to infer three-dimensional sight information, where the three-dimensional sight information is angle information and depth information. Wherein, the monocular sight filter comprises: An autoregressive integrated moving average model is used to take the sight direction of the current frame as input and output the filtered sight direction of the current frame based on the model parameters; A multi-layer perceptron takes the sight line direction of the current frame and the filtered sight line direction of the current frame as input, and outputs the optimized model parameters to the autoregressive integrated moving average model, The autoregressive integrated moving average model performs filtering processing on the line of sight direction of the next frame based on the optimized model parameters.

2. The method according to claim 1, characterized in that The pre-trained multi-layer perceptron network includes: An input layer configured to use the left eye gaze direction, the right eye gaze direction and the facial features as input; A first hidden layer is configured to include a first sublayer and a second sublayer, wherein the left eye sight direction and the facial features in the input layer form a full connection with the first sublayer, and the right eye sight direction and the facial features in the input layer form a full connection with the second sublayer; A second hidden layer is configured to form a full connection with the first hidden layer; and The output layer is configured to form a full connection with the second hidden layer.

3. The method according to claim 1, characterized in that The step of respectively estimating the left eye sight direction and the right eye sight direction based on the eye socket area and the corresponding iris area comprises: dividing each of the orbital regions into four quadrants equal to one another; Calculating the integral value of the corresponding iris area falling in each of the four quadrants; and The left eye sight line direction and the right eye sight line direction are estimated based on the integrated values.

4. A device for line of sight detection based on a monocular camera, characterized in that: The device comprises: A feature extraction unit is configured to extract facial features of a facial region and an eye region included in the facial region based on an image acquired by a monocular camera, wherein the eye region includes an eye socket region and an iris region; A monocular sight line estimation unit, configured to estimate the left eye sight line direction and the right eye sight line direction respectively based on the eye socket area and the corresponding iris area; The three-dimensional sight line estimation unit is configured to filter the left eye sight line direction and the right eye sight line direction through a monocular sight line filter, and estimate three-dimensional sight line information based on the filtered left eye sight line direction, the filtered right eye sight line direction and the facial features using a pre-trained multi-layer perceptron network, the three-dimensional sight line information being angle information and depth information. Wherein, the monocular sight filter comprises: An autoregressive integrated moving average model is used to take the sight direction of the current frame as input and output the filtered sight direction of the current frame based on the model parameters; A multi-layer perceptron takes the sight line direction of the current frame and the filtered sight line direction of the current frame as input, and outputs the optimized model parameters to the autoregressive integrated moving average model, The autoregressive integrated moving average model performs filtering processing on the line of sight direction of the next frame based on the optimized model parameters.

5. The device according to claim 4, characterized in that The pre-trained multi-layer perceptron network includes: An input layer configured to use the left eye gaze direction, the right eye gaze direction and the facial features as input; A first hidden layer is configured to include a first sublayer and a second sublayer, wherein the left eye sight direction and the facial features in the input layer form a full connection with the first sublayer, and the right eye sight direction and the facial features in the input layer form a full connection with the second sublayer; A second hidden layer is configured to form a full connection with the first hidden layer; and The output layer is configured to form a full connection with the second hidden layer.

6. The device according to claim 4, characterized in that The step of respectively estimating the left eye sight direction and the right eye sight direction based on the eye socket area and the corresponding iris area comprises: dividing each of the orbital regions into four quadrants equal to one another; Calculating the integral value of the corresponding iris area falling in each of the four quadrants; and The left eye sight line direction and the right eye sight line direction are estimated based on the integrated values.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for line of sight detection based on a monocular camera as described in any one of claims 1 to 3 is implemented.

8. A computer device, characterized in that: The computer device comprises: processor; A memory storing a computer program, which, when executed by a processor, implements the method for line of sight detection based on a monocular camera as described in any one of claims 1 to 3.

Images