System and method for realizing sight line estimation and attention analysis based on recursive convolutional neural network

Abstract

The invention discloses a system for realizing sight line estimation and attention analysis based on a recursive convolutional neural network. The system comprises a sight line feature extraction module, a sight line regression module, a sight line drop point mapping module and an attention visualization and analysis module. The invention also relates to a corresponding method which comprises the following steps: simultaneously extracting the apparent characteristics of the eyes and the posture characteristics of the head during implementation, and carrying out spatial domain characteristic fusion; for continuous multi-frame sight line features, performing joint coding on time sequence features of a gazing behavior through a Bi-LSTM network layer, completing time domain feature fusion, and further performing regression to obtain a sight line vector of a middle frame; the invention further provides a sight line drop point resolving method based on the monocular camera, the sight line drop point coordinates can be obtained in real time, and the method is not limited by scenes; data support is provided by a bottom layer module, and abundant real-time sight tracking visualization and relevant sight parameter visualization forms are provided in an attention visualization and analysis module. According to the technical scheme, the accuracy and stability requirements of use scenes can be met, and the application scenes are wide.

Description

technical field [0001] The invention belongs to the technical field of computer vision, and specifically refers to a system and a method for realizing line of sight estimation and attention analysis based on a recursive convolutional neural network. Background technique [0002] The eye is the main organ for human beings to obtain information from the outside world. The direction of human sight contains a wealth of potential information. By detecting the direction of sight, we can identify the position where people's attention is concentrated, and then infer the area of ​​interest in the current space and Things, this information has great value in business evaluation and disease diagnosis. In addition, the direction of sight is also important information for the machine to understand human intentions. The accurate estimation of the direction of sight can provide strong technical support for human-computer interaction and bring users a non-contact human-computer interaction ...

AI Technical Summary

Problems solved by technology

This type of method is, for example, Chinese patent authorization number CN 106599994, application publication date 2017.04.26, patent name: a method of line of sight estimation based on deep regression network, using a single eye image as the input of a 5-layer deep regression neural network to estimate the line of sight direction , the problem with this method is that only eye features are considered, and it cannot handle the unconstrained line of sight estimation task of the head well.

Chinese patent publication number CN 110795982, publication date 2019.01.04, patent name: A method for estimating apparent line of sight based on human body posture analysis, in which the CNN network is used to extract the features of human head posture and body posture, and finally return to the line of sight Direction, because the eye information is not introduced, the details of eye movement are lost, so it will inevitably cause a large line of sight estimation error

Chinese patent publication number CN 111680546 A, application publication date 2020.09.18, patent name: attention detection method, device, electronic equipment and storage medium, the invention patent uses two branches of CNN network to extract head posture features and eye features respectively Features and fusion, but it finally establishes a binary classification model for attention detection, that is, it can only obtain the binary classification result of whether the subject is gazing at a specific target area; this invention converts the line of sight estimation problem into a binary classification solution, although Compared with the regression model, the calculation is simplified, but because it is impossible to estimate the position of the line of sight in real time, the usage scenarios of this technical solution are greatly limited, resulting in the inability to complete the complete attention analysis task; Instead, it directly establishes the binary classification relationship between the appearance of the face and the specific target area, which makes the model highly dependent on the camera arrangement and the position selection of the gaze target. Therefore, when facing complex and changeable target areas to be detected in real scenes, , which brings difficulty to the training and deployment of the model, and is less practical

In addition, most of the existing methods are based on the convolutional neural network to extract the features of a single frame of the face or eyes, and then use the regression model to find the direction of the line of sight. Due to the dynamics and variability of the line of sight estimation, this method is less robust. Poor, there are cases where the results are unstable during use

And in the line of sight estimation task, the ultimate goal is to determine the location of the user's line of sight in the current scene for subsequent analysis. However, the current solution for 3D line of sight estimation only gives the line of sight direction represented by a two-dimensional angle vector , has not established its mapping relationship with the final sight point, so it cannot constitute an end-to-end system

Images