A gaze point detection method based on attention mechanism and deep prior

Abstract

This invention relates to a gaze point detection method based on an attention mechanism and depth prior, comprising: extracting features from a head image in a scene map to obtain gaze features; predicting the gaze direction based on the gaze features; and encoding the gaze direction to generate a gaze field; inputting the scene map, scene depth map, and gaze field into a feature pyramid network with a ResNet network as its backbone to obtain the salient objects in the gaze region. This invention can improve the accuracy of gaze point detection.

Description

Technical Field

[0001] This invention relates to the field of gaze detection technology, and in particular to a gaze detection method based on attention mechanisms and depth priors. Background Technology

[0002] Gaze detection aims to detect the area or object a person is gazing at in an image. Gaze behavior is an important component of human social activity, clearly expressing human interest, feelings, and intentions. We can infer a great deal of potential information from the object of a person's gaze; therefore, gaze detection has become an important tool for assessing behavioral intentions and predicting human behavior in various scenarios. For example, in retail scenarios, we can infer a consumer's level of interest in products based on their eye contact with different items, and analyze what information (food ingredients, price, production date, etc.) most attracts their attention. Gaze detection has been applied to various fields such as human-computer interaction, disability or disease behavior pattern analysis, game design, and virtual reality.

[0003] When people infer where another person is looking, they typically first observe the person's head and eyes to estimate their field of vision, and then predict what they are looking at by inferring salient objects in their viewpoint. Gaze detection algorithms often consist of two modules: a gaze prediction module and a saliency detection module. The gaze prediction module uses information such as local facial images and face position to learn the gaze direction through a neural network, encoding and generating a gaze direction field. The saliency detection module concatenates the gaze direction field with the original image, uses a deep neural network to learn scene features, and outputs the gaze detection result through heatmap regression.

[0004] Existing gaze prediction modules input head images into a ResNet-50 neural network for feature extraction. A linear layer outputs a 1×512 feature vector, and a linear layer outputs a 1×256 feature vector for the head position coordinates (x, y). These two feature vectors are concatenated and transformed into a 1×2 gaze direction vector by a linear layer, which is then encoded to generate a multi-scale field of view (FOV). Since the head position features are extracted using a linear transformation on a two-dimensional (x, y) coordinate, they contain limited information and lack global context. In facial feature extraction, more attention needs to be paid to features in the eye region, and the ResNet-50 neural network cannot reflect this attentional allocation.

[0005] Existing saliency detection modules concatenate the multi-scale gaze field with the original image along the channel direction and input it into a neural network (such as a Feature Pyramid Network (FPN)) for heatmap regression. The point with the maximum value in the heatmap is considered the final gaze point. This approach lacks an understanding of scene depth and struggles to accurately capture spatial information. For example, along the subject's gaze direction, there may be two or more candidate objects at different depths, making it impossible to determine which candidate object the gaze point ultimately falls on using this method. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a gaze point detection method based on attention mechanism and depth prior, which can improve the accuracy of gaze point detection.

[0007] The technical solution adopted by this invention to solve its technical problem is: to provide a gaze point detection method based on attention mechanism and depth prior, comprising:

[0008] The gaze features are extracted from the head image in the scene image. The gaze direction is predicted based on the gaze features, and the gaze field is generated by encoding based on the gaze direction.

[0009] The scene map, scene depth map, and gaze field are input into a feature pyramid network with ResNet as the backbone to obtain the salient objects in the gaze region.

[0010] The process of extracting gaze features from the head image in the scene graph, predicting the gaze direction based on the gaze features, and encoding the gaze direction to generate a gaze field includes:

[0011] Extract the head image from the scene image;

[0012] The head image is input into a ResNet-50 neural network for feature extraction to obtain the first feature;

[0013] The head position in the scene image is displayed with white pixels, and the rest of the position is displayed with black pixels to obtain a binary image. Three consecutive max pooling operations are performed on the binary image, and then features are extracted through convolution kernels to obtain the second feature.

[0014] The head position coordinates are processed through a linear layer to obtain the third feature;

[0015] The first feature, the second feature, and the third feature are concatenated to obtain the gaze feature;

[0016] The gaze direction is obtained by linearly transforming the gaze characteristics, and then the gaze direction is encoded to obtain a multi-scale gaze field.

[0017] The ResNet-50 neural network incorporates an attention strategy to achieve adaptive learning of the importance of eye features.

[0018] The gaze field is represented by a cone-shaped region with the head position as the vertex, and the probability that point P is the gaze point is expressed as: Where G represents the line connecting point P and the head position; Let λ represent the gaze direction and λ be the angle of the gaze field. When the angle between the line connecting point P and the head position and the gaze direction is greater than 90°, the probability of point P becoming the gaze point is 0.

[0019] The scene depth map is obtained by inputting the scene image into a depth estimation network.

[0020] In the feature pyramid network with ResNet as the backbone, a permutation attention module is added to the ResNet network.

[0021] The displacement attention module includes:

[0022] The grouping section is used to group the input features, dividing the input features into G sub-features along the channel direction, and each sub-feature group is further divided into a first sub-feature and a second sub-feature.

[0023] The channel attention branch is used to perform global average pooling on the first sub-feature to obtain the global information of the first sub-feature, generate the channel feature map, and output it through the activation function to obtain the first output feature;

[0024] The spatial attention branch is used to perform group normalization on the second sub-feature to obtain spatial statistics, and then output them through an activation function to obtain the second output feature;

[0025] The concatenation section is used to concatenate the first output feature and the second output feature to obtain the output feature.

[0026] The activation function used in both the channel attention branch and the spatial attention branch is the Sigmoid activation function.

[0027] Beneficial effects

[0028] Due to the adoption of the above technical solutions, this invention has the following advantages and positive effects compared with the prior art: This invention encodes the head position using a binary image, performs dimensionality reduction through three consecutive max pooling operations, extracts features using convolutional kernels, and performs linear transformation after fusing multiple features, thus performing deeper feature extraction of head position information; This invention introduces an attention strategy when extracting features from the head image, realizing adaptive learning of the importance of facial features, strengthening feature representation capabilities, and improving the accuracy of gaze direction prediction; This invention concatenates the scene depth map, the original scene image, and the gaze field in the channel direction and inputs them into a deep learning network, which can extract effective scene depth information. Furthermore, by adding a permutation attention module to the deep learning network, it can learn the importance (weight) of each set of features in the spatial and channel dimensions, assigning greater weight to feature expressions with richer information while suppressing useless feature expressions. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the attention mechanism and depth prior gaze point detection method according to the embodiments of the present invention;

[0030] Figure 2 This is a general framework diagram of the model in the embodiments of the present invention;

[0031] Figure 3 This is a block diagram of the line-of-sight prediction module in an embodiment of the present invention;

[0032] Figure 4 This is a block diagram of the saliency detection module in an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of the displacement attention module in an embodiment of the present invention. Detailed Implementation

[0034] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0035] The embodiments of the present invention relate to a gaze point detection method based on attention mechanisms and depth priors, the principle of which is as follows: Figure 1 As shown, the gaze prediction module requires the coordinates of the eye center and a binary map of the head's position within the entire scene map. The saliency detection module utilizes existing depth estimation methods such as MiDaS to obtain prior depth information from the original scene map, which is then input into the gaze field detection network to obtain the final gaze point heatmap.

[0036] The present invention specifically includes the following steps:

[0037] Step 1: Extract features from the head image in the scene image to obtain gaze features, predict the gaze direction based on the gaze features, and encode the gaze direction to generate a gaze field.

[0038] Step 2: Input the scene map, scene depth map and gaze field into the feature pyramid network with ResNet as the backbone to obtain the salient objects in the gaze region.

[0039] like Figure 2 As shown, this embodiment employs a two-stage gaze point detection network architecture, including a head branch and a scene branch. The head branch (i.e., the gaze direction prediction module) aims to predict the gaze area or direction of the human eye using head image features and the head's position information within the entire image; while the scene branch (i.e., the saliency detection module) uses the features of the entire scene image and the already predicted gaze area to locate salient objects in the gaze area, i.e., the gaze point of the human eye. Step 1 can be implemented through the gaze direction prediction module, and step 2 can be implemented through the saliency detection module.

[0040] The gaze direction prediction module in this embodiment utilizes a convolutional neural network to extract facial texture features, fuses eye position information and head geometric position information in the scene, and maps them to the gaze direction to predict the gaze direction. The gaze direction is then encoded to generate a gaze field. Specifically, this includes: extracting a head image from the scene image; inputting the head image into a ResNet-50 neural network for feature extraction to obtain a first feature; displaying the head position in the scene image with white pixels and the remaining positions with black pixels to obtain a binary image; performing three consecutive max-pooling operations on the binary image, and then extracting features through convolutional kernels to obtain a second feature; processing the head position coordinates through a linear layer to obtain a third feature; concatenating the first, second, and third features to obtain the gaze feature; performing a linear transformation on the gaze feature to obtain the gaze direction, and then encoding the gaze direction to obtain a multi-scale gaze field.

[0041] like Figure 3 As shown, the cropped head image is scaled to a ratio of 224×224 and input into a ResNet-50 neural network for feature extraction. The output is a 1×1024 feature vector, which is then reduced to a 1×512 feature vector F through a linear layer. f (i.e., the first characteristic).

[0042] The head position is encoded using a binary image, the size of which depends on the scene. Figure 1The head region is represented by white pixels, and the rest of the image is filled with black pixels. Dimensionality reduction and flattening are performed using three consecutive max-pooling operations. Then, features are extracted using 3×3 convolutional kernels, outputting a 1×256 feature vector F. h (i.e., the second characteristic). Head position (h) x ,h y The coordinates are passed through a linear layer to output a 1×256 feature vector F. e (i.e., the third characteristic).

[0043] Since humans primarily focus their gaze on the eyes, it's known that the eyes contain richer and more important information. Therefore, this implementation introduces an attention strategy to pay more attention to eye texture features. Specifically, an attention module is applied to the ResNet-50 network to achieve adaptive learning of the importance of eye features in facial features. Then, F... f F h F e These three features are concatenated into a 1×1024 feature vector, which is then linearly transformed to output a 1×2 gaze direction vector. Finally, through encoding, a multi-scale gaze field (FOV) is output.

[0044] The gaze field of the target person is determined by a field based on head position H(h). x ,h y The cone region represented by vertices P(p) is given by a point P(p). x ,p y If we disregard scene content, the probability that point P is a gaze point should be proportional to the angle θ between the line connecting points H and P and the predicted gaze direction. Therefore, a cosine function is used to describe the mapping from angle to probability value. (Predicted gaze direction) The line connecting points P and H is G = (p x -h x ,p y -h y The probability that point P is the gaze point is .

[0045]

[0046] When the angle between the two lines is greater than 90°, the probability of it becoming the gaze point is 0. The angle of the gaze field is controlled by λ, where Sim(P,λ) = [Sim(P)]. λ The larger λ is, the smaller the gaze field angle.

[0047] like Figure 4 As shown, the saliency detection module in this embodiment first obtains the scene depth map I by passing the scene image I through the depth estimation network MiDaS. d Then, scene map I and scene depth map I. dThe multi-scale gaze field is fed into the Feature Pyramid Network (FPN) with ResNet as the backbone. A permutation attention module is added to the ResNet network, and a channel-space attention module is introduced after the feature layers at different scales to correct the features and improve the feature representation capability.

[0048] like Figure 5 As shown, the permutation attention module first groups the input features, grouping the input features X∈R... C ×H×W The features are divided into G groups along the channel direction, represented by the following formula:

[0049] X = [X1, X2, ..., X G ], X n ∈R C / G×H×W

[0050] Among them, sub-feature X n They were then divided into two groups, X n1 Input channel attention branch, X n2 Input space attention branch.

[0051] The channel attention branch uses global average pooling (GAP) to obtain X. n1 Global information is used to generate channel feature maps S. n1 The formula is as follows:

[0052]

[0053] X n1 The output X′ is obtained through the channel attention branch. n1 This is achieved using the Sigmoid activation function, with the specific formula as follows:

[0054] X′ n1 =σ(W n1 ·S n1 +b n1 )·X n1

[0055] Among them, W n1 ∈R C / 2G×1×1 b n1 ∈R C / 2G×1×1 Used to scale and move channel feature map S n1 .

[0056] Spatial attention branch first on X n2 Spatial statistics S are obtained using group normalization (GroupNorm, GN). n2 The formula is as follows:

[0057] S n2 =GN(X) n2)

[0058] Then, X′ is output through the Sigmoid activation function. n2 The specific formula is as follows:

[0059] X′ n2 =σ(W n2 ·S n2 +b n2 )·X n2

[0060] Among them, W n2 ∈R C / 2G×1×1 b n2 ∈R C / 2G×1×1 The spatial statistics S used for scaling and movement n2 .

[0061] Finally, output X′ from both branches n1 、X′ n2 By splicing, we obtain X′ n =[X′ n1 ,X′ n2 ]∈R C / G×H×W X′ n =[X′ n1 ,X′ n2 ]∈R C / G×H×W After all sub-features have been trained, cross-group aggregation is performed to obtain the final output with the same size as the input feature X.

[0062] It is easy to see that this invention encodes the head position using a binary image, performs dimensionality reduction through three consecutive max pooling operations, extracts features using convolutional kernels, and then performs a linear transformation after fusing multiple features, thus performing a deeper feature extraction of head position information. This invention introduces an attention strategy when extracting features from the head image, achieving adaptive learning of the importance of facial features, strengthening feature representation capabilities, and improving the accuracy of gaze direction prediction. This invention concatenates the scene depth map, the original scene image, and the gaze field along the channel direction and inputs them into a deep learning network, which can extract effective scene depth information. Furthermore, by adding a permutation attention module to the deep learning network, it can learn the importance (weight) of each set of features in the spatial and channel dimensions, assigning greater weight to feature expressions with richer information, while suppressing useless feature expressions.

Claims

1. A gaze point detection method based on attention mechanism and depth prior, characterized in that, include: Feature extraction is performed on the head image in the scene to obtain gaze features. The gaze direction is predicted based on these features, and a gaze field is generated by encoding the gaze direction, including: Extract the head image from the scene image; The head image is input into a ResNet-50 neural network for feature extraction to obtain the first feature; The head position in the scene image is displayed with white pixels, and the rest of the position is displayed with black pixels to obtain a binary image. Three consecutive max pooling operations are performed on the binary image, and then features are extracted through convolution kernels to obtain the second feature. The head position coordinates are processed through a linear layer to obtain the third feature; The first feature, the second feature, and the third feature are concatenated to obtain the gaze feature; The gaze features are linearly transformed to obtain the gaze direction, and then the gaze direction is encoded to obtain a multi-scale gaze field. The scene map, scene depth map, and gaze field are input into a feature pyramid network with a ResNet network as the backbone to obtain the salient objects in the gaze region. The feature pyramid network with the ResNet network as the backbone incorporates a permutation attention module, which includes: The grouping section is used to group the input features, dividing the input features into G sub-features along the channel direction, and each sub-feature group is further divided into a first sub-feature and a second sub-feature. The channel attention branch is used to perform global average pooling on the first sub-feature to obtain the global information of the first sub-feature, generate the channel feature map, and output it through the activation function to obtain the first output feature; The spatial attention branch is used to perform group normalization on the second sub-feature to obtain spatial statistics, and then output them through an activation function to obtain the second output feature; The concatenation section is used to concatenate the first output feature and the second output feature to obtain the output feature.

2. The gaze point detection method based on attention mechanism and depth prior according to claim 1, characterized in that, The ResNet-50 neural network incorporates an attention strategy to achieve adaptive learning of the importance of eye features.

3. The gaze point detection method based on attention mechanism and depth prior according to claim 1, characterized in that, The gaze field is represented by a cone-shaped region with the head position as the vertex, and the probability that point P is the gaze point is expressed as: Where G represents the line connecting point P and the head position; Let λ represent the gaze direction and λ be the angle of the gaze field. When the angle between the line connecting point P and the head position and the gaze direction is greater than 90°, the probability of point P becoming the gaze point is 0.

4. The gaze point detection method based on attention mechanism and depth prior according to claim 1, characterized in that, The scene depth map is obtained by inputting the scene image into a depth estimation network.

5. The gaze point detection method based on attention mechanism and depth prior according to claim 1, characterized in that, The activation function used in both the channel attention branch and the spatial attention branch is the Sigmoid activation function.

Images