Individual specific appearance-based fixation point estimation method
By extracting and integrating facial and eye features, using convolutional neural network and Transformer model, combining multi-head attention mechanism and adaptive group normalization module, the problem of insufficient individual-specific appearance information is solved and the focus estimation is improved accuracy and model availability.
Patent Information
- Application Number
- CN202411683215.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-01-10
AI Technical Summary
The existing gaze point estimation methods lack individual-specific appearance information, and their accuracy is low, affecting the reliability of gaze behavior analysis.
By extracting facial and eye position information features, combining convolutional neural networks and Transformer models, using multi-head attention mechanisms and adaptive group normalization modules, the facial and eye features are integrated to estimate the coordinates of the gaze point.
Effectively extract individual-specific appearance information, improve the accuracy of gaze point estimation, reduce the amount of model parameters, and enhance the usability of the application.
Smart Images
Figure CN119296162A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision research, and in particular to a method for estimating a gaze point based on individual-specific appearance, which can effectively extract individual-specific appearance information and estimate the individual gaze point based on the information. Background Art
[0002] Gaze behavior can reflect a person's psychological or physiological state, and its results can provide a basis for the required decision in many scenarios. Gaze estimation, as a prerequisite for gaze behavior analysis, is widely used in many fields, such as psychological research, medical diagnosis, human-computer interaction, etc.
[0003] Significant differences in appearance between different individuals reduce the accuracy of individual gaze point estimation. Existing methods use individual calibration tasks to improve the performance of gaze point estimation models. Although this method can effectively improve the accuracy of individual gaze estimation, the additional calibration task brings inconvenience to practical applications. Some methods do not use individual calibration during the training process of the model. However, without individual calibration, due to the lack of individual-specific appearance information, previous gaze point estimation methods have low accuracy, affecting the reliability of gaze behavior analysis.
[0004] In order to make the model fully extract individual-specific appearance information, more attention should be paid to the appearance parts that are related to gaze and have significant individual differences during feature extraction. Some individual-specific appearances will cause unnecessary interference to gaze estimation, and the influence of these factors should be eliminated during the model extraction process. Summary of the invention
[0005] The purpose of the present invention is to address the problem that the prior art lacks individual-specific appearance information when estimating gaze points, and to provide a gaze point estimation method based on individual-specific appearance that can effectively extract individual-specific appearance information and estimate individual gaze points based on this information.
[0006] The present invention comprises the following steps:
[0007] 1) Extract the face and eye position information features using all vertices of the face and eye bounding boxes;
[0008] 2) Taking the facial image as input, calculate the facial features containing the global information of the face;
[0009] 3) Taking the eye image as input, extract monocular features of the left and right eye images respectively;
[0010] 4) Integrate the monocular features of the left and right eyes extracted in step 3) and perform adaptive feature fusion to obtain eye features;
[0011] 5) Integrate the facial features calculated in step 2), the eye features obtained in step 4), and the position features extracted in step 1) to calculate and obtain the final gaze point coordinate estimate.
[0012] In step 1), the face and eye bounding boxes refer to rectangular frames used to mark the face and left and right eye parts after face and eye detection is performed on the image, with a total of 3 bounding boxes.
[0013] In step 1), all vertices of the face and eye bounding boxes refer to 4 vertices of all rectangular frames, totaling 12 vertices.
[0014] In step 2), the facial image is used as input to calculate facial features containing global face information. The specific steps may be:
[0015] Step 2.1: Input the facial image into the convolutional neural network (CNN) to extract the local features of the facial image.
[0016] Step 2.2: Input the local features extracted by CNN into the Transformer model, and use the multi-head attention mechanism (MSA) and multi-layer perceptron (MLP) to extract the global correlation features of the face. Layer normalization (LN) should be performed after each layer of Transformer block.
[0017] Step 2.3: Pass the global facial correlation features output by Transformer through the fully connected layer to obtain the facial feature vector.
[0018] In step 3), the eye image is used as input, and monocular features are extracted from the left and right eye images respectively. The specific steps are as follows:
[0019] Step 3.1: The left and right eye images are respectively passed through a monocular feature extraction model (containing five convolutional layers), where the last three layers are respectively preceded by polarized self-attention modules.
[0020] Step 3.2: The polarized self-attention module is calculated as described, including channel-based self-attention and space-based self-attention.
[0021] Step 3.3: Use the Adaptive Group Normalization (AGN) module to calibrate the monocular features, where the position feature vector can be extracted from step 1 and the face feature vector can be extracted from step 2).
[0022] In step 3), in the process of extracting monocular features from the left and right eye images respectively, the features calculated by the third and fifth convolutional layers are calibrated using adaptive group normalization modules for subsequent eye feature stacking.
[0023] In step 4), the left and right eye monocular features extracted in step 3) are integrated, and the features calculated by the third convolution layer in the monocular feature extraction model refer to the features extracted by the monocular image through the first three convolution layers (including the polarization self-attention module), and the features calculated by the fifth convolution layer refer to the features extracted by the monocular image through the first five convolution layers (including the polarization self-attention module).
[0024] In step 5), the gaze point coordinate estimation refers to estimating the two-dimensional gaze point coordinates of an individual in a prediction space, where the prediction space takes the camera center as the origin and the horizontal and vertical coordinates are constructed by the physical distance to the camera center.
[0025] The present invention proposes a method for estimating gaze points based on individual-specific appearance, which effectively extracts individual-specific appearance information and estimates individual gaze points. Compared with the prior art, the present invention has the following outstanding technical effects:
[0026] 1. This paper proposes a new gaze point estimation model, which combines the attention mechanism and adaptive group normalization to effectively extract individual-specific appearance information related to gaze.
[0027] 2. The present invention proposes a new monocular feature extraction model, which can better extract relevant information such as the gaze direction and shape appearance of a monocular.
[0028] 3. The present invention ensures model performance while maintaining the model parameter quantity at a low level, reducing equipment load and enhancing model availability.
[0029] 4. The present invention realizes the effective extraction of individual-specific appearance information through the steps of position feature extraction, facial feature extraction, eye feature extraction and fusion, provides a better performance method for gaze point estimation, and is of great significance to the application of computer vision. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 This is a diagram of the method architecture of the present invention;
[0031] Figure 2 This is a schematic diagram of the monocular feature extraction model structure;
[0032] Figure 3 Schematic diagram of the polarized self-attention module structure;
[0033] Figure 4 Schematic diagram of the adaptive group normalization module structure. DETAILED DESCRIPTION
[0034] The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0035] The present invention can use public face gaze point datasets, such as MPIIGaze, EYEDIAP, etc. These datasets contain images of multiple individuals and corresponding gaze point coordinates. Face detection algorithms (such as MTCNN, Haar features, etc.) can be used to detect faces and obtain the bounding boxes of the face and eyes.
[0036] like Figure 1 , the embodiment of the present invention includes the following steps:
[0037] 1) Extract face and eye position information features using all vertices of the face and eye bounding boxes:
[0038] Step 1: Take all vertices of the face and eye bounding boxes as input and pass through four fully connected layers to calculate the feature vector representing the face and eye position information.
[0039] 2) Take the facial image as input and calculate the facial features containing the global information of the face:
[0040] Step 1: Take the preprocessed face image as input and extract the local features of the face image through CNN. Convolutional neural network CNN is a common model used for image recognition and classification in deep learning, which can automatically extract useful features from images.
[0041] Step 2: Take the local facial features extracted by CNN as input and use transformer to extract global facial correlation features.
[0042] The calculation method of a single layer in transformer is as follows:
[0043] f face '=MSA(LN(F face ))+F face
[0044] f face =MLP(LN(f face '))+f face '
[0045] Among them, LN is the layer normalization operation, MSA is the multi-head attention mechanism, and MLP is the multi-layer perceptron.
[0046] Step 3: Take the global facial correlation features as input and pass them through the fully connected layer (FC) to calculate the facial feature vector.
[0047] 3) Take the eye image as input and perform monocular feature extraction on the left and right eye images respectively:
[0048] Step 1: Let the left and right eye images pass through the monocular feature extraction model (see Figure 2 ), calculate the features corresponding to each eye.
[0049] The monocular feature extraction model contains five convolutional layers, and a polarized self-attention module (PSA_pBlock) is stacked in front of the last three layers.
[0050] The polarized self-attention module consists of two parts: channel-only self-attention and spatial-only self-attention. (See Figure 3 )The calculation method of the polarized self-attention module is as follows:
[0051] PSA_p(f eye )=A ch (f eye ) ch f eye +A sp (f eye ) sp f eye
[0052] in,· ch is a channel-based multiplication operation, sp is a space-based multiplication operation, and + is an element-based addition operation. ch (·) is calculated as follows:
[0053] A ch (f eye )=F SG [W z (σ1(W v (f eye ))×F SM (σ2(W q (f eye )))))]
[0054] Among them, W q , W v and W z are 1×1 convolutional layers, σ1 and σ2 are two tensor reshaping operations, and F SM (·) is a SoftMax operation, and × is a matrix dot product operation.
[0055] A sp (·) is calculated as follows:
[0056] A sp (f eye )=F SG [σ3(F SM (σ1(F GP (W q (f eye ))))×σ2(W v (f eye )))]
[0057] Among them, W q and W v are the standard 1×1 convolutional layers, σ1, σ2, and σ3 are three tensor reshaping operations, and F SM (·) is a SoftMax operation, F GP (·) is a global pooling operation, and × is a matrix dot product operation.
[0058] Step 2: Use the adaptive group normalization module to recalibrate the monocular features. The structural diagram of the adaptive group normalization module is shown in Figure 4 .
[0059] The adaptive group normalization module is calculated as follows:
[0060] [W shift ,W scale ]=LeakyReLU(L(f Rects ,f face ))
[0061] f eye_out =W scale *GN(f eye_in )+W shift
[0062] Among them, f Rects is the position feature vector, f face is the facial feature vector. L is the fully connected layer. W shift is the offset parameter, W scale is the scaling parameter. GN is the group normalization operation.
[0063] 4) Integrate the monocular features of the left and right eyes extracted in step 3) and perform adaptive feature fusion to obtain eye features:
[0064] Step 1: For each eye, select the features calculated by the third and fifth convolutional layers in the monocular feature extraction model and stack them, and stack the features of the left and right eyes again.
[0065] Step 2: Take the stacked left and right eye features as input and perform adaptive eye feature fusion through the polarized self-attention module.
[0066] Step 3: Normalize the fused eye features using the adaptive group normalization module.
[0067] Step 4: Take the standardized eye features as input and extract features through a neural convolution layer.
[0068] Step 5: Use the polarized self-attention module to integrate the extracted features with the eye features again.
[0069] Step 6: Take the re-fused eye features as input and pass them through a fully connected layer to calculate the eye feature vector.
[0070] 5) Integrate the facial features calculated in step 2), the eye features obtained in step 4), and the position features extracted in step 1) to calculate the final gaze point coordinate estimate:
[0071] Step 1: Concatenate the facial feature vector, eye feature vector and position feature vector.
[0072] Step 2: Take the concatenated feature vector as input and pass it through two fully connected layers (FC) to calculate the gaze point coordinate estimate.
[0073] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A method for estimating gaze points based on individual-specific appearance, characterized in that The following steps are involved: 1) Extract the face and eye position information features using all vertices of the face and eye bounding boxes; 2) Taking the facial image as input, calculate the facial features containing the global information of the face; 3) Taking the eye image as input, extract monocular features of the left and right eye images respectively; 4) Integrate the monocular features of the left and right eyes extracted in step 3) and perform adaptive feature fusion to obtain eye features; 5) Integrate the facial features calculated in step 2), the eye features obtained in step 4), and the facial and eye position features extracted in step 1) to calculate and obtain the final gaze point coordinate estimate.
2. A method for estimating gaze points based on individual-specific appearance as claimed in claim 1, characterized in that In step 1), the face and eye bounding boxes refer to rectangular frames used to mark the face and left and right eye parts after face and eye detection is performed on the image, with a total of 3 bounding boxes.
3. A method for estimating gaze points based on individual-specific appearance as claimed in claim 1, characterized in that In step 1), all vertices of the face and eye bounding boxes refer to 4 vertices of all rectangular frames, totaling 12 vertices.
4. A method for estimating gaze points based on individual-specific appearance as claimed in claim 1, characterized in that In step 2), the facial image is taken as input to calculate the facial features containing the global information of the face. The specific steps may be: Step 2.1: Input the facial image into the convolutional neural network (CNN) to extract local features of the facial image; Step 2.2: Input the local features extracted by CNN into the Transformer model, and use the multi-head attention mechanism MSA and the multi-layer perceptron MLP to extract the global correlation features of the face; each layer of the Transformer block should be followed by a layer normalization LN operation; Step 2.3: Pass the global facial correlation features output by Transformer through the fully connected layer to obtain the facial feature vector.
5. A method for estimating gaze points based on individual-specific appearance as claimed in claim 1, characterized in that In step 3), the eye image is used as input, and monocular features are extracted from the left and right eye images respectively. The specific steps are as follows: Step 3.1: The left and right eye images are respectively passed through a monocular feature extraction model, which contains five convolutional layers, of which the last three layers are respectively preceded by polarized self-attention modules; Step 3.2: The polarized self-attention module is calculated as described, including channel-based self-attention and space-based self-attention; Step 3.3: Use the adaptive group normalization module AGN to calibrate the monocular features, where the position feature vector can be extracted from step 1 and the face feature vector can be extracted from step 2).
6. A method for estimating gaze points based on individual-specific appearance as claimed in claim 1, characterized in that In step 3), in the process of extracting monocular features from the left and right eye images respectively, the features calculated by the third and fifth convolutional layers are calibrated using adaptive group normalization modules for subsequent eye feature stacking.
7. A method for estimating gaze points based on individual-specific appearance as claimed in claim 1, characterized in that In step 4), the left and right eye monocular features extracted in step 3) are integrated. The features calculated by the third convolutional layer in the monocular feature extraction model refer to the features extracted by the monocular image through the first three convolutional layers, and the features calculated by the fifth convolutional layer refer to the features extracted by the monocular image through the first five convolutional layers.
8. A method for estimating gaze points based on individual-specific appearance as claimed in claim 1, characterized in that In step 5), the gaze point coordinate estimation refers to estimating the two-dimensional gaze point coordinates of an individual in a prediction space, where the prediction space takes the camera center as the origin and the horizontal and vertical coordinates are constructed by the physical distance to the camera center.