End-to-end eye contact identification method and system based on full face information guidance

Abstract

The invention discloses an end-to-end eye contact recognition method and system based on full face information guidance, and the method comprises the following steps: obtaining full face image data, carrying out the eye state type marking, and carrying out the data set division; constructing an eye contact recognition model of double-flow feature cross fusion based on full-face key point guidance; training the eye contact recognition model to obtain a trained eye contact recognition model; and performing classification identification based on the trained eye contact identification model to obtain probability vectors of eye contact identification, and selecting an eye contact state category corresponding to a maximum probability value in the probability vectors as an eye contact state identification result. According to the invention, eye contact recognition is carried out based on full face information guidance, face global information is provided by using face key points, and through cross fusion and guidance between the image feature flow and the face key point information flow, the recognition accuracy under the condition that the face is shielded or is in an extreme posture environment is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an end-to-end eye contact recognition method and system guided by full-face information. Background Art

[0002] In order to be more adaptable to practical applications, many deep learning-based methods have been explored for eye contact detection in natural and unconstrained environments. In 2017, Chong et al. [CHONG E, CHANDA K, YE Z, et al. Detecting gaze towards eyes in natural social interactions and its use in child assessment [J]. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2017, 1(3): 1-20] used a head-mounted camera to analyze the facial information captured by the camera to infer the head posture, thereby automatically estimating the head posture and predicting eye contact; Yu et al. [

[11] Mitsuzumi Y, Nakazawa A. Eye contact detection algorithms using deep learning and generative adversarial networks [C] / / 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2018: 3927-3931.] used a small number of detected eye images to perform eye contact detection using a semi-supervised learning algorithm; Müller et al. [

[12] Müller P, Huang MX, Zhang X, et al. Robust eye contact Detection in natural multi-person interactions using gaze and speaking behavior[C] / / Proceedings of the2018ACMSymposium on Eye Tracking Research&Applications.2018:1-10.]Eye contact detection using the correlation between gaze and speaking.

[0003] However, most of the above methods are non-end-to-end training and learning methods, relying on pre-calculated head poses and facial key points, or pre-cropping the eyes before training. Therefore, they cannot better perform eye contact detection when key point detection or eye area positioning fails due to factors such as face occlusion or extreme poses. Summary of the invention

[0004] In order to overcome the defects and shortcomings of the prior art, the present invention provides an end-to-end eye contact recognition method and system based on full-face information guidance. The present invention performs eye contact recognition based on full-face information guidance, which can provide more global information than the method of predicting the eye parts in advance and then recognizing them, effectively alleviate the similarity between classes, use facial key points to provide global facial information, and improve the recognition accuracy in environments where the face is blocked or in extreme postures through the cross-fusion and guidance between the image feature stream and the facial key point information stream.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides an end-to-end eye contact recognition method based on full-face information guidance, comprising the following steps:

[0007] Obtain full-face image data, mark eye state categories, and divide the data set;

[0008] Construct an eye contact recognition model based on dual-stream feature cross-fusion guided by full-face key points, including a face image feature stream extraction module, a face key point information stream extraction module, a feature cross-fusion module, and a classification module;

[0009] The face image feature stream extraction module is used to extract face image features, the face key point information stream extraction module is used to extract face key point features, the feature cross fusion module is used to perform a cross fusion operation on face image features and face key point features based on cross fusion multi-head attention, and the classification module is used to map the output features of the feature cross fusion module to the dimension of the target category to obtain eye contact recognition results;

[0010] Training the eye contact recognition model to obtain a trained eye contact recognition model;

[0011] Classification and recognition are performed based on the trained eye contact recognition model to obtain a probability vector for eye contact recognition, and the eye contact state category corresponding to the maximum probability value in the probability vector is selected as the recognition result of the eye contact state.

[0012] As a preferred technical solution, the facial image feature stream extraction module uses the first four convolution blocks of ResNet50 as the backbone network, resizes the output image of the backbone network, and obtains facial image features.

[0013] As a preferred technical solution, the face key point information flow extraction module uses the facial feature point detector Mobile-FaceNet to obtain the face key point feature X L ∈R P×D , where P represents the number of tiles and D represents the feature dimension.

[0014] As a preferred technical solution, the feature cross-fusion module is used to perform a cross-fusion operation on the face image features and the face key point features based on the cross-fusion multi-head attention, specifically including:

[0015] The face image feature stream is mapped into three image matrices through three linear transformations, which are represented as follows: I , key matrix K I Sum value matrix V I ;

[0016] The facial key point information flow is mapped into three image matrices through three linear transformations, which are represented as follows: query matrix Q L , key matrix K L and the value matrix V L ;

[0017] Face image features X I and facial key point features X L After the feature cross fusion module, the corresponding output X is obtained Iout and output X Lout , specifically expressed as:

[0018] X I '=CFMSA I (Q L, K I, V I )+X I

[0019] X Iout =MLP(Norm(X I '))+X I '

[0020] X' L =CFMSA L (Q I, K L ,V L )+X L

[0021] X Lout =MLP(Norm(X' L ))+X' L

[0022] Among them, CFMSA I (·) represents the cross-fusion multi-head attention module corresponding to the face image features, CFMSA L (·) represents the cross-fusion multi-head attention module corresponding to the facial key point features, Norm(·) is the normalization operator, and MLP(·) represents the multi-layer perceptron.

[0023] As a preferred technical solution, the output of the cross-fusion multi-head attention module is described as the following mapping function:

[0024]

[0025]

[0026] in, represents the normalized scaling factor, and Softmax represents the Softmax function.

[0027] As a preferred technical solution, the classification module includes a global average pooling layer, a linear layer, and a softmax classification layer;

[0028] The facial image feature X I and facial key point features X L The output X obtained after the feature cross fusion module Iout and output X Lout Perform fusion to obtain fusion features;

[0029] The fused features are input into the classification module, the global average pooling layer performs average pooling operation on the spatial dimension of the feature map, the linear layer maps the features to the dimension of the target category, and the softmax classification layer outputs the probability of eye contact recognition to obtain the eye contact recognition result.

[0030] The present invention also provides an end-to-end eye contact recognition system guided by full-face information, including: a full-face image data acquisition module, a data labeling module, a data set division module, an eye contact recognition model construction module, a model training module, and a classification recognition module;

[0031] The full face image data acquisition module is used to acquire full face image data;

[0032] The data labeling module is used to label the full-face image data with eye state categories;

[0033] The data set division module is used to divide the labeled full face image data into data sets;

[0034] The eye contact recognition model construction module is used to construct an eye contact recognition model based on double-stream feature cross-fusion guided by full-face key points, including a face image feature stream extraction module, a face key point information stream extraction module, a feature cross-fusion module, and a classification module;

[0035] The face image feature stream extraction module is used to extract face image features, the face key point information stream extraction module is used to extract face key point features, the feature cross fusion module is used to perform a cross fusion operation on face image features and face key point features based on cross fusion multi-head attention, and the classification module is used to map the output features of the feature cross fusion module to the dimension of the target category to obtain eye contact recognition results;

[0036] The model training module is used to train the eye contact recognition model to obtain a trained eye contact recognition model;

[0037] The classification and recognition module is used to perform classification and recognition based on the trained eye contact recognition model to obtain a probability vector of eye contact recognition, and select the eye contact state category corresponding to the maximum probability value in the probability vector as the recognition result of the eye contact state.

[0038] As a preferred technical solution, the feature cross-fusion module is used to perform a cross-fusion operation on the face image features and the face key point features based on the cross-fusion multi-head attention, specifically including:

[0039] The face image feature stream is mapped into three image matrices through three linear transformations, which are represented as follows: I , key matrix K I Sum value matrix V I ;

[0040] The facial key point information flow is mapped into three image matrices through three linear transformations, which are represented as follows: query matrix Q L , key matrix K L and the value matrix V L ;

[0041] Face image features X I and facial key point features X L After the feature cross fusion module, the corresponding output X is obtained Iout and output X Lout , specifically expressed as:

[0042] X I '=CFMSA I (Q L ,K I ,V I )+X I

[0043] X Iout =MLP(Norm(X I '))+X I '

[0044] X' L =CFMSA L (Q I ,K L, V L )+X L

[0045] X Lout =MLP(Norm(X' L ))+X' L

[0046] Among them, CFMSA I (·) represents the cross-fusion multi-head attention module corresponding to the face image features, CFMSA L (·) represents the cross-fusion multi-head attention module corresponding to the facial key point features, Norm(·) is the normalization operator, and MLP(·) represents the multi-layer perceptron.

[0047] As a preferred technical solution, the output of the cross-fusion multi-head attention module is described as the following mapping function:

[0048]

[0049]

[0050] in, represents the normalized scaling factor, and Softmax represents the Softmax function.

[0051] As a preferred technical solution, the classification module includes a global average pooling layer, a linear layer, and a softmax classification layer;

[0052] The facial image feature X I and facial key point features X L The output X obtained after the feature cross fusion module Iout and output X Lout Perform fusion to obtain fusion features;

[0053] The fused features are input into the classification module, the global average pooling layer performs average pooling operation on the spatial dimension of the feature map, the linear layer maps the features to the dimension of the target category, and the softmax classification layer outputs the probability of eye contact recognition to obtain the eye contact recognition result.

[0054] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0055] (1) The end-to-end eye contact recognition of the present invention solves the problem that most existing studies rely on pre-calculated head postures, or pre-crop the eye area before training, which leads to errors in eye contact state recognition. The present invention only needs to input the full-face RGB image to effectively recognize the eye contact state, alleviating the inter-class similarity caused by local recognition.

[0056] (2) The present invention performs eye contact recognition based on full-face information guidance, uses facial key points to provide global facial information, and improves recognition accuracy in environments where the face is obscured or in extreme postures through cross-fusion and guidance between image feature streams and facial key point information streams.

[0057] (3) The feature cross-fusion module of the present invention can effectively allow the image stream and the key point information stream to guide each other, and realize the global correlation between the dual-stream features through attention, which can improve the indicativeness of the eye state. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of an end-to-end eye contact recognition method guided by full face information according to the present invention;

[0059] Figure 2 A schematic diagram of the network architecture of the eye contact recognition model with cross-fusion of dual-stream features of the present invention;

[0060] Figure 3 This is a schematic diagram of the network architecture of the feature cross-fusion module of the present invention;

[0061] Figure 4 It is a schematic diagram for visualizing and comparing the facial image feature flow, facial key point information flow, and cross-fusion flow feature results of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] Example 1

[0064] like Figure 1As shown, this embodiment provides an end-to-end eye contact recognition method based on full-face information guidance. This embodiment is different from the method of using pre-calculated head posture and facial key point information or pre-cropping the eye part before training. This embodiment only needs to input the full-face RGB image, and utilizes the cross-fusion and guided learning between the facial image feature stream and the facial key point information stream. The eye contact recognition result of this embodiment can achieve the comprehensive performance improvement of accuracy, F-measure, and area under the ROC curve (AUC);

[0065] The method specifically comprises the following steps:

[0066] S1: Acquire full face image data and perform preprocessing;

[0067] S11: First, obtain multiple full-face image data, and mark the eye state category of each face image according to "whether there is eye contact", "yes" is marked as "1", and "no" is marked as "0", and the full-face image and its corresponding eye state category are used as the initial eye contact image data set;

[0068] S2: Divide the initial eye contact image dataset into a training dataset and a test dataset;

[0069] In this embodiment, the image sizes in the divided eye contact image dataset are adjusted to 224×224 and then data enhancement is performed to obtain enhanced eye contact image training dataset and test dataset;

[0070] S3: Construct an eye contact recognition model based on the cross-fusion of two-stream features guided by full-face key points;

[0071] In this embodiment, the model adopts a dual-stream architecture consisting of a face image feature stream and a face key point information stream. The face image feature stream can significantly extract important features of parts such as the eyes. The face key point information stream refers to the key points on the face image, providing a sparse representation of the key areas of the face. The face key point information stream can provide global information of the entire head and face, accurately locate the significant areas related to the eye contact area and provide relative position information of various parts of the face. The image feature stream learning can be guided by the cross-fusion network to explore the correlation between the key points and the image features. The specific steps include:

[0072] S31: Figure 2 As shown, an eye contact recognition model based on double-stream feature cross-fusion guided by full-face key points is constructed, which includes four modules: face image feature stream extraction module, face key point information stream extraction module, feature cross-fusion module, and classification module;

[0073] S32: Define the input image X∈RH×W×3 , where H and W are the height and width of the image respectively. Then the input image X∈R H×W×3 The face image feature stream extraction module and the face key point information stream extraction module are input respectively. The face image feature stream extraction module uses the first four convolutional blocks of ResNet50 as the backbone network, and the output image size is B×2048×7×7. Then, the face image feature X can be obtained by reshaping the size. I ∈R P×D The facial key point information flow extraction module uses the facial feature point detector Mobile-FaceNet to obtain the facial key point feature X L ∈R P×D , where P represents the number of patches and D represents the feature dimension. During the training process, the weights of the facial feature point detector Mobile-FaceNet are frozen to maintain the appropriate output of the facial key point information flow;

[0074] S33: This embodiment is improved based on transformer encoders, and a feature cross-fusion module for exchanging the face image feature stream and the face key point information stream is constructed, and the query matrix Q is exchanged to realize the cross-fusion and guided learning between the face image feature stream and the face key point information stream;

[0075] Combination Figure 2 As shown, using the extracted facial image feature X I and facial key point features X L , through the cross-fusion operation of the multi-head attention module MHA, the face key point information flow can provide some prior knowledge of salient areas to guide the learning of the face image feature flow. Relatively speaking, the face image feature flow also provides the representation of the face key point information flow with global context information from the image features, promoting feature collaboration;

[0076] In this embodiment, for the face image feature stream, the input X is transformed into I ∈R P×D Mapped into 3 image matrices: query matrix Q I , key matrix K I Sum value matrix V I Similarly, for the facial key point information flow, input X L ∈R P ×D Is mapped into three landmark matrices: query matrix Q L , key matrix K L and the value matrix V L ;

[0077] like Figure 3As shown, MatMul represents matrix multiplication, Scale represents scaling, and the scaled attention score is converted into a probability distribution through the Softmax function. This step ensures that the sum of all attention weights is 1, and the value can be weighted and combined by weighted summation to generate the final output. The specific formula is as follows:

[0078] Q J =X I W Q1 ;

[0079] K I =X I W K1 ;

[0080] V I =X I W V1 ;

[0081] Q L =X L W Q2 ;

[0082] K L =X L W K2 ;

[0083] V L =X L W V2 ;

[0084] Among them, W Q1 , W Q2 , W K1 , W K2 , W V1 , W V2 ∈R D×D , indicating the corresponding weight value

[0085] W Q1 , W Q2 , W K1 , W K2 , W V1 , W V2 ∈R D×D ;

[0086] The output can then be described as the following mapping function:

[0087]

[0088]

[0089] in, represents the normalized scaling factor to prevent extremely small gradients, query matrix QI and the query matrix Q L Exchange between the face image feature stream and the face key point information stream. By doing so, the image features are given the salient areas provided by the landmark features, and on the other hand, the key point features are provided with global information from the image features;

[0090] S34: Given a face image feature X I and facial key point features X L In the case of , the corresponding output X in the feature cross fusion module Iout and output X Lout It is expressed as:

[0091] X I '=CFMSA I (Q L ,K I ,V I )+X I

[0092] X Iout =MLP(Norm(X I '))+X I '

[0093] X' L =CFMSA L (Q I, K L, V L )+X L

[0094] X Lout =MLP(Norm(X' L ))+X' L

[0095] Among them, CFMSA(·) represents the cross-fusion multi-head attention module, combined with Figure 3 As shown, it is a dual-stream cross-fusion multi-head attention module (Cross-Fusion MSA) that realizes the image feature stream and the face key point information stream. Norm (·) is a normalization operator, MLP (·) represents a multi-layer perceptron, and the number of attention heads in the multi-head self-attention module of this embodiment is preferably 8;

[0096] S35: Get the fused feature X out =X Iout +X Lout After that, the fused feature X is pooled through the global average pooling layer. outAfter processing, the spatial dimension of the feature map is averaged and pooled to obtain "1×1", thereby retaining the global context information. Finally, through the linear layer, the feature is mapped to the dimension of the target category, and the eye contact recognition result is finally obtained. Specifically, the eye contact recognition result refers to the result (YES or NO) obtained by a single image through a two-stream feature cross-fusion model. Eye contact can be understood as "look at me, YES or NO";

[0097] S4: training the eye contact recognition model based on the eye contact image training data set obtained in step S2 until a set number of training times is reached to obtain a trained eye contact recognition model;

[0098] S5: Classify and recognize the eye contact image test data set using the trained network model weights; specifically, input the eye contact images in the eye contact image test data set into the trained eye contact recognition model, and output a probability vector through the softmax classification module to represent the probability of whether eye contact is recognized, and select the eye contact state category corresponding to the maximum probability value in the probability vector as the recognition result of the eye contact state in the eye contact image test data set. After obtaining the recognition result of each eye contact image in the test data set, the recognition and classification effect of the trained eye contact recognition model can be evaluated.

[0099] In this embodiment, the probability of eye contact recognition and the identification and classification of the test set refer to the accuracy (percentage) of the eye contact results obtained by the two-stream feature cross-fusion model in the batch data. The test set is used to measure the accuracy of the two-stream feature cross-fusion model.

[0100] In this embodiment, the fused feature Xout is input into the softmax classification module, which includes a global average pooling layer, a linear layer, and a softmax classification layer. The linear layer obtains the classification dimension [16,2], where 2 represents the number of classifications. The softmax classification layer converts it into a probability. For example, if the probability of "Yes" is 0.8 and the probability of "No" is 0.3, then the result of this image is "yes", which is "1". The results of the test set are then aggregated to calculate the accuracy.

[0101] The following is a further description of the technical effects of the present invention in combination with simulation experiments:

[0102] 1. Simulation conditions and contents:

[0103] Using the above technical solution, in order to verify the effectiveness of the eye contact recognition method based on full-face information guidance proposed in the present invention, experiments were conducted on the public dataset OFDIW-HF. The OFDIW-HF dataset has great diversity in terms of different age levels of individuals, complexity of image scenes, degree of occlusion of faces, emotional distribution of individuals, gender distribution of individuals, and facial orientation of individuals. OFDIW-HF contains 9924 training images and 3309 test images;

[0104] All experiments were conducted on the same experimental platform, that is, in the hardware environment of RTX 2080 GPU and the software environment of Window 10. All experiments were developed based on the PyTorch deep learning framework. The batch size of the model training was set to 16, the loss function used was SGD, the initial learning rate was 0.001, the learning rate decay was 5E-5, and the number of training epochs was set to 200.

[0105] 2. Analysis of simulation results:

[0106] The experiment compares the proposed eye contact recognition method based on full face information with VGG16[Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014. ArXiv: 1409.1556.], Densenet121[Huang G, Liu Z, Pleiss G, et al. Convolutional networks with dense connectivity. IEEE Trans Pattern Anal Mach Intell, 2019. doi: 10.1109 / TPAMI.2019.2918284.], Shufflenet[Zhang X, Zhou X, Lin M, et al. Shufflenet: An extremely efficient convolutional neural network for mobile devices[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2018:6848-6856.], and then compared with similar gaze estimation methods Full_face[Zhang X,Sugano Y,Fritz M,et al.It'swritten all over your face:Full-face appearance-based gaze estimation[C] / / Proceedings of the IEEE conference on computer vision and pattern recognitionworkshops.2017:51-60.] and L2CS-Net[Abdelrahman AA,Hempel T,Khalifa A,etal.L2CS-Net:fine-grained gaze estimation in unconstrained environments[J].arXiv preprint arXiv:2203.03339,2022.]. When compared with similar gaze estimation methods, the experiment simply modifies the prediction module of the model for eye contact image recognition, as shown in Table 1 below, and the experimental results are obtained:

[0107] Table 1 Experimental results of eye contact recognition method based on full face information guidance

[0108]

[0109]

[0110] As shown in Table 1, the experimental results of the model on the OFDIW-HF dataset show that the proposed method achieves the best results in Accuracy, F1 and AUC on the OFDIW-HF dataset, where ACC = 0.8456, F1 = 0.9004, and AUC = 0.9055. The experimental results show that the eye contact recognition method guided by full-face information can effectively identify eye contact behavior, and this end-to-end method improvement brings significant performance improvement.

[0111] The eye contact recognition method based on full face information guidance proposed in the present invention has two streams: image feature stream and face key point information stream. The attention distribution maps of the two streams are visualized and analyzed respectively. Figure 4 As shown in the figure, for the attention map of the face key point stream (III), the highlighted area usually comes from the area involved in the key point. However, the face key point stream also obtains some global attention from the image feature stream that does not belong to the key point area; for the image stream attention map (IV), the highlighted area mostly comes from the eye area, but because of the guidance of the face key point information stream and the cross-fusion of features, the attention is also concentrated on some important areas related to the key points. From the visualization results, the attention areas learned by the two streams are different but complementary. Through the cross-fusion and guidance between features, the proposed model can improve the indicativeness of the eye state, and can be widely used in fields such as eye contact social behavior analysis in the process of social interaction in natural unconstrained environments, and promote robots to perform human-computer interaction and communication.

[0112] Example 2

[0113] This embodiment provides an end-to-end eye contact recognition system guided by full face information, which is used to implement the end-to-end eye contact recognition method guided by full face information of the above embodiment 1. The system includes: a full face image data acquisition module, a data labeling module, a data set division module, an eye contact recognition model construction module, a model training module, and a classification recognition module;

[0114] In this embodiment, the full face image data acquisition module is used to acquire full face image data;

[0115] In this embodiment, the data labeling module is used to label the full-face image data with eye state categories;

[0116] In this embodiment, the data set division module is used to divide the labeled full-face image data into data sets;

[0117] In this embodiment, the eye contact recognition model construction module is used to construct an eye contact recognition model based on a dual-stream feature cross-fusion guided by full-face key points, including a face image feature stream extraction module, a face key point information stream extraction module, a feature cross-fusion module, and a classification module;

[0118] In this embodiment, the face image feature stream extraction module is used to extract face image features, the face key point information stream extraction module is used to extract face key point features, the feature cross fusion module is used to perform a cross fusion operation on face image features and face key point features based on cross fusion multi-head attention, and the classification module is used to map the output features of the feature cross fusion module to the dimension of the target category to obtain eye contact recognition results;

[0119] In this embodiment, the model training module is used to train the eye contact recognition model to obtain a trained eye contact recognition model;

[0120] In this embodiment, the classification and recognition module is used to perform classification and recognition based on the trained eye contact recognition model to obtain a probability vector of eye contact recognition, and select the eye contact state category corresponding to the maximum probability value in the probability vector as the recognition result of the eye contact state.

[0121] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. An end-to-end eye contact recognition method based on full-face information guidance, characterized in that: The steps include: Obtain full-face image data, mark eye state categories, and divide the data set; Construct an eye contact recognition model based on dual-stream feature cross-fusion guided by full-face key points, including a face image feature stream extraction module, a face key point information stream extraction module, a feature cross-fusion module, and a classification module; The face image feature stream extraction module is used to extract face image features, the face key point information stream extraction module is used to extract face key point features, the feature cross fusion module is used to perform a cross fusion operation on face image features and face key point features based on cross fusion multi-head attention, and the classification module is used to map the output features of the feature cross fusion module to the dimension of the target category to obtain eye contact recognition results; Training the eye contact recognition model to obtain a trained eye contact recognition model; Classification and recognition are performed based on the trained eye contact recognition model to obtain a probability vector for eye contact recognition, and the eye contact state category corresponding to the maximum probability value in the probability vector is selected as the recognition result of the eye contact state.

2. The end-to-end eye contact recognition method based on full face information guidance according to claim 1, characterized in that: The facial image feature stream extraction module uses the first four convolutional blocks of ResNet50 as the backbone network, reshapes the output image of the backbone network, and obtains facial image features.

3. The end-to-end eye contact recognition method based on full face information guidance according to claim 1, characterized in that: The facial key point information flow extraction module uses the facial feature point detector Mobile-FaceNet to obtain the facial key point feature X L ∈R P×D , where P represents the number of tiles and D represents the feature dimension.

4. The end-to-end eye contact recognition method based on full-face information guidance according to claim 1, characterized in that: The feature cross-fusion module is used to perform a cross-fusion operation on the face image features and the face key point features based on the cross-fusion multi-head attention, specifically including: The face image feature stream is mapped into three image matrices through three linear transformations, which are represented as follows: I , key matrix K I Sum value matrix V I ; The facial key point information flow is mapped into three image matrices through three linear transformations, which are represented as follows: query matrix Q L , key matrix K L and the value matrix V L ; Face image features X I and facial key point features X L After the feature cross fusion module, the corresponding output X is obtained Iout and output X Lout , specifically expressed as: X′ I =CFMSA I (Q L ,K I ,V I )+X I X Iout =MLP(Norm(X′ I ))+X′ I X′ L =CFMSA L (Q I ,K L ,V L )+X L X Lout =MLP(Norm(X' L ))+X' L Among them, CFMSA I (·) represents the cross-fusion multi-head attention module corresponding to the face image features, CFMSA L (·) represents the cross-fusion multi-head attention module corresponding to the facial key point features, Norm(·) is the normalization operator, and MLP(·) represents the multi-layer perceptron.

5. The end-to-end eye contact recognition method based on full-face information guidance according to claim 4, characterized in that: The output of the cross-fusion multi-head attention module is described as the following mapping function: in, represents the normalized scaling factor, and Softmax represents the Softmax function.

6. The end-to-end eye contact recognition method based on full-face information guidance according to claim 1, characterized in that: The classification module includes a global average pooling layer, a linear layer, and a softmax classification layer; The facial image feature X I and facial key point features X L The output X obtained after the feature cross fusion module Iout and output X Lout Perform fusion to obtain fusion features; The fused features are input into the classification module, the global average pooling layer performs average pooling operation on the spatial dimension of the feature map, the linear layer maps the features to the dimension of the target category, and the softmax classification layer outputs the probability of eye contact recognition to obtain the eye contact recognition result.

7. An end-to-end eye contact recognition system guided by full face information, characterized in that: include: Full-face image data acquisition module, data labeling module, data set division module, eye contact recognition model construction module, model training module, classification and recognition module; The full face image data acquisition module is used to acquire full face image data; The data labeling module is used to label the full-face image data with eye state categories; The data set division module is used to divide the labeled full face image data into data sets; The eye contact recognition model construction module is used to construct an eye contact recognition model based on double-stream feature cross-fusion guided by full-face key points, including a face image feature stream extraction module, a face key point information stream extraction module, a feature cross-fusion module, and a classification module; The face image feature stream extraction module is used to extract face image features, the face key point information stream extraction module is used to extract face key point features, the feature cross fusion module is used to perform a cross fusion operation on face image features and face key point features based on cross fusion multi-head attention, and the classification module is used to map the output features of the feature cross fusion module to the dimension of the target category to obtain eye contact recognition results; The model training module is used to train the eye contact recognition model to obtain a trained eye contact recognition model; The classification and recognition module is used to perform classification and recognition based on the trained eye contact recognition model to obtain a probability vector of eye contact recognition, and select the eye contact state category corresponding to the maximum probability value in the probability vector as the recognition result of the eye contact state.

8. The end-to-end eye contact recognition system based on full-face information guidance according to claim 7, characterized in that: The feature cross-fusion module is used to perform a cross-fusion operation on the face image features and the face key point features based on the cross-fusion multi-head attention, specifically including: The face image feature stream is mapped into three image matrices through three linear transformations, which are represented as follows: I , key matrix K I Sum value matrix V I ; The facial key point information flow is mapped into three image matrices through three linear transformations, which are represented as follows: query matrix Q L , key matrix K L and the value matrix V L ; Face image features X I and facial key point features X L After the feature cross fusion module, the corresponding output X is obtained Iout and output X Lout , specifically expressed as: X′ I =CFMSA I (Q L ,K I ,V I )+X I X Iout =MLP(Norm(X I '))+X I ' X' L =CFMSA L (Q I ,K L ,V L )+X L X Lout =MLP(Norm(X' L ))+X' L Among them, CFMSA I (·) represents the cross-fusion multi-head attention module corresponding to the face image features, CFMSA L (·) represents the cross-fusion multi-head attention module corresponding to the facial key point features, Norm(·) is the normalization operator, and MLP(·) represents the multi-layer perceptron.

9. The end-to-end eye contact recognition system based on full-face information guidance according to claim 7, characterized in that: The output of the cross-fusion multi-head attention module is described as the following mapping function: in, represents the normalized scaling factor, and Softmax represents the Softmax function.

10. The end-to-end eye contact recognition system based on full face information guidance according to claim 7, characterized in that: The classification module includes a global average pooling layer, a linear layer, and a softmax classification layer; The facial image feature X I and facial key point features X L The output X obtained after the feature cross fusion module Iout and output X Lout Perform fusion to obtain fusion features; The fused features are input into the classification module, the global average pooling layer performs average pooling operation on the spatial dimension of the feature map, the linear layer maps the features to the dimension of the target category, and the softmax classification layer outputs the probability of eye contact recognition to obtain the eye contact recognition result.

Images