An eye contact recognition method based on local and global feature mining

Abstract

The application discloses an eye contact recognition method based on local and global feature mining, which comprises the following steps: acquiring full-face image data, and taking the full-face image data and corresponding eye contact state categories as initial eye contact image data sets; dividing the initial eye contact image data sets into training data sets and test data sets; constructing an eye contact recognition network model based on local and global feature mining; training the eye contact recognition network model based on the eye contact image training data sets until a set training number is reached, obtaining a trained eye contact recognition model; and classifying and recognizing the eye contact image test data sets by using the trained eye contact recognition model weight. The global learning enhanced features provided by the application can learn more diversified features, so that more subtle and deeper information of key parts such as eyes can be obtained, and the accuracy of eye state discrimination is provided.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision detection, and more particularly, to an eye contact recognition method based on local and global feature mining. BACKGROUND

[0002] Eye contact is an important basis for face-to-face social interaction. In the process of social interaction, the social behavior of making eye contact with the interactive object is one of the earliest social skills of human beings, and eye contact has many important functions in social interaction, such as establishing relationships between partners, expressing interest and attention, and identifying intentions. From the existing research, the exploration of social behavior is largely dependent on the face, which carries the expression of emotional interaction. In social interaction, the content of the face of the social object in the natural and unrestricted situation is more dependent on the interaction between partners than the content of the unresponsive face (such as pictures or videos) described in typical experiments, is more comprehensive and dynamic, and can more truly reflect the social behavior characteristics of the social interaction process.

[0003] Abnormal eye contact and gaze patterns are often a signal of many medical or mental disorders, including autism spectrum disorder (ASD), hyperactivity, Williams syndrome, social anxiety / behavior inhibition, and oppositional defiant disorder. For ASD, the Autism Diagnostic Observation Schedule (ADOS) continuously tracks and quantifies eye contact, and the reduction of eye contact has been included in the DSM-5 ASD diagnostic criteria, and is also the focus of early screening and treatment. In real life, the recognition of eye contact social behavior in a natural and unrestricted environment can be widely used in the fields of eye contact social behavior analysis in the process of social interaction, disease discovery, etc., and it also has the value of promoting human-robot interaction and communication.

[0004] In natural social interaction, the recognition of eye contact social behavior has important value, but the efficient and simple detection and recognition method is still in continuous exploration. The current eye contact recognition method can be divided into three kinds, one is to use one or more cameras to record social interaction activities, and then manually judge the eye contact in the interaction activity record by artificial. Obviously, this method has great subjectivity, and because it is a pure manual operation, it is time-consuming and labor-intensive, making it difficult to be widely used in large data recognition and detection; the second is to use wearable eye tracking device to recognize eye contact social behavior, but this method is expensive and limits the use scene, and also affects the natural movement in social activities; the third is to recognize and detect based on deep learning method, this method can be applied to large-scale social activity data eye contact behavior recognition analysis, and has objectivity and effectiveness. From the current research, the eye contact social behavior recognition method based on deep learning shows great potential and broad development prospects in real scene application, which can more accurately and objectively analyze and understand human social behavior, and thus plays an important role in various practical scenes, and provides support for intelligent analysis of social communication.

[0005] Studies have shown that in complex natural scenes, eye contact recognition and detection errors mainly occur when it is difficult to observe the eyes of social objects from the observation image. In this case, the extracted features cannot provide enough information for accurate prediction. In fact, under natural and unconstrained conditions, the movement of the head is complex and variable, and some extreme head positions are more difficult to observe the eye part, or in face-to-face communication, social personnel wear glasses and other items to block the eye part, so that the eye state cannot be well recognized and judged.

[0006] It is worth noting that existing researches mostly focus on the features of a single recognition area, ignoring the relative relationship between different areas in the whole face image. However, in eye contact behavior recognition, the relationship between different discriminant areas of the face contains rich relative information of the areas, which can to some extent make up for the information loss of the face features in complex environments.

[0007] The eye contact recognition method for learning enhanced features and mining regional relationships. In order to solve the problem of eye feature or face information loss in eye contact recognition under complex postures (such as complex changes of head posture and occlusion of eye part) in natural environment, an eye contact recognition method for learning enhanced features and mining regional relationships is constructed. SUMMARY

[0008] The technical problem to be solved by the present application is to provide an eye contact recognition method based on local feature and global feature mining to solve the above-mentioned problems of the prior art.

[0009] The technical scheme of the present application is:

[0010] An eye contact recognition method based on local and global feature mining, comprising the following steps:

[0011] S1: Obtain full-face image data, and use the full-face image data and its corresponding eye contact state category as an initial eye contact image data set;

[0012] S2: Divide the initial eye contact image data set into a training data set and a test data set;

[0013] S3: Construct an eye contact recognition network model based on local and global feature mining;

[0014] S4: Train the eye contact recognition network model based on the eye contact image training data set obtained in step S2 until a set number of training times is reached, to obtain a trained eye contact recognition model;

[0015] S5: Use the trained eye contact recognition model to classify and recognize the eye contact image test data set;

[0016] Input the eye contact image in the eye contact image test data set into the trained eye contact recognition model, and output a probability vector through a softmax classification module, representing the probability of eye contact recognition, select the eye contact state category corresponding to the maximum probability value in the probability vector as the recognition result of the eye contact state in the eye contact image test data set, and obtain the recognition result of each eye contact image in the test data set.

[0017] Preferably, the eye contact state category corresponding to the full-face image data in S1 is based on "whether there is eye contact" to label the eye contact state category of each face image, "yes" is labeled as "1", and "no" is labeled as "0", and the full-face image and its corresponding eye contact state category are used as the initial eye contact image data set.

[0018] Preferably, after dividing the training data set and the test data set, the image size in the eye contact image data set is adjusted to 224x224, and then data enhancement is performed respectively to obtain the enhanced eye contact image training data set and the test data set.

[0019] Preferably, the implementation process of the eye contact recognition network model in S3 is:

[0020] S31: Global feature extraction: define the input full-face image X∈R H×W×3 , where H and W are the height and width of the image, respectively, and then extract the global feature of the full-face image X∈R H×W×3ResNet50 first four convolutional blocks are used as backbone network for global feature extraction, and the output global feature X of the full face image has a size of Bx2048x7x7;

[0021] S32: Feature enhancement: the global feature X of the full face image is enhanced, and the secondary discriminant region is enhanced to obtain diversified features containing main discriminant features and confusing features as the final features and obtain a confidence score;

[0022] S33: Region navigation: a series of rectangular regions {R1 ′ ,R2 ′ ,…R A ′} are proposed by using the region proposal network FPN according to the obtained global feature X of the full face image, the region proposal network performs dimension reduction and down-sampling on the input global feature X of the full face image to form multi-scale feature maps, and the features extracted by each layer of feature maps focus on different receptive fields, thereby capturing information of different size targets; each region has a score indicating the information amount of the region, as follows, where A is the number of anchor points, and I(R i ) is the i-th element in the sorted information list:

[0023] I(R1)≥I(R2)≥…≥I(R A ) (4)

[0024] S34: graph construction: using multi-scale feature maps from different layers, the information amount of the region is generated in different scales and scales; then, non-maximum suppression (NMS) is used to screen the regions according to their information amount, and the first M regions {R1,R2,...,R M} with the most abundant information amount are selected; after the number M of the selected region module is selected, the face local image X M ∈R H×W×3 is obtained through size resizing, and the region feature is obtained through feature extraction, which is used to represent the node feature BxMx2048, where B is the batch size, M is the number of nodes in the graph, and 2048 is the feature dimension of each node; using the obtained node feature, the graph construction (GCN) is obtained from the unstructured data, which has the ability to infer the region relationship, and then the graph is processed layer by layer through the multi-layer GCN of the graph construction, and the node feature is gradually aggregated and updated to mine the relative relationship between different regions; the output of each layer of GCN is the updated node feature matrix, and the update formula is generally as follows:

[0025]

[0026] where represents the node feature matrix of the l-th layer, A represents the adjacency matrix of the graph, and represents the learnable weights and biases, and σ represents the activation function;

[0027] S35: Eye contact recognition network model construction: Get the relevant information of the last layer of GCN, map the features to the dimension of the target category through the linear layer, and combine it with the confidence score of the enhanced feature learning module to finally obtain the eye contact recognition result.

[0028] Preferably, in step S32, feature enhancement of the global feature X of the full face image is achieved by combining the original learning branch and the enhanced learning branch.

[0029] The original learning branch and the enhanced learning branch share the global feature X of the full face image, but the inputs of the two branches are different;

[0030] The original learning branch generates the class activation map and locates the main identification area. After convolving the feature F by the 1×1 convolution layer, the class activation map is obtained. , where C represents the number of categories, W and H represent the width and height of the image respectively; then Perform global average pooling to obtain the identified class activation map , where C represents the number of categories and p is the recognition vector The maximum index of , identifying vector as follows:

[0031] (1)

[0032] Where g(·) is the global average pooling;

[0033] The reinforcement learning branch uses the class activation map Generate a mask to suppress the top-k discriminant positions of F; since the feature information of the top-k positions is suppressed, the branch focuses on other secondary distinguishable features, achieving the purpose of enhancing learning features;

[0034] Let B be the slave The obtained mask and the input of the reinforcement learning branch are , as follows, where Represents the element-wise multiplication of two tensors:

[0035] (2)

[0036] Then the output of the reinforcement learning branch is obtained After that; let M be the output of the enhanced feature learning module, the obtained M is:

[0037] (3)

[0038] Finally, the confidence score is obtained after the average pooling layer

[0039] Compared with the prior art, the technical effects of the present application are:

[0040] (1) The global learning enhanced feature module proposed in the present application can learn more diversified features, thereby obtaining more subtle and deeper information of key parts such as eyes, and providing accuracy of eye state discrimination;

[0041] (2) The eye contact recognition method for mining regional relationships proposed in the present application pays attention to and explores the relative relationship between different regions, rather than only focusing on the features of a single recognition region. In the case of lack of facial information in a complex environment, the information is supplemented through the relationship between regions, and the behavior of the attribute object is learned, which can effectively solve the problem of misjudgment of eye contact recognition caused by complex changes in head posture and occlusion of the eye part by the environment in a natural environment, and improve the performance of eye contact recognition. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 Flowchart of the eye contact recognition method based on local and global feature mining

[0043] Figure 2 Schematic diagram of the eye contact recognition network model based on local and global feature mining.

[0044] Figure 3 Schematic diagram of the enhanced feature learning process. DETAILED DESCRIPTION

[0045] The drawings are only used for illustrative purposes and cannot be understood as limiting the present patent; in order to better illustrate the present embodiment, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product;

[0046] It is understandable for those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted. The technical solutions of the present application will be further described below in combination with the drawings and embodiments.

[0047] As shown in Figure 1 , the present embodiment provides an end-to-end eye contact recognition method based on local and global feature mining. Unlike the method that only focuses on the features of a single recognition region, the present embodiment only needs to input a full-face RGB image, and uses the relative information between the local and global discrimination regions of the face to recognize eye contact behavior. The eye contact recognition result of the present embodiment can improve the comprehensive performance of accuracy (Accuracy) and F value (F-measure).

[0048] The method of the present application specifically includes the following steps:

[0049] S1: Obtain full-face image data and pre-process;

[0050] S11: First, obtain multiple full-face image data, and label the eye contact state category of each face image according to "whether there is eye contact", "yes" is labeled as "1", and "no" is labeled as "0". The full-face image and its corresponding eye contact state category are used as an initial eye contact image data set;

[0051] S2: Divide the initial eye contact image data set into a training data set and a test data set;

[0052] In this embodiment, after adjusting the image size in the divided eye contact image data set to 224x224, data enhancement is performed respectively to obtain an enhanced eye contact image training data set and a test data set;

[0053] S3: Construct an eye contact recognition network model based on local and global feature mining;

[0054] In this embodiment, the recognition method constructs an eye contact recognition network model that learns enhanced features and mines regional relationships, as shown in Figure 2 The eye contact recognition network model mainly consists of a global enhanced feature learning module and a region navigation and graph construction module.

[0055] The specific steps include:

[0056] S31: Define the input full-face image X e R H×W×3 , where H and W are the height and width of the image, respectively, and then input the input full-face image X e R H×W×3 to the global feature extraction network, respectively. The global feature extraction network uses the first four convolutional blocks of ResNet50 as the backbone network, and outputs the full-face image global feature X with a size of Bx2048x7x7.

[0057] S32: Input the image global feature X to the enhanced learning network. The purpose of the enhanced learning network is to enable the network to extract more diversified features of the eye key parts, as shown in Figure 2 The enhanced learning network consists of an original learning branch and an enhanced learning branch. The enhanced learning network generates an inhibition mask based on a class activation map, suppresses the most distinguishable regions, and obtains diversified features containing main discriminative features and confusing features by enhancing the secondary discriminative regions, thereby serving as the final features.

[0058] The original learning branch and the enhanced learning branch share the full-face image global feature X, but the inputs of the two branches are different.

[0059] The original learning branch aims to generate a class activation map and locate the main discriminative regions. After the extracted features F are convolved by a 1x1 convolutional layer, a class activation map , where C represents the number of categories, W and H represent the width and height of the image respectively. Perform global average pooling to obtain the identified class activation map , where C represents the number of categories and p is the recognition vector The maximum index of , identifying vector as follows:

[0060] (1)

[0061] Where g(·) is the global average pooling. For the reinforcement learning branch, use Generate a mask to suppress the top-k discriminant positions of F. Since the feature information of the top-k positions is suppressed, the branch focuses on other secondary distinguishable features, achieving the purpose of enhancing learning features.

[0062] Let B be the slave The obtained mask and the input of the reinforcement learning branch are , as follows, where Represents the element-wise multiplication of two tensors:

[0063] (2)

[0064] Then the output of the reinforcement learning branch is obtained After that. Let M be the output of the reinforcement learning network, and the obtained M is:

[0065] (3)

[0066] Finally, the confidence score is obtained after the average pooling layer.

[0067] S33: Combine Figure 2 As shown in the figure, the region navigation and graph construction module obtains the global features X of the full face image from the global feature network as shallow features of the image. The region navigation network generates object proposals from the image based on the shallow feature expressions and constructs the object graph structure based on the locations of the interaction regions in the image. Furthermore, features are extracted from the regions and mapped to the node features of the graph structure to form a complete graph data. The graph data not only contains the image feature information but also the relative positional relationships between image regions, which can effectively identify eye contact social behaviors in complex posture scenes.

[0068] The region navigation network uses the region proposal network (FPN) to propose a series of rectangular regions based on the obtained features {R1 ′ ,R2 ′ ,…R A ′}, the region proposal network reduces the dimension and down-samples the input global feature X to form multi-scale feature maps, and each layer of feature maps extracts features focusing on different receptive fields, thereby capturing information of different size targets. Each region has a score indicating the amount of information in the region, as follows, where A is the number of anchor points, I(R i ) is the i-th element in the sorted information list:

[0069] I(R1) ≥ I(R2) ≥... ≥ I(R A ) (4)

[0070] S34: Using multi-scale feature maps from different layers, the network can generate the amount of information of regions in different scales and proportions. To reduce region redundancy, non-maximum suppression (NMS) is used to screen regions according to their information amount, and the top M regions {R1, R2,..., RM} with the most information amount can be selected. M After selecting the number M of regions, the face local image X M ∈R H×W×3 is obtained through size resizing. The region features are obtained through the feature extraction network, which are used to represent node features BxMx2048, where B is the batch size, M is the number of nodes in the graph, and 2048 is the feature dimension of each node. Using the obtained node features, the graph construction module (GCN) can infer and learn from unstructured data, and has the ability to infer regional relationships. Then the graph is processed layer by layer through the multi-layer GCN of the graph construction module, and the node features are gradually aggregated and updated to mine the relative relationships between different regions. The output of each layer of GCN is the updated node feature matrix, and the update formula is generally as follows:

[0071]

[0072] where represents the node feature matrix of the l-th layer, A represents the adjacency matrix of the graph, and represent the learnable weights and biases, and σ represents the activation function;

[0073] S35: Obtain the relevant information of the last layer of GCN, map the features to the dimension of the target class through a linear layer, and combine the confidence score of the enhanced feature learning module to finally obtain the eye contact recognition result;

[0074] S4: Train the eye contact recognition model based on the eye contact image training data set obtained in step S2 until the set number of training times is reached, and obtain the trained eye contact recognition model;

[0075] S5: using the trained network model weight to classify and identify the eye contact image test data set; specifically, input the eye contact image in the eye contact image test data set into the trained eye contact recognition model, and output a probability vector through the softmax classification module, indicating the probability of eye contact recognition, select the eye contact state category corresponding to the maximum probability value in the probability vector as the recognition result of the eye contact state in the eye contact image test data set, after obtaining the recognition result of each eye contact image in the test data set, the recognition and classification effect of the trained eye contact recognition model can be evaluated.

[0076] The technical effects of the present application are further described below in combination with simulation tests:

[0077] 1. Simulation conditions and contents:

[0078] In order to verify the effectiveness of the eye contact recognition method based on local and global feature mining proposed by the present application, comparative experiments are carried out, all experiments are carried out in the same experimental platform, that is, in the hardware environment of RTX 2080 GPU and the software environment of Window 10, and all experiments are developed based on the PyTorch deep learning framework. The experiment uses the public data set OFDIW-HF, which has great diversity in individual age level, image scene complexity, face occlusion degree, individual emotion distribution, individual gender distribution, and individual face orientation. OFDIW-HF contains 9924 training images and 3309 test images;

[0079] The batchsize of model setting training is uniformly 16, the loss function uses SGD, the initial learning rate is 0.001, the learning rate decay is 5E -5 , the training iteration number epoch is uniformly set to 100, and the region proposal number M is 6.

[0080] 2. Simulation process description:

[0081] The data set is trained and tested according to the set batch size (batchsize=16), then the picture data is input to the network model in the format of [16,3,224,224] (3 represents the channel number of the image, and 224x224 is the height and width of the image), the simulation specific process is shown in Table 1, and the classification result with the size of 16x2 is obtained through the feature fusion output of the model, corresponding to 2 eye contact state categories;

[0082] Table 1 Model simulation process

[0083]

[0084] 3. Simulation result analysis:

[0085] The eye contact recognition method based on local and global feature mining provided in the application is different from some regional attention or regional information mining network methods in recent years, such as:

[0086] NTS-Net: [Yang Z, Luo T, Wang D, et al. Learning to navigate for fine-grained classification [C] / / Proceedings of the European conference on computer vision (ECCV). 2018: 420-435.],

[0087] MMAL-Net: [

[31] Zhang F, Li M, Zhai G, et al. Multi-branch and multi-scale attention learning for fine-grained visual categorization [C] / / MultiMedia Modeling: 27th International Conference, MMM 2021, Prague, Czech Republic, June 22-24, 2021, Proceedings, Part I 27. Springer International Publishing, 2021: 136-147.],

[0088] RARN: [

[32] Li X, Li C, Tian B, et al. Learning Discriminative Features with Region Attention and Refinement Network for Facial Expression Recognition in the Wild [C] / / 2022 26th International Conference on Pattern Recognition (ICPR). IEEE, 2022: 1113-1119.],

[0089] F-Net: [

[33] Nie X, Chai B, Wang L, et al. Learning enhanced features and inferring twice for fine-grained image classification[J]. Multimedia Tools and Applications, 2023, 82(10): 14799-14813.

[0090] Compared with the above method, as shown in the table, the eye contact recognition method for learning enhanced features and mining regional relationships used in the study achieved the optimal result: Accuracy = 0.8550, F1 = 0.9058. The experimental results show that the method proposed in the embodiment can significantly improve the performance of eye contact social behavior recognition by mining the relationship between different regions of the face.

[0091] Table 1 Comparison results of eye contact recognition methods based on local and global feature mining

[0092]

[0093] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. For those skilled in the art, on the basis of the above description, other different forms of changes or variations can also be made. Here, it is not necessary and impossible to exhaust all the embodiments. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. An eye contact recognition method based on local and global feature mining, characterized in that, The method comprises the following steps: S1: obtaining full-face image data, and taking the full-face image data and its corresponding eye contact state category as an initial eye contact image data set; S2: dividing the initial eye contact image data set into a training data set and a test data set; S3: constructing an eye contact recognition network model based on local and global feature mining; S4: training the eye contact recognition network model based on the eye contact image training data set obtained in step S2 until a set number of training times is reached, to obtain a trained eye contact recognition model; S5: using the trained eye contact recognition model to classify and recognize the eye contact image test data set; The implementation process of the eye contact recognition network model of S3 is as follows: S31: global feature extraction: input the full-face image X ∈ R H×W×3 where H and W are the height and width of the image, respectively, global feature extraction is performed on the full-face image to obtain the full-face image global feature X; S32: feature enhancement: using an original learning branch and an enhanced learning branch, The original learning branch and the enhanced learning branch share global features X of the full-face image; The original learning branch is to generate class activation map and locate the main discriminative region, and after the feature X is convolved by a 1x1 convolution layer, a class activation map is obtained , where C represents the number of classifications; then global average pooling is performed to obtain the recognized class activation map , p is the maximum index of the recognition vector , and the recognition vector is as follows: （1） Where g(·) is a global average pooling; The reinforcement learning branch is to utilize class activation maps A mask is generated to suppress top-k discriminative positions of the feature X; B is the mask obtained from the input to the enhanced learning branch is obtained as where , denotes the element-wise multiplication of two tensors: Subsequently obtaining an output of the reinforcement learning branch ; let M be an output of the enhanced feature learning module: （3） Finally, the confidence score is obtained through the average pooling layer; S33: regional navigation: a series of rectangular regions {R1 ′ ,R2 ′ ,…R A ′} are proposed according to the obtained global features X of the full face image using RPN, RPN reduces and down-samples the input global features X of the full face image to form multi-scale feature maps, the features extracted by each layer of the feature maps focus on different receptive fields, thereby capturing information of different size targets; each region has a score indicating the amount of information of the region; S34: graph construction: adopt non-maximum suppression, select the top M regions {R1, R2,..., R M} with the most information; obtain the local face image X M ∈R H×W×3 by size remolding, obtain the region feature by feature extraction, which is used to represent the node feature BxMx2048, where B is the batch size, M is the number of nodes in the graph, and 2048 is the feature dimension of each node; obtain the graph construction by using the node feature, process the graph layer by layer by the multi-layer GCN of the graph construction, gradually aggregate and update the node feature, and mine the relative relationship between different regions; the update formula is: ， wherein denotes the node feature matrix of the l-th layer, A denotes the adjacency matrix of the graph, and denotes the learnable weights and biases, and σ denotes the activation function; S35: eye contact recognition network model construction: obtaining the relevant information of the last GCN layer, mapping the features to the dimension of the target category through a linear layer, combining the confidence score of the enhanced feature learning module, and finally obtaining the eye contact recognition result.

2. The method of claim 1, wherein, The eye contact state category corresponding to the full-face image data in S1 is based on "whether there is eye contact" to label the eye contact state category of each face image, "yes" is labeled as "1", and "no" is labeled as "0". The full-face image and its corresponding eye contact state category are taken as the initial eye contact image data set.

3. The method of claim 2, wherein, After adjusting the image size in the eye contact image data set to 224x224 after dividing the training data set and the test data set in S2, data enhancement is performed on the eye contact image data set, to obtain an enhanced eye contact image training data set and an enhanced eye contact image test data set.

4. The method of claim 1, wherein, The eye contact image in the eye contact image test data set is input into the trained eye contact recognition model, and a probability vector is output through a softmax classification module, representing the probability of eye contact recognition. The eye contact state category corresponding to the maximum probability value in the probability vector is selected as the recognition result of the eye contact state in the eye contact image test data set.

Images