A head-body coordination based gaze estimation method

Abstract

The application relates to the technical field of computer vision and pattern recognition, and discloses a head-body coordination-based gaze estimation method, which comprises the following steps: image input, feature extraction, speed information mapping, feature fusion and context association, model training and testing, performance optimization, gaze direction estimation, data enhancement and standardization, and method verification; a cascade Bayesian framework is used to map the images of the head and the body into three-dimensional directions and confidence degrees subject to von Mises-Fisher distribution. By analyzing the postures of the body and the head, the gaze direction can be estimated; in addition, two multi-branch structure feature extractors are used in the model to extract head and body features, and a Bi-LSTM recurrent neural network is combined to process the problem of gaze change in a short time. The training of the method is carried out on two public data sets GAFA and 3DPW, and the method is tested in daily monitoring images, and has high accuracy and robustness.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and pattern recognition, and particularly relates to a head-body coordination based gaze estimation method. BACKGROUND

[0002] Human gaze direction is an important indicator for analyzing human behavior, which reflects the attention level and cognitive state of various visual stimuli in the environment. In order to better understand the mechanism of human gaze, scholars need to better measure the angle of human gaze to analyze the information about human consciousness contained in the gaze, so how to realize a high-precision gaze estimation method is a research hotspot in the field of computer vision in recent years. Gaze estimation method refers to collecting human face or human eye through camera, using image processing means to obtain eye coordinates and determining eye gaze point position or gaze vector direction through calculation, which has a very wide application in human-computer interaction, virtual reality, medicine, saliency detection, video monitoring and fatigue test and many other fields. Early gaze estimation methods mainly use corneal reflection-based methods and eye model-based methods. Such methods depend on expensive special equipment and accurate data calibration (such as optical axis, corneal radius and pupil radius), and are usually limited by the surrounding environment, it is difficult to get high-resolution pictures, and the target of appearance-based gaze estimation is to learn a mapping from image I to gaze vector θ φ where θ x , φ are the yaw angle, pitch angle and roll angle, and g represents the components of the true value in the eye coordinate system [g x , g z ], respectively. Since the roll angle of the gaze is almost meaningless in practice, only the yaw angle and pitch angle of the gaze are considered in the present application. Early appearance-based methods generally learn the mapping function for individual experimenters, directly inputting human eye or human face images into a deep neural network, and the output is the corresponding gaze direction, which requires a lot of time to collect a large number of personal training samples. Although the model for individuals has good accuracy, it needs to collect a large amount of personal data each time, and the model does not have generalization. Later, most researchers used a multi-person cross-training method to solve the generalization problem of the model.

[0003] The gaze estimation in real environment reflects the interaction information of the user's eyes with the environment, and the real-time estimation of the user's gaze becomes quite difficult with the change of the environment or the user's attention. Although many deep learning-based gaze estimation models have been proposed, the existing accuracy and robustness still cannot meet the actual application requirements. Although considerable progress has been made in the research of gaze estimation, especially by using neural networks, most appearance-based methods rely too much on clear head or eye images, which cannot accurately reflect the subtle changes in a person's attention in a real scene. When the user's head is blocked or the image of the eyes is not clear, the appearance-based method relying only on the image of the eyes or the head may not be able to accurately estimate the real gaze. A standard surveillance camera may not be able to clearly capture the details of a single eye, but it is enough to record the overall person.

[0004] In view of this, a gaze estimation method based on head-body coordination is proposed, which estimates the gaze from the whole posture of the person, rather than just the head and eyes. The direction of the gaze is often closely related to the posture of the body and the head. By simulating the correlation between the head, the body and the gaze direction, the invention can estimate the general direction of the gaze when the tester's eyes are blocked or the tester is facing away from the camera. For example, when the user is attracted by something in the scene, the body or the head will rotate appropriately to keep the center of the eyes directly towards it. This behavior ensures that the vectors of the gaze, the head and the body generally point to the interesting target. Due to the coordination between the head, the body and the eyes, we can estimate the general direction of the gaze even if the eyes cannot see. In addition, the invention proposes to use a feature extractor with two multi-branch structures to solve the problem of accurately extracting direction features in background noise, and combines a Bi-LSTM recurrent neural network to solve the ambiguity caused by the change of the gaze in a short time. The invention balances the gaze direction at each time point by confidence, and designs a gaze time-varying correlation structure based on Bi-LSTM to correctly derive the relevance of the gaze in the video frame, so as to better obtain the optimal gaze prediction in a short time. SUMMARY

[0005] The invention aims to solve the above technical problems of the prior art, such as over-reliance on clear images, generalization problem, real-time and robustness, the gaze may change in a short time, and the existing method may have difficulty in handling such dynamic changes, resulting in a decrease in the accuracy of gaze estimation, and the accurate extraction of gaze direction features cannot be effectively realized in complex background noise.

[0006] To achieve the above purpose, the invention provides the following technical solutions:

[0007] A gaze estimation method based on head-body coordination, the method comprising the following steps:

[0008] S1, image input:

[0009] Input the video stream to be detected, select 7 consecutive frames of body images as a set of inputs for the body branch network, obtain the head images of the same frame number from the body images by head positioning and generating masks, and input them as the inputs of the head branch network; input the speed vector of each frame into the speed branch and map it into a tensor with the same dimension as the output of the body and head feature extractor; remove invalid background and body pixel interference to obtain clean head branch network input;

[0010] S2, feature extraction:

[0011] Body feature extraction: use a body feature extractor based on ConvNeXt to process the input body image sequence, extract features from the body image to represent the posture and motion of the body;

[0012] Head feature extraction: use a head feature extractor based on EfficientNet-B0 to process the input head image; extract features from the head image to represent the posture and direction of the head;

[0013] S3, speed information mapping:

[0014] Map the speed information of each frame in step S1 to the same dimension as the body and head features, and perform multi-scale fusion with the high-dimensional features extracted by the body and head branch networks to obtain fused features;

[0015] S4, feature fusion and context association:

[0016] Input the fused features in step S3 into the Bi-LSTM network, use Bi-LSTM to simulate the characteristics of human brain memory process, and learn the association between the previous and subsequent frames; integrate image information at multiple time points to learn the coordination of body, head and gaze in long time series;

[0017] S5, model training and testing:

[0018] Train and test the model on two public datasets GAFA and 3DPW; perform label conversion and head mask definition on the 3DPW data, and divide the training set and test set of GAFA and 3DPW;

[0019] S6, performance optimization:

[0020] Debug the head branch network under different depth coefficients and width coefficients, optimize the performance through model compounding and scaling, and introduce random depth and Kaiming initialization methods to improve the robustness of the model;

[0021] S7, gaze direction estimation:

[0022] mapping the images of the head and body into three-dimensional directions and confidence levels subject to von Mises-Fisher distribution; adjusting the three-dimensional directions of the head and body by outputting the confidence-weighted adjustment of both to obtain the final gaze output;

[0023] S8, data enhancement and standardization:

[0024] The sample data is graphically enhanced and standardized to improve the generalization ability of the model;

[0025] S9, method verification:

[0026] Method verification is performed on the collected monitoring video to obtain the body, head, and gaze yaw angle, pitch angle, roll angle, and three-dimensional angle error of the user image.

[0027] As preferred, the workflow of the body feature extractor based on ConvNeXt is as follows:

[0028] Input 7-frame sequence, dimension [32, 21, 192, 256];

[0029] Obtain high-dimensional features by downsampling;

[0030] Use GAP to average filter the channel dimension, reducing the feature to [32x21, 768];

[0031] Map to [32x21, 1280] dimension.

[0032] As preferred, the workflow of the head feature extractor based on EfficientNet-B0 is as follows:

[0033] Use the binary mask of the head bounding box to determine the head position;

[0034] Process the head image through stacked Mbconv blocks and extract high-dimensional features;

[0035] Use GAP to obtain the average features of each channel.

[0036] As preferred, in step S7, the images of the head and body are mapped into three-dimensional directions and confidence levels subject to von Mises-Fisher distribution through a cascading Bayesian framework;

[0037] The body branch network and the head branch network are responsible for extracting object features and learning a mapping of a probability distribution

[0038] where X b , X h , v, j∈[1, 7] represent the body image, head image, motion speed, and jth frame, respectivelyb / h / g = [x b / h / g , y b / h / g , z b / h / g ] respectively represent the mean direction of the body, head or gaze; ^ represents the corresponding estimated value; k b / h / g ∈ [0, +∞) respectively represent the concentration of the distribution of the body direction, head direction or gaze direction; as the value of k b / h / g increases, the likelihood of the estimated direction being μ b / h / g also increases;

[0039] The von Mises-Fisher distribution of the gaze is formulated as follows:

[0040]

[0041] where, represent the 7-frame direction distribution of the body, head and gaze after Bi-LSTM respectively; FC g represents the fully connected layer that outputs the gaze probability distribution; = Concat represents the output of the two branch networks after confidence k b / h weighting and splicing.

[0042] As a preference, the probability mapping process based on the Bayesian framework is as follows:

[0043] The von Mises-Fisher distribution is a statistical distribution of a spherical direction, taking the estimated value of the direction as the central value of the von Mises-Fisher distribution, and calculating the maximum likelihood value between it and the true value of the direction μ and the concentration estimate value of the distribution

[0044] When k is 0, the directions of the body, head and gaze are uniformly distributed;

[0045] When k , the directions of the body, head and gaze follow the Dirac distribution; at this time, the estimated direction of the model is exactly consistent with the ground truth μ;

[0046] The general definition of the von Mises-Fisher distribution is shown in the following formula;

[0047]

[0048] where, is the modified Bessel function of order k ; μ represents the prior true value of the direction; and respectively represent the direction and confidence output of our model; then, the modified Bessel equation is derived as follows:

[0049] wherein,! denotes factorial, For the convenience of calculation, we set to 3 to push out the probability distribution used in the model designed by the application, that is, the input image and the velocity vector will be mapped to a three-dimensional direction subject to the following formula:

[0050]

[0051] The cascade network parameters are learned by using maximum likelihood estimation; according to the formula The loss function is defined as the negative log-likelihood value of the vMF distribution, as shown in the following formula:

[0052] wherein, sinh is the hyperbolic sine function, and ln denotes the logarithm with natural base.

[0053] As preferred, the training and testing of the model are carried out on two public data sets GAFA and 3DPW, and the testing is carried out in the monitoring images collected daily; the detailed description of the training mode is as follows:

[0054] Dataset description:

[0055] GAFA dataset: containing 882,000 frames of monitoring video, capturing the behavior of free movement in 5 daily environments, including kitchen, library, laboratory, living room and courtyard, the GAFA dataset provides three-dimensional gaze direction, head and body annotation;

[0056] 3DPW dataset: composed of handheld videos, containing rich ground, real 3D human posture and shape, the part used for training includes video sequence and IMU (inertial measurement unit) data to provide additional motion information;

[0057] Image preprocessing: since the original label of the 3DPW dataset is not suitable for the current task and the head mask is not defined, label conversion and head mask definition are required; the training set and test set of the GAFA and 3DPW datasets are divided; label normalization and image standardization are carried out to obtain the body and head images of each user; the image size in the 3DPW dataset is adjusted to the same size as the GAFA dataset ([3x256x192]);

[0058] Model training environment: distributed data parallel training is used on two 12G NVIDIA RTX 3080Ti GPUs to improve the training efficiency;

[0059] Training parameter settings: In the PyTorch framework, the Adam optimizer is used for training; the batch size is set to 32, that is, 32 samples are processed each time the training iteration is performed; the learning rate is set to 0.0001, which controls the step size of parameter updates; the decay rate is set to 0.0001, which gradually reduces the learning rate as the training progresses, to help the model converge; the training period (epoch) is set to 100, that is, the model is trained on the entire training set for 100 times; these training parameters and settings ensure that the model can effectively learn from the data and achieve good performance on the gaze estimation task.

[0060] As a preferred, in step S9, when testing and verifying the model using daily monitoring videos, the specific requirements to be met are as follows:

[0061] Video duration: The video duration captured by the camera needs to be greater than 3 seconds to ensure that enough image sequences of sufficient duration can be obtained, which can ensure that the model has enough data to estimate the gaze direction, especially in dynamic scenes.

[0062] Head position: Head position information needs to be provided to generate a head mask, which is very important for extracting head images, as it helps to accurately separate the head region from the body image, thereby providing input for the head feature extractor;

[0063] Moving speed vector: If the moving speed vector of the target person in each frame of image cannot be provided, set the speed vector to 0, which can simplify the model input, but may affect the accuracy of the model in dynamic environments.

[0064] Preprocessing module:

[0065] The preprocessing module is used to perform the following tasks:

[0066] Crop 7 frames x 3 video sequences that meet the preset rules from each video, where 7 frames represent 7 consecutive images, and 3 represents 3 consecutive video channels (e.g. RGB), forming a complete image sequence; all images in the cropped sequence are standardized; standardization usually includes adjusting the pixel values of the images to the range of 0 to 1 to help the model converge faster and improve performance.

[0067] As a preferred, in step S4, in the correlation of the front and back frame image features, the structure of Bi-LSTM contains two LSTM layers, and the specific principle of LSTM is as follows:

[0068]

[0069] where x t is the input data matrix of the layer, σ, tanh is the activation function, Wf , W i , W c is a weight matrix, b o , b C , b i is each bias, h is the output of each layer of LSTM; the input data matrix x t and the hidden output h t-1 of the previous LSTM cell;

[0070] First, the forgetting gate f t filters out the features related to the gaze direction of the previous frame but obviously irrelevant to the current target; while the memory gate is used g t and i t function to extract the line of sight information in the input x t of the current moment and the hidden layer input h t-1 of the previous moment;

[0071] Then, follow the formula C t = f t *C t-1 +i t *g t to control these memories, that is, after merging the information of the forgetting gate and the input gate, the cell state C t of the current moment is obtained;

[0072] Next, through the output gate o t = σ(W o ·[h t-1 ,x t ]+b o ) to get a normalized weight o t ;

[0073] Finally, according to the formula h t = o t *tanh(C t ) to get the hidden output h t of the current cell, and after merging the hidden layer outputs of the other 6 LSTM cells, the output containing context information is obtained and sent to a channel attention layer to enhance the time correlation between features.

[0074] Compared with the prior art, the technical effects and advantages of the present application are:

[0075] The head-body coordination-based gaze estimation method proposes to input 7 frames of body images and head mask images as inputs into a body feature extractor based on ConvNeXt and a head feature extractor based on EfficientNet-B0 to extract high-dimensional features of the images, and map the speed vectors of each frame through a linear layer and splice the outputs of the two feature extractors; then the context information of the 7 frames of face images is obtained by using Bi-LSTM and channel attention layer; then the three-dimensional gaze direction and confidence of the head and body are solved through two fully connected layers; finally, the directions are spliced after being adjusted by the confidence of the body and the head, and then input into Bi-LSTM and fully connected layer to output the final gaze direction and confidence. The present application also uses a loss function vmf loss based on von Mises-Fisher distribution to solve the problem that the update gradient of the traditional MAE or MSE loss function is always the same, which leads to local optimization or non-convergence;

[0076] The head-body coordination-based gaze estimation method models the direction of the gaze as a three-dimensional von Mises-Fisher distribution, and represents the uncertainty of the estimation through the concentration of the distribution. We use multiple datasets to evaluate our method and test it in collected surveillance videos. The results show that the method proposed in the present application can stably estimate the gaze direction in various situations, even if the target person is in a poor viewing angle or is blocked by an obstacle. Under the conditions of meeting the real-time, stability, accuracy and universality, the technical problems such as over-reliance on clear eye images and slow training speed of traditional methods are solved, and the most effective gaze estimation is realized by using only the resources of the terminal device. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 The flowchart of the head-body coordination-based gaze estimation method of the present application;

[0078] Figure 2 The model structure diagram of the head-body coordination-based gaze estimation method of the present application;

[0079] Figure 3 The visualization UI interface diagram of the present application. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0081] The following will be described in conjunction with the drawings in the embodiments of the present application, Figures 1-3Further details of the application are described below,

[0082] The embodiment of the application discloses a head-body coordination based gaze estimation method, which comprises the following steps:

[0083] S1, image input:

[0084] Input the video stream to be detected, select 7 consecutive body images as a set of inputs of the body branch network, obtain the head images of the same frame number from the body images by head positioning and generating a mask, and input the head images as the inputs of the head branch network; input the speed vector of each frame into the speed branch and map it into a tensor with the same dimension as the output of the body and head feature extractor; remove invalid background and body pixel interference to obtain clean head branch network input;

[0085] S2, feature extraction:

[0086] Body feature extraction: a body feature extractor based on ConvNeXt is used to process the input body image sequence, and features are extracted from the body image to represent the posture and motion of the body;

[0087] Head feature extraction: an EfficientNet-B0-based head feature extractor is used to process the input head image; features are extracted from the head image to represent the posture and direction of the head;

[0088] S3, speed information mapping:

[0089] Map the speed information of each frame in step S1 to the same dimension as the body and head features, and perform multi-scale fusion with the high-dimensional features extracted by the body and head branch networks to obtain fused features;

[0090] S4, feature fusion and context association:

[0091] Input the fused features in step S3 into a Bi-LSTM network, use Bi-LSTM to simulate the characteristics of the human brain memory process, and learn the association between the previous and subsequent frames; integrate image information at multiple time points to learn the coordination of the body, head and gaze in a long time sequence;

[0092] S5, model training and testing:

[0093] Train and test the model on two public datasets GAFA and 3DPW; perform label conversion and head mask definition on the 3DPW data, and divide the training set and test set of GAFA and 3DPW;

[0094] S6, performance optimization:

[0095] Debugging head branch network under different depth coefficients and width coefficients, optimizing performance through model composite scaling, introducing random depth and Caymin initialization method to improve model robustness;

[0096] S7, line-of-sight direction estimation:

[0097] Mapping the images of the head and body into three-dimensional directions and confidence levels subject to von Mises-Fisher distribution; adjusting the three-dimensional directions of the head and body by outputting the confidence levels of the head and body to obtain the final gaze output;

[0098] S8, data augmentation and standardization:

[0099] Performing graphical enhancement and standardization processing on sample data to improve the generalization ability of the model;

[0100] S9, method verification:

[0101] Method verification is performed on the collected monitoring video to obtain the body, head and gaze yaw angle, pitch angle, roll angle and three-dimensional angle error of the user image.

[0102] For the complex background and body pixels of different sizes in the collected video, the body feature extractor is designed based on ConvNeXt. By introducing random depth, the branch structure in the extractor is deleted with a certain probability to ensure that the extractor can learn information of different depths and improve the robustness of the model;

[0103] The high-dimensional features of the head are extracted by the branch network based on EfficientNet-B0. The same random depth as in the body feature extractor is also used to make the branch network learn head features of different depths. At the same time, the head branch network based on EfficientNet-B0 under different depth coefficients and width coefficients is debugged, and the composite scaling of the model is used to ensure that the head branch network can obtain appropriate fine-grained and depth features and improve the accuracy of the overall model;

[0104] The head-body coordination-based gaze estimation method adopts a ConvNeXt-based body feature extractor and an EfficientNet-B0-based head feature extractor with a multi-branch structure, which helps to obtain deeper significant features and facilitates the use of random depth to probabilistically delete a small part of the branch to explore the performance of the model at different depths. The multi-branch structure of each ConvNeXt block enables the network to integrate features from different scales. At the same time, each sub-path of the ConvNeXt block has a certain chance of being deleted. Therefore, the body extractor of the present application has strong robustness for fusing body pixels of different depths and different scales (whether the tester is close to the camera). In addition, the use of depth separable convolution and droppath will facilitate the lightweight design of the model.

[0105] Specifically, the working process of the ConvNeXt-based body feature extractor is as follows:

[0106] Input 7 frames of sequence, dimension [32, 21, 192, 256];

[0107] Obtain high-dimensional features by downsampling;

[0108] Use GAP to average filter the channel dimension, reducing the feature to [32x21, 768];

[0109] Map to [32x21, 1280] dimension.

[0110] For the input of the above head extractor, the present application uses the binary mask of the head bounding box to determine the position of the head. First, the head image is processed by the stacked Mbconv block and high-dimensional features are extracted, and then the same GAP as the body extractor is used to obtain the average features of each channel. Our head extractor is mainly designed based on EfficientNet-B0. The depth convolution used in each MBConv module of EfficientNet-B0 greatly reduces the number of parameters of the head extractor. We use the comprehensive scaling method of EfficientNet-B0 to coordinate the width and depth, so as to extract features with appropriate granularity and depth. Similar to the body extractor, we use random depth to probabilistically delete branches in each Mbconv block to study its performance at different depths. The use of random depth can prevent overfitting while ensuring the robustness of our model on head images of different scales.

[0111] Specifically, the working process of the head feature extractor based on EfficientNet-B0 is as follows:

[0112] Determine the head position using the binary mask of the head bounding box;

[0113] The head image is processed and high-dimensional features are extracted by stacked Mbconv blocks;

[0114] The average feature of each channel is obtained using GAP.

[0115] The gaze estimation method based on head-body coordination has been trained on the GAFA dataset and the 3DPW dataset, which contains tens of thousands of labeled images, dozens of users, and dozens of indoor and outdoor scenes. The invention not only focuses on the appearance of the face and eyes, but also considers the overall posture of a person. The cascaded Bayesian framework designed by the invention can map the images of the head and body into three-dimensional directions and confidence that obey the von Mises-Fisher distribution. The cascaded network designed by the invention encodes the relevance of the gaze direction in each frame to the past and future head and body directions in the time scale. The influence of the three-dimensional directions of the head and body on the final gaze output is adjusted by the confidence weighting output.

[0116] In step S7, the images of the head and body are mapped into three-dimensional directions and confidence that obey the von Mises-Fisher distribution by the cascaded Bayesian framework;

[0117] The body branch network and the head branch network are responsible for extracting object features and learning a mapping of a probability distribution

[0118] where X b , X h , v, j ∈ [1, 7] represent the body image, head image, motion speed and the j-th frame respectively; μ b / h / g = [x b / h / g , y b / h / g , z b / h / g ] represent the average direction of the body, head or gaze respectively; ^ represents the corresponding estimated value; k b / h / g ∈ [0, +∞) represent the distribution concentration of the body direction, head direction or gaze direction respectively; as the value of k b / h / g increases, the likelihood of the estimated direction being μ b / h / g also increases;

[0119] The formula of the von Mises-Fisher distribution of the gaze is as follows:

[0120]

[0121] where, represent the 7-frame direction distribution of the body, head and gaze after Bi-LSTM respectively; FC g represents the fully connected layer that outputs the gaze probability distribution; = Concat represents the two branch network outputs spliced by confidence k b / h weighting.

[0122] Specifically, the probability mapping process based on the Bayesian framework is as follows:

[0123] The Von Mises-Fisher distribution is a spherical directional statistical distribution, taking the estimated value of the direction as the central value of the Von Mises-Fisher distribution, and calculating the maximum likelihood value and the concentration estimate value of the distribution between the direction and the real value μ of the ground

[0124] When is 0, the directions of the body, head and gaze are uniformly distributed;

[0125] When , the directions of the body, head and gaze follow the Dirac distribution; however, in real dynamic scenes, due to the problems of occlusion of the head and body, incomplete input, background interference, etc., the Von Mises-Fisher distribution cannot be completely converted into the Dirac distribution. However, the purpose of the present application is to design a cascaded Bayesian framework to construct a probability model in which the body, head and gaze are as close as possible to the real direction distribution, and the concentration estimate value output by the method of the present application reaches 2.6 or more. This means that the direction estimate value output by the model designed in the present application is very likely to be near the real value, and the output direction approximately obeys the Dirac distribution. At this time, the estimated direction of the model is exactly the same as the ground truth μ;

[0126] The general definition of the Von Mises-Fisher distribution is as follows:

[0127]

[0128] wherein, is the modified Bessel function of order ; μ represents the prior real value of the direction; and respectively represent the direction and confidence output of our model; then, the modified Bessel equation is derived as follows:

[0129] wherein,! represents the factorial, In order to simplify the calculation, we set to 3 to derive the probability distribution used in the model designed in the present application, i.e., the input image and velocity vector will be mapped to a three-dimensional direction obeying the following equation:

[0130]

[0131] The cascade network parameters are learned using maximum likelihood estimation; according to the formula The loss function is defined as the negative log-likelihood of the vMF distribution, as shown in the following formula:

[0132] Where sinh is the hyperbolic sine function, and ln represents the logarithm with natural base.

[0133] As a preferred, the training and testing of the model are carried out on two public datasets, GAFA and 3DPW, and the testing is carried out on the monitoring images collected daily; the detailed description of the training mode is as follows:

[0134] Dataset description:

[0135] GAFA dataset: contains 882,000 frames of surveillance video, capturing behaviors of free movement in 5 daily environments, including kitchen, library, laboratory, living room and courtyard, the GAFA dataset provides three-dimensional gaze direction, head and body annotations;

[0136] 3DPW dataset: composed of handheld videos, containing rich ground, real 3D human poses and shapes, the part used for training includes video sequences and IMU (Inertial Measurement Unit) data to provide additional motion information;

[0137] Image preprocessing: since the original labels of the 3DPW dataset are not suitable for the current task and the head mask is not defined, label conversion and head mask definition are needed; the training set and test set of GAFA and 3DPW datasets are divided; label normalization and image standardization are performed to obtain the body and head images of each user; the image size in the 3DPW dataset is adjusted to the same size as the GAFA dataset ([3x256x192]);

[0138] Model training environment: distributed data parallel training is used on two 12G NVIDIA RTX 3080Ti GPUs to improve training efficiency;

[0139] Training parameter settings: training is carried out using Adam optimizer in PyTorch framework; batch size is set to 32, i.e. 32 samples are processed per training iteration; learning rate is set to 0.0001, which is used to control the step size of parameter update; decay rate is set to 0.0001, which is used to gradually reduce the learning rate as the training progresses, to help the model converge; training period (epoch) is set to 100, i.e. the model is trained completely on the entire training set for 100 times; these training parameters and settings ensure that the model can effectively learn from the data and achieve good performance on the gaze estimation task.

[0140] Specifically, in step S9, when using daily monitoring videos to test and verify the model, the specific requirements to be met are as follows:

[0141] Video duration: The video duration captured by the camera needs to be greater than 3 seconds to ensure that enough image sequences of sufficient duration can be obtained, which can ensure that the model has enough data to estimate the gaze direction, especially in dynamic scenes.

[0142] Head position: Head position information needs to be provided to generate a head mask, which is very important for extracting head images, as it helps to accurately separate the head region from the body image, thereby providing input for the head feature extractor;

[0143] Moving speed vector: If the moving speed vector of the target person in each frame of image cannot be provided, set the speed vector to 0, which can simplify the model input, but may affect the accuracy of the model in dynamic environments.

[0144] Preprocessing module:

[0145] The preprocessing module is used to perform the following tasks:

[0146] Crop 7 frames x 3 video sequences that meet the preset rules from each video, where 7 frames represent 7 consecutive images, and 3 represents 3 consecutive video channels (e.g. RGB), forming a complete image sequence; standardize all images in the cropped sequence; standardization usually includes adjusting the pixel values of the images to the range of 0 to 1 to help the model converge faster and improve performance.

[0147] Specifically, in step S4, when correlating the features of the front and back frames, the structure of Bi-LSTM contains two LSTM layers, and the specific principle of LSTM is as follows:

[0148]

[0149] where x t is the input data matrix of the layer, σ, tanh is the activation function, W f , W i , W C are weight matrices, b o , b C , b i are the biases of each item, h is the output of each LSTM layer; the input data matrix x t and the hidden output h t-1 of the previous LSTM cell.

[0150] First, the forget gate f tFeatures that are related to the gaze direction of the previous frame but are clearly unrelated to the current target are filtered out; while memory gates use g... t and i t The function will input x at the current time. t The input h of the hidden layer at the previous time step t-1 Extract the gaze information from the image;

[0151] Then, following formula C t =f t *C t-1 +i t *g t To control these memories, that is, to obtain the current cell state C by merging the information from the forget gate and the input gate. t ;

[0152] Next, through the output gate o t =σ(W o ·[h t-1 ,x t +b o A normalized weight o is obtained t ;

[0153] Finally, according to formula h t =o t *tanh(C t Get the hidden output h of the current cell. t After merging the hidden layer outputs of the other 6 LSTM cells, an output containing contextual information is obtained. It is then fed into a channel attention layer to enhance the temporal correlation between features.

[0154] like Figure 2 As shown, the gaze estimation network consists of three modules: a body branch network based on ConvNeXt and Bi-LSTM, a head branch network based on EfficientNet-B0 and Bi-LSTM, and a gaze estimation module based on confidence-weighted feature fusion. The body branch network based on ConvNeXt and Bi-LSTM removes background noise from the image, extracts high-dimensional body orientation features, and learns the contextual relationships between consecutive frames. The head branch network based on EfficientNet-B0 and Bi-LSTM employs a comprehensive scaling method to balance width and depth, thereby extracting features with appropriate fine-grainedness and depth and connecting the contextual information of seven consecutive frames. Finally, the obtained body and head vectors are weighted by confidence and fed into the gaze estimation module based on fully connected layers and Bi-LSTM to output a 3D gaze that follows a von Mises-Fischer distribution.

[0155] like Figure 2As shown, the body feature extractor based on ConvNeXt mainly consists of 4 ConvNeXt blocks, a normalization layer, a down-sampling layer, a global average pooling and a linear layer. The body orientation features extracted by the extractor are sent to a bidirectional long short-term memory network (Bi-LSTM) after being fused with other high-dimensional features to obtain the correlation of the front and rear frames. In order to calculate the spatial attention, the output layer of the branch network is globally averaged along the channel axis to generate an efficient feature vector. It has been proved that the pooling operation along the channel axis can effectively highlight the significant information of the features. In addition, the present application also uses random depth probabilistic deletion of ConvNeXt blocks to improve the generalization of the model.

[0156] For the input of the head extractor of the present application, a binary mask of the head bounding box is used to determine the position of the head. The head image is first processed by a stacked Mbconv block to extract high-dimensional features, and then the same global average pooling as the body extractor is used to obtain the average features of each channel. The depth convolution used in the MBConv module greatly reduces the number of parameters of the head extractor. We use the comprehensive scaling method of efficient net to coordinate the width and depth, so as to extract features with fine granularity and depth. Similar to the body extractor, the present application uses droppath to randomly delete branches in each Mbconv block, allowing the model to learn the orientation features at different depths. The use of random depth can prevent overfitting while ensuring that the model designed by the present application is robust to head images of different scales;

[0157] The present application uses the GAFA dataset and the 3DPW dataset to train and test the entire model, and verifies the method on the collected surveillance video data. As shown in Figure 3 As shown, the visualization images of the body, head and line-of-sight three-dimensional orientation of each frame of target person are shown. As shown in Figure 3 As shown, the UI interface designed by the present application can output the visualization of each frame and the yaw angle, pitch angle and angle error of the body, head and line-of-sight. As shown in Table 1, the angle error obtained by the method described in the present embodiment is the smallest compared with the currently popular line-of-sight estimation methods on the GAFA dataset.

[0158] Table 1 Comparison of the body, head and line-of-sight angle error of the line-of-sight estimation method based on head-body coordination of the present application and other methods on the GAFA dataset

[0159]

[0160] In the data input of the present application, the input video is divided into multiple 7-frame 3-channel image sequences, and a batch of 32 is used, that is, a body picture with a shape of [32x21x192x256] size. The head image with the same shape is obtained by the head mask. The input is trained in the proposed gaze estimation model, and the sample data is also subjected to graphic enhancement and standardization processing before training. The present application can also selectively determine whether to input the velocity vector. The head-body coordination-based gaze estimation method proposed in the present application achieves an angle error of less than 22.6° and a confidence of up to 2.62 on the 3DPW and GAFA test sets.

[0161] In the extraction of body features and head features of the present application, the traditional convolutional layer or fully connected layer has more or less information loss problem in information transmission. ConvNeXt and EfficientNet-B0 are a kind of deep convolutional neural network with residual structure, which can effectively alleviate the gradient dispersion problem caused by network depth, and can learn more complex and rich image feature representation, which is beneficial to the gaze estimation task. And ConvNeXt and EfficientNet-B0 both use depth separable convolution and droppath, which will be conducive to the lightweight design of the model. For real-time gaze estimation tasks, global spatial information and context information are also very important, therefore the present application proposes to use the inherent SE attention mechanism of EfficientNet-B0 and the depth separable convolution of ConvNeXt to obtain the features of the direction vector contained in the image, and use Bi-LSTM combined with feature fusion to obtain the context information of the gaze.

[0162] Finally, it should be noted that: the above only for the preferred embodiments of the present application, and not for limiting the present application, although the present application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, it still can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features, any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application, should be included in the protection scope of the present application.

Claims

1. A head-body coordination based line-of-sight estimation method, characterized by, The method comprises the following steps: S1, image input: Input the video stream to be detected, select 7 consecutive body images as a set of inputs for the body branch network, obtain the head images of the same frame number from the body images by head positioning and generating a mask, and input the head images as inputs for the head branch network; input the speed vector of each frame into the speed branch and map it into a tensor with the same dimension as the output of the body and head feature extractors; S2, feature extraction: Body feature extraction: use a body feature extractor based on ConvNeXt to process the input body image sequence, extract features from the body images to represent the posture and movement of the body; Head feature extraction: use a head feature extractor based on EfficientNet-B0 to process the input head image; extract features from the head image to represent the posture and direction of the head; S3, speed information mapping: Map the speed information of each frame in step S1 to the same dimension as the body and head features, and perform multi-scale fusion with the high-dimensional features extracted by the body and head branch networks to obtain fused features; S4, feature fusion and context association: Input the fused features in step S3 into a Bi-LSTM network, use Bi-LSTM to simulate the characteristics of the human brain memory process, and learn the association between the previous and subsequent frames; integrate image information at multiple time points to learn the coordination of the body, head, and gaze in a long time sequence; S5, model training and testing: Train and test the model on two public datasets GAFA and 3DPW; perform label conversion and head mask definition on the 3DPW data, and divide the training set and test set of GAFA and 3DPW; S6, performance optimization: Debug the head branch network under the depth coefficient and width coefficient, optimize the performance through model scaling, and introduce random depth and Kaiming initialization methods to improve the robustness of the model; S7, gaze direction estimation: Map the head and body images to three-dimensional directions and confidence levels that follow the von Mises-Fisher distribution; adjust the three-dimensional directions of the head and body by weighting their confidence levels to obtain the final gaze output; S8, data augmentation and standardization: Perform graphical enhancement and standardization on the sample data to improve the generalization ability of the model; S9, method verification: Verify the method on the collected monitoring video, and obtain the yaw angle, pitch angle, roll angle, and three-dimensional angle error of the user's body, head, and gaze.

2. The head-body coordination based line-of-sight estimation method of claim 1, wherein: The working process of the body feature extractor based on ConvNeXt is as follows: Input a 7-frame sequence with a dimension of [32, 21, 192, 256]; Obtain high-dimensional features by downsampling; Use GAP to average filter the channel dimension and reduce the features to [32×21, 768]; Map to a dimension of [32×21, 1280].

3. The head-coordination based line-of-sight estimation method of claim 1, wherein: The working process of the head feature extractor based on EfficientNet-B0 is as follows: Use the binary mask of the head bounding box to determine the head position; Process the head image through stacked Mbconv blocks and extract high-dimensional features; The average feature of each channel is obtained using GAP.

4. The head-coordination-based line-of-sight estimation method of claim 1, wherein: In step S7, the images of the head and body are mapped into three-dimensional directions and confidence levels subject to von Mises-Fisher distribution through a cascaded Bayesian framework; The body branch network and the head branch network are responsible for extracting object features and learning a mapping of a probability distribution where X b , X h , v, j ∈ [1, 7] represent the body image, head image, motion speed and the j-th frame, respectively; μ b / h / g = [x b / h / g , y b / h / g , z b / h / g ] represent the average direction of the body, head or gaze, respectively; ^ represents the corresponding estimated value; k b / h / g ∈ [0, +∞) represent the distribution concentration of the body direction, head direction or gaze direction, respectively; as the value of k b / h / g increases, the likelihood of the estimated direction being μ b / h / g also increases; The formula of the von Mises-Fisher distribution of gaze is as follows: where, represent the 7-frame orientation distribution of the body, head and gaze after Bi-LSTM, respectively; FC g denotes a fully connected layer that outputs the gaze probability distribution; Concat denotes the weighted concatenation of the outputs of the two branch networks. b / h weighted concatenation.

5. The head-coordination-based line-of-sight estimation method of claim 4, wherein: The probability mapping process based on the Bayesian framework is as follows: The von Mises-Fisher distribution is a statistical distribution on the surface of a sphere with an estimated value of the direction The central value of the von Mises-Fisher distribution is taken as the true value of the direction μ and the maximum likelihood value and the concentration estimate of the distribution are found When For 0, the body, head and gaze direction are uniformly distributed; When the body, head and gaze direction follow a Dirac distribution; at this time, the estimated direction of the model coincides exactly with the ground truth μ; The general definition of the von Mises-Fisher distribution is as follows: where, is the modified Bessel function of order ; m represents the prior true value of the direction; and and denote the direction and confidence output of our model, respectively; then, plugging into the modified Bessel equation yields the following equation: where! denotes the factorial, The Set to 3 pushes out the probability distribution employed by the design model, and the input image and velocity vector will be mapped as obeying the three-dimensional direction shown in the following equation; The cascade network parameters are learned using maximum likelihood estimation; according to the formula The loss function is defined as the negative log-likelihood of the vMF distribution, as shown in the following formula: Wherein, sinh is the hyperbolic sine function, and ln represents the logarithm with natural base.

6. The head-coordination-based line-of-sight estimation method of claim 1, wherein: The training and testing of the model are carried out on two public data sets GAFA and 3DPW, and the testing is carried out on the monitoring images collected in daily life; the training mode is as follows: Dataset description: GAFA dataset: contains 882,000 frames of monitoring video, captures the behavior of free movement in 5 daily environments, including kitchen, library, laboratory, living room and courtyard, the GAFA dataset provides three-dimensional gaze direction, head and body annotation; 3DPW dataset: composed of handheld videos, containing rich ground, real 3D human posture and shape, the part used for training includes video sequences and IMU data; Image preprocessing: divide the training set and test set of GAFA and 3DPW datasets; perform label normalization and image standardization to obtain the body and head images of each user; adjust the image size in the 3DPW dataset to the same size as the GAFA dataset ([3x256x192]); Model training environment: use distributed data parallel training on two 12G NVIDIA RTX 3080Ti GPUs; Training parameter setting: use Adam optimizer in PyTorch framework for training; batch size is set to 32, learning rate is set to 0.0001, decay rate is set to 0.0001, and training period is set to 100.

7. The head-coordination based line-of-sight estimation method of claim 1, wherein: In step S9, when testing and verifying the model using daily monitoring video, the specific requirements to be met are as follows: Video duration: the video duration collected by the camera needs to be greater than 3 seconds; Head position: head position information needs to be provided to generate a head mask; Moving speed vector: if the moving speed vector of the target person in each frame of image cannot be provided, set the speed vector to 0; Preprocessing module: the preprocessing module is used to perform the following tasks: crop 7 frames x 3 video sequences that meet the preset rules from each video to form a complete image sequence; standardize all images in the cropped sequence.

8. The head-coordination-based line-of-sight estimation method of claim 1, wherein: In step S4, in the association of the features of the front and rear frame images, the structure of Bi-LSTM contains two LSTM layers, and the specific principle of LSTM is as follows: where x t is the input data matrix of the layer, σ, tanh are activation functions, W f , W i , W C are weight matrices, b o , b C , b i are the respective biases, and h is the output of each LSTM layer; the input data matrix x t and the hidden output h t-1 of the previous LSTM cell. First, the forget gate f t filters out the features related to the gaze direction of the previous frame but apparently irrelevant to the current target; while the memory gate g t filters out the features related to the gaze direction of the previous frame but apparently irrelevant to the current target; while the memory gate g t extracts the gaze information from the current input x t and the input h t-1 of the hidden layer at the previous time step. Then, the formula C t = f t * C t-1 + i t * g t is followed to control these memories, i.e. the information of the merge forget gate and the input gate is obtained after the current time cell state C t ; Next, a normalized weight o t = σ(W o · [h t-1 , x t ]+b o ) is obtained t ; Finally, the hidden output h t of the current cell is obtained according to the formula h t = o t *tanh(C t ) and the output containing context information is obtained after merging the hidden layer outputs of the other 6 LSTM cells and sending it to a channel attention layer to enhance the time correlation between features.

Images