Sight line estimation method based on head-body coordination

Abstract

The invention relates to the technical field of computer vision and pattern recognition, and discloses a head-body coordination-based sight line estimation method, which comprises the following steps of image input, feature extraction, speed information mapping, feature fusion and context association, model training and testing, performance optimization, sight line direction estimation, data enhancement and standardization and method verification. Using a cascaded Bayesian framework, images of the head, body are mapped to a three-dimensional direction and confidence compliant with the von Mises-Fisher distribution. By analyzing the body and head postures, the sight line direction can be estimated, in addition, the model adopts two feature extractors of a multi-branch structure to extract head and body features, and a Bi-LSTM recurrent neural network is combined to solve the problem that the sight line changes in a short time. Training of the method is carried out on two public data sets GAFA and 3DPW, testing is carried out in daily monitoring images, and high accuracy and robustness are achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and pattern recognition, and in particular relates to a sight line estimation method based on head-body coordination. Background Art

[0002] The direction of human gaze is an important indicator for analyzing human behavior, which reflects the attention level and cognitive state of various visual stimuli in the environment. In order to better understand the mechanism of human gaze, scholars need to better measure the human gaze angle to analyze the information of human consciousness contained in the gaze. Therefore, how to achieve a high-precision gaze estimation method has been a research hotspot in the field of computer vision in recent years. The gaze estimation method refers to the acquisition of human faces or eyes by cameras, the use of image processing methods, the acquisition of eye coordinates and the determination of the position of the human eye gaze point or the direction of the gaze vector by calculation. This technology has been widely used in many fields such as human-computer interaction, virtual reality, medicine, saliency detection, video surveillance and fatigue testing. Early gaze estimation methods mainly used methods based on corneal reflection and methods based on eyeball models. Such methods rely on expensive special equipment and precise data calibration (such as optical axis, corneal radius and pupil radius), and are usually limited by the influence of the surrounding environment. It is difficult to obtain high-resolution images. The goal of appearance-based gaze estimation is mainly to learn an image I to the gaze vector Mapping Among them, φ is the yaw angle, pitch angle and roll angle, and g represents the angle in the eye coordinate system [g x , g x , g z ] is the component of the true value in the image. Since the roll angle of the line of sight is almost meaningless in practice, only the yaw angle and pitch angle of the line of sight are considered in the present invention. Early appearance-based methods generally learn the mapping function for the individual experimenter, directly input the human eye or face image into the deep neural network, and the output is the corresponding line of sight direction. This requires time to collect a large number of personal training samples. Although the model for individuals has good accuracy, each use requires a large amount of personal data collection, and the model is not generalizable. Later, most researchers used multi-person cross-training methods to solve the generalization problem of the model.

[0003] Gaze estimation in real environments reflects the interaction information between the user's eyes and the environment. As the environment or the user's attention changes, real-time estimation of the user's gaze becomes quite difficult. Although many deep learning-based gaze estimation models have been proposed, the existing accuracy and robustness still cannot meet the needs of practical applications. Although considerable progress has been made in gaze estimation research, especially by using neural networks, most appearance-based methods overly rely on clear head or eye images, which cannot accurately reflect the subtle changes in a person's attention in real scenes. When the user's head is occluded or the image of the eye is unclear, appearance-based methods that only rely on images of the eye or head may not accurately estimate the real gaze. A standard surveillance camera may not be able to clearly capture the details of a single eye, but it is sufficient to record a person as a whole.

[0004] In view of this, a method for estimating gaze based on head-body coordination is proposed. The present invention estimates gaze from the posture of the whole person, not just the head and eyes. The direction of gaze is often closely related to the posture of the body and head. By simulating the correlation between the head, body and gaze direction, the present invention can estimate the approximate direction of gaze when the tester's eyes are blocked or the tester is facing away from the camera. For example, when the user is attracted by something in the scene, the body or head will rotate appropriately to keep the center of the eyes directly facing it. This behavior ensures that the vectors of gaze, head and body roughly point to the interesting target. Due to the coordination between the head, body and eyes, we can estimate the general direction of gaze even if the eyes cannot see. In addition, the present invention proposes a feature extractor with two multi-branch structures to solve the problem of difficulty in accurately extracting directional features in background noise, and combines the Bi-LSTM recurrent neural network to solve the ambiguity caused by the change of sight in a short time. The present invention weighs the sight direction at each time point by confidence, and designs a sight time-varying correlation structure based on Bi-LSTM to correctly derive the correlation of sight in video frames, so as to better obtain the optimal sight prediction in a short time. Summary of the invention

[0005] The present invention aims to solve the technical problems of excessive reliance on clear images, generalization problems, lack of real-time performance and robustness in the above-mentioned prior arts; the line of sight may change in a short period of time, and the existing methods may find it difficult to handle such dynamic changes, resulting in a decrease in the accuracy of line of sight estimation, and the inability to effectively extract line of sight direction features in complex background noise.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A line of sight estimation method based on head-body coordination, the method comprising the following steps:

[0008] S1. Image input:

[0009] Input the video stream to be detected, select 7 consecutive frames of body images as a set of inputs of the body branch network, obtain head images of the same number of frames from the body images by locating the head and generating masks, and use them as inputs of the head branch network; input the velocity vector of each frame into the velocity branch and map it to a tensor of the same dimension as the output of the body and head feature extractors; remove invalid background and body pixel interference to obtain a clean input for the head branch network;

[0010] S2. Feature extraction:

[0011] Body feature extraction: Use a ConvNeXt-based body feature extractor to process the input body image sequence and extract features from the body image to represent the body posture and movement;

[0012] Head feature extraction: Use the head feature extractor based on EfficientNet-B0 to process the input head image; extract features from the head image to characterize the posture and direction of the head;

[0013] S3, speed information mapping:

[0014] Map the speed information of each frame in step S1 to the same dimension as the body and head features, and perform multi-scale fusion with the high-dimensional features extracted by the body and head branch networks to obtain fused features;

[0015] S4. Feature fusion and context association:

[0016] Input the fused features in step S3 into the Bi-LSTM network, and use the Bi-LSTM to simulate the characteristics of the human brain memory process to learn the correlation between the previous and next frames; integrate the image information at multiple moments to learn the coordination of the body, head and gaze in a long time series;

[0017] S5. Model training and testing:

[0018] The model is trained and tested on two public datasets, GAFA and 3DPW. The 3DPW data is labeled and converted, the head mask is defined, and the training and test sets are divided for GAFA and 3DPW.

[0019] S6. Performance optimization:

[0020] Debug the head branch network under different depth coefficients and width coefficients, optimize the performance through compound scaling of the model, and introduce random depth and Kaiming initialization methods to improve the robustness of the model;

[0021] S7. Line of sight direction estimation:

[0022] Mapping the head and body images into three-dimensional directions and confidences that obey the von Mises-Fisher distribution; weighting and adjusting the three-dimensional directions of the head and body by outputting the confidences of the head and body to obtain the final gaze output;

[0023] S8. Data enhancement and standardization:

[0024] Perform graphical enhancement and standardization on sample data to improve the generalization ability of the model;

[0025] S9. Method validation:

[0026] The method is verified on the collected surveillance video to obtain the yaw angle, pitch angle, roll angle and three-dimensional angle error of the body, head and gaze of the user image.

[0027] As a preference, the workflow of the ConvNeXt-based body feature extractor is as follows:

[0028] Input 7-frame sequence, dimension is [32, 21, 192, 256];

[0029] Obtain high-dimensional features through downsampling;

[0030] Use GAP to average filter the channel dimension and reduce the features to [32×21,768];

[0031] Mapped to [32×21,1280] dimensions.

[0032] As a preferred method, the workflow of the head feature extractor based on EfffficientNet-B0 is as follows:

[0033] Determine the head position using a binary mask of the head bounding box;

[0034] The head image is processed through stacked Mbconv blocks and high-dimensional features are extracted;

[0035] GAP was used to obtain the average features for each channel.

[0036] Preferably, in step S7, the head and body images are mapped into three-dimensional directions and confidences that obey the von Mises-Fisher distribution through a cascaded Bayesian framework;

[0037] The two body and head branch networks are responsible for extracting object features and learning a probability distribution mapping

[0038]

[0039] Among them, X b , X h, v, j∈[1,7] represent body image, head image, motion speed and jth frame respectively; μ b / h / g =[x b / h / g ,y b / h / g , z b / h / g ] represent the average direction of body, head or gaze respectively; ^ represents the corresponding estimated value; k b / h / g ∈[0, +∞) represent the distribution concentration of body direction, head direction or gaze direction respectively; as k b / h / g The value increases, and the estimated direction is μ b / h / g The likelihood of also increasing;

[0040] The formula for the staring von Mises-Fisher distribution is as follows:

[0041]

[0042] in, Represents the 7-frame orientation distribution of body, head and gaze after Bi-LSTM; FC g =Concat represents the fully connected layer that outputs the gaze probability distribution; =Concat represents the confidence k b / h The two branch network outputs of weighted concatenation.

[0043] As a preference, the probability mapping process based on the Bayesian framework is as follows:

[0044] The von Mises–Fisher distribution is a spherical distribution with an estimated value of As the central value of the von Mises-Fisher distribution, the maximum likelihood between it and the direction ground truth μ and the distribution concentration estimate are calculated

[0045] when When it is 0, the directions of body, head and gaze are evenly distributed;

[0046] when When , the directions of the body, head, and gaze follow a Dirac distribution; at this time, the model's estimated direction Exactly consistent with the ground truth μ;

[0047] The general definition of the von Mises-Fisher distribution is shown below;

[0048]

[0049] in, for The modified Bessel function of order; μ represents the a priori true value of the direction; and Respectively represent the direction and confidence output of our model; then, bring it into the modified Bessel equation to derive the following equation:

[0050] Among them, ! represents the factorial form, For ease of calculation we will Setting it to 3 leads to the probability distribution adopted by the model designed by the present invention, that is, the input image and velocity vector will be mapped to obey the three-dimensional direction shown in the following formula:

[0051]

[0052] Use maximum likelihood estimation to learn the cascade network parameters; according to the formula The loss function is defined as the negative log-likelihood of the vMF distribution, as shown in the following formula:

[0053] Here, sinh is the hyperbolic sine function, and ln represents the logarithm with the natural logarithm as the base.

[0054] As a preference, the model is trained and tested on two public datasets, GAFA and 3DPW, and tested on surveillance images collected daily; the detailed description of the training method is as follows:

[0055] Dataset Description:

[0056] GAFA dataset: Contains 882,000 frames of surveillance video, capturing free movement in five everyday environments, including kitchen, library, laboratory, living room and courtyard. The GAFA dataset provides annotations of 3D gaze direction, head and body.

[0057] 3DPW dataset: It consists of handheld videos, containing rich ground, real 3D human poses and shapes. The training part includes video sequences and IMU (inertial measurement unit) data to provide additional motion information.

[0058] Image preprocessing: Since the original labels of the 3DPW dataset are not suitable for the current task and the head mask is not defined, label conversion and head mask definition are required; the GAFA and 3DPW datasets are divided into training sets and test sets; label normalization and image standardization are performed to obtain the body and head images of each user; the image size in the 3DPW dataset is adjusted to the same size as the GAFA dataset ([3×256×192]);

[0059] Model training environment: Use distributed data parallel training on two 12G NVIDIA RTX 3080Ti GPUS to improve training efficiency;

[0060] Training parameter settings: The Adam optimizer is used for training in the PyTorch framework; the batch size is set to 32, that is, 32 samples are processed in each training iteration; the learning rate is set to 0.0001 to control the step size of parameter update; the decay rate is set to 0.0001 to gradually reduce the learning rate as the training progresses to help the model converge; the training cycle (epoch) is set to 100, that is, the model is fully trained 100 times on the entire training set; these training parameters and settings ensure that the model can effectively learn from the data and achieve good performance on the line of sight estimation task.

[0061] Preferably, in step S9, when using daily surveillance videos to test and verify the model, the specific requirements that need to be met are as follows:

[0062] Video duration: The video length captured by the camera must be greater than 3 seconds to ensure that a sufficient length of image sequence can be obtained. This ensures that the model has enough data to estimate the direction of sight, especially in dynamic scenes.

[0063] Head position: Head position information is required to generate a head mask. The head mask is very important for extracting head images because it helps to accurately separate the head region from the body image, thereby providing input for the head feature extractor.

[0064] Moving speed vector: If the moving speed vector of the target person in each frame image cannot be provided, the speed vector is set to 0. This can simplify the model input but may affect the accuracy of the model in a dynamic environment.

[0065] Preprocessing module:

[0066] The preprocessing module is used to perform the following tasks:

[0067] A 7-frame × 3 video sequence that complies with preset rules is cropped from each video, where 7 frames represent 7 consecutive images and 3 represents 3 consecutive video channels (such as RGB) to form a complete image sequence; all images in the cropped sequence are normalized; normalization usually includes adjusting the pixel values ​​of the image to the range of 0 to 1 to help the model converge faster and improve performance.

[0068] Preferably, in step S4, in associating the features of the previous and next frame images, the structure of Bi-LSTM includes two LSTM layers, and the specific principle of LSTM is shown in the following formula:

[0069]

[0070] Among them, x t is the input data matrix of this layer, σ, tanh is the activation function, Wf , W i , W C is the weight matrix, b o 、b C 、b i is the bias, h is the output of each layer of LSTM; input data matrix x t and the hidden output h of the previous LSTM cell t-1 ;

[0071] First, through the forget gate f t Filter out features that are related to the gaze direction of the previous frame but are obviously irrelevant to the current target; the memory gate uses g t and i t The function takes the current time as input x t and the input h of the hidden layer at the previous moment t-1 Extract the sight information from the

[0072] Then, follow Formula C t =f t *C t-1 +i t *g t To control these memories, that is, to combine the information of the forget gate and the input gate to obtain the current cell state C t ;

[0073] Next, through the output gate o t =σ(W o ·[h t-1 , x t ]+b o ) to get a normalized weight o t ;

[0074] Finally, according to the formula h t =o t *tanh(C t ) Get the hidden output h of the current cell t , and merge the hidden layer outputs of the other 6 LSTM cells to obtain the output containing context information And send it to a channel attention layer to enhance the temporal correlation between features.

[0075] Compared with the prior art, the technical effects and advantages of the present invention are:

[0076] This line of sight estimation method based on head-body coordination proposes to take 7 frames of body images and head mask images as input, and input them into the body feature extractor based on ConvNeXt and the head feature extractor based on EfffficientNet-B0 to extract high-dimensional features of the image, and map the velocity vector of each frame through a linear layer and perform feature splicing with the output of the two feature extractors; then use Bi-LSTM and channel attention layer to obtain the context information of the 7 frames of face images; then use two fully connected layers to solve the three-dimensional line of sight direction and confidence of the head and body; finally, adjust the direction output by the confidence of the body and head, splice the two, and then input them into Bi-LSTM and fully connected layers to output the final line of sight direction and confidence. The present invention also uses a loss function vmf loss based on von Mises-Fisher distribution to solve the problem that the update gradient of the traditional MAE or MSE loss function is always the same, resulting in falling into a local optimum or failure to converge;

[0077] The gaze estimation method based on head-body coordination models their directions as a three-dimensional von Mises-Fisher distribution and expresses the uncertainty of the estimation by the concentration of the distribution. We use multiple datasets to evaluate our method and test it on daily surveillance videos collected. The results show that the proposed method can stably estimate the gaze direction in various situations, even if the target person is in a bad viewing angle or blocked by obstacles. Under the conditions of meeting the requirements of real-time, stability, accuracy and versatility, it solves the technical problems of traditional methods such as over-reliance on clear eye images and slow training speed, and uses the limited resources of terminal devices to achieve the most effective gaze estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 is a flow chart of a sight line estimation method based on head-body coordination of the present invention;

[0079] Figure 2 A model structure diagram of the sight line estimation method based on head-body coordination of the present invention;

[0080] Figure 3 This is a visualization UI interface diagram of the present invention. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] The following is combined with Figure 1-3To further explain this application in detail,

[0083] The present application embodiment discloses a line of sight estimation method based on head-body coordination, the method comprising the following steps:

[0084] S1. Image input:

[0085] Input the video stream to be detected, select 7 consecutive frames of body images as a set of inputs of the body branch network, obtain head images of the same number of frames from the body images by locating the head and generating masks, and use them as inputs of the head branch network; input the velocity vector of each frame into the velocity branch and map it to a tensor of the same dimension as the output of the body and head feature extractors; remove invalid background and body pixel interference to obtain a clean input for the head branch network;

[0086] S2. Feature extraction:

[0087] Body feature extraction: Use a ConvNeXt-based body feature extractor to process the input body image sequence and extract features from the body image to represent the body posture and movement;

[0088] Head feature extraction: Use the head feature extractor based on EfficientNet-B0 to process the input head image; extract features from the head image to characterize the posture and direction of the head;

[0089] S3, speed information mapping:

[0090] Map the speed information of each frame in step S1 to the same dimension as the body and head features, and perform multi-scale fusion with the high-dimensional features extracted by the body and head branch networks to obtain fused features;

[0091] S4. Feature fusion and context association:

[0092] Input the fused features in step S3 into the Bi-LSTM network, and use the Bi-LSTM to simulate the characteristics of the human brain memory process to learn the correlation between the previous and next frames; integrate the image information at multiple moments to learn the coordination of the body, head and gaze in a long time series;

[0093] S5. Model training and testing:

[0094] The model is trained and tested on two public datasets, GAFA and 3DPW. The 3DPW data is labeled and converted, the head mask is defined, and the training and test sets are divided for GAFA and 3DPW.

[0095] S6. Performance optimization:

[0096] Debug the head branch network under different depth coefficients and width coefficients, optimize the performance through compound scaling of the model, and introduce random depth and Kaiming initialization methods to improve the robustness of the model;

[0097] S7. Line of sight direction estimation:

[0098] Mapping the head and body images into three-dimensional directions and confidences that obey the von Mises-Fisher distribution; weighting and adjusting the three-dimensional directions of the head and body by outputting the confidences of the head and body to obtain the final gaze output;

[0099] S8. Data enhancement and standardization:

[0100] Perform graphical enhancement and standardization on sample data to improve the generalization ability of the model;

[0101] S9. Method validation:

[0102] The method is verified on the collected surveillance video to obtain the yaw angle, pitch angle, roll angle and three-dimensional angle error of the body, head and gaze of the user image.

[0103] Aiming at the complex background and body pixels of different sizes in the collected video, the present invention designs a body feature extractor based on ConvNeXt. By introducing random depth, the branch structure in the extractor is deleted probabilistically to ensure that the extractor can learn information of different depths and improve the robustness of the model;

[0104] The branch network based on EfficientNet-B0 is used to extract high-dimensional features of the head. The present invention also uses the same random depth as that in the body feature extractor to allow the branch network to learn head features of different depths. At the same time, the present invention also debugs the head branch network based on EfficientNet-B0 under different depth coefficients and width coefficients, and through compound scaling of the model, ensures that the head branch network can obtain appropriate fine-grained and deep features and improve the accuracy of the overall model;

[0105] The body feature extractor based on ConvNeXt and the head feature extractor based on EfffficientNet-B0 used in the line of sight estimation method based on head-body coordination have a multi-branch structure, which helps to obtain deeper significant features and is also convenient for using random depth to probabilistically delete a small number of branches to explore the model performance at different depths. The multi-branch structure of each ConvNeXt block enables the network to integrate features from different scales. At the same time, each subpath of the ConvNeXt block has a certain chance of being deleted. Therefore, the body extractor of the present invention is more robust for fusing different depth features with body pixels of different scales (regardless of whether the tester is close to the camera). In addition, the use of depth-separable convolution and droppath will be conducive to the lightweight design of the model.

[0106] Specifically, the workflow of the ConvNeXt-based body feature extractor is as follows:

[0107] Input 7-frame sequence, dimension is [32, 21, 192, 256];

[0108] Obtain high-dimensional features through downsampling;

[0109] Use GAP to average filter the channel dimension and reduce the features to [32×21,768];

[0110] Mapped to [32×21,1280] dimensions.

[0111] For the input of the above-mentioned head extractor, the present invention uses the binary mask of the head bounding box to determine the position of the head. The head image is first processed and high-dimensional features are extracted by stacking Mbconv blocks, and then the average features of each channel are obtained using the same GAP as the body extractor. Our head extractor is mainly designed based on EfffficientNet-B0. The deep convolution adopted by each MBConv module of EfffficientNet-B0 greatly reduces the number of parameters of the head extractor. We use the comprehensive scaling method of EfffficientNet-B0 to coordinate width and depth, so as to extract features with suitable fine-grainedness and depth. Similar to the body extractor, we use random depth to probabilistically delete branches in each Mbconv block to study its performance at different depths. This use of random depth can prevent overfitting while ensuring the robustness of our model on head images of different scales.

[0112] Specifically, the workflow of the head feature extractor based on EfffficientNet-B0 is as follows:

[0113] Determine the head position using a binary mask of the head bounding box;

[0114] The head image is processed through stacked Mbconv blocks and high-dimensional features are extracted;

[0115] GAP was used to obtain the average features for each channel.

[0116] The gaze estimation method based on head-body coordination has been trained on the GAFA dataset and the 3DPW dataset, and has been trained and tested on these two datasets containing tens of thousands of labeled images, more than a dozen users, and dozens of indoor and outdoor scenes. The present invention not only focuses on the appearance of the face and eyes, but also considers the overall posture of a person. The cascaded Bayesian framework designed by the present invention can map the images of the head and body into three-dimensional directions and confidences that obey the von Mises-Fisher distribution. The cascade network designed by the present invention encodes the correlation of the gaze direction in each frame to the past and future head and body directions on a time scale. The influence of the three-dimensional directions of the head and body on the final gaze output is then weighted and adjusted by the output confidence of the head and body.

[0117] In step S7, the head and body images are mapped into three-dimensional directions and confidences that obey the von Mises-Fisher distribution through a cascaded Bayesian framework;

[0118] The two body and head branch networks are responsible for extracting object features and learning a probability distribution mapping

[0119] Among them, X b , X h , v, j∈[1,7] represent body image, head image, motion speed and jth frame respectively; μ b / h / g =[x b / h / g ,y b / h / g , z b / h / g ] represent the average direction of body, head or gaze respectively; ^ represents the corresponding estimated value; k b / h / g ∈[0, +∞) represent the distribution concentration of body direction, head direction or gaze direction respectively; as k b / h / g The value increases, and the estimated direction is μ b / h / g The likelihood of also increasing;

[0120] The formula for the staring von Mises-Fisher distribution is as follows:

[0121]

[0122] in, Represents the 7-frame orientation distribution of body, head and gaze after Bi-LSTM; FC g =Concat represents the fully connected layer that outputs the gaze probability distribution; =Concat represents the confidence k b / h The two branch network outputs of weighted concatenation.

[0123] Specifically, the probability mapping process based on the Bayesian framework is as follows:

[0124] The von Mises–Fisher distribution is a spherical distribution with an estimated value of As the central value of the von Mises-Fisher distribution, the maximum likelihood between it and the direction ground truth μ and the distribution concentration estimate are calculated

[0125] when When it is 0, the directions of body, head and gaze are evenly distributed;

[0126] when When the body, head and gaze directions follow the Dirac distribution; however, in real dynamic scenes, due to the occlusion of the head and body, incomplete input, background interference and other problems, the von Mises-Fisher distribution cannot be completely converted into the Dirac distribution. However, the purpose of the present invention is to design a cascaded Bayesian framework to construct a probability model in which the body, head and gaze are as close to the real direction distribution as possible, and the concentration estimation value output by the method proposed in the present invention for most test images is Reaching 2.6 or more. This means that the direction estimation value output by the model designed by the present invention is very likely to be close to the true value, and the output direction approximately obeys the Dirac distribution. At this time, the estimated direction of the model Exactly consistent with the ground truth μ;

[0127] The general definition of the von Mises-Fisher distribution is shown below;

[0128]

[0129] in, for The modified Bessel function of order; μ represents the a priori true value of the direction; and Respectively represent the direction and confidence output of our model; then, bring it into the modified Bessel equation to derive the following equation:

[0130] Among them, ! represents the factorial form, For ease of calculation we will Setting it to 3 leads to the probability distribution adopted by the model designed by the present invention, that is, the input image and velocity vector will be mapped to obey the three-dimensional direction shown in the following formula:

[0131]

[0132] Use maximum likelihood estimation to learn the cascade network parameters; according to the formula The loss function is defined as the negative log-likelihood of the vMF distribution, as shown in the following formula:

[0133] Here, sinh is the hyperbolic sine function, and ln represents the logarithm with the natural logarithm as the base.

[0134] As a preference, the model is trained and tested on two public datasets, GAFA and 3DPW, and tested on surveillance images collected daily; the detailed description of the training method is as follows:

[0135] Dataset Description:

[0136] GAFA dataset: Contains 882,000 frames of surveillance video, capturing free movement in five everyday environments, including kitchen, library, laboratory, living room and courtyard. The GAFA dataset provides annotations of 3D gaze direction, head and body.

[0137] 3DPW dataset: It consists of handheld videos, containing rich ground, real 3D human poses and shapes. The training part includes video sequences and IMU (inertial measurement unit) data to provide additional motion information.

[0138] Image preprocessing: Since the original labels of the 3DPW dataset are not suitable for the current task and the head mask is not defined, label conversion and head mask definition are required; the GAFA and 3DPW datasets are divided into training sets and test sets; label normalization and image standardization are performed to obtain the body and head images of each user; the image size in the 3DPW dataset is adjusted to the same size as the GAFA dataset ([3×256×192]);

[0139] Model training environment: Use distributed data parallel training on two 12G NVIDIA RTX 3080Ti GPUS to improve training efficiency;

[0140] Training parameter settings: The Adam optimizer is used for training in the PyTorch framework; the batch size is set to 32, that is, 32 samples are processed in each training iteration; the learning rate is set to 0.0001 to control the step size of parameter update; the decay rate is set to 0.0001 to gradually reduce the learning rate as the training progresses to help the model converge; the training cycle (epoch) is set to 100, that is, the model is fully trained 100 times on the entire training set; these training parameters and settings ensure that the model can effectively learn from the data and achieve good performance on the line of sight estimation task.

[0141] Specifically, in step S9, when using daily surveillance videos to test and verify the model, the specific requirements that need to be met are as follows:

[0142] Video duration: The video length captured by the camera must be greater than 3 seconds to ensure that a sufficient length of image sequence can be obtained. This ensures that the model has enough data to estimate the direction of sight, especially in dynamic scenes.

[0143] Head position: Head position information is required to generate a head mask. The head mask is very important for extracting head images because it helps to accurately separate the head region from the body image, thereby providing input for the head feature extractor.

[0144] Moving speed vector: If the moving speed vector of the target person in each frame image cannot be provided, the speed vector is set to 0. This can simplify the model input but may affect the accuracy of the model in a dynamic environment.

[0145] Preprocessing module:

[0146] The preprocessing module is used to perform the following tasks:

[0147] A 7-frame × 3 video sequence that complies with preset rules is cropped from each video, where 7 frames represent 7 consecutive images and 3 represents 3 consecutive video channels (such as RGB) to form a complete image sequence; all images in the cropped sequence are normalized; normalization usually includes adjusting the pixel values ​​of the image to the range of 0 to 1 to help the model converge faster and improve performance.

[0148] Specifically, in step S4, when associating the features of the previous and next frame images, the structure of Bi-LSTM includes two LSTM layers. The specific principle of LSTM is shown in the following formula:

[0149]

[0150] Among them, x t is the input data matrix of this layer, σ, tanh is the activation function, W f , W i , W C is the weight matrix, b o 、b C 、b i is the bias, h is the output of each layer of LSTM; input data matrix x t and the hidden output h of the previous LSTM cell t-1 ;

[0151] First, through the forget gate f tFilter out features that are related to the gaze direction of the previous frame but are obviously irrelevant to the current target; the memory gate uses g t and i t The function takes the current time as input x t and the input h of the hidden layer at the previous moment t-1 Extract the sight information from the

[0152] Then, follow Formula C t =f t *C t-1 +i t *g t To control these memories, that is, to combine the information of the forget gate and the input gate to obtain the current cell state C t ;

[0153] Next, through the output gate o t =σ(W o ·[h t-1 , x t ]+b o ) to get a normalized weight o t ;

[0154] Finally, according to the formula h t =o t *tanh(C t ) Get the hidden output h of the current cell t , and merge the hidden layer outputs of the other 6 LSTM cells to obtain the output containing context information And send it to a channel attention layer to enhance the temporal correlation between features.

[0155] like Figure 2 As shown, the sight estimation network is divided into three modules: a body branch network based on ConvNeXt and Bi-LSTM, a head branch network based on EfficientNet-B0 and Bi-LSTM, and a sight estimation module based on confidence-weighted feature fusion. The body branch network based on ConvNeXt and Bi-LSTM is used to remove the background noise of the image, extract high-dimensional body direction features, and learn the contextual connection between the previous and next frames. The head branch network based on EfficientNet-B0 and Bi-LSTM uses a comprehensive scaling method to balance width and depth, thereby extracting features with both appropriate fine-grainedness and depth and can connect the contextual information of the previous and next 7 frames of images. Finally, the obtained body and head vectors are confidence-weighted and sent to the sight estimation module based on the fully connected layer and Bi-LSTM to output a three-dimensional sight line that obeys the von Mises-Fisher distribution.

[0156] like Figure 2As shown, the ConvNeXt-based body feature extractor is mainly composed of 4 ConvNeXt blocks, a normalization layer, a downsampling layer, a global average pooling and a linear layer. The body direction feature extracted by the extractor is sent to a bidirectional long short-term memory network (Bi-LSTM) after being fused with other high-dimensional features to obtain the correlation between the directions of the previous and next frames; in order to calculate the spatial attention, the present invention performs global average pooling along the channel axis in the output layer of the branch network to generate an efficient feature vector. It has been proved that pooling operations along the channel axis can effectively highlight the significant information of the features. In addition, the present invention also uses random depth to probabilistically delete ConvNeXt blocks to improve the generalization of the model.

[0157] For the input of the head extractor of the present invention, we use the binary mask of the head bounding box to determine the position of the head. First, the head image is processed by stacked Mbconv blocks to extract high-dimensional features, and then the average features of each channel are obtained using the same global average pooling as the body extractor. The deep convolution adopted by the MBConv module greatly reduces the number of parameters of the head extractor. We use the comprehensive scaling method of efficient nets to coordinate width and depth to extract features with fine-grainedness and depth. Similar to the body extractor, the present invention uses droppath to randomly delete branches in each Mbconv block, allowing the model to learn directional features at different depths. The use of random depth can prevent overfitting while ensuring that the model designed by the present invention obtains robustness on head images of different scales;

[0158] The present invention uses the GAFA dataset and the 3DPW dataset to train and test the entire model, and verifies the method on the collected surveillance video data. Figure 3 As shown in Figure 1, the three-dimensional visual images of the target person’s body, head, and sight line in each frame are displayed. Figure 3 As shown, the UI interface designed by the present invention can output the visualization of each frame as well as the yaw angle, pitch angle and angle error of the body, head and sight. As shown in Table 1, the angle error obtained by the method described in this embodiment is the smallest compared with the currently popular sight estimation method on the GAFA dataset.

[0159] Table 1 Comparison of the body, head and sight angle errors of the sight line estimation method based on head-body coordination in the present invention on the GAFA dataset with other methods

[0160]

[0161] In the data input of the present invention, the input video is divided into multiple 7-frame 3-channel image sequences, and a batch size of 32 is used, that is, a body image with a shape of [32×21×192×256]. A head image of the same shape is obtained by the head mask. The image is input into the line of sight estimation model proposed in the present invention for training, and before that, the sample data is also subjected to graphic enhancement and normalization processing. The present invention can also selectively decide whether to input a velocity vector. The line of sight estimation method based on head-body coordination proposed in the present invention achieves an angle error of less than 22.6° and a confidence level of up to 2.62 on the 3DPW and GAFA test sets.

[0162] In the extraction of body features and head features of the present invention, the traditional convolutional layer or fully connected layer will more or less have the problem of information loss during information transmission. ConvNeXt and EfficientNet-B0 are deep convolutional neural networks with residual structures, which can effectively alleviate the gradient diffusion problem caused by the depth of the network, and can learn more complex and richer image feature representations, which is beneficial for the line of sight estimation task. And both ConvNeXt and EfficientNet-B0 use deep separable convolution and droppath, which will be conducive to the lightweight design of the model. For real-time line of sight estimation tasks, global spatial information and contextual information are also very important. Therefore, the present invention proposes to use the SE attention mechanism inherent in EfffficientNet-B0 and the deep separable convolution of ConvNeXt to obtain the characteristics of the direction vector contained in the image, and use Bi-LSTM combined with feature fusion to obtain the contextual information of the line of sight.

[0163] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A line of sight estimation method based on head-body coordination, characterized in that: The method comprises the following steps: S1. Image input: Input the video stream to be detected, select 7 consecutive frames of body images as a set of inputs to the body branch network, obtain head images of the same number of frames from the body images by locating the head and generating masks, and use them as the input of the head branch network; input the velocity vector of each frame into the velocity branch and map it to a tensor of the same dimension as the output of the body and head feature extractors; S2. Feature extraction: Body feature extraction: Use a ConvNeXt-based body feature extractor to process the input body image sequence and extract features from the body image to represent the body posture and movement; Head feature extraction: Use the head feature extractor based on EfficientNet-B0 to process the input head image; extract features from the head image to characterize the posture and direction of the head; S3, speed information mapping: Map the speed information of each frame in step S1 to the same dimension as the body and head features, and perform multi-scale fusion with the high-dimensional features extracted by the body and head branch networks to obtain fused features; S4. Feature fusion and context association: Input the fused features in step S3 into the Bi-LSTM network, and use the Bi-LSTM to simulate the characteristics of the human brain memory process to learn the correlation between the previous and next frames; integrate the image information at multiple moments to learn the coordination of the body, head and gaze in a long time series; S5. Model training and testing: The model is trained and tested on two public datasets, GAFA and 3DPW. The 3DPW data is labeled and converted, the head mask is defined, and the training and test sets are divided for GAFA and 3DPW. S6. Performance optimization: Debug the head branch network under the depth coefficient and width coefficient, optimize the performance through compound scaling of the model, and introduce random depth and Kaiming initialization methods to improve the robustness of the model; S7. Line of sight direction estimation: Mapping the head and body images into three-dimensional directions and confidences that obey the von Mises-Fisher distribution; weighting and adjusting the three-dimensional directions of the head and body by outputting the confidences of the head and body to obtain the final gaze output; S8. Data enhancement and standardization: Perform graphical enhancement and standardization on sample data to improve the generalization ability of the model; S9. Method validation: The method is verified on the collected surveillance video to obtain the yaw angle, pitch angle, roll angle and three-dimensional angle error of the body, head and gaze of the user image.

2. The method for estimating sight lines based on head-body coordination according to claim 1, characterized in that: The workflow of the ConvNeXt-based body feature extractor is as follows: Input 7-frame sequence, dimension is [32, 21, 192, 256]; Obtain high-dimensional features through downsampling; Use GAP to average filter the channel dimension and reduce the features to [32×21,768]; Mapped to [32×21,1280] dimensions.

3. The method for estimating sight lines based on head-body coordination according to claim 1, characterized in that: The workflow of the head feature extractor based on EfffficientNet-B0 is as follows: Determine the head position using a binary mask of the head bounding box; The head image is processed through stacked Mbconv blocks and high-dimensional features are extracted; GAP was used to obtain the average features for each channel.

4. The method for estimating sight lines based on head-body coordination according to claim 1, characterized in that: In step S7, the head and body images are mapped into three-dimensional directions and confidences that obey the von Mises-Fisher distribution through a cascaded Bayesian framework; The two body and head branch networks are responsible for extracting object features and learning a probability distribution mapping Among them, X b , X h , v, j∈[1,7] represent body image, head image, motion speed and jth frame respectively; μ b / h / g =[x b / h / g ,y b / h / g , z b / h / g ] represent the average direction of body, head or gaze respectively; ^ represents the corresponding estimated value; k b / h / g ∈[0, +∞) represent the distribution concentration of body direction, head direction or gaze direction respectively; as k b / h / g The value increases, and the estimated direction is μ b / h / g The likelihood of also increasing; The formula for the staring von Mises-Fisher distribution is as follows: in, Represents the 7-frame orientation distribution of body, head and gaze after Bi-LSTM; FC g =Concat represents the fully connected layer that outputs the gaze probability distribution; =Concat represents the confidence k b / h The two branch network outputs of weighted concatenation.

5. The method for estimating sight lines based on head-body coordination according to claim 4, characterized in that: The probability mapping process based on the Bayesian framework is as follows: The von Mises–Fisher distribution is a spherical distribution with an estimated value of As the central value of the von Mises-Fisher distribution, the maximum likelihood between it and the direction ground truth μ and the estimated concentration of the distribution are obtained when When it is 0, the directions of body, head and gaze are evenly distributed; when When , the directions of the body, head, and gaze follow a Dirac distribution; in this case, the model’s estimated direction Exactly consistent with the ground truth μ; The general definition of the von Mises-Fisher distribution is shown below; in, for The modified Bessel function of order; μ represents the a priori true value of the direction; and Respectively represent the direction and confidence output of our model; then, bring it into the modified Bessel equation to derive the following equation: Among them, ! represents the factorial form, Will Set to 3 to deduce the probability distribution used by the design model, the input image and velocity vector will be mapped to obey the three-dimensional direction as shown below; Use maximum likelihood estimation to learn the cascade network parameters; according to the formula The loss function is defined as the negative log-likelihood of the vMF distribution, as shown in the following formula: Here, sinh is the hyperbolic sine function, and ln represents the logarithm with the natural logarithm as the base.

6. The method for estimating sight lines based on head-body coordination according to claim 1, characterized in that: The model is trained and tested on two public datasets, GAFA and 3DPW, and tested on daily surveillance images. The training method is as follows: Dataset description: GAFA dataset: contains 882,000 frames of surveillance video, capturing free movement in five daily environments, including kitchen, library, laboratory, living room and courtyard. The GAFA dataset provides annotations of 3D gaze direction, head and body; 3DPW dataset: It consists of handheld videos, contains rich ground, real 3D human posture and shape, and the part used for training includes video sequences and IMU data; Image preprocessing: Divide the GAFA and 3DPW datasets into training sets and test sets; perform label normalization and image standardization to obtain body and head images of each user; resize the images in the 3DPW dataset to the same size as the GAFA dataset ([3×256×192]); Model training environment: Use distributed data parallel training on two 12G NVIDIA RTX 3080Ti GPUS; Training parameter setting: The Adam optimizer is used for training in the PyTorch framework; the batch size is set to 32, the learning rate is set to 0.0001, the decay rate is set to 0.0001, and the training cycle is set to 100.

7. The method for estimating sight lines based on head-body coordination according to claim 1, characterized in that: In step S9, when using daily surveillance videos to test and verify the model, the specific requirements that need to be met are as follows: Video duration: The video captured by the camera must be longer than 3 seconds; Head position: Head position information is required to generate a head mask; Moving speed vector: If the moving speed vector of the target person in each frame image cannot be provided, the speed vector is set to 0; Preprocessing module: The preprocessing module is used to perform the following tasks: crop a 7-frame × 3 video sequence that complies with preset rules from each video to form a complete image sequence; standardize all images in the cropped sequence.

8. The method for estimating sight lines based on head-body coordination according to claim 1, characterized in that: In step S4, the features of the previous and next frame images are associated. The structure of Bi-LSTM includes two LSTM layers. The specific principle of LSTM is shown in the following formula: Among them, x t is the input data matrix of this layer, σ, tanh is the activation function, W f , W i , W C is the weight matrix, b o 、b C 、b i is the bias, h is the output of each layer of LSTM; input data matrix x t and the hidden output h of the previous LSTM cell t-1 ; First, through the forget gate f t Filter out features that are related to the gaze direction of the previous frame but are obviously irrelevant to the current target; the memory gate uses g t and i t The function takes the current time as input x t and the input h of the hidden layer at the previous moment t-1 Extract the sight information from the Then, follow Formula C t =f t *C t-1 +i t *g t To control these memories, that is, to combine the information of the forget gate and the input gate to obtain the current cell state C t ; Next, through the output gate o t =σ(W o ·[h t-1 , x t ]+b o ) to get a normalized weight o t ; Finally, according to the formula h t =o t *tanh(C t ) Get the hidden output h of the current cell t , and merge the hidden layer outputs of the other 6 LSTM cells to obtain the output containing context information And send it to a channel attention layer to enhance the temporal correlation between features.

Images