Head pose estimation method and system based on structure-equivariant self-supervised learning

Abstract

The present invention provides a head posture estimation method and system based on structured equivariant self-supervised learning, which belongs to the field of head posture estimation in computer vision. The method includes: inputting batch images to establish a training set X; sending the batch images to a feature extractor network for learning, selecting samples in the training set X as anchor images, constructing a neighbor graph G about the anchor image in the feature space and performing refinement adjustment; and i The face is divided into T non-overlapping patches and mapped into the feature space. The differences between different patch positions are calculated, and the anchor image features are trained and adjusted. The trained and adjusted anchor image features are input into the softmax function, converted into three angle posture vectors, and the probability distribution data of facial posture labels is calculated. The label with the largest probability is selected as the estimated result of the anchor image, and the head posture estimation model is established. The head posture estimation model is used to process the image to be analyzed, obtain the estimation result, and calculate the Euler angle.

Description

Technical Field

[0001] The present invention relates to the field of head posture estimation in computer vision, and in particular to a head posture estimation method and system based on structure-isomorphic self-supervised learning. Background Art

[0002] Head pose estimation aims to predict the continuous rotational orientation (pitch, yaw, and roll) of the human head based on a given facial RGB image. It plays a crucial role in many potential computer vision applications, such as human-computer interaction, attention modeling, and behavioral analysis. In recent years, much work has focused on addressing related issues in head pose estimation. However, due to the ambiguity of head pose labels and the nonlinear relationships in facial data caused by expression, occlusion, gender, and ethnicity, the performance of algorithms remains limited. In recent years, the development of head pose estimation algorithms has shifted from traditional pattern recognition methods to deep neural network approaches.

[0003] The intrinsic relationship between the regular structure and structural variability of the face and head motion plays a crucial role in face pose estimation. The human head can be modeled as a non-substantial, rigid object where each pixel changes during motion. However, a single RGB image can only represent the position coordinates of facial pixels, not the three-dimensional shape. In head pose estimation, most models suffer from head pose uncertainty and often fail in outdoor conditions. This is mainly because faces captured in the real world vary significantly in shape and occlusion due to differences in expression, gender, and ethnicity. Head rotation can distort or hide discriminative features, making it impossible to correctly estimate the pose in the rotational space. Traditional head pose estimation methods typically utilize facial landmarks and depth information to regress head pose. Since these approaches require high prediction accuracy of facial landmarks, they may introduce biased predictions in head pose estimation. Summary of the Invention

[0004] In view of this, the present invention provides a head posture estimation method and system based on structural equivariance self-supervised learning, which realizes structural equivariance through a self-supervised learning strategy of external attributes and potential local associations of facial neighbors, and simulates the posture changes of head rotation based on structural equivariance estimation. In the absence of external labels, three different directional features are extracted in a self-supervised manner, thereby effectively describing the posture.

[0005] The technical solution adopted by the embodiment of the present invention to solve the technical problem is:

[0006] A head pose estimation method based on structure-equivariant self-supervised learning, comprising:

[0007] Step S1: input batch images and establish training set X = {xi The batch processed images are enhanced images that have been pre-processed by face detection and alignment operations;

[0008] Step S2: feeding the batch images in the training set X into a feature extractor network for learning to obtain facial semantic features of the batch images, selecting samples in the training set X as anchor images, further constructing a neighbor graph G about the anchor images in the feature space, and refining and adjusting the neighbor graph G using a pose-invariant loss function, so that each sample in the training set X controls different latent semantic attributes to enhance the diversity of feature learning;

[0009] Step S3: each of the samples x i The image is divided into T non-overlapping patches and mapped into the feature space. The local space equivariant loss function is used to calculate the difference between the positions of different patches in the refined and adjusted neighborhood graph G. The anchor feature of the anchor image is trained and adjusted based on a single patch, so as to extend the features under different semantic factors to local images of different postures.

[0010] Step S4: Input the anchor features of the trained and adjusted anchor image into the softmax function and convert them into three angle posture vectors [y i ,p i ,r i ] and calculate the probability distribution data of facial posture labels, select the label with the largest probability as the head posture estimation result of the anchor image anchor, and the head posture estimation model is established;

[0011] Step S5: Input the image to be analyzed as a new anchor image anchor into the head pose estimation model, process the image to be analyzed according to steps S2-S4, obtain the facial pose label probability distribution data and head pose estimation result of the image to be analyzed, and convert the three angle pose vectors of the image to be analyzed into Euler angles.

[0012] Preferably, the step S2 includes:

[0013] Step S21, sending the batch processed images into a feature extractor network for learning to obtain facial semantic features of the batch processed images;

[0014] Step S22: construct an initial neighbor graph G = {V, E} with node v as the anchor image anchor, where V is a set of nodes, E is a set of edges, and the images corresponding to the nodes at both ends of the edge have the same visual structure under the same semantics. The initial neighbor graph G has A subgraphs, and the nodes that constitute the same subgraph are neighbor images of the anchor image anchor, and all have the same visual structure under the same semantics;

[0015] Step S23, select one of the subgraphs G a Perform pose invariant loss calculation, a∈[1,A], let the subgraph G a The node set in is represented as {n1,...,n i}, the initial neighbor graph G does not belong to the subgraph G a The node set inside is represented as U, and the result L is calculated by the posture invariant loss function i :

[0016]

[0017] Where u∈{n1,...,n i} represents the subgraph G a The node set in G, u′∈U represents a non-subgraph G a The node set within, τ is the temperature coefficient, x i is the original sample;

[0018] Step S24, according to the result L of the posture invariant loss function i Adjusting the spatial distances of the nodes in the initial neighbor graph G so that features with the same latent semantic factors in the feature space are clustered together and features with different latent semantic factors are kept apart;

[0019] Step S25, repeating steps S23-S24, based on each of the subgraphs G a The pose invariance loss is calculated and the spatial distance is adjusted. After A times of adjustment, the refined and adjusted neighbor graph G is obtained. The refined and adjusted neighbor graph G captures the correlation between different semantic factors in the image, thereby achieving better visual feature representation.

[0020] Preferably, the step S3 includes:

[0021] Step S31, dividing each sample into T non-overlapping patches and mapping them into the feature space, and converting them into structurally equivariant facial features, wherein the sample x i Divide into T non-overlapping patches And mapped to the feature space to get the patch feature In the feature space Pj (t) The corresponding patch feature is F j (t) , t∈[1,T], i∈[1,N], j∈[1,N], N is the total number of samples in the training set;

[0022]

[0023] Among them, F i (k) Denotes the anchor patch P i (k) The characteristics of F j (k) represents the patch P at the same position as the anchor point j (k) The characteristics of F j (t) represents a patch P at a different position than the anchor point j (t) characteristic, τ is the temperature coefficient; F i (t) Refers to the anchor image except F i (k) Other patches;

[0024] Step S33, repeating steps S32-S33, according to the loss result L of the local space equivalent loss function e Adjust the anchor patch P in the feature space i (k) With patch P i (t) , and {u1,u2,...u N-1}, and finally use structural equivariance to learn the compensatory features of the facial structure.

[0025] Preferably, the step S4 includes:

[0026] Step S41, converting the anchor image features trained and adjusted in step S33 into a posture vector [y i ,p i ,r i ], where y i 、p i 、r i They correspond to the three direction vectors of yaw angle, pitch angle and roll angle respectively;

[0027] Step S42: mapping the posture vector [y i ,p i ,ri ] is input into the sotamax function to obtain the probability distribution of facial posture labels, where the distribution is normalized to ensure

[0028]

[0029]

[0030]

[0031] Among them, l i ∈{l1,l2,…,l m} represents the binned pitch angle within the range of ±99°, m is the number of bins, σ represents the standard deviation of the Gaussian distribution and the finest granularity of the attitude angle;

[0032] Step S43, select y(l i ) value, p(l i ) value, r(l i ) value is taken as the head pose estimation result of the anchor image anchor.

[0033] The present invention also provides a head posture estimation system based on structured equivariant self-supervised learning, which is the implementation subject of the above method, including:

[0034] Face detection and alignment module: input batch images, establish training set X = {x i The batch images are enhanced images that have been pre-processed by face detection and alignment operations;

[0035] Feature extraction module: The batch images in the training set X are fed into the feature extractor network for learning, the facial semantic features of the batch images are obtained, samples in the training set X are selected as anchor images, a neighbor graph G about the anchor image is further constructed in the feature space, and the neighbor graph G is refined and adjusted by the pose invariant loss function, so that each sample in the training set X controls different potential semantic attributes to enhance the diversity of feature learning; each sample x is i The image is divided into T non-overlapping patches and mapped into the feature space. The local space equivariant loss function is used to calculate the difference between the positions of different patches in the refined and adjusted neighborhood graph G. The anchor feature of the anchor image is trained and adjusted based on a single patch, so as to extend the features under different semantic factors to local images of different postures.

[0036] The head posture estimation module is used to input the anchor image features after training and adjustment into the softmax function and convert them into three angle posture vectors [y i ,p i ,r i ] and calculate the probability distribution data of facial posture labels, and select the label with the largest probability as the head posture estimation result of the anchor image anchor;

[0037] A head pose visualization module is used to visually display the head pose estimation results of the anchor image anchor; and to visually display the head pose estimation results of the image to be analyzed processed by the head pose estimation system.

[0038] Preferably, the feature extraction module is used to:

[0039] Sending the batch processed images into a feature extractor network for learning to obtain facial semantic features of the batch processed images;

[0040] Construct an initial neighbor graph G = {V, E} with node v as the anchor image anchor, where V is a set of nodes and E is a set of edges. The images corresponding to the nodes at both ends of the edge have the same visual structure under the same semantics. The initial neighbor graph G has A subgraphs. The nodes that constitute the same subgraph are neighbor images of the anchor image anchor and all have the same visual structure under the same semantics.

[0041] Select one of the subgraphs G a Perform pose invariant loss calculation, a∈[1,A], let the subgraph G a The node set in is represented as {n1,...,n i}, the initial neighbor graph G does not belong to the subgraph G a The node set inside is represented as U, and the result L is calculated by the posture invariant loss function i :

[0042]

[0043] Where u∈{n1,...,n i} represents the subgraph G a The node set in G, u′∈U represents a non-subgraph G a The node set within, τ is the temperature coefficient, x i is the original sample;

[0044] According to the result of the pose invariant loss function L iAdjusting the spatial distances of the nodes in the initial neighbor graph G so that features with the same latent semantic factors in the feature space are clustered together and features with different latent semantic factors are kept apart;

[0045] Repeat steps S23-S24, based on each of the subgraphs G a The pose invariance loss is calculated and the spatial distance is adjusted. After A times of adjustment, the refined and adjusted neighbor graph G is obtained. The refined and adjusted neighbor graph G captures the correlation between different semantic factors in the image, thereby achieving better visual feature representation.

[0046] Preferably, the feature extraction module is used to:

[0047] Each sample is divided into T non-overlapping patches and mapped into the feature space, and converted into structurally equivariant facial features, where the sample x i Divide into T non-overlapping patches And mapped to the feature space to get the patch feature In the feature space P j (t) The corresponding patch feature is F j (t) , t∈[1,T], i∈[1,N], j∈[1,N], N is the total number of samples in the training set;

[0048]

[0049] Among them, F i (k) Denotes the anchor patch P i (k) The characteristics of F j (k) represents the patch P at the same position as the anchor point j (k) The characteristics of F j (t) represents a patch P at a different position than the anchor point j (t) characteristic, τ is the temperature coefficient; F i (t) Refers to patches at different positions of the anchor image;

[0050] Repeat steps S32-S33, and calculate the loss result L according to the local space equivalent loss function. e Adjust the anchor patch P in the feature space i (k) and the rest of the patches P in the anchor image anchor i (t)、{u1,u2,...u N-1}, and finally use structural equivariance to learn the compensatory features of the facial structure.

[0051] Preferably, the head posture estimation module is used to:

[0052] Convert the anchor image features after training adjustment into the pose vector [y i ,p i ,r i ], where y i 、p i 、r i They correspond to the three direction vectors of yaw angle, pitch angle and roll angle respectively;

[0053] The pose vector [y i ,p i ,r i ] is input into the sotamax function to obtain the probability distribution of facial posture labels, where the distribution is normalized to ensure

[0054]

[0055]

[0056]

[0057] Among them, l i ∈{l1,l2,…,l m} represents the binned pitch angle within the range of ±99°, m is the number of bins, σ represents the standard deviation of the Gaussian distribution and the finest granularity of the attitude angle;

[0058] Select y(l i ) value, p(l i ) value, r(l i ) value is taken as the head pose estimation result of the anchor image anchor.

[0059] It can be seen from the above technical solution that in the head posture estimation method based on structure equivariant self-supervised learning provided by the embodiment of the present invention, a batch of images is input to establish a training set X = {x i}, where the batch image is an enhanced image preprocessed by face detection and alignment operations; the batch images in the training set X are fed into the feature extractor network for learning to obtain the facial semantic features of the batch images, and the samples in the training set X are selected as anchor images. The neighbor graph G about the anchor image is further constructed in the feature space, and the neighbor graph G is refined and adjusted by the pose invariant loss function, so that each sample in the training set X controls different potential semantic attributes to enhance the diversity of feature learning; each sample x is i The image is divided into T non-overlapping patches and mapped into the feature space. The local space equivariant loss function is used to calculate the difference between the positions of different patches in the refined and adjusted neighbor graph G. The anchor image features are trained and adjusted based on a single patch to extend the features under different semantic factors to local images with different postures. The trained and adjusted anchor image features are input into the softmax function and converted into three angle posture vectors [y i ,p i ,r i ] and calculate the probability distribution data of facial posture labels, select the label with the largest probability as the head posture estimation result of the anchor image anchor, and the head posture estimation model is established; the image to be analyzed is input into the head posture estimation model as a new anchor image anchor for processing, and the facial posture label probability distribution data of the image to be analyzed is obtained, and the three angle posture vectors of the image to be analyzed are converted into Euler angles. Through the method of the present invention, structural equivariance is achieved through a self-supervised learning strategy of external attributes and potential local associations of facial neighbors, and the posture change of the simulated head rotation is estimated based on the structural equivariance. In the absence of external labels, three different directional features are extracted in a self-supervised manner, thereby effectively describing the posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Flowchart of the head pose estimation method based on structure-equivariant self-supervised learning.

[0061] Figure 2 Schematic diagram of the head pose estimation method based on structure-equivariant self-supervised learning.

[0062] Figure 3 It is a neighbor graph.

[0063] Figure 4 For comparative learning.

[0064] Figure 5 This is the system framework diagram.

[0065] Figure 6 This is an example picture of the system main interface.

[0066] Figure 7 This is an example diagram of the training module interface.

[0067] Figure 8 An example diagram of the head posture visualization module interface.

[0068] Figure 9 Visualize the experimental results. DETAILED DESCRIPTION

[0069] The technical solutions and technical effects of the present invention are further described in detail below with reference to the accompanying drawings of the present invention.

[0070] Local features are inspired by discriminative features and facial attributes. They contain semantic features and spatial information of specific areas. Local features are not affected by occlusion and posture changes. The impact of local features on head pose estimation in facial images is different. Since head rotation is a rigid change, each pixel is changing, and the relative positions of facial attributes are different. Based on this, we propose the following hypothesis: the head pose predicted by a single local feature is inaccurate, while the combination of local features has a significant impact on head pose. Therefore, we decompose the face into a set of patches that are repeatable and can provide semantic and structural information to describe the motion of rigid objects. However, a single patch is not acceptable because a single patch is often interfered by noise (background). What is obtained from the local features is the underlying potential spatial and semantic information related to the whole.

[0071] Leveraging the fact that structural equivariance can model pose variations due to head rotation, we employ a self-supervised strategy to disentangle pose-related information. To address this issue, we design a proxy task to extract three different orientation features in a self-supervised manner without external labels. This focuses more on instance-level discrimination rather than randomly augmenting samples to construct views.

[0072] To address these challenges, this paper proposes a self-supervised facial pose estimation architecture that achieves structural equivariance through a self-supervised learning strategy based on the external attributes and latent local associations of facial neighbors, and develops a corresponding head pose estimation system. Specifically, a complete neighborhood graph is constructed to capture the external attributes of facial neighbors, where different latent semantic attributes are assigned to each subgraph. A set of proxy tasks based on subgraphs with different attributes is designed to encourage the model to learn the underlying relationships between local features under pose variations.

[0073] like Figure 1 As shown, the present invention provides a head posture estimation method based on structure equivariance self-supervised learning, the steps comprising:

[0074] Step S1: input batch images and establish training set X = {x i}, where the batch images are enhanced images preprocessed by face detection and alignment operations;

[0075] Step S2: The batch images in the training set X are fed into the feature extractor network for learning to obtain facial semantic features of the batch images. Samples in the training set X are selected as anchor images. A neighbor graph G of the anchor images is further constructed in the feature space. The neighbor graph G is refined and adjusted using a pose-invariant loss function, so that each sample in the training set X controls different latent semantic attributes to enhance the diversity of feature learning.

[0076] Step S3: each sample x i The image is divided into T non-overlapping patches and mapped into the feature space. The local space equivariant loss function is used to calculate the difference between different patch positions in the refined and adjusted neighborhood graph G. The anchor image features are trained and adjusted based on a single patch, thus extending the features under different semantic factors to local images of different postures.

[0077] Step S4: Input the anchor features of the trained and adjusted anchor image into the softmax function and convert them into three angle posture vectors [y i ,p i ,r i ] and calculate the probability distribution data of facial posture labels, select the label with the largest probability as the head posture estimation result of the anchor image anchor, and the head posture estimation model is established;

[0078] Step S5: Input the image to be analyzed as the new anchor image anchor into the head posture estimation model, process the image to be analyzed according to steps S2-S4, obtain the facial posture label probability distribution data of the image to be analyzed, and convert the three angle posture vectors of the image to be analyzed into Euler angles.

[0079] like Figure 2 As shown in Figure 1, we first start with the input batch image and feed it into the feature extractor network E(·) for learning. After obtaining these facial features, we first construct a complete neighborhood graph (Neighborhood Graph G), where each subgraph provides potential semantic attributes, which are determined by the appearance and shape information of the face. In addition, in order to refine the relationship between different neighbors, we hope that each subgraph is dominated by a series of external attributes of the face (such as expression, gender, and occlusion), such as Figure 3 shown. Figure 3In the neighbor graph, each adjacent face is assigned to multiple subgraphs with different semantics. We cluster to obtain subspaces representing different semantics (expression, gender, and identical Euler angles), and construct a neighbor graph adjacent to the original image based on facial attributes. Our model aims to decouple related information by leveraging structural equivariance under the same semantics.

[0080] The specific implementation steps of step S2, which constructs a neighbor graph and adjusts it based on semantic features, include:

[0081] Step S21, sending the batch processed images into the feature extractor network for learning to obtain facial semantic features of the batch processed images;

[0082] Step S22: construct an initial neighbor graph G = {V, E} with node v as the anchor image anchor, where V is a set of nodes and E is a set of edges. The images corresponding to the nodes at both ends of the edge have the same visual structure under the same semantics. The initial neighbor graph G has A subgraphs. The nodes that make up the same subgraph are neighbor images of the anchor image anchor and all have the same visual structure under the same semantics.

[0083] Step S23, select one of the subgraphs G a Perform pose invariant loss calculation, a∈[1,A], and let the subgraph G a The node set in is represented as {n1,...,n i}, the initial neighbor graph G does not belong to the subgraph G a The node set inside is represented as U, and the result L is calculated by the posture invariant loss function i :

[0084]

[0085] Where u∈{n1,...,n i} represents the subgraph G a The node set in G, u′∈U represents a non-subgraph a The node set within, τ is the temperature coefficient, x i is the original sample;

[0086] Step S24, according to the result L of the posture invariant loss function i Adjust the spatial distance of each node in the initial neighbor graph G so that features with the same latent semantic factors in the feature space are clustered together and features with different latent semantic factors are kept away from each other;

[0087] Step S25, repeating steps S23-S24, based on each subgraph G aThe pose invariance loss is calculated and the spatial distance is adjusted. After A adjustments, the refined and adjusted neighbor graph G is obtained. The refined and adjusted neighbor graph G captures the correlation between different semantic factors in the image, thereby achieving better visual feature representation.

[0088] Reference Figure 2 In the second row, the model divides the image into local patches and maps them into the feature space. Patch features from the same location in the figure are marked with the same color. To avoid confusion, we analyze a pair of local patches, and the other patch features are marked in gray. Based on a given sub-image, the local space equivariant loss discovers the differences in facial structure in different semantic scenarios. The specific implementation of step S3, which segments the sample and performs anchor image anchor feature training and adjustment on a patch-by-patch basis, includes:

[0089] Step S31: Divide each sample into T non-overlapping patches and map them into the feature space, converting them into structurally equivariant facial features, where the sample x i Divide into T non-overlapping patches And mapped to the feature space to get the patch feature In the feature space P j (t) The corresponding patch feature is F j (t) , t∈[1,T], i∈[1,N], j∈[1,N], N is the total number of samples in the training set;

[0090] Step S32: Let sample x i Represents the anchor image anchor, {u1,u2,...u N-1} represents the training set X except sample x i For the sample set other than , the kth patch image in the anchor image anchor is regarded as the target patch feature F i (k) , will {u1,u2,...u N-1} each sample j and P i (k) Patch P at the same location j (k) The corresponding feature F j (k) As a positive sample, in the feature space, except F j (k) 、F i (k) All patch features other than are considered as negative samples, and the loss result L is calculated by the local space equivalent loss function. e :

[0091]

[0092] Among them, F i (k) Denotes the anchor patch P i (k) The characteristics of F j (k) Represents the non-anchor patch P at the same position j (k) The characteristics of F j (t) Represents patches P at different locations of non-anchor points j (t) characteristic, τ is the temperature coefficient; F i (t) Refers to the anchor image except F i (k) Other patches;

[0093] Step S33, repeat steps S32-S33, and calculate the loss result L of the local space equivalent loss function e Adjust the anchor patch P in the feature space i (k) With patch P i (t) , and {u1,u2,...u N-1}, and finally use structural equivariance to learn the compensatory features of the facial structure.

[0094] Please refer to Figure 4 An example of contrastive learning is shown in Figure 2. After learning the subspace, positive and negative sample pairs can be constructed for contrastive learning. The goal of contrastive learning is to push images with similar visual structure closer together, thereby achieving natural semantic clustering. Simultaneously, this approach models invariance within different instances while capturing latent spatial and semantic information in local features. In this way, the model learns correspondences between different facial identities, where images with similar visual structure are encouraged to be identical while their pose representations remain unchanged.

[0095] To make pose invariant across different latent semantic factors, our model encourages features belonging to the same sub-image to be similar and maximizes semantic similarity between all other sub-spaces. We then perform contrastive learning using local patch pairs sampled on different sub-images (instead of the same sub-image), as Figure 3As shown. In this way, we can construct the underlying local associations of the face, such as the spatial correspondence between the eyes and nose, rather than just homogeneous visual features. The pose-invariant loss discovers the same visual structure under the same semantics. Finally, to reduce the ambiguity between head pose labels, we convert a series of facial pose labels into a mean-variance label distribution in each direction to better fit the real-world situation. The specific implementation steps of step S4 include:

[0096] Step S41, converting the anchor image features trained and adjusted in step S33 into a posture vector [y i ,p i ,r i ], where y i 、p i 、r i They correspond to the three direction vectors of yaw angle, pitch angle and roll angle respectively;

[0097] Step S42: mapping the posture vector [y i ,p i ,r i ] is input into the sotamax function to obtain the probability distribution of facial posture labels, where the distribution is normalized to ensure

[0098]

[0099]

[0100]

[0101] Among them, l i ∈{l1,l2,…,l m} represents the binned pitch angle within the range of ±99°, m is the number of bins, σ represents the standard deviation of the Gaussian distribution and the finest granularity of the attitude angle;

[0102] Step S43, select y(l i ) value, p(l i ) value, r(l i ) value is taken as the head pose estimation result of the anchor image anchor.

[0103] The method of the present invention focuses on constructing a head pose estimation model, starting with an input batch of images and feeding it into a feature extractor network E(·). After obtaining these facial features, a neighborhood graph is first constructed, where each subgraph provides potential semantic attributes. The pose invariant loss discovers the same visual structure under the same semantics. At the same time, the model divides the image into local patches and maps them into the feature space. Patch features from the same position in the figure are marked with the same color. To avoid confusion, we analyze a pair of local patches and mark the other patch features in gray. Based on a given subgraph, the local space equivalence loss discovers the differences in facial structure under different semantic scenarios.

[0104] Based on the above head posture estimation method and model, the present invention provides a corresponding head posture estimation system. The system design framework diagram is as follows: Figure 5 As shown, the system of the present invention mainly consists of four parts: a face detection and alignment module, a feature extraction module, a head posture estimation module and a head posture visualization module.

[0105] Face detection and alignment module: input batch images, establish training set X = {x i};Batch images are enhanced images after preprocessing by face detection and alignment operations;;

[0106] Feature extraction module: The batch images in the training set X are fed into the feature extractor network for learning, the facial semantic features of the batch images are obtained, the samples in the training set X are selected as anchor images, the neighbor graph G about the anchor images is further constructed in the feature space, and the neighbor graph G is refined and adjusted through the pose invariant loss function, so that each sample in the training set X controls different potential semantic attributes to enhance the diversity of feature learning; each sample x is i The image is divided into T non-overlapping patches and mapped into the feature space. The local space equivariant loss function is used to calculate the difference between different patch positions in the refined and adjusted neighborhood graph G. The anchor image features are trained and adjusted based on a single patch, thus extending the features under different semantic factors to local images of different postures.

[0107] The head posture estimation module is used to input the anchor image features after training and adjustment into the softmax function and convert them into three angle posture vectors [y i ,p i ,r i ] and calculate the probability distribution data of facial posture labels, and select the label with the largest probability as the head posture estimation result of the anchor image anchor;

[0108] The head pose visualization module is used to visualize the head pose estimation results of the anchor image anchor; and to visualize the head pose estimation results of the image to be analyzed processed by the head pose estimation system.

[0109] The feature extraction module is used to:

[0110] Send the batch images into the feature extractor network for learning to obtain the facial semantic features of the batch images;

[0111] Construct an initial neighbor graph G = {V, E} with node v as the anchor image, where V is a set of nodes and E is a set of edges. The images corresponding to the nodes at both ends of the edge have the same visual structure under the same semantics. The initial neighbor graph G has A subgraphs. The nodes that make up the same subgraph are neighbor images of the anchor image, and all have the same visual structure under the same semantics.

[0112] Select one of the subgraphs G a Perform pose invariant loss calculation, a∈[1,A], and let the subgraph G a The node set in is represented as {n1,...,n i}, the initial neighbor graph G does not belong to the subgraph G a The node set inside is represented as U, and the result L is calculated by the posture invariant loss function i :

[0113]

[0114] Where u∈{n1,...,n i} represents the subgraph G a The node set in G, u′∈U represents a non-subgraph a The node set within, τ is the temperature coefficient, x i is the original sample;

[0115] According to the result of the pose invariant loss function L i Adjust the spatial distance of each node in the initial neighbor graph G so that features with the same latent semantic factors in the feature space are clustered together and features with different latent semantic factors are kept away from each other;

[0116] Repeat the above steps S23-S24, based on each subgraph G a The pose invariance loss is calculated and the spatial distance is adjusted. After A adjustments, the refined and adjusted neighbor graph G is obtained. The refined and adjusted neighbor graph G captures the correlation between different semantic factors in the image, thereby achieving better visual feature representation.

[0117] The feature extraction module is used to:

[0118] Each sample is divided into T non-overlapping patches and mapped into the feature space, converted into structurally equivariant facial features, where the sample x i Divide into T non-overlapping patches And mapped to the feature space to get the patch feature In the feature space P j (t) The corresponding patch feature is F j (t) , t∈[1,T], i∈[1,N], j∈[1,N], N is the total number of samples in the training set;

[0119]

[0120] Among them, F i (k) Denotes the anchor patch P i (k) The characteristics of F j (k) Represents the non-anchor patch P at the same position j (k) The characteristics of F j (t) Represents patches P at different locations of non-anchor points j (t) characteristic, τ is the temperature coefficient; F i (t) Refers to patches at different locations of the anchor image;

[0121] Repeat the above steps S32-S33, and calculate the loss result L of the local space equivalent loss function. e Adjust the anchor patch P in the feature space i (k) and the rest of the patches P in the anchor image anchor i (t) 、{u1,u2,...u N-1}, and finally use structural equivariance to learn the compensatory features of the facial structure.

[0122] The head pose estimation module is used to:

[0123] Convert the anchor image features after training adjustment into the pose vector [y i ,p i ,r i ], where y i 、p i 、r i They correspond to the three direction vectors of yaw angle, pitch angle and roll angle respectively;

[0124] The pose vector [y i ,p i ,t i ] is input into the sotamax function to obtain the probability distribution of facial posture labels, where the distribution is normalized to ensure

[0125]

[0126]

[0127]

[0128] Among them, l i ∈{l1,l2,…,l m} represents the binned pitch angle within the range of ±99°, m is the number of bins, σ represents the standard deviation of the Gaussian distribution and the finest granularity of the attitude angle;

[0129] Select y(l i ) value, p(l i ) value, r(l i ) value is taken as the head pose estimation result of the anchor image anchor.

[0130] The system provided by the present invention is based on the PyTorch framework and is developed using python+pyqt5. The overall programming environment of the system is ubuntu18.04, and the programming language of the algorithm model and the system interface UI design is unified in Python. The main functions of the system are model training, model testing and result visualization (such as Figure 6 、 7 , 8). The model training module can train the model by modifying the image path and related training parameters, and monitor the log files during training. The head posture verification module verifies the performance of the model by testing and converging the training model. The head posture visualization module inputs a video in any format and plots the Euler angles of the head posture in the video through the Matlabplot library. MCTNN is mainly used for face detection and face alignment modules of input images, which belongs to the normalization processing of data. After correctly estimating the head posture of the human body, the Euler angles are plotted with the help of the matlabplot library, and the head posture is visualized in the video. Figure 9This visualization shows the head pose estimation results for the AFLW2000 dataset using the first protocol. The three lines represent the front, downward, and side of the face, respectively. This system implements the key functions of head pose estimation with a user-friendly interface, simple operation, and stability and reliability.

[0131] Our method is compared with the state-of-the-art methods. As can be seen from the results, our model achieves competitive performance. The specific contributions and advantages can be summarized as follows:

[0132] (1) Traditional methods, such as FAN, Dlib, and 3DDFA, introduce excessive external information by utilizing facial key points or 3D facial models to help predict head pose. Therefore, the performance of head pose estimation is highly dependent on facial key points and external 3D facial models. Our method directly predicts head pose from RGB images without using external information, which mitigates the impact of introducing external information and is more convenient in the real world.

[0133] (2) Keypoint-free methods such as FSA-Net use feature aggregation by grouping pixel-level features encoded with spatial information. However, although it utilizes feature aggregation, this method cannot clearly separate pose-related information from complex information.

[0134] (3) ASGLearning is based on the probability distribution of rotation. The gap between our method and ASG is very small. ASG calculates the sample points on the sphere through the probability density distribution during the training phase, effectively utilizing the supervisory signal of the pose label. Our method uses structural equivariance to capture more information related to the pose without introducing additional supervisory information. To better demonstrate the effectiveness of our method, we visualize the predicted regression pose. The results show that the angles predicted by our model are closer to the actual situation.

[0135] The above disclosure is only a preferred embodiment of the present invention, and it is certainly not intended to limit the scope of the present invention. A person skilled in the art can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A head pose estimation method based on structure-isovariant self-supervised learning, characterized in that: include: Step S1: input batch images and establish training set X = {x i The batch processed images are enhanced images that have been pre-processed by face detection and alignment operations; Step S2: feeding the batch images in the training set X into a feature extractor network for learning to obtain facial semantic features of the batch images, selecting samples in the training set X as anchor images, further constructing a neighbor graph G about the anchor images in the feature space, and refining and adjusting the neighbor graph G using a pose-invariant loss function, so that each sample in the training set X controls different latent semantic attributes to enhance the diversity of feature learning; Step S3: each of the samples x i The image is divided into T non-overlapping patches and mapped into the feature space. The local space equivariant loss function is used to calculate the difference between the positions of different patches in the refined and adjusted neighborhood graph G. The anchor feature of the anchor image is trained and adjusted based on a single patch, so as to extend the features under different semantic factors to local images of different postures. Step S4: Input the anchor features of the trained and adjusted anchor image into the softmax function and convert them into three angle posture vectors [y i ,p i ,r i ] and calculate the probability distribution data of facial posture labels, select the label with the largest probability as the head posture estimation result of the anchor image anchor, and the head posture estimation model is established; Step S5: Input the image to be analyzed as a new anchor image anchor into the head pose estimation model, process the image to be analyzed according to steps S2-S4, obtain the facial pose label probability distribution data and head pose estimation result of the image to be analyzed, and convert the three angle pose vectors of the image to be analyzed into Euler angles.

2. The head pose estimation method based on structure-isovariant self-supervised learning according to claim 1, characterized in that: The step S2 comprises: Step S21, sending the batch processed images into a feature extractor network for learning to obtain facial semantic features of the batch processed images; Step S22: construct an initial neighbor graph G = {V, E} with node v as the anchor image anchor, where V is a set of nodes, E is a set of edges, and the images corresponding to the nodes at both ends of the edge have the same visual structure under the same semantics. The initial neighbor graph G has A subgraphs, and the nodes that constitute the same subgraph are neighbor images of the anchor image anchor, and all have the same visual structure under the same semantics; Step S23, select one of the subgraphs G a Perform pose invariant loss calculation, a∈[1,A], let the subgraph G a The node set in is represented as {n1,...,n i }, the initial neighbor graph G does not belong to the subgraph G a The node set inside is represented as U, and the result L is calculated by the posture invariant loss function i : Where u∈{n1,...,n i } represents the subgraph G a The node set in G, u′∈U represents a non-subgraph G a The node set within, τ is the temperature coefficient, x i is the original sample; Step S24, according to the result L of the posture invariant loss function i Adjusting the spatial distances of the nodes in the initial neighbor graph G so that features with the same latent semantic factors in the feature space are clustered together and features with different latent semantic factors are kept apart; Step S25, repeating steps S23-S24, based on each of the subgraphs G a The pose invariance loss is calculated and the spatial distance is adjusted. After A times of adjustment, the refined and adjusted neighbor graph G is obtained. The refined and adjusted neighbor graph G captures the correlation between different semantic factors in the image, thereby achieving better visual feature representation.

3. The head pose estimation method based on structure-isovariant self-supervised learning according to claim 2, characterized in that: The step S3 comprises: Step S31, dividing each sample into T non-overlapping patches and mapping them into the feature space, and converting them into structurally equivariant facial features, wherein the sample x i Divide into T non-overlapping patches And mapped to the feature space to get the patch features In the feature space P j (t) The corresponding patch feature is F j (t) , t∈[1,T], i∈[1,N], j∈[1,N], N is the total number of samples in the training set; Among them, F i (k) Denotes the anchor patch P i (k) The characteristics of F j (k) represents the patch P at the same position as the anchor point j (k) The characteristics of F j (t) represents a patch P at a different position than the anchor point j (t) characteristic, τ is the temperature coefficient; F i (t) Refers to the anchor image except F i (k) Other patches; Step S33, repeating steps S32-S33, according to the loss result L of the local space equivalent loss function e Adjust the anchor patch P in the feature space i (k) With patch P i (t) , and {u1,u2,...u N-1 }, and finally use structural equivariance to learn the compensatory features of the facial structure.

4. The head posture estimation method based on structure-isovariant self-supervised learning according to claim 3, characterized in that: The step S4 comprises: Step S41, converting the anchor image features trained and adjusted in step S33 into a posture vector [y i ,p i ,r i ], where y i 、p i 、r i They correspond to the three direction vectors of yaw angle, pitch angle and roll angle respectively; Step S42, mapping the posture vector [y i ,p i ,r i ] is input into the sotamax function to obtain the probability distribution of facial posture labels, where the distribution is normalized to ensure Among them, l i ∈{l1,l2,…,l m } represents the binned pitch angle within the range of ±99°, m is the number of bins, σ represents the standard deviation of the Gaussian distribution and the finest granularity of the attitude angle; Step S43, select y(l i ) value, p(l i ) value, r(l i ) value is taken as the head pose estimation result of the anchor image anchor.

5. A head pose estimation system based on structure-isovariant self-supervised learning, implementing the method of any one of claims 1 to 4, characterized in that: include: Face detection and alignment module: input batch images, establish training set X = {x i The batch processed images are enhanced images that have been pre-processed by face detection and alignment operations; Feature extraction module: feeding the batch images in the training set X into a feature extractor network for learning, obtaining facial semantic features of the batch images, selecting samples in the training set X as anchor images, further constructing a neighbor graph G about the anchor images in the feature space, and refining the neighbor graph G using a pose-invariant loss function, so that each sample in the training set X controls different latent semantic attributes to enhance the diversity of feature learning; Each of the samples x i The image is divided into T non-overlapping patches and mapped into the feature space. The local space equivariant loss function is used to calculate the difference between the positions of different patches in the refined and adjusted neighborhood graph G. The anchor feature of the anchor image is trained and adjusted based on a single patch, so as to extend the features under different semantic factors to local images of different postures. The head posture estimation module is used to input the anchor image features after training and adjustment into the softmax function and convert them into three angle posture vectors [y i ,p i ,r i ] and calculate the probability distribution data of facial posture labels, and select the label with the largest probability as the head posture estimation result of the anchor image anchor; A head posture visualization module is used to visualize the head posture estimation result of the anchor image anchor; And visually display the head posture estimation results of the image to be analyzed processed by the head posture estimation system.

6. The head pose estimation system based on structure-isovariant self-supervised learning according to claim 5, characterized in that: The feature extraction module is used to: Sending the batch processed images into a feature extractor network for learning to obtain facial semantic features of the batch processed images; Construct an initial neighbor graph G = {v, E} with node v as the anchor image anchor, where V is a set of nodes and E is a set of edges. The images corresponding to the nodes at both ends of the edge have the same visual structure under the same semantics. The initial neighbor graph G has A subgraphs. The nodes that constitute the same subgraph are neighbor images of the anchor image anchor and all have the same visual structure under the same semantics. Select one of the subgraphs G a Perform pose invariant loss calculation, a∈[1,A], let the subgraph G a The node set in is represented as {n1,...,n i }, the initial neighbor graph G does not belong to the subgraph G a The node set inside is represented as U, and the result L is calculated by the posture invariant loss function i : Where u∈{n1,...,n i } represents the subgraph G a The node set in G, u′∈U represents a non-subgraph G a The node set within, τ is the temperature coefficient, x i is the original sample; According to the result of the pose invariant loss function L i Adjusting the spatial distances of the nodes in the initial neighbor graph G so that features with the same latent semantic factors in the feature space are clustered together and features with different latent semantic factors are kept apart; Repeat steps S23-S24 in claim 2, based on each of the subgraphs G a The pose invariance loss is calculated and the spatial distance is adjusted. After A times of adjustment, the refined and adjusted neighbor graph G is obtained. The refined and adjusted neighbor graph G captures the correlation between different semantic factors in the image, thereby achieving better visual feature representation.

7. The head pose estimation system based on structure-isovariant self-supervised learning according to claim 6, characterized in that: The feature extraction module is used to: Each sample is divided into T non-overlapping patches and mapped into the feature space, and converted into structurally equivariant facial features, where the sample x i Divide into T non-overlapping patches And mapped to the feature space to get the patch features In the feature space P j (t) The corresponding patch feature is F j (t) , t∈[1,T], i∈[1,N], j∈[1,N], N is the total number of samples in the training set; Among them, F i (k) Denotes the anchor patch P i (k) The characteristics of F j (k) represents the patch P at the same position as the anchor point j (k) The characteristics of F j (t) represents a patch P at a different position than the anchor point j (t) characteristic, τ is the temperature coefficient; F i (t) Refers to patches at different positions of the anchor image; Repeat steps S32-S33 in claim 3, and calculate the loss result L according to the local space equivalent loss function. e Adjust the anchor patch P in the feature space i (k) and the rest of the patches P in the anchor image anchor i (t) 、{u1,u2,...u N-1 }, and finally use structural equivariance to learn the compensatory features of the facial structure.

8. The head pose estimation system based on structure-isovariant self-supervised learning according to claim 7, characterized in that: The head posture estimation module is used to: Convert the anchor image features after training adjustment into the pose vector [y i ,p i ,r i ], where y i 、p i 、r i They correspond to the three direction vectors of yaw angle, pitch angle and roll angle respectively; The pose vector [y i ,p i ,r i ] is input into the sotamax function to obtain the probability distribution of facial posture labels, where the distribution is normalized to ensure Among them, l i ∈{l1,l2,…,l m } represents the binned pitch angle within the range of ±99°, m is the number of bins, σ represents the standard deviation of the Gaussian distribution and the finest granularity of the attitude angle; Select y(l i ) value, p(l i ) value, r(l i ) value is taken as the head pose estimation result of the anchor image anchor.

Images