Head pose estimation method based on visual transformer

Abstract

The application belongs to the field of computer vision, and particularly relates to a head pose estimation method based on a visual Transformer, comprising the following steps: constructing a head pose estimation model; obtaining an image to be detected, inputting the image into the trained head pose estimation model, and obtaining a head pose estimation result; the head pose estimation model comprises a visual Transformer pre-training network module, a multi-scale Transformer encoder module, a CNN module and a prediction head module; the head pose estimation method can introduce a flipped image through an image flipping operation, can provide self-supervised information for an original image, and through this way, the model can learn to reverse features of pose changes, so that the model can capture the similarity under different poses, and further improve the accuracy and stability of head pose estimation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a head pose estimation method based on visual Transformer. Background Technology

[0002] Head pose estimation (HPE) has been widely applied in human-computer interaction, autonomous driving, and classroom monitoring. Currently, many methods focus on representing head pose or orientation and have achieved impressive performance. However, HPE methods in classroom scenarios often neglect issues such as the size of the human head in surveillance video and inaccuracies caused by different camera placements. Furthermore, HPE tasks often ignore the inherent facial relationships, such as the inherent symmetry of facial information. Therefore, this invention uses flipped images to provide self-supervised information to the original image. However, since some precise facial relationships are difficult to learn with existing CNN architectures, the Transformer architecture has a remarkable ability to learn long-range, high-level relationships between image patches, enabling better or more accurate facial information.

[0003] In the field of multi-person head pose estimation, single-scale methods face challenges beyond common difficulties such as aliasing and scale inconsistencies. They also involve the accurate separation and identification of pose information from multiple individuals. Due to the significant differences in scale, position, and pose variations among different heads, single-scale methods often struggle to adequately adapt to this diversity, potentially leading to a sharp decline in pose estimation accuracy. Furthermore, single-scale methods may fail to accurately handle conflicting information such as head overlap, occlusion, and mutual interference, resulting in errors in the output. Simultaneously, single-scale methods typically fail to comprehensively consider the global features of multi-person head poses, tending to be limited to local information and leading to insufficient understanding of the overall pose. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention proposes a head pose estimation method based on visual Transformer. The method includes: constructing a head pose estimation model; acquiring an image to be detected; inputting the image into the trained head pose estimation model to obtain a head pose estimation result; the head pose estimation model includes a visual Transformer pre-trained network module, a multi-scale Transformer encoder module, a CNN module, and a predictive head module.

[0005] Training the head pose estimation model includes: acquiring a head pose dataset and performing a random scaling transformation on the data in the dataset to obtain a training set; flipping the data in the training set and inputting the original image and the flipped image into the visual Transformer pre-trained network module and CNN module respectively for feature extraction to obtain the corresponding feature serialization representation; adding a learnable d-dimensional vector to the feature serialization representation to obtain a fused sequence; inputting the fused sequence into multiple multi-scale Transformer encoder modules and obtaining the interactive feature vector through multi-scale interaction strategies and image enhancement strategies; fusing the interactive feature vector with the feature vector extracted by the CNN module; converting the fused features of the original image and the flipped image into rotation matrices, multiplying the rotation matrix of the flipped image with the learnable matrix, and then fusing it with the rotation matrix of the original image; inputting the fused rotation matrix into the prediction head module to obtain the head pose estimation result; calculating the model's loss function, adjusting the model parameters, and completing the model training when the loss function value is minimized.

[0006] The beneficial effects of this invention are:

[0007] This invention designs a multi-scale fusion strategy and an image enhancement strategy based on the Transformer network architecture. First, by introducing a folded image through an image folding operation, self-supervised information is provided to the original image. In this way, the model can learn the inverse features of pose changes, enabling it to capture similarities across different poses and further improving the accuracy and stability of head pose estimation. Simultaneously, this invention adds a lightweight CNN branch to the original Transformer architecture to provide multi-scale information to the original features. Through the gradual fusion of original and multi-scale features, the model adapts to head images at different scales, allowing it to capture subtle changes and features in head pose more comprehensively and accurately, thus achieving more precise pose estimation and prediction. Attached Figure Description

[0008] Figure 1 This is a framework diagram of a head pose estimation method based on visual Transformer according to the present invention;

[0009] Figure 2 This is a structural diagram of the multi-scale Transformer encoder module of the present invention;

[0010] Figure 3 This is a structural diagram of the attention interaction module of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] A head pose estimation method based on visual Transformer is disclosed. The method includes: constructing a head pose estimation model; acquiring an image to be detected, inputting the image into the trained head pose estimation model, and obtaining a head pose estimation result; the head pose estimation model includes a visual Transformer pre-trained network module, a multi-scale Transformer encoder module, a CNN module, and a predictive head module.

[0013] Training the head pose estimation model includes: acquiring a head pose dataset and performing a random scaling transformation on the data in the dataset to obtain a training set; flipping the data in the training set and inputting the original image and the flipped image into the visual Transformer pre-trained network module and CNN module respectively for feature extraction to obtain the corresponding feature serialization representation; adding a learnable d-dimensional vector to the feature serialization representation to obtain a fused sequence; inputting the fused sequence into multiple multi-scale Transformer encoder modules and obtaining the interactive feature vector through multi-scale interaction strategies and image enhancement strategies; fusing the interactive feature vector with the feature vector extracted by the CNN module; converting the fused features of the original image and the flipped image into rotation matrices, multiplying the rotation matrix of the flipped image with the learnable matrix, and then fusing it with the rotation matrix of the original image; inputting the fused rotation matrix into the prediction head module to obtain the head pose estimation result; calculating the model's loss function, adjusting the model parameters, and completing the model training when the loss function value is minimized.

[0014] In this embodiment, as Figures 1-3As shown, the head pose estimation model framework illustrates the specific implementation of this invention. In this method, the multi-scale feature fusion strategy is applied in parallel with two branches. Specifically, in the Transformer, for the attention mechanism in each layer, two features are input each time: one from the Transformer branch and one from the CNN branch. The calculation result obtained from the attention matrix is ​​simultaneously passed to the feedforward layer and the CNN branch. After being combined with the channel attention from the CNN branch, it is input to the next layer. At the end of the model, the predicted rotation matrix from the Transformer branch and the predicted rotation matrix from the flipped image of the CNN branch are extracted and fused. Finally, the Euler angles are predicted through an MLP layer. Through this design, this invention can fully utilize the multi-scale feature fusion strategy and image enhancement strategy, effectively utilizing multi-scale information and facial symmetry information to enable the model to capture subtle changes and features in head pose more comprehensively and accurately, thereby achieving more accurate pose estimation and prediction. The invention is based on a general Transformer model compression strategy; therefore, in other head pose estimation tasks, corresponding strategies can also be flexibly applied as needed.

[0015] A specific implementation of a head pose estimation method based on visual Transformer, the specific process of which includes:

[0016] S1: Perform random scaling transformation on the existing head pose dataset and use it to train the head pose estimation model.

[0017] S2: Construct a head pose estimation model; the model includes a visual Transformer pre-trained network module, a multi-scale Transformer encoder module, a CNN module, and a prediction head module;

[0018] S3: Design a multi-scale visual Transformer model strategy suitable for the above model; the strategy includes a multi-scale interaction strategy and an image enhancement strategy;

[0019] S4: Input the images from the training dataset into the head pose estimation model to train the model;

[0020] S5: Input the image of the head to be detected into the trained model to obtain the Euler angle of the head in the image.

[0021] In this embodiment, training the model includes:

[0022] S41: Input the head image and the folded head image into the visual Transformer pre-trained network module and the CNN network module for feature extraction to obtain a series of feature maps. Then, the feature maps of different scales extracted by the CNN network module are fused together. After that, a global average pooling layer is used to reduce the spatial dimension to form a feature vector of a fixed size, and a serialized representation is obtained.

[0023] S42: The obtained feature vector sequence is added to 11 learnable d-dimensional vectors, which are used to represent 11 basic directional regions to enhance the model's ability to judge different orientations of the human head in the head image.

[0024] S43: The fused sequence is fed into a series of multi-scale Transformer modules to obtain feature representations with multi-scale information. Each multi-scale Transformer module consists of a cross attention mechanism, a squeeze-excitation module, and a feedforward neural network. For this series of multi-scale Transformer modules, a multi-scale feature fusion strategy is adopted to gradually integrate multi-scale information into the features extracted by the visual Transformer, so that the model can capture subtle changes and features of head pose more comprehensively and accurately.

[0025] S44: The cross Attention module performs an attention operation on the query from the CNN branch and the feature vector encoded by the visual Transformer encoder module to obtain a feature representation related to the query.

[0026] S45: Fuse the feature vectors after interaction with the feature vectors in the CNN branch;

[0027] S46: Convert the feature vectors of the original image and the flipped image into a predicted rotation matrix. Multiply the rotation matrix of the flipped image with the learnable matrix and then fuse it with the rotation matrix of the original image.

[0028] S47: Use the fused prediction rotation matrix as input to the MLP to predict the Euler angle of the head.

[0029] S48: The geodesic distance loss, contrast loss, and orientation token loss are used as the final loss functions of the head pose estimation model. During model training, the model is trained by minimizing the loss functions.

[0030] For the cross-attention operation in the multi-scale Transformer module, the attention matrix is:

[0031]

[0032] Among them, Qc The query is composed of feature maps from a CNN branch, where d represents the dimension of the embedding matrix and h represents the index of the attention head.

[0033] CNN branch output matrix A c It can be obtained from the following formula:

[0034] A c =Softmax(SELayer(A c ))

[0035] The fused attention matrix is ​​A e for:

[0036]

[0037] in and W s A respectively c and A s The weight matrix.

[0038] By multiplying the predicted rotation matrix of the flipped image with a learnable weight matrix and then fusing it with the predicted rotation matrix of the original image, symmetry information can be effectively utilized to enhance the model's representational ability and improve the accuracy of head pose estimation. The fused 3D predicted rotation matrix M g for:

[0039]

[0040] Where W is the learnable weight matrix, and M... x To predict the rotation matrix for a flipped image, M y This is the original image rotation prediction matrix.

[0041] The model's prediction is represented by a rotation matrix, and geodesic distance is used as the loss between two 3D rotations. By applying geodesic distance as the loss between the original image prediction rotation matrix and the flipped image prediction rotation matrix, the comparison loss is:

[0042]

[0043] Where M1 is the original image prediction rotation matrix, M2 is the flipped image prediction rotation matrix, and tr is the trace of the matrix.

[0044] The actual predicted loss is:

[0045]

[0046] Where M is the true rotation matrix, M g This is the predicted rotation matrix after fusion.

[0047] Information can be encoded into direction tokens using direction token loss, which is defined as the geodesic distance relative to the corresponding direction region. Therefore, the direction token loss is defined as:

[0048]

[0049] Where 11 represents the number of basic orientation regions, and M is the true rotation matrix. i Let M be the predicted rotation matrix for the i-th region, and I(M,i) be an indicator function used to determine whether the true head pose is located in the i-th base region. I(M,i) can be expressed as:

[0050]

[0051] The total loss consists of the contrast loss, the true prediction loss, and the direction token loss. The total loss can be expressed as:

[0052] L=αL1+βL g +(1-β)L ori

[0053] Here, α and β represent hyperparameters used to balance the weights between different losses.

[0054] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A head pose estimation method based on visual Transformer, characterized in that, include: A head pose estimation model is constructed; the image to be detected is acquired, and the image is input into the trained head pose estimation model to obtain the head pose estimation result; the head pose estimation model includes a visual Transformer pre-trained network module, a multi-scale Transformer encoder module, a CNN module, and a prediction head module; Training the head pose estimation model includes: acquiring a head pose dataset and performing a random scaling transformation on the data in the dataset to obtain a training set; flipping the data in the training set and inputting the original image and the flipped image into the visual Transformer pre-trained network module and the CNN module respectively for feature extraction to obtain the corresponding feature serialization representation; adding a learnable d-dimensional vector to the feature serialization representation to obtain a fusion sequence; inputting the fusion sequence into multiple multi-scale Transformer encoder modules and obtaining the interactive feature vector through multi-scale interaction strategies and image enhancement strategies; fusing the interactive feature vector with the feature vector extracted by the CNN module; converting the fusion features of the original image and the flipped image into rotation matrices, multiplying the rotation matrix of the flipped image with the learnable matrix, and then fusing it with the rotation matrix of the original image; inputting the fused rotation matrix into the prediction head module to obtain the head pose estimation result; calculating the model's loss function, adjusting the model parameters, and completing the model training when the loss function value is minimized; The fusion of the interactive feature vector with the feature vector extracted by the CNN module includes: multiplying the predicted rotation matrix of the flipped image by the learnable weight matrix and then fusing it with the original image's predicted rotation matrix; the fused 3D predicted rotation matrix M g for: Where W is the learnable weight matrix, and M... x To predict the rotation matrix for a flipped image, M y The original image rotation prediction matrix; By applying geodesic distance as the loss between the original image prediction rotation matrix and the flipped image prediction rotation matrix, the comparison loss is: Where M1 is the original image prediction rotation matrix, M2 is the flipped image prediction rotation matrix, and tr is the trace of the matrix; The actual predicted loss is: Where M is the true rotation matrix, M g Predict the rotation matrix after fusion; The loss of the direction token is: Where 11 represents the number of basic orientation regions, and M is the true rotation matrix. i Let I(M,i) be the prediction rotation matrix for the i-th region, and let I(M,i) be an indicator function. The total loss is: L=αL1+βL g +(1-β)L ori Here, α and β represent hyperparameters.

2. The head pose estimation method based on visual Transformer according to claim 1, characterized in that, The Visual Transformer pre-trained network module processes the input image by: extracting features from the head image using the Visual Transformer feature extractor, and adding the resulting feature vector sequence to 11 learnable d-dimensional vectors.

3. The head pose estimation method based on visual Transformer according to claim 1, characterized in that, The CNN module processes the input image by: inputting the image into the CNN module, extracting three feature maps of different dimensions, feeding the different feature maps into a linear layer for dimensionality reduction, fusing the dimensionality-reduced features, and adding the fused features to 11 learnable d-dimensional vectors.

4. The head pose estimation method based on visual Transformer according to claim 1, characterized in that, The multi-scale Transformer encoder module consists of a cross attention mechanism, a squeeze-excitation module, and a feedforward neural network.

5. The head pose estimation method based on visual Transformer according to claim 1, characterized in that, The multi-scale interaction strategy for multi-scale feature interaction includes: passing the feature vector output by the CNN module through a normalization layer and inputting it into the attention interaction layer; using the self-attention mechanism to use the feature vector as the query vector; using the vector extracted by the feature extraction module as the key vector and value vector; and performing self-attention operation on the query vector, key vector, and value vector.

6. The head pose estimation method based on visual Transformer according to claim 1, characterized in that, The process of fusing the interactive feature vector with the feature vector extracted by the CNN module includes: adding the feature vector of the left branch of the CNN module before it passes through the linear layer to the interactive feature vector; feeding the feature vector after the self-attention operation into the right branch for transpose and multiplication; and finally feeding the feature vectors of the left and right branches into the next module after passing through the normalization layer.

7. The head pose estimation method based on visual Transformer according to claim 1, characterized in that, Converting the fusion features of the original image and the fusion features of the flipped image into rotation matrices involves: two feature vectors output by the multi-scale Transformer module, one derived from the original head image and the other from the flipped image. After processing, the predicted rotation matrices for the original image and the predicted rotation matrices for the flipped image are obtained.

Images