Head posture estimation method based on visual Transform

Abstract

The invention belongs to the field of computer vision, and particularly relates to a head posture estimation method based on vision Transform, which comprises the following steps: constructing a head posture estimation model; obtaining a to-be-detected image, and inputting the image into the trained head posture estimation model to obtain a head posture estimation result; the head posture estimation model comprises a visual Transform pre-training network module, a multi-scale Transform encoder module, a CNN (Convolutional Neural Network) module and a prediction head module; according to the method, the folded image is introduced through the image folding operation, self-supervision information can be provided for the original image, in this way, the model can learn reverse features of posture changes, the model can capture similarities under different postures, and the accuracy and stability of head posture estimation are further improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a head posture estimation method based on visual Transformer. Background Art

[0002] Head pose estimation (HPE) has been widely used in fields such as human-computer interaction, autonomous driving, and classroom monitoring. At present, many methods focus on the representation of head pose or direction and have achieved impressive performance. The head pose estimation method in the classroom scene often ignores the size of the head in the monitoring video and the inaccurate estimation caused by the different camera positions. In the head pose estimation task, the intrinsic facial part relationship of the face, such as the intrinsic relationship of facial information symmetry, is often ignored. Therefore, the present invention uses the folded image to provide self-supervisory information for the original image. However, since some precise facial relationships are difficult to learn by the existing CNN architecture, the Transformer architecture has the extraordinary ability to learn long-distance and high-level relationships between image blocks, which can better or more accurate facial information.

[0003] In the field of multi-person head pose estimation, the challenges faced by single-scale methods include not only common difficulties such as aliasing and scale inconsistency, but also the correct separation and identification of multiple individual head pose information. Due to the huge differences in the scale, position, and pose of different people's heads, single-scale methods often find it difficult to fully adapt to this diversity, which may lead to a sharp drop in the accuracy of pose estimation. In addition, for problems such as head overlap, occlusion, and mutual interference, single-scale methods may not be able to accurately handle these conflicting information, resulting in errors in the output results. At the same time, single-scale methods usually find it difficult to fully consider the global characteristics of multi-person head poses, and are easily limited to local information, resulting in insufficient understanding of the overall pose. Summary of the invention

[0004] In order to solve the problems existing in the above prior art, the present invention proposes a head posture estimation method based on visual Transformer, which comprises: constructing a head posture estimation model; acquiring an image to be detected, inputting the image into a trained head posture estimation model, and obtaining a head posture estimation result; the head posture estimation model comprises a visual Transformer pre-trained network module, a multi-scale Transformer encoder module, a CNN module, and a prediction head module;

[0005] Training the head posture estimation model includes: obtaining a head posture data set, and performing random scale transformation on the data in the data set to obtain a training set; flipping the data in the training set, and inputting the original image and the flipped image into the visual Transformer pre-training network module and the CNN module respectively for feature extraction to obtain the corresponding feature serialization representation; adding a learnable d-dimensional vector to the feature serialization representation to obtain a fused sequence; inputting the fused sequence into multiple multi-scale Transformer encoder modules, and obtaining an interactive feature vector through a multi-scale interaction strategy and an image enhancement strategy; fusing the interactive feature vector with the feature vector extracted by the CNN module; converting the fused features of the original image and the fused features of the flipped image into a rotation matrix, and multiplying the rotation matrix of the flipped image with the learnable matrix and fusing it with the rotation matrix of the original image; inputting the fused rotation matrix into the prediction head module to obtain the head posture estimation result; calculating the loss function of the model, adjusting the model parameters, and completing the model training when the loss function value is minimized.

[0006] Beneficial effects of the present invention:

[0007] The present invention designs a multi-scale fusion strategy and an image enhancement strategy based on the Transformer network architecture. First, by introducing a folded image through an image folding operation, self-supervision information can be provided for the original image. In this way, the model can learn the reverse features of posture changes, so that the model can capture the similarities under different postures, and further improve the accuracy and stability of head posture estimation. At the same time, the present invention adds a lightweight CNN branch to the original Transformer architecture to provide multi-scale information for the original features. Through the gradual fusion of the original features and the multi-scale features, the model is adapted to head images of different scales. The model can capture the subtle changes and features of the head posture more comprehensively and accurately, thereby achieving more accurate posture estimation and prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a framework diagram of a head posture estimation method based on visual Transformer of the present invention;

[0009] Figure 2 It is a structural diagram of the multi-scale Transformer encoder module of the present invention;

[0010] Figure 3 This is a structural diagram of the attention interaction module of the present invention. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] A head posture estimation method based on visual Transformer, the method comprising: constructing a head posture estimation model; acquiring an image to be detected, inputting the image into a trained head posture estimation model, and obtaining a head posture estimation result; the head posture estimation model comprises a visual Transformer pre-trained network module, a multi-scale Transformer encoder module, a CNN module, and a prediction head module.

[0013] Training the head posture estimation model includes: obtaining a head posture data set, and performing random scale transformation on the data in the data set to obtain a training set; flipping the data in the training set, and inputting the original image and the flipped image into the visual Transformer pre-training network module and the CNN module respectively for feature extraction to obtain the corresponding feature serialization representation; adding a learnable d-dimensional vector to the feature serialization representation to obtain a fused sequence; inputting the fused sequence into multiple multi-scale Transformer encoder modules, and obtaining an interactive feature vector through a multi-scale interaction strategy and an image enhancement strategy; fusing the interactive feature vector with the feature vector extracted by the CNN module; converting the fused features of the original image and the fused features of the flipped image into a rotation matrix, and multiplying the rotation matrix of the flipped image with the learnable matrix and fusing it with the rotation matrix of the original image; inputting the fused rotation matrix into the prediction head module to obtain the head posture estimation result; calculating the loss function of the model, adjusting the model parameters, and completing the model training when the loss function value is minimized.

[0014] In this embodiment, if Figure 1 to Figure 3As shown, the head posture estimation model framework shows the specific implementation of the present invention. In this method, the multi-scale feature fusion strategy is applied in parallel with two branches. Specifically, in the Transformer, for the attention mechanism in each layer, two features are input each time, one from the Transformer branch and the other from the CNN branch. The calculation result obtained from the self-attention matrix is ​​simultaneously passed to the feedforward layer and the CNN branch, and after being combined with the channel attention through the CNN branch, it is input to the next layer of branches. At the end of the model, the predicted rotation matrix of the Transformer branch and the predicted rotation matrix of the folded image of the CNN branch are taken out, fused, and finally the Euler angle is predicted through the MLP layer. Through such a design, the present invention can make full use of the multi-scale feature fusion strategy and image enhancement strategy, and effectively use multi-scale information and facial symmetry information to enable the model to capture the subtle changes and features of the head posture more comprehensively and accurately, thereby achieving more accurate posture estimation and prediction. The invention is based on a general Transformer model compression strategy, so in other head posture estimation tasks, the corresponding strategy can also be flexibly applied as needed.

[0015] A specific implementation of a head posture estimation method based on a visual Transformer, the specific process includes:

[0016] S1: Randomly scale the existing head pose dataset and use it to train the head pose estimation model;

[0017] S2: construct a head posture estimation model; the model includes a visual Transformer pre-trained network module, a multi-scale Transformer encoder module, a CNN module, and a prediction head module;

[0018] S3: Design a multi-scale visual Transformer model strategy suitable for the above model; the strategy includes a multi-scale interaction strategy and an image enhancement strategy;

[0019] S4: Input the images in the training data set into the head posture estimation model to perform model training;

[0020] S5: Input the head image to be detected into the trained model, and obtain the Euler angle size of the head in the head image.

[0021] In this embodiment, training the model includes:

[0022] S41: The head image and the folded head image are input into the visual Transformer pre-trained network module and the CNN network module for feature extraction to obtain a series of feature maps. The feature maps of different scales extracted by the CNN network module are then fused. After that, a global average pooling layer is used to reduce the spatial dimension to form a feature vector of a fixed size to obtain a serialized representation.

[0023] S42: adding the obtained feature vector sequence to 11 learnable d-dimensional vectors, which are used to represent 11 basic direction areas, so as to enhance the model's ability to judge different orientations of the human head in the human head image.

[0024] S43: The fused sequence is sent to a series of multi-scale Transformer modules to obtain feature representations with multi-scale information. Each multi-scale Transformer module consists of a cross attention mechanism, a squeeze-excitation module, and a feed-forward neural network. For this series of multi-scale Transformer modules, a multi-scale feature fusion strategy is adopted to gradually integrate multi-scale information into the features extracted by the visual Transformer, so that the model can capture subtle changes and features of head posture more comprehensively and accurately.

[0025] S44: The cross attention module performs an attention operation on the query from the CNN branch and the feature vector encoded by the visual Transformer encoder module to obtain feature representations related to the query;

[0026] S45: Fuse the feature vector after interaction with the feature vector in the CNN branch;

[0027] S46: converting the feature vectors of the original image and the folded image into a predicted rotation matrix, and multiplying the rotation matrix of the folded image with the learnable matrix and fusing it with the rotation matrix of the original image.

[0028] S47: Use the fused predicted rotation matrix as the input of the MLP to predict the Euler angle size of the head.

[0029] S48: The geodesic distance loss, contrast loss, and direction token loss are used as the final loss function of the head posture estimation model. During the model training process, the model training is completed by minimizing the loss function.

[0030] For the criss-cross attention operation in the multi-scale Transformer module, its attention matrix is:

[0031]

[0032] Among them, Qc represents the query composed of feature maps from the CNN branch, d represents the dimension of the embedding matrix, and h represents the index of the attention head.

[0033] CNN branch output matrix A c It can be obtained by the following formula:

[0034] A c =Softmax(SELayer(A c ))

[0035] The fused attention matrix is ​​A e for:

[0036]

[0037] in and W s A c and A s The weight matrix of .

[0038] By multiplying the predicted rotation matrix of the folded image with the learnable weight matrix and fusing it with the predicted rotation matrix of the original image, the symmetric information can be effectively used to enhance the representation ability of the model and improve the accuracy of head posture estimation. The fused 3D predicted rotation matrix M g for:

[0039]

[0040] Where W is the learnable weight matrix, M x Predict the rotation matrix for the flipped image, M y Rotate the prediction matrix for the original image.

[0041] The prediction of the model is a rotation matrix representation, and the geodesic distance is used as the loss between two 3D rotations. By applying the geodesic distance as the loss between the two rotation matrices to the original image prediction rotation matrix and the folded image prediction rotation matrix, the contrast loss is:

[0042]

[0043] Where M1 is the predicted rotation matrix of the original image, M2 is the predicted rotation matrix of the folded image, and tr is the trace of the matrix.

[0044] The true prediction loss is:

[0045]

[0046] Among them, M is the true rotation matrix, M g is the predicted rotation matrix after fusion.

[0047] Information can be encoded into the orientation tokens via the orientation token loss, which is defined as the geodesic distance relative to the corresponding orientation region. Therefore, the orientation token loss is defined as:

[0048]

[0049] Among them, 11 is the number of basic direction regions, M is the true rotation matrix, and M i is the predicted rotation matrix of the i-th region, and I(M,i) is an indicator function used to determine whether the true head posture is located in the i-th basic region. I(M,i) can be expressed as:

[0050]

[0051] The total loss can be composed of contrast loss, true prediction loss and direction token loss. The total loss can be expressed as:

[0052] L=αL1+βL g +(1-β)L ori

[0053] Among them, α and β represent hyperparameters, which are used to balance the weights between different losses.

[0054] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A head posture estimation method based on visual Transformer, characterized in that: include: Constructing a head posture estimation model; obtaining an image to be detected, inputting the image into the trained head posture estimation model, and obtaining a head posture estimation result; the head posture estimation model includes a visual Transformer pre-trained network module, a multi-scale Transformer encoder module, a CNN module, and a prediction head module; Training the head posture estimation model includes: obtaining a head posture data set, and performing random scale transformation on the data in the data set to obtain a training set; flipping the data in the training set, and inputting the original image and the flipped image into the visual Transformer pre-training network module and the CNN module respectively for feature extraction to obtain the corresponding feature serialization representation; adding a learnable d-dimensional vector to the feature serialization representation to obtain a fused sequence; inputting the fused sequence into multiple multi-scale Transformer encoder modules, and obtaining an interactive feature vector through a multi-scale interaction strategy and an image enhancement strategy; fusing the interactive feature vector with the feature vector extracted by the CNN module; converting the fused features of the original image and the fused features of the flipped image into a rotation matrix, and multiplying the rotation matrix of the flipped image with the learnable matrix and fusing it with the rotation matrix of the original image; inputting the fused rotation matrix into the prediction head module to obtain the head posture estimation result; calculating the loss function of the model, adjusting the model parameters, and completing the model training when the loss function value is minimized.

2. The head posture estimation method based on visual Transformer according to claim 1, characterized in that: The visual Transformer pre-trained network module processes the input image by: using the visual Transformer feature extractor to extract features from the head image, and adding the obtained feature vector sequence to 11 learnable d-dimensional vectors.

3. The head posture estimation method based on visual Transformer according to claim 1, characterized in that: The CNN module processes the input image by: inputting the image into the CNN module, extracting feature maps of three different dimensions, and sending different feature maps to the linear layer for dimensionality reduction; fusing the reduced features, and adding the fused features to 11 learnable d-dimensional vectors.

4. The head posture estimation method based on visual Transformer according to claim 1, characterized in that: The multi-scale Transformer encoder module consists of a cross attention mechanism and a squeeze-excitation module and a feed-forward neural network.

5. The head posture estimation method based on visual Transformer according to claim 1, characterized in that: The multi-scale interaction strategy is used to interact with multi-scale features, including: the feature vector output by the CNN module is input into the attention interaction layer after passing through the normalization layer, the feature vector is used as the query vector using the self-attention mechanism, the vector extracted by the feature extraction module is used as the key vector and the value vector, and the query vector, key vector and value vector are subjected to self-attention operation.

6. The head posture estimation method based on visual Transformer according to claim 1, characterized in that: The fusion of the feature vector after interaction with the feature vector extracted by the CNN module includes: adding the feature vector of the left branch of the CNN module before passing through the linear layer and the feature vector after interaction; sending the feature vector after the self-attention operation to the right branch for transposition multiplication, and finally sending the feature vectors of the left and right branches to the next module after passing through the normalization layer.

7. The head posture estimation method based on visual Transformer according to claim 1, characterized in that: Converting the fusion features of the original image and the flipped image into a rotation matrix includes: two feature vectors output after the multi-scale Transformer module, one is obtained from the original head image, and the other is obtained from the flipped image. After processing, the predicted rotation matrix of the original image and the predicted rotation matrix of the flipped image are obtained.

8. The head posture estimation method based on visual Transformer according to claim 1, characterized in that: The fused rotation matrix is: Among them, W is the learnable weight matrix, M x Predict the rotation matrix for the flipped image, M y Rotate the prediction matrix for the original image.

9. The head posture estimation method based on visual Transformer according to claim 1, characterized in that: The loss functions include geodesic distance loss, contrast loss, and direction token loss.

Images