Head posture estimation method based on fine-tuning CLIP

Abstract

The invention provides a head posture estimation method based on fine-tuning CLIP, and the method comprises the steps: obtaining a to-be-detected image, inputting the to-be-detected image into a trained head posture estimation model, and obtaining a head posture estimation result; the head posture estimation model comprises a visual coding module, a text coding module, a feature fusion module and a posture prediction module; the visual coding module adopts a visual Transform pre-trained in a CLIP model as an image encoder, and is responsible for extracting multi-level and global visual feature representation from an input image; the text coding module adopts a text editor and is used for generating corresponding text features according to a preset head posture text prompt; the feature fusion module is used for performing cross-modal fusion on the visual features and the text features to obtain fusion features; the attitude prediction module is used for processing the fused features to obtain a head attitude angle; according to the method, the head posture estimation model is designed based on the fine tuning CLIP network architecture, and the discrimination capability of the model is effectively enhanced by combining an image contrast learning mechanism while introducing text description as semantic guidance.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a head pose estimation method based on fine-tuned CLIP. Background Technology

[0002] Head pose estimation (HPE) has been widely applied in fields such as human-computer interaction, autonomous driving, and classroom monitoring. Currently, many methods focus on representing head pose or orientation and have achieved impressive performance. Its core objective is to accurately estimate the orientation of a human head in three-dimensional space from one or more images, typically described by three Euler angles: pitch, yaw, and roll.

[0003] Traditional head pose estimation methods primarily rely on purely visual features and can be broadly categorized into three types: methods based on facial keypoints, methods based on global face regression, and methods based on 3D deformable models. In recent years, global face regression methods have become mainstream. These methods typically use convolutional neural networks (CNNs) or Transformer models to directly regress Euler angles or rotation matrices from the input image. However, these methods have inherent limitations: firstly, they heavily rely on large-scale, high-quality, and accurately labeled datasets, which are costly to annotate; secondly, the model lacks explicit semantic guidance during the learning process, making it difficult to fully understand the high-level concept of "head pose." For example, the model can learn the relationship between pixels and angle values, but it cannot understand the semantic connotations of "looking left" or "looking up" like humans do. This lack of semantic understanding leads to poor performance when faced with scenarios not fully covered by the training data (such as novel occlusions or extreme lighting). Essentially, these methods only utilize information from image modalities and fail to introduce semantically rich prior knowledge that the model can reference. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention proposes a head pose estimation method based on fine-tuned CLIP. This method includes: constructing a head pose estimation model CLIP; acquiring a target image and inputting it into the trained head pose estimation model to obtain a head pose estimation result; the head pose estimation model includes a visual encoding module, a text encoding module, a feature fusion module, and a pose prediction module; the visual encoding module uses a pre-trained visual Transformer from the CLIP model as an image encoder, responsible for extracting multi-level, global visual feature representations from the input image; the text encoding module uses a text editor to generate corresponding text features based on preset head pose text prompts; the feature fusion module performs cross-modal fusion of visual and text features to obtain fused features; and the pose prediction module processes the fused features to obtain the head pose angle.

[0005] The beneficial effects of this invention are:

[0006] This invention designs a head pose estimation model based on a fine-tuned CLIP network architecture. By introducing textual descriptions as semantic guidance and combining them with an image contrast learning mechanism, the model's discriminative ability is effectively enhanced. This method enables the model to make more reliable judgments by fully utilizing semantic prior information when facing complex scenes such as changes in lighting and partial occlusion, significantly improving the model's adaptability and stability. Furthermore, by constructing reasonable text prompt templates, the model can still learn effectively under unsupervised or weakly supervised conditions, greatly reducing data collection and annotation costs and expanding the applicability of this method in resource-constrained scenarios. Attached Figure Description

[0007] Figure 1 This is a block diagram of a head pose estimation method based on fine-tuned CLIP according to the present invention. Detailed Implementation

[0008] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0009] A head pose estimation method based on fine-tuned CLIP is disclosed. The method includes: constructing a head pose estimation model CLIP; acquiring a target image and inputting it into the trained head pose estimation model to obtain a head pose estimation result; the head pose estimation model includes a visual encoding module, a text encoding module, a feature fusion module, and a pose prediction module; the visual encoding module uses a pre-trained visual Transformer from the CLIP model as an image encoder, responsible for extracting multi-level, global visual feature representations from the input image; the text encoding module uses a text editor to generate corresponding text features based on preset head pose text prompts; the feature fusion module performs cross-modal fusion of visual and text features to obtain fused features; and the pose prediction module processes the fused features to obtain the head pose angle.

[0010] In this embodiment, a head pose estimation method based on fine-tuned CLIP is presented. This method introduces text descriptions as semantic supervision and leverages the powerful cross-modal understanding capabilities of the CLIP model to inject directional semantic information (such as "forward" and "left") and degree information (such as "slightly" and "strongly") of the head into visual features, thereby achieving a more accurate and robust estimation of head pose. The method includes: constructing a head pose estimation model; acquiring an image A to be detected, an image B with Euler angles close to A, and the true Euler angles of A; inputting the images into the trained head pose estimation model to obtain the head pose estimation result; the head pose estimation model includes an image encoder (visual Transformer), a text encoder, a feature fusion module, and a pose prediction module.

[0011] Training the head pose estimation model includes: acquiring a head pose dataset; converting angles within the same pose range into corresponding text descriptions using text templates (e.g., "an image of a head with its face slightly turned to the left") according to different Euler angle true values, thus obtaining a training set; inputting the image and corresponding description into a visual Transformer pre-trained network module and a text encoder module for feature extraction, obtaining the corresponding feature sequence representation; and inputting the image features and text features into a feature fusion module, which is the key innovation of this invention. This module receives sets of visual and text features and injects semantic information into the visual features through an efficient cross-modal interaction mechanism.

[0012] The specific implementation involves: calculating the cosine similarity between visual features and each text feature; normalizing these similarities using the Softmax function to construct a set of attention weights; using these weights to perform a weighted summation of all text features to obtain a semantic context vector. This vector aggregates the pose semantic information most relevant to the input image. Finally, the original visual features are concatenated or added to this semantic context vector to output a semantically enhanced fused feature. The fused feature is then input into the pose prediction module, which consists of a multilayer perceptron (MLP). This module receives the fused feature and regresses the three Euler angles of the final head pose—pitch, yaw, and roll—through a fully connected layer. The L1 loss between the predicted angles and the true angle labels, and the contrast loss between the predicted angles from the original image and those from similar images are calculated. Gradient descent is then used for backpropagation to update the model's trainable parameters until the model converges.

[0013] In this embodiment, as Figure 1 As shown, the head pose estimation model framework illustrates the specific implementation of this invention.

[0014] S1: Obtain publicly available head pose datasets (such as 300W-LP, AFLW2000, etc.). One of the key innovations of this invention lies in the introduction of text supervision. We divide the head pose into different pose intervals based on the true values ​​of the Euler angles, and define one or more text templates for each interval. According to the different true Euler angle values, angles within the same pose interval are converted into corresponding text descriptions using the text templates (for example, when the yaw angle is close to 0°, the text is: "An image of a head with its face slightly turned to the left"). In this way, a corresponding text description is generated for each training image, forming an image-text pair training set. This not only provides continuous Euler angle supervision but also introduces semantically rich category information.

[0015] S2: Construct a head pose estimation model; the model includes an image encoder (visual Transformer), a text encoder, a feature fusion module, and a pose prediction module;

[0016] S3: Employs the Visual Transformer from the CLIP model as the backbone network to extract rich, multi-level visual features from the input image. Input image Output visual feature sequence ,in The length of the image patch sequence. For feature dimensions;

[0017] S4: The Text Transformer from the CLIP model is used as the backbone network to extract semantic features from text descriptions. Input text Output text feature set ,in , The number of text descriptions.

[0018] S5: The key innovation of this invention lies in inputting image features and text features into the feature fusion module. This module receives sets of visual and text features and injects semantic information into the visual features through an efficient cross-modal interaction mechanism.

[0019] S6: To find the semantic information most relevant to the input image, we compute visual features. With each text feature The similarity between them. First, we will consider the visual feature sequences. The features are aggregated into a global visual feature vector through a global average pooling layer.

[0020]

[0021] S7: In the feature fusion module, calculate With each text feature The cosine similarity between them is normalized using the Softmax function to construct a set of attention weights.

[0022]

[0023] in It is an optional temperature parameter used to control the sharpness of the attention distribution.

[0024] Cosine similarity The calculation is as follows:

[0025]

[0026] in This is a learnable weight matrix used to adjust the contribution of the semantic context vector.

[0027] S8: Use the weights to sum all the text features to obtain a semantic context vector, which aggregates the pose semantic information most relevant to the input image.

[0028]

[0029] S9: The original global visual features With semantic context vectors The mixture is fused and added using a residual connection method, and a learnable weight matrix is ​​introduced. This adjusts the contribution of semantic information, ultimately outputting semantically enhanced fusion features. :

[0030]

[0031] S10: Input the fused features into the attitude prediction module, which consists of a multilayer perceptron (MLP). The module receives the fused features and regresses the three Euler angles of the final head attitude through a fully connected layer, as well as the L1 loss between the predicted angles: pitch angle, yaw angle, and roll angle.

[0032] S11: Calculate the predicted Euler angles With real angle tags The L1 smoothing loss is preferred because it is less sensitive to outliers than MSE and is smoother than L1 loss, which is beneficial for training stability.

[0033]

[0034] Backpropagation using the gradient descent algorithm is employed to calculate the gradient and update all trainable parameters in the model until the model converges, at which point the loss is calculated. Similarly (A is the original image, and B is an image with Euler angles similar to the original image).

[0035]

[0036] in, For L1 smoothing loss, The loss is the loss when converting the original image to an image with a similar Euler angle.

[0037] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A fine-tuning CLIP-based head pose estimation method, characterized in that, The head pose estimation model CLIP is constructed; an image to be detected is obtained, and the image to be detected is input into the trained head pose estimation model to obtain a head pose estimation result; the head pose estimation model comprises a visual encoding module, a text encoding module, a feature fusion module, and a pose prediction module; the visual encoding module adopts a pre-trained visual Transformer in the CLIP model as an image encoder, and is responsible for extracting multi-level and global visual feature representations from an input image; The text encoding module adopts a text editor to generate corresponding text features according to a preset head pose text prompt; the feature fusion module is used for cross-modal fusion of visual features and text features to obtain fused features; The pose prediction module is used for processing the fused features to obtain a head pose angle. The training data when the head pose estimation model CLIP is trained adopts a head pose data set; according to a true value of a head pose Euler angle, the head pose Euler angle is divided into different pose intervals, and one or more text templates are defined for each interval; angles in the same pose interval are converted into corresponding text descriptions through the text templates according to different true values of the Euler angles, and an image-text pair training set is obtained.

2. The fine-tuning CLIP-based head pose estimation method of claim 1, wherein, The visual encoding module processes the input image, including: pre-processing the input head image, and inputting the pre-processed head image into a visual encoder; the visual encoder is based on a Vision Transformer architecture, divides an image into an image block sequence and adds a position embedding, extracts features through multiple Transformer layers, and finally outputs a global image feature vector at a [CLS] marker position at the beginning of the sequence as a visual feature representation of the head pose.

3. The fine-tuned CLIP-based head pose estimation method of claim 1, wherein, The text encoding module processes the input data, including: constructing a text prompt set covering head pose angle information; inputting each text prompt in the text prompt set into a text encoder of the CLIP model; the text encoder is based on a Transformer architecture, tokenizes the text and adds a position embedding, encodes the context through multiple Transformer layers, and finally outputs a corresponding text feature vector at a [CLS] marker position at the beginning of the sequence; all text feature vectors are combined to form a text feature space for similarity matching with the visual features.

4. The fine-tuned CLIP-based head pose estimation method of claim 1, wherein, The feature fusion module processes the input data, including: calculating the similarity between the visual features and each text feature to obtain an initial similarity score; normalizing the similarity score through a Softmax function to generate an attention weight distribution; weighting and fusing the text features according to the attention weight distribution to obtain semantic enhanced features; and splicing the semantic enhanced features and the original visual features to obtain a fused feature vector.

5. The fine-tuned CLIP-based head pose estimation method of claim 1, wherein, ​ 6. The fine-tuned CLIP-based head pose estimation method of claim 1, wherein, The pose prediction module comprises a first fully connected layer, an activation function layer and a second fully connected layer connected in sequence; the first fully connected layer is responsible for projecting the input high-dimensional fusion feature to a low-dimensional feature space; the activation function layer adopts a ReLU activation function; and the second fully connected layer maps the feature to a three-dimensional output space, corresponding to the predicted values of three Euler angles respectively.

7. The fine-tuned CLIP-based head pose estimation method of claim 1, wherein, Training the head pose estimation model CLIP comprises: in an initial training stage, freezing the pre-training parameters of the visual encoder and the text encoder in the CLIP model, and only training the image adapter, the text adapter, the feature fusion module and the pose prediction module; in a later training stage, the freezing of the CLIP model parameters is released, and the overall model parameters are fine-tuned end-to-end with a small learning rate, and the training of the model is completed by adopting a staged training strategy.

8. The fine-tuned CLIP-based head pose estimation method of claim 7, wherein, The head pose estimation model CLIP adopts a smooth L1 loss function when being trained.

Images