Head pose detection method and device based on improved repvgg network

Abstract

The application discloses a head posture detection method and device based on an improved RepVGG network, and relates to the technical field of posture detection. The method comprises the following steps: inputting a to-be-detected head image into a head posture detection model to obtain the head posture of the to-be-detected head image; wherein the head posture detection model is obtained by training an improved RepVGG network by using facial images with different postures; and the improved RepVGG network is obtained by adding a feature denoising module and an attention information fusion module to a RepVGG network. The application can improve the detection accuracy of head posture detection by constructing and training the improved RepVGG.

Description

Technical Field

[0001] This application relates to the field of posture detection technology, and in particular to a head posture detection method and device based on an improved RepVGG network. Background Technology

[0002] Head pose estimation, which aims to calculate the three-dimensional orientation of a human head in an image or video, is a fundamental problem widely studied in computer vision and has broad practical value in multiple fields. In driver state detection, head pose estimation can be combined with face detection to monitor driver alertness, thereby improving driving safety. In human-computer interaction, it can help systems understand the user's gaze and attention direction, adjusting the viewing angle based on the user's head pose for a more realistic experience. In surveillance applications, head pose estimation can help identify abnormal behavior in crowds, such as real-time identification of student behavior in a classroom, allowing teachers to adjust teaching methods to improve student engagement. Head pose estimation technology enhances the functionality and interactivity of various technologies and devices, greatly expanding the scope and depth of modern technology applications.

[0003] Facial keypoint-based and keypoint-free methods are two main approaches to head pose estimation. Keypoint-based methods primarily regress head pose angles by extracting facial keypoints from a facial coordinate detector. HHP-Net automatically calculates keypoints using a 2D pose estimation algorithm and outputs head pose represented by yaw, pitch, and roll. Barra et al. first predicted the positions of 68 facial keypoints, then assigned each keypoint to a specific sector, and used the obtained overall information to construct a feature vector to infer head pose. Gupta et al. used an uncertainty graph in the form of a 2D soft-localization heatmap on five facial keypoints and regressed head pose using a convolutional neural network. EVA-GCN constructed a keypoint connectivity graph and used a graph convolutional network to learn the nonlinear correlation between the true head pose and the distribution of facial keypoints. H-CNN iteratively refines facial keypoints, while head pose estimation is a branch task of the overall task. However, facial keypoint-based methods heavily rely on the quality of keypoint detection; even slight inaccuracies in keypoint localization can adversely affect head pose estimation, and learning the relationship between the true head pose and the geometric distribution of facial keypoints is also a very tedious task. Furthermore, the facial key point-based approach requires a dataset of facial key points, which increases the workload before model training.

[0004] Non-keypoint methods directly estimate head pose from facial images without detecting keypoints, reducing execution time. RAFA automatically captures facial details by introducing a self-attention mechanism into the network structure. Kuhnke et al. proposed a semi-supervised learning strategy based on relative pose consistency regularization for head pose estimation. TriNet proposed a deep network model that predicts three vectors with orthogonality constraints and used mean absolute error (MAEV) to evaluate performance. MFDNet proposed a three-branch head pose model with a triple module and a matrix Fisher distribution module. It is a robustly learnable head pose embedding method. ARHPE introduces a Lorentz distribution to assign different weights to yaw and pitch directions. Compared with traditional end-to-end frameworks, the proposed method can predict head pose angles in cases of incorrect labeling by utilizing asymmetric relational cues from the input image. In recent years, non-facial keypoint-based head pose estimation methods have become a promising research area, with better generalization ability and simple processing flow. Therefore, this embodiment uses a non-facial keypoint-based method for facial pose estimation.

[0005] With the development of deep learning, head pose detection technology has also advanced, and the detection effect has been greatly improved. However, existing detection schemes suffer from problems such as poor robustness of collected features, insufficient ability to capture key features, and an imbalance between model stability and accuracy. Summary of the Invention

[0006] The purpose of this application is to provide a head pose detection method and device based on an improved RepVGG network, which completes head pose detection based on the improved RepVGG and without facial key points, thereby improving the detection accuracy of head pose detection.

[0007] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a head pose detection method based on an improved RepVGG network, including: Obtain the image of the head to be tested; The head image to be tested is input into the head pose detection model to obtain the head pose of the head image to be tested; the head pose detection model is obtained by training an improved RepVGG network using facial images of different poses; the improved RepVGG network is obtained by adding a feature denoising module and an attention information fusion module to the RepVGG network.

[0008] Optionally, the improved RepVGG network includes: a feature denoising module, a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, an attention information fusion module, a fifth feature extraction module, a fully connected structure, and a Schmidt orthogonal function, connected in sequence.

[0009] Optionally, the feature denoising module includes: a first convolutional operation layer, a batch normalization layer, a ReLU activation function layer, and a first residual operation layer connected in sequence; The input terminal of the convolutional operation layer is the head image to be tested or the facial images of different poses trained; The second input terminal of the first residual operation layer is used to input the head image to be tested or the facial images of different poses trained; The first residual operation layer is also connected to the input of the first feature extraction module.

[0010] Optionally, the first feature extraction module includes a first residual structure; The second feature extraction module includes a second residual structure and N1 third residual structures connected in sequence; The third feature extraction module includes a fourth residual structure and N2 fifth residual structures connected in sequence; The fourth feature extraction module includes a sixth residual structure and N3 seventh residual structures connected in sequence; The fifth feature extraction module includes an eighth residual structure; The first residual structure, the second residual structure, the fourth residual structure, the sixth residual structure, and the eighth residual structure are all residual structures containing a 1×1 convolution matrix; The third residual structure, the fifth residual structure, and the seventh residual structure are all residual structures that include 1×1 convolution and identity mapping.

[0011] Optionally, the attention information fusion module includes an improved attention module, a second residual operation layer, a ReLU activation function layer, a second convolution operation layer, a first Sigmoid function layer, a first multiplication operation layer, a second multiplication operation layer, and a third residual operation layer; The input of the improved attention module, the first input of the second residual operation layer, and the first input of the first multiplication operation layer are all connected to the output of the fourth feature extraction module. The first output of the improved attention module is connected to the second input of the second residual operation layer; The second output of the improved attention module is connected to the first input of the second multiplication layer; The output of the second residual operation layer is connected to the input of the ReLU activation function layer; The output of the ReLU activation function layer is connected to the input of the second convolution operation layer. The output of the second convolutional layer is connected to the input of the first sigmoid function layer; The first output of the first Sigmoid function layer is connected to the second input of the first multiplication layer. The second output of the first Sigmoid function layer is connected to the second input of the second multiplication layer. The output of the second multiplication layer is connected to the first input of the third residual layer; The output of the first multiplication layer is connected to the second input of the third residual layer. The output of the third residual operation layer is connected to the input of the fifth feature extraction module.

[0012] Optionally, the improved attention module includes: a first branch, a second branch, a third branch, and a fourth residual operation layer; The output terminals of the first branch, the second branch, and the third branch are all connected to the input terminal of the fourth residual operation layer. The output of the fourth residual operation layer is connected to the second input of the second residual operation layer and the first input of the second multiplication operation layer, respectively.

[0013] Optionally, the first branch includes a third convolutional operation layer; The third branch includes a fourth convolutional operation layer; The second branch includes an average pooling layer, a fifth convolutional operation layer, a second sigmoid function layer, and a third multiplication operation layer; The input of the average pooling layer and the first input of the third multiplication layer are both connected to the output of the fourth feature extraction module. The output of the average pooling layer is connected to the input of the fifth convolution operation layer. The output of the fifth convolutional operation layer is connected to the input of the second sigmoid function layer. The output of the second Sigmoid function layer is connected to the second input of the third multiplication layer.

[0014] Optionally, before acquiring the head image to be tested, the following steps are also included: Build an improved RepVGG network; The rotation matrix corresponding to multiple training head images is determined to be the true rotation matrix; the training head images are facial images in different poses; Multiple training head images are input into the improved RepVGG network to obtain the prediction rotation matrix corresponding to the multiple training head images; The value of the hybrid loss function is determined based on multiple true rotation matrices and multiple predicted rotation matrices; the hybrid loss function is determined by weighted summation of geodesic distance and scaled Frobenius norm. Adjust the parameters of the improved RepVGG network and return to the step "input multiple training head images into the improved RepVGG network to obtain the prediction rotation matrix corresponding to the multiple training head images" until the number of iterations equals the preset number of iterations, and determine the improved RepVGG network at the last iteration as the head pose detection model.

[0015] Optionally, the hybrid loss function is: in, For a mixed loss function, For the true rotation matrix, To predict the rotation matrix, For geodesic distance, For geodesic distance weighting, It is the Frobenius norm. Scaling factor These are the scaled Frobenius norm weights.

[0016] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described head pose detection method based on the improved RepVGG network.

[0017] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a head pose detection method and device based on an improved RepVGG network. An improved RepVGG network (i.e., the RepVGG-DP framework) is obtained by adding a feature denoising module and an attention information fusion module to the RepVGG network. The feature denoising module (ImD) obtains robust features by processing the residual connections of the original features using only a combination of convolutional layers, batch normalization, and ReLU activation functions, effectively promoting contextual interaction and thus enhancing the model's learning ability. The feature denoising module is simple in design and can be easily integrated into various CNN architectures. Its lightweight nature allows for seamless end-to-end training with standard CNNs, with negligible additional costs. The attention information fusion module dynamically adjusts the fusion weight of the original and processed features to adaptively adjust the importance of features based on the content of the feature map itself, enhancing the model's ability to capture key features. Compared to traditional attention mechanisms, it provides a more fine-grained adjustment method, thereby improving the model's performance in complex visual tasks.

[0018] Furthermore, this application proposes a hybrid loss function that combines the advantages of both geodesic distance and Frobenius norm, and adjusts the Frobenius scale to better match the geodesic distance. Compared to a single loss function, this hybrid loss function maintains the accuracy of posture assessment while adapting more sensitively to posture changes, significantly improving the performance of head posture estimation tasks. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a head pose detection method based on an improved RepVGG network in one embodiment of this application; Figure 2 This is a schematic diagram of a feature denoising module in one embodiment of this application; Figure 3 This is a schematic diagram of the attention information fusion module in one embodiment of this application; Figure 4 This is a schematic diagram of an improved attention module in one embodiment of this application; Figure 5 This is a schematic diagram of the RepVGG network training phase in the prior art; Figure 6 This is a schematic diagram of the inference stage of the RepVGG network in the prior art; Figure 7 This is a schematic diagram of the first residual structure in one embodiment of this application; Figure 8 This is a schematic diagram of the third residual structure in one embodiment of this application; Figure 9 This is a schematic diagram of an improved RepVGG network in one embodiment of this application; Figure 10 This is an overall network framework diagram in one embodiment of this application; Figure 11 This is a schematic diagram of the training loss of the 300W-LP dataset and the BIWI training set in one embodiment of this application; Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] In one exemplary embodiment, such as Figure 1 As shown, a head pose detection method based on an improved RepVGG network is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both. In this embodiment, the method is described using a server as an example, and includes the following steps: steps 101 to 102. Wherein: Step 101: Obtain the head image to be tested.

[0024] Step 102: Input the head image to be tested into the head pose detection model to obtain the head pose of the head image. The head pose detection model is obtained by training an improved RepVGG network with facial images of different poses; the improved RepVGG network is obtained by adding a feature denoising module and an attention information fusion module to the RepVGG network.

[0025] The improved RepVGG network includes: a feature denoising module, a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, an attention information fusion module, a fifth feature extraction module, a fully connected structure, and a Schmidt orthogonal function, connected in sequence.

[0026] The feature denoising module includes: a first convolutional operation layer, a batch normalization layer, a ReLU activation function layer, and a first residual operation layer connected in sequence; the input end of the convolutional operation layer is the head image to be tested or the facial images of different poses trained; the second input end of the first residual operation layer is used to input the head image to be tested or the facial images of different poses trained; the first residual operation layer is also connected to the input end of the first feature extraction module.

[0027] During the acquisition of input images, feature quality can be affected by various factors, such as poor lighting conditions or low camera quality, ultimately leading to system misjudgments and reduced detection accuracy. Improving the quality of input features for head pose estimation tasks not only enhances the recognizability of key features but also strengthens the robustness of the algorithm, thereby improving the overall detection accuracy and reliability of the system.

[0028] The characteristics containing noise are shown in formula (1), where Indicates potential clean features. Represents noise. Features that indicate the presence of noise are used to remove noise using feature enhancement techniques. To store a given clean feature.

[0029] Residual learning has been widely used in denoising techniques. The main idea is to utilize the residual structure to remove noise. Substituted data for absolutely clean features This embodiment designs a novel noise extraction residual block (i.e., a feature denoising module) that subtracts the residual features generated in the output from the input noisy image to obtain a clean result. In other words, instead of learning a complex mapping from complete features to another feature, the feature enhancement module learns residual features, which discard the noise and recover high-frequency details. The structure of the feature enhancement module is as follows: Figure 2As shown. The input to this module is the acquired image. For grayscale images, the number of input channels is 1, and for color images, the number of input channels is 3. The output channel is always equal to the input channel. The initial convolutional layer uses Conv+BN+ReLU to extract shallow features, where "+" indicates sequential function, that is, the processing order is convolution operation, batch normalization, and ReLU activation function. After the features pass through the convolutional layer with 3×3 convolutional kernels, they are then subjected to batch normalization to normalize the data, and finally the ReLU activation function is executed to improve the stability and learning efficiency of the model. In order to improve the learning ability of the denoising module, another convolutional layer "Conv" is added after the initial convolutional layer. The two layers are combined to extract significant features to enhance the denoising performance. Subsequently, the residual structure is used in the last layer to better construct clean features, as shown in formula (2).

[0030] in, As input features, The features after denoising. A function of Conv+BN+ReLU For convolution operations with 3×3 kernels, "" refers to residual operations used to construct clean features.

[0031] Combining the feature enhancement module with the RepVGG network can make the features input into the model cleaner, enhance feature robustness, and improve the model's feature learning ability.

[0032] Furthermore, the attention information fusion module includes an improved attention module, a second residual operation layer, a ReLU activation function layer, a second convolution operation layer, a first sigmoid function layer, a first multiplication operation layer, a second multiplication operation layer, and a third residual operation layer. The input terminals of the improved attention module, the first input terminals of the second residual operation layer, and the first input terminals of the first multiplication operation layer are all connected to the output terminal of the fourth feature extraction module. The first output terminal of the improved attention module is connected to the second input terminal of the second residual operation layer. The second output terminal of the improved attention module is connected to the first input terminal of the second multiplication operation layer. The output terminal of the second residual operation layer is connected to the ReLU activation function layer. The input of the ReLU activation function layer is connected to the input of the second convolution operation layer; the output of the second convolution operation layer is connected to the input of the first Sigmoid function layer; the first output of the first Sigmoid function layer is connected to the second input of the first multiplication operation layer; the second output of the first Sigmoid function layer is connected to the second input of the second multiplication operation layer; the output of the second multiplication operation layer is connected to the first input of the third residual operation layer; the output of the first multiplication operation layer is connected to the second input of the third residual operation layer; and the output of the third residual operation layer is connected to the input of the fifth feature extraction module.

[0033] The improved attention module includes a first branch, a second branch, a third branch, and a fourth residual operation layer. The outputs of the first branch, the second branch, and the second branch are all connected to the input of the fourth residual operation layer. The output of the fourth residual operation layer is connected to the second input of the second residual operation layer and the first input of the second multiplication operation layer. The first branch includes a third convolution operation layer; the third branch includes a fourth convolution operation layer; the second branch includes an average pooling layer, a fifth convolution operation layer, a second sigmoid function layer, and a third multiplication operation layer. The inputs of the average pooling layer and the first input of the third multiplication operation layer are both connected to the output of the fourth feature extraction module; the output of the average pooling layer is connected to the input of the fifth convolution operation layer; the output of the fifth convolution operation layer is connected to the input of the second sigmoid function layer; and the output of the second sigmoid function layer is connected to the second input of the third multiplication operation layer.

[0034] Specifically, head pose estimation tasks often require focusing on specific regions of the head in complex scenes or datasets to capture key features. Introducing attention mechanisms can help models better adapt to these situations and improve their generalization ability. Attention mechanisms in deep learning models simulate the human visual and cognitive attention process, giving neural networks information processing capabilities. They can distinguish key points and flexibly capture the connections between global and local information. The goal is to enable the model to identify the target region requiring attention, increase the weight of this region, highlight important and useful features, and suppress or ignore irrelevant features.

[0035] This embodiment designs a novel attention information fusion module. Building upon the spatial and channel attention implemented in the ParNet Attention module (i.e., the improved attention module), it adds a layer of finer-grained spatial dimension attention. This allows the model to focus on key information even when processing complex and subtle structures in images, improving the model's robustness. The structure of the attention information fusion module is as follows: Figure 3 As shown.

[0036] The improved attention module is a novel attention module that includes multiple parallel sub-networks, each responsible for extracting feature information at different levels, such as... Figure 4 As shown in the figure. This parallel structure enables the model to better capture multi-scale feature information, improving its representational power and performance.

[0037] In the first branch, the input features are weighted by a 1×1 convolution. The second branch contains a residual structure that passes the input features through an average pooling layer (AvgPool) to reduce the oversensitivity of the convolutional layers to location and preserve background information. It is then adjusted by a 1×1 convolution, activated by the Sigmoid function, and multiplied with the original feature map to obtain the weights of the relevant information. The third branch expands the receptive range of the feature map through a 3×3 convolution. Finally, the feature information from the three branches is merged to generate the output features. To enable the model to handle various large-scale and complex datasets and increase the nonlinearity of the network, the SiLU activation function is used instead of the ReLU function. The calculation process of this module is shown in formula (3): I (3) in, Indicates average pooling. and These represent 3×3 and 1×1 convolutions, respectively. This represents the output of the ParNet Attention module. This represents the features input to the attention module. This module uses multi-branch fusion to adjust the network's focus on the image, expand the perceptual range, and enhance the feature extraction capability of the backbone network.

[0038] The core idea of ​​the attention information fusion module is to introduce an additional processing step on top of the original ParNet Attention module, by combining the original input features. and features after attention processing The weights are processed using a novel ReLU activation function and convolutional layers, and finally a dynamic weight map is generated using the Sigmoid function. To adjust and The fusion ratio emphasizes the complementarity between different feature maps, that is, how to effectively combine the detailed information of the original features and the contextual information of the processed features to achieve the optimal feature representation. Guided by the original input and Multiplication and ParNet Attention output and weight graph In the multiplication calculation, the module not only considers the features enhanced by the attention mechanism but also retains the information of the original input features, and the fusion of the two is dynamically adjusted. This fusion strategy helps maintain the model's sensitivity to the original information while adding information enhanced by the attention mechanism, thus improving the richness and accuracy of feature representation. The calculation process of this module is shown in formula (4): in, This represents a 3×3 convolution. This represents the output of the ParNet Attention module. This represents the features input into the attention information fusion module. This represents the result obtained after adjusting the feature weights through the attention information fusion module.

[0039] The attention information fusion module emphasizes feature complementarity and content-driven adaptive adjustment, enabling the model to not only process and understand feature information more effectively, but also to have high flexibility and adaptability, making it particularly suitable for the accurate feature understanding and large pose angle capture required for head pose estimation tasks.

[0040] Furthermore, the first feature extraction module includes a first residual structure; the second feature extraction module includes a second residual structure and N1 third residual structures connected in sequence; the third feature extraction module includes a fourth residual structure and N2 fifth residual structures connected in sequence; the fourth feature extraction module includes a sixth residual structure and N3 seventh residual structures connected in sequence; and the fifth feature extraction module includes an eighth residual structure. The first, second, fourth, sixth, and eighth residual structures are all residual structures containing a 1×1 convolution matrix; and the third, fifth, and seventh residual structures are all residual structures containing a 1×1 convolution and an identity mapping.

[0041] Specifically, the architecture of the RepVGG network is as follows: Figure 5 and Figure 6 As shown, N1, N2, and N3 represent the number of layers in these structures. The core of the RepVGG network uses only ReLU and 3×3 convolutions, without employing compound scaling, manual fine-tuning, automatic search, or other complex designs to instantiate a specific architecture. Its core feature is the use of multi-branch structures to enhance representational power during training, while transforming the network into a pure convolutional structure during inference using structural reparameterization. This design significantly improves inference efficiency.

[0042] The first, second, fourth, sixth, and eighth residual structures are all RepVGG-R1; RepVGG-R1 is a residual structure containing a 1×1 convolution matrix, such as... Figure 7 As shown. The third, fifth, and seventh residual structures are all RepVGG-R2. RepVGG-R2 is a residual structure containing 1×1 convolutions and identity mappings, as shown. Figure 8 As shown.

[0043] RepVGG-DP (i.e., Improved RepVGG Network) selects RepVGG-B1g2, which is equivalent to ResNet50, as its base model because it has appropriate inference speed and satisfactory performance. The structural block diagram of RepVGG-B1g2 during the training and inference phases is shown below. Figure 4 As shown, it has five feature extraction stages. Stage 1 (i.e., the first feature extraction module) and Stage 5 (i.e., the fifth feature extraction module) have only 1 layer, while Stage 2 (i.e., the second feature extraction module), Stage 3 (i.e., the third feature extraction module) and Stage 4 (i.e., the fourth feature extraction module) have 4 layers, 6 layers and 16 layers respectively, for a total of 28 layers. The specific network structure is shown in Table 1.

[0044] Table 1 RepVGG-B1g2 Model Architecture

[0045] For tasks like head pose estimation, optimizing deep features is crucial because accurate pose estimation requires a comprehensive understanding of details and high-level semantic information. Therefore, RepVGG-DP adds an attention fusion module between Stage 4 and Stage 5 of RepVGG-B1g2, enabling the model to better process and optimize deep features that already contain rich semantic information. This module effectively utilizes limited visual processing resources, allowing the model to focus on head pose information while ignoring irrelevant background information, thus improving overall performance. Furthermore, RepVGG-DP adds the feature enhancement module ImD before Stage 1 to ensure the quality of features input into the model and reduce the computational burden on the model in subsequent processing. The improved RepVGG network architecture is as follows: Figure 9 As shown.

[0046] In the final stage of designing the RepVGG-DP model, global pooling and a fully connected FC-layer structure with six output units were used to output the 6DoF pose vector of the head pose, and the Gram-Schmidt orthogonal function was applied. The transformation is converted into a predicted rotation matrix, and finally, the training loss is calculated using the improved loss function. The overall network architecture is as follows: Figure 10 As shown.

[0047] Before acquiring the head image to be tested, this implementation also includes a training process for the improved RepVGG network, including steps 103-107.

[0048] Step 103: Construct the improved RepVGG network.

[0049] Step 104: Determine that the rotation matrix corresponding to multiple training head images is the true rotation matrix; the training head images are facial images in different poses.

[0050] Step 105: Input multiple training head images into the improved RepVGG network to obtain the prediction rotation matrix corresponding to the multiple training head images.

[0051] Step 106: Determine the value of the mixed loss function based on multiple true rotation matrices and multiple predicted rotation matrices; the mixed loss function is determined by weighted summation of geodesic distance and scaled Frobenius norm.

[0052] Step 107: Adjust the parameters of the improved RepVGG network and return to the step "Input multiple training head images into the improved RepVGG network to obtain the prediction rotation matrix corresponding to the multiple training head images" until the number of iterations equals the preset number of iterations, and determine the improved RepVGG network in the last iteration as the head pose detection model.

[0053] The hybrid loss function is: in, For a mixed loss function, For the true rotation matrix, To predict the rotation matrix, For geodesic distance, For geodesic distance weighting, It is the Frobenius norm. Scaling factor These are the scaled Frobenius norm weights.

[0054] In head pose detection tasks, the choice of loss function is crucial to the effectiveness of model training and final performance. Traditional loss functions, such as mean squared error (MSE), may not adequately capture the complexity and diversity in head pose prediction. To address this issue, this embodiment proposes a novel loss function that combines geodesic distance and a scaled Frobenius norm with different weights. This hybrid loss function aims to fully leverage the advantages of both geodesic distance and the Frobenius norm to improve the accuracy and robustness of head pose detection.

[0055] Hempel et al. used geodesic distance to measure the shortest path between two 3D rotations, taking into account the geometric properties of the pose space, which can more accurately reflect the true differences between head poses. Compared with traditional loss functions, especially when the pose changes are large, geodesic distance can better handle the nonlinear problems in pose estimation. The definition of the geodesic distance between two rotation matrices is given in Equation (5): (5) The Frobenius norm, as a matrix norm, is used to quantify the differences between matrix elements, as shown in Equation (6). In head pose detection, incorporating the Frobenius norm as part of the loss function helps to emphasize the overall difference between the predicted and true matrices. This approach is particularly suitable for evaluating the accuracy of pose estimation, especially when predicting the rotation matrix of head pose.

[0056] (6) By combining the geodesic distance with the scaled Frobenius norm, the loss function proposed in this embodiment balances the stability and accuracy of the model in head pose estimation. The improved loss function is shown in Equation (7): (7) in, , These are the weight coefficients of the two loss functions. and Let these represent the true rotation matrix and the predicted rotation matrix, respectively. The formula follows... : The ratio combines geodesic distance and Frobenius norm loss, while utilizing... As a scaling factor, the Frobenius norm loss value better matches the scale of change in geodesic distance during training.

[0057] The total loss function proposed in this embodiment incorporates both angular and distance information from the rotation matrix. This hybrid approach allows the model to enhance its learning of head pose geometry during training using geodesic distance, particularly effective when handling large-angle rotations, resulting in more consistent predictions in geometric space. Simultaneously, adjusting the Frobenius norm to a scale similar to that of the geodesic distance helps to balance the weights of both in the loss function, preventing either from having an excessive impact on gradient updates. This better guides network learning, increases sensitivity to subtle differences in the rotation matrix, and improves the detection accuracy of minor pose changes.

[0058] 1. Dataset.

[0059] To ensure the fairness of the comparison of experimental results, this embodiment uses three public face datasets: 300W-LP, AFLW-2000, and BIWI.

[0060] 300W-LP: Contains over 60,000 facial samples collected from multiple databases, further increased to 122,450 samples through image flipping, designed to increase the number of challenging samples with extreme poses. Labels provide ground truth values ​​in Euler angle format, with yaw angles ranging from ±89°. In this embodiment, Euler angles are converted to matrix form for training.

[0061] ALFW-2000: This dataset contains the first 2000 images from the ALFW dataset, with 68 3D labels re-annotated using a 3D model. Therefore, this dataset contains accurate, fine-grained pose annotations and is a leading candidate test set for head pose estimation tasks. It includes a variety of samples with different lighting and occlusion conditions. Yaw angles range from ±120°, and roll and pitch angles range from ±90°.

[0062] BIWI: The dataset was collected in a laboratory environment using a Kinect v2 device to record RGB-D videos of different subjects in various head poses. It comprises 24 datasets from 20 individuals (14 men and 6 women). It contains approximately 15,000 frames with yaw angles ranging from ±75°, pitch angles from ±60°, and roll angles from ±50°. In this dataset, the head occupies only a small area in the images; in this embodiment, the third-party face detection program MTCNN was used for facial region segmentation.

[0063] For the sake of experimental validity, all data were retained only for samples with Euler angles between -99° and 99°.

[0064] 2. Evaluation indicators.

[0065] In head attitude assessment tasks, Mean Absolute Error (MAE) is a key evaluation metric. MAE measures the average deviation between the predicted and actual attitudes, typically expressed in degrees (°). Head attitude angles are represented by the symbols yaw, pitch, and roll, respectively. The set of Euler angles for the true labels is represented as follows: The set of Euler angles predicted by the model is represented as Therefore, the formula for calculating this indicator is defined as follows: The advantage of MAE lies in its intuitiveness and ease of calculation; a lower MAE value indicates better predictive accuracy. When evaluating head pose accuracy, MAE provides a clear metric to determine the model's performance in real-world applications.

[0066] By using MAE as a performance measurement tool, this embodiment can evaluate and compare the performance of different models, determine the merits of head pose estimation algorithms, and thus determine the overall reliability and effectiveness of the system.

[0067] 3. Model training.

[0068] Experiment 1: To ensure a fair comparison of results, the dataset was trained on the 300W-LP dataset and tested on the ALFW2000 and BIWI datasets.

[0069] Experiment 2: Using the BIWI dataset, randomly divide it into a 70% training set and a 30% test set.

[0070] This embodiment uses PyTorch to implement the proposed network structure. The initial learning rate of the backbone network is 1e-5, and the initial learning rate of the final fully connected layer is 1e-4. The batch size is set to 64, and the Adam optimizer is used to train the network for 32 epochs. The experimental environment is shown in Table 2 below.

[0071] Table 2 Experimental Environment Configuration Table

[0072] 4. Analysis of loss function parameters.

[0073] For the parameter analysis of the loss function, Experiment 1 was used to fine-tune the weight parameters α and β, as well as the scaling factor γ, of the improved loss function to determine the optimal parameters for head pose estimation. As shown in Table 3, although the MAE value of AFLW2000 reached its lowest when the geodesic distance and Frobenius norm α and β were 0.7 and 0.3 respectively, and γ was 1, it was only 0.01° lower than the predicted MAE value when γ was √2 with the same values ​​of α and β, but 0.15° higher for the BIWI dataset. This reflects that the loss function with weight parameters of 0.7 and 0.3 and a scaling factor of √2 can maximize the comprehensive utilization of angle and distance information, and the Frobenius norm change scale can also best match the geodesic distance, achieving the most balanced model performance. Therefore, the improved loss function selects weight parameters α=0.7, β=0.3, and a scaling factor γ=√2 as the optimal weights for training the head pose estimation loss function.

[0074] Table 3 shows the weighting coefficients and scaling factors tested on the AFLW2000 and BIWI datasets.

[0075] Figure 11 The graph shows the loss variation of the RepVGG-DP model during training with the improved loss function in Experiments 1 and 2. As the number of epochs increases, the fluctuation in training loss decreases. For the 300W-LP dataset, the loss fluctuation is small and tends to stabilize in the later stages, indicating that the model achieves a good fit on the training set and can predict the labels of the training data well. However, for the BIWI training set, the accuracy reaches its lowest point at the 25th epoch, at which point the model tends to overfit.

[0076] 5. Ablation experiment.

[0077] To verify the effectiveness of the RepVGG-DP network, this embodiment conducted relevant ablation experiments. The same parameters were used in each group of experiments to ensure fairness. The experimental results are shown in Tables 4 and 5. Table 4 shows the results of training the model on the 300W-LP dataset and testing it on the AFLW2000 and BIWI datasets, respectively. Table 5 shows the results when training and testing were performed using 70% and 30% of the BIWI dataset, respectively. Among these, ImD, PNB, and... These represent the ImD feature enhancement module, the PNB information fusion module, and the improved loss function, respectively.

[0078] Table 4 compares different model architectures on the AFLW2000 and BIWI datasets.

[0079] Table 5 compares different model structures on the BIWI test set.

[0080] As shown in Tables 4 and 5, although RepVGG-DP does not always achieve the best results in all three angles, it achieves the best overall performance in all three angles. This indicates that it has stable prediction of pose, strong generalization ability, and improves the performance of head pose estimation algorithms.

[0081] Ablation experiments showed that the original RepVGG model had the highest overall evaluation index (MAE). The model's learning of head pose geometry was enhanced, improving the detection accuracy of subtle pose changes. This resulted in significant improvements in the AFLW2000 dataset with extreme poses, but it did not optimize the BIWI dataset with small pose angles when used alone. The ImD feature enhancement module provides clearer and more accurate feature representations, improving performance. Table 4 shows a reduction in MAE values ​​of 0.05° and 0.11° compared to the original RepVGG model in Experiment 1, while Table 5 shows a reduction in MAE value of 0.17° in Experiment 2. The ability to extract features was enhanced by adding only the PNB information fusion module. However, its performance on the AFLW2000 dataset is mediocre because this dataset contains many extreme poses and a large amount of information. By simultaneously introducing the ImD and attention information fusion modules, the model optimizes feature extraction and representation capabilities at multiple levels and completes contextual interaction, ultimately effectively processing and understanding image data. This results in a more balanced and superior experimental outcome compared to introducing only a single module. Furthermore, by introducing an improved loss function on top of the ImD and attention information fusion modules... Specifically, in experiments based on the RepVGG-DP model, detection on the AFLW2000 dataset showed significant improvement, decreasing by 0.17°. Simultaneously, feature optimization reduced the error by 0.12° on the BIWI dataset and 0.08° on the BIWI test set. This model demonstrated the best overall performance during the evaluation period, reducing the mean absolute error (MAE) by 8.5% and 4.4% in Experiment 1 and by 11.2% in Experiment 2 compared to the original model. This result reflects the significant improvement in the performance of the introduced loss function for head pose estimation models. Training and evaluation with models of different configurations showed that each improvement introduced into RepVGG resulted in a reduction in the model's mean absolute error, confirming the positive impact of each technique on improving model performance.

[0082] This framework not only achieved good prediction results for images with different poses and lighting conditions on the 300W-LP and AFLW2000 datasets, but also demonstrated excellent prediction performance for smaller face images on the BIWI dataset, with overall clear and accurate results. Furthermore, the framework achieved an FPS of 72.4 frames per second in the experimental environment shown in Table 2, achieving real-time estimation capabilities that meet the needs of daily life.

[0083] 6. Comparative experiment.

[0084] To demonstrate the model's sophistication, this embodiment compares it with 12 recent head pose estimation methods, including FAN, OsGG-Net, and EVA-GCN, which are head pose estimation methods based on facial key points, and the rest, which are representative methods without facial key points. The comparison results of the two sets of experiments are shown in Tables 6 and 7, respectively.

[0085] Table 6 compares recent methods on the AFLW2000 and BIWI datasets under Experiment 1.

[0086] Table 7. Comparison of Experiment 2 method with recent methods on the BIWI test set.

[0087] As shown in Table 6, keypoint-based methods are most prone to estimation bias due to their inherent limitations. The classic FAN method achieved the largest mean absolute error across the entire table. OsGG-Net and EVA-GCN performed better than other methods from the same year, such as MFDNet and HeadPosr EH38, indicating that facial keypoint-based methods are not the optimal choice.

[0088] Among the keypoint-free head pose estimation methods compared, HopeNet is generally considered the benchmark model, proposing two multi-loss ResNet50 networks with different regression coefficients (α=1 and α=2). Compared to the HopeNet method, the method in this embodiment reduces the MAE by 37.0% on the AFLW2000 dataset (Table 6), 36.9% on the BIWI dataset, and 24.1% on the BIWI test set (Table 7), indicating that the method has higher accuracy. Furthermore, although in Experiment 1, the mean absolute error of Yaw is greater than DirectMHP-M on the AFLW2000 dataset and greater than 6DRepnet on the BIWI dataset, and in Experiment 2, the mean absolute errors of Yaw and Roll are greater than TokenHPE-E, the method in this embodiment has lower MAE values ​​in both sets of experiments, demonstrating the best model performance. Especially on the challenging AFLW2000 dataset, which contains many unpredictable images, the method in this embodiment still significantly outperforms most of the comparison methods, performing 0.09° slower than the 6DRepnet method, which is the best performing method on this dataset in Table 6. For smaller face images in the BIWI dataset, the method in this embodiment is 0.04° slower than TokenHPE-E, which is the best performing method on this dataset in the table. This verifies the theoretical advancement and effectiveness of the proposed technical strategy and demonstrates its strong potential in practical applications.

[0089] Head pose estimation has broad practical value in multiple fields, providing crucial information for various applications by analyzing and understanding the orientation and angle of the human head. This paper analyzes existing problems in current head pose estimation tasks and selects a computer vision detection method based on facial landmark-free methods to achieve head pose detection. The proposed RepVGG-DP uses RepVGG as the base model and designs an improved method by adding an ImD feature enhancement module and a PNB information fusion module. Furthermore, it replaces the MSE loss function with a new loss function that proportionally fuses geodesic distance and scaled Frobenius norm. Experiments show that the proposed method outperforms other methods in overall MAE performance, demonstrating the effectiveness of head pose detection.

[0090] While a robust backbone can improve prediction results, the current model structure still has redundancy compared to some ultra-lightweight models. Future work could consider making the network lighter so that it can achieve similar performance on low-configuration devices.

[0091] This application also provides an application scenario in which the aforementioned head pose detection method based on the improved RepVGG network is applied. Specifically, the head pose detection method based on the improved RepVGG network provided in this embodiment can be applied in a car driving safety warning scenario. The car driving safety warning scenario includes: an image acquisition stage, a dangerous driving behavior recognition stage, and a warning stage; the head pose detection method based on the improved RepVGG network provided in this embodiment belongs to the pose recognition stage within the dangerous driving behavior recognition stage.

[0092] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 12 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores topic-specific data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a head pose detection method based on an improved RepVGG network.

[0093] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0094] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0095] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0096] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0097] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0098] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0099] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] This embodiment uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application; at the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. In summary, the content of this specification should not be construed as a limitation of this application.

Claims

1. A head pose detection method based on an improved RepVGG network, characterized in that, include: Obtain the image of the head to be tested; The head image to be tested is input into the head pose detection model to obtain the head pose of the head image to be tested. The head pose detection model is obtained by training an improved RepVGG network using facial images of different poses. The improved RepVGG network is obtained by adding a feature denoising module and an attention information fusion module to the RepVGG network. The feature denoising module includes: a first convolutional operation layer, a batch normalization layer, a ReLU activation function layer, and a first residual operation layer connected in sequence; The input terminal of the convolutional operation layer is the head image to be tested or the facial images of different poses trained; The second input terminal of the first residual operation layer is used to input the head image to be tested or the facial images of different poses trained; The first residual operation layer is also connected to the input of the first feature extraction module in the improved RepVGG network; The attention information fusion module includes an improved attention module, a second residual operation layer, a ReLU activation function layer, a second convolution operation layer, a first Sigmoid function layer, a first multiplication operation layer, a second multiplication operation layer, and a third residual operation layer; The input of the improved attention module, the first input of the second residual operation layer, and the first input of the first multiplication operation layer are all connected to the output of the fourth feature extraction module in the improved RepVGG network. The first output of the improved attention module is connected to the second input of the second residual operation layer; The second output of the improved attention module is connected to the first input of the second multiplication layer; The output of the second residual operation layer is connected to the input of the ReLU activation function layer; The output of the ReLU activation function layer is connected to the input of the second convolution operation layer. The output of the second convolutional layer is connected to the input of the first sigmoid function layer; The first output of the first Sigmoid function layer is connected to the second input of the first multiplication layer. The second output of the first Sigmoid function layer is connected to the second input of the second multiplication layer. The output of the second multiplication layer is connected to the first input of the third residual layer; The output of the first multiplication layer is connected to the second input of the third residual layer. The output of the third residual operation layer is connected to the input of the fifth feature extraction module in the improved RepVGG network; The improved attention module includes: a first branch, a second branch, a third branch, and a fourth residual operation layer; The output terminals of the first branch, the second branch, and the third branch are all connected to the input terminal of the fourth residual operation layer. The output of the fourth residual operation layer is connected to the second input of the second residual operation layer and the first input of the second multiplication operation layer, respectively.

2. The head pose detection method based on the improved RepVGG network according to claim 1, characterized in that, The improved RepVGG network includes: a feature denoising module, a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, an attention information fusion module, a fifth feature extraction module, a fully connected structure, and a Schmidt orthogonal function, connected in sequence.

3. The head pose detection method based on the improved RepVGG network according to claim 2, characterized in that, The first feature extraction module includes a first residual structure; The second feature extraction module includes a second residual structure and N1 third residual structures connected in sequence; The third feature extraction module includes a fourth residual structure and N2 fifth residual structures connected in sequence; The fourth feature extraction module includes a sixth residual structure and N3 seventh residual structures connected in sequence; The fifth feature extraction module includes an eighth residual structure; The first residual structure, the second residual structure, the fourth residual structure, the sixth residual structure, and the eighth residual structure are all residual structures containing a 1×1 convolution matrix; The third residual structure, the fifth residual structure, and the seventh residual structure are all residual structures that include 1×1 convolution and identity mapping.

4. The head pose detection method based on the improved RepVGG network according to claim 3, characterized in that, The first branch includes a third convolutional operation layer; The third branch includes a fourth convolutional operation layer; The second branch includes an average pooling layer, a fifth convolutional operation layer, a second sigmoid function layer, and a third multiplication operation layer; The input of the average pooling layer and the first input of the third multiplication layer are both connected to the output of the fourth feature extraction module. The output of the average pooling layer is connected to the input of the fifth convolution operation layer. The output of the fifth convolutional operation layer is connected to the input of the second sigmoid function layer. The output of the second Sigmoid function layer is connected to the second input of the third multiplication layer.

5. The head pose detection method based on the improved RepVGG network according to claim 4, characterized in that, Before acquiring the image of the head to be tested, the following steps are also included: Build an improved RepVGG network; The rotation matrix corresponding to multiple training head images is determined to be the true rotation matrix; the training head images are facial images in different poses; Multiple training head images are input into the improved RepVGG network to obtain the prediction rotation matrix corresponding to the multiple training head images; The value of the hybrid loss function is determined based on multiple true rotation matrices and multiple predicted rotation matrices; the hybrid loss function is determined by weighted summation of geodesic distance and scaled Frobenius norm. Adjust the parameters of the improved RepVGG network and return to the step "input multiple training head images into the improved RepVGG network to obtain the prediction rotation matrix corresponding to the multiple training head images" until the number of iterations equals the preset number of iterations, and determine the improved RepVGG network at the last iteration as the head pose detection model.

6. The head pose detection method based on the improved RepVGG network according to claim 5, characterized in that, The hybrid loss function is: in, For a mixed loss function, For the true rotation matrix, To predict the rotation matrix, For geodesic distance, For geodesic distance weighting, It is the Frobenius norm. Scaling factor These are the scaled Frobenius norm weights.

7. A computer device, comprising: The memory and processor contain a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the head pose detection method based on the improved RepVGG network according to any one of claims 1-6.

Images