Pupil canthus point identification method based on improved RTMPose

Abstract

The invention relates to the technical field of computer vision, in particular to a pupil and eye corner point recognition method based on improved RTMPose, and the method comprises the steps: obtaining a to-be-recognized face image; the to-be-recognized face image is input into a preset recognition model, pupil and canthus point recognition results are obtained, the recognition model is obtained based on training of a training set, the training set comprises face images marked with pupils and canthus points, and the recognition model is constructed through an improved RTMPose network. According to the method, the identification model is constructed by introducing the SimAM attention module and the optimization loss function into the RTMPose network, so that the identification precision and the model robustness can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a pupil and eye corner point recognition method based on improved RTMPose. Background Art

[0002] Pupil and eye corner point recognition is a key task in the field of computer vision, with a wide range of application backgrounds and practical value. Accurate pupil and eye corner point recognition plays a vital role in many fields such as medical diagnosis, eye tracking, and intelligent driving. First, the pupil and eye corner points are small in size and have variable shapes in the image, which requires the algorithm to have high-precision detection capabilities for small targets. Second, the texture of the eye area is complex and the lighting conditions are variable, which increases the difficulty of recognition.

[0003] At present, pupil point recognition algorithms are mainly divided into two categories: morphological-based methods and deep learning-based methods. Morphological-based methods require that the image be binarized and filtered first, and then a corner detection algorithm is used to detect many corners when performing corner detection on the face image. Finally, the results of the corner detection must be screened, but the screening algorithms are generally more complicated and the effect is not very good.

[0004] There are many pupil center localization algorithms based on deep convolutional neural networks (DCNN) based on deep learning methods. One of the most popular structures is the U-Net convolutional neural network. This network structure uses convolution and pooling to encode the image, then upsamples the decoded image based on the feature map generated by the encoding, and segments the decoded image to extract the pupil area. Convolutional neural networks such as U-Net perform well in pupil center localization tasks, but they may be difficult to generalize to different scenarios or data with different distributions. The real-time performance and models of convolutional neural networks such as U-Net need to be optimized and improved.

[0005] Therefore, a pupil and eye corner point recognition method based on improved RTMPose is proposed. Summary of the invention

[0006] The purpose of the present invention is to address the problem of low recognition accuracy of existing pupil and eye corner point recognition technologies, and to provide a pupil and eye corner point recognition method based on improved RTMPose, thereby improving the recognition accuracy and robustness of the recognition model by introducing the SimAM attention module and optimizing the loss function.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] The pupil and eye corner point recognition method based on improved RTMPose includes:

[0009] Obtaining a face image to be recognized;

[0010] The face image to be identified is input into a preset recognition model to obtain pupil and eye corner recognition results, wherein the recognition model is obtained based on a training set, the training set includes face images with pupils and eye corners marked, and the recognition model is constructed by improving the RTMPose network.

[0011] Optionally, the training set includes: volunteer face images and public face images with pupils and eye corners annotated after adjusting lighting conditions, facial angles, and simulating occlusion.

[0012] Optionally, before training the recognition model based on the training set, the training set is also preprocessed, and the preprocessing includes: scaling, normalization, cropping and padding operations.

[0013] Optionally, constructing the recognition model by improving the RTMPose network includes:

[0014] A SimAM attention module is introduced into the CSPLayer of the RTMPose network. The SimAM attention module is used to obtain attention weights by calculating local self-similarity of feature maps, and to enhance features of the feature maps through the attention weights.

[0015] Optionally, the SimAM attention module obtains attention weights by calculating local self-similarity of the feature map, and performs feature enhancement on the feature map by using the attention weights, including:

[0016] Input the feature map into the SimAM attention module, and calculate the mean, variance, and weighting parameters of each channel in the SimAM attention module;

[0017] The attention weight of each pixel position is calculated based on the mean, variance, and weighting parameter, and the attention weight is applied to the feature map to obtain an enhanced feature map.

[0018] Optionally, the mean is calculated as follows:

[0019]

[0020] Among them, μ c is the mean of the cth channel, X c,i,j is the feature map with height i and width j in the cth channel, where H and W are the height and width of the input feature map;

[0021] The variance is calculated as follows:

[0022]

[0023] in, is the variance of the cth channel;

[0024] The calculation method of the weighted parameter is:

[0025]

[0026] Among them, α c and β c is the weighting parameter, ∈ is a constant.

[0027] Optionally, the method for calculating the attention weight of each pixel position based on the mean, variance, and weighting parameter is:

[0028] A c，i,j =α c (X c，i，j -μ c )+β c

[0029] Among them, A c,i,j is the attention weight for each pixel position.

[0030] Optionally, the method of applying the attention weight to the feature map to obtain an enhanced feature map is:

[0031] Y c，i，j =A c，i,j X c，i，j

[0032] Among them, Y c,i,j is the enhanced feature map.

[0033] Optionally, constructing the recognition model by improving the RTMPose network also includes: using Focal EIOU Loss as a loss function of the RTMPose network.

[0034] The beneficial effects of the present invention are:

[0035] The pupil and eye corner point recognition method based on improved RTMPose provided by the present invention can improve the accuracy and robustness of the recognition model by introducing the SimAM attention module into RTMPose and optimizing the loss function of RTMPose. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 This is a flow chart of a pupil eye corner point recognition method based on improved RTMPose according to an embodiment of the present invention;

[0038] Figure 2 Schematic diagram of the recognition model structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] This embodiment provides a pupil and eye corner point recognition method based on improved RTMPose, including:

[0042] The pupil and eye corner point recognition method based on improved RTMPose includes:

[0043] Obtaining a face image to be recognized;

[0044] The face image to be identified is input into a preset recognition model to obtain pupil and eye corner recognition results, wherein the recognition model is obtained based on a training set, the training set includes face images with pupils and eye corners marked, and the recognition model is constructed by improving the RTMPose network.

[0045] Furthermore, the training set includes: volunteer face images and public face images with pupils and eye corners annotated after adjusting lighting conditions, facial angles, and simulating occlusion.

[0046] Before training the recognition model based on the training set, the training set is also preprocessed, and the preprocessing includes: scaling, normalization, cropping and padding operations.

[0047] Furthermore, constructing the recognition model by improving the RTMPose network includes:

[0048] A SimAM attention module is introduced into the CSPLayer of the RTMPose network. The SimAM attention module is used to obtain attention weights by calculating local self-similarity of feature maps, and to enhance features of the feature maps through the attention weights.

[0049] Furthermore, constructing the recognition model by improving the RTMPose network also includes: using Focal EIOU Loss as a loss function of the RTMPose network.

[0050] Combine the following Figure 1 , Figure 2 The pupil eye corner point recognition method based on improved RTMPose provided in this embodiment is specifically described, including the following contents:

[0051] Step 1, build a data set;

[0052] Facial images are collected by recruiting volunteers, and high-quality facial images are formed by adjusting lighting conditions, facial angles, and simulating occlusion to construct the first dataset.

[0053] We also screened out images that met the annotation requirements from multiple open source and public datasets, and further processed them to construct a second dataset.

[0054] After image preprocessing of the first and second datasets, the final dataset is constructed. Preprocessing includes: scaling operations to ensure that the image size matches the model input layer; normalization processing to adjust the pixel values ​​of the image to the range used in model training to reduce the impact of data distribution differences on model recognition performance; cropping and padding to eliminate irrelevant background information and highlight the target area. Through preprocessing, the original image can be converted into a format that the model can efficiently process, laying the foundation for subsequent feature extraction and recognition.

[0055] Step 2: Build a recognition model;

[0056] RTMPose is a simple and efficient real-time human pose estimation framework. RTMPose continuously downsamples feature maps and uses k (number of key points) key point representations as the basis for sub-pixel scale classification of the head. This achieves a streamlined feature extraction and coordinated classification process, eliminating unnecessary operations. RTMPose uses the CSPNext framework as the backbone image feature extraction component. CSPNext uses deep convolution to strike a balance between speed and accuracy.

[0057] RTMPose consists of a backbone and a classification head for feature extraction. The backbone network consists of a convolutional module (ConvModule), a cross-stage local layer (CSPLayer), and a spatial pyramid pooling bottleneck layer (SPPFBottleneck). First, ConvModule, as the basic module for network downsampling and channel adjustment, consists of Conv2d, BatchNormalization, and SiLU activation functions. CSPLayer enhances the feature extraction capability of the network through cascading. SPPFBottleneck pools feature maps of different scales to effectively capture semantic information in the image, thereby improving model performance. The head part includes a 7×7 convolution, FC (fully connected layer), GAU (gated attention unit), and a coordinate classifier. The 7×7 convolution layer reduces the channel to k (the number of key points) feature maps, which generate representations of k key points. The FC layer expands the representation of the key points to the required dimension, which is controlled by hyperparameters. GAU uses self-attention to further integrate global and local spatial information. Finally, the classifier classifies the coordinates of the representations of the k key points in the horizontal and vertical directions to achieve key point positioning.

[0058] Based on the RTMPose network, a compression and excitation module is introduced in CSPLayer. In this embodiment, a SimAM attention module is used, and the SimAM attention module is used to obtain the attention weight by calculating the local self-similarity of the feature map.

[0059] The core idea of ​​the SimAM attention module is based on the local self-similarity of images. In an image, adjacent pixels usually have strong similarities, while the similarities between distant pixels are weaker. The SimAM attention module uses this feature to generate attention weights by calculating the similarity between each pixel in the feature map and its neighboring pixels.

[0060] The implementation and workflow of the SimAM attention module can be summarized into the following steps:

[0061] (1) Input feature map: The input feature map is denoted by X, and its shape is C*H*W, where C is the number of channels, and H and W are the height and width of the feature map.

[0062] (2) Calculate the mean μ of each channel c , the formula is as follows:

[0063]

[0064] (3) Calculate the variance of each channel The formula is as follows:

[0065]

[0066] (4) Calculate the weighting parameter α c and β c :in ∈ is a small constant used to avoid division by zero errors.

[0067] (5) Calculate the attention weight A for each pixel position c,i,j , the formula is as follows:

[0068] A c，i,j =α c (X c，i，j -μ c )+β c

[0069] (6) Apply the attention weight to the original feature map to obtain the enhanced feature map:

[0070] Y c,i，j =A c,i,j X c，i，j

[0071] The loss function of RTMPose during training was optimized. By adjusting the weights of different parts in the loss function, the model paid more attention to the recognition accuracy of pupils and eye corners. The improved positioning loss function of RTMPose is Focal EIOULoss, and the calculation formula is:

[0072] L Focal-EIOU =IOU γ L EIOU

[0073] EIOU not only inherits the excellent characteristics of CIOU loss, but also significantly improves the convergence speed while ensuring positioning accuracy. In addition, in order to effectively deal with the sample imbalance problem often encountered in bounding box regression tasks, EIOU introduces Focal Loss. This measure further enhances the robustness of EIOU in dealing with complex scenes and diverse targets, making the positioning effect of the model even better.

[0074] In summary, Focal EIOU Loss can explicitly measure the overlap area between the prediction box and the label box, the difference in the center point position and the side length, and handle the problem of sample imbalance. The IOU between the target box and the prediction box is used to reweight the EIOU loss. High-quality samples with high IOU are given greater weight, while low-quality samples are suppressed.

[0075] Step 3, recognition model training;

[0076] Initialize the recognition model, set parameters and initial state. The recognition model is built and trained under the deep learning framework Pytorch, i9-12900KF CPU, Nvidia GTX 3080Ti. The data set is input into the recognition model. Through multiple rounds of iteration and optimization, the recognition model can gradually learn the feature representation of pupils and eye corners, thereby improving its performance in related tasks. Save the recognition model after training.

[0077] Step 4, processing of recognition results;

[0078] Obtain a face image to be identified, input the face image to be identified into a saved recognition model, and obtain pupil and eye corner point recognition results, wherein the recognition result is a face image with a prediction box of the pupil and eye corner point.

[0079] Post-processing techniques such as non-maximum suppression are used to fine-tune the recognition results and remove redundant prediction boxes, making the recognition results more accurate and reliable.

[0080] Finally, the processed recognition results are output in the form of text and image. The text output records the location information of the pupil and eye corner in detail, which is convenient for subsequent data analysis and processing; while the image output intuitively displays the recognition results by drawing a marking box on the original image, making the results easier to understand and use.

[0081] Step 5, identification model testing and evaluation;

[0082] RTMPose performance evaluation:

[0083] An independent test set is used to evaluate the performance of the trained recognition model. The evaluation indicators include positioning accuracy, positioning speed, recall rate, false positive rate, etc. Positioning accuracy is usually measured by calculating the deviation between the model's predicted position and the actual marked position.

[0084] RTMPose robustness test:

[0085] The recognition model is tested for robustness, including its performance under different lighting conditions, noise interference, posture changes, etc. This helps to evaluate the stability and reliability of the model in practical applications.

[0086] RTMPose evaluation combination and assessment:

[0087] In the evaluation of the pupil and eye corner positioning model, we tested our own labeled datasets and used the pixel-level accuracy in image processing technology to evaluate the accuracy of the model positioning. At the same time, we also considered the generalization ability of the model in different scenarios, that is, whether the model can show stable performance in different datasets.

[0088] The comparison data of the recognition model of this embodiment and other models are shown in Table 1:

[0089] Table 1

[0090]

[0091]

[0092] From the above content, it can be seen that the recognition model constructed by improving RTMPose in this implementation has the following advantages:

[0093] High-precision positioning: The model can accurately locate the position of the pupil and the corner of the eye, meeting the high accuracy requirements in practical applications.

[0094] Real-time performance: The model meets real-time requirements in terms of processing speed and is suitable for scenarios that require fast response, such as human-computer interaction, eye tracking, etc.

[0095] Strong robustness: The model is robust to changes in illumination, noise interference, posture changes, etc., and can work stably in complex environments.

[0096] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. The pupil and eye corner point recognition method based on improved RTMPose is characterized by: include: Obtain a face image to be recognized; The face image to be identified is input into a preset recognition model to obtain pupil and eye corner recognition results, wherein the recognition model is obtained based on a training set, the training set includes face images with pupils and eye corners marked, and the recognition model is constructed by improving the RTMPose network.

2. The pupil and eye corner point recognition method based on improved RTMPose according to claim 1, characterized in that: The training set includes: volunteer face images and public face images with pupils and eye corners marked after adjusting lighting conditions, facial angles, and simulating occlusion.

3. The pupil and eye corner point recognition method based on improved RTMPose according to claim 1, characterized in that: Before training the recognition model based on the training set, the training set is also preprocessed, and the preprocessing includes: scaling, normalization, cropping and padding operations.

4. The pupil corner point recognition method based on improved RTMPose according to claim 1 is characterized in that: The recognition model is constructed by improving the RTMPose network, including: A SimAM attention module is introduced into the CSPLayer of the RTMPose network. The SimAM attention module is used to obtain attention weights by calculating local self-similarity of feature maps, and to enhance features of the feature maps through the attention weights.

5. The pupil and eye corner point recognition method based on improved RTMPose according to claim 4 is characterized in that: The SimAM attention module obtains attention weights by calculating the local self-similarity of the feature map, and performs feature enhancement on the feature map by using the attention weights, including: Input the feature map into the SimAM attention module, and calculate the mean, variance, and weighting parameters of each channel in the SimAM attention module; The attention weight of each pixel position is calculated based on the mean, variance, and weighting parameter, and the attention weight is applied to the feature map to obtain an enhanced feature map.

6. The pupil and eye corner point recognition method based on improved RTMPose according to claim 5, characterized in that: The mean is calculated as follows: Among them, μ c is the mean of the cth channel, X c，i，j is the feature map with height i and width j in the cth channel, where H and W are the height and width of the input feature map; The variance is calculated as follows: in, is the variance of the cth channel; The calculation method of the weighted parameter is: Among them, α c and β c is the weighting parameter, ∈ is a constant.

7. The pupil and eye corner point recognition method based on improved RTMPose according to claim 6, characterized in that: The method for calculating the attention weight of each pixel position based on the mean, variance, and weighting parameters is: A c，i，j =a c (X c，i，j -m c )+b c Among them, A c，i，j is the attention weight for each pixel position.

8. The pupil and eye corner point recognition method based on improved RTMPose according to claim 7, characterized in that: The method of applying the attention weight to the feature map to obtain the enhanced feature map is: AND c，i，j =A c，i，j X c，i，j Among them, Y c,i,j is the enhanced feature map.

9. The pupil and eye corner point recognition method based on improved RTMPose according to claim 4, characterized in that: Constructing the recognition model by improving the RTMPose network also includes: using FocalEIOU Loss as the loss function of the RTMPose network.

Images