Facial expression recognition method for driving scene
Through the dynamic expression recognition model of lighting invariant lighting, combined with low-level and advanced feature learning modules, the problem of insufficient robustness of facial expression recognition technology when lighting changes in driving scenes is solved, the recognition accuracy and adaptability are improved, and the complexity is improved. Facial expression recognition performance under lighting conditions.
Patent Information
- Application Number
- CN202411495488.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing facial expression recognition technology lacks robustness and adaptability to changes in lighting conditions in driving scenarios, resulting in a degradation of recognition performance under night or under variable lighting conditions, making it difficult to apply in real driving environments.
The dynamic expression recognition model of lighting is adopted, combined with the low-level lighting is constant expression feature learning module and the advanced dynamic significant expression feature learning module, and extract and Classify facial expression characteristics to enhance the generalization ability of the model under different lighting conditions.
Improves the accuracy and reliability of facial expression recognition under complex lighting conditions, improves the practicality in safe driving applications, can better identify unseen facial expression data, and reduces features within the same expression category Distance, expand the characteristic distance between different expression categories.
Smart Images

Figure CN119028004A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image recognition technology, and in particular to a facial expression recognition method for driving scenarios. Background Art
[0002] Facial expression recognition aims to estimate a person's emotional state using visual information, and has important application value in many human-computer interaction systems such as medical assistance, driving assistance, distance education, and social robotics. Currently, common facial expression recognition methods have shown promising results, but their reliance on visible light images remains a significant limitation that hinders their application in driving scenarios. The inconsistent lighting conditions encountered in environments such as nighttime or tunnels may significantly affect the performance of visible light-based facial expression recognition techniques. This dependence limits the effectiveness and reliability of common facial expression recognition techniques when dealing with variable lighting conditions in real driving environments.
[0003] The facial expression recognition task in driving scenarios is different from the common facial expression recognition task. The most significant difference between them is the lighting conditions. In driving scenarios, there are two very different lighting conditions, corresponding to daytime and nighttime. Typically, cameras capture visible spectrum (VIS) images during the daytime and near infrared (NIR) images at night. One view is that in order to cope with nighttime scenes, a separate facial expression recognition method can be trained using NIR facial image data. However, this view does not take into account the changes in lighting conditions that may occur during a driving process, such as the transition period between dawn and dusk, or the dynamic lighting conditions caused by street lights and car lights. Therefore, these models usually lack the robustness and adaptability required to cope with unpredictable and rapidly changing lighting conditions in real driving environments.
[0004] Facial expression recognition technology for driving scenarios should focus on developing models that are robust to lighting changes, which may involve developing hybrid models that can seamlessly integrate visible and near-infrared data, or implementing advanced pre-processing techniques to normalize facial features under different lighting conditions. By addressing these challenges, facial expression recognition technology can achieve higher accuracy and reliability, thereby improving its practicality in safe driving applications. Summary of the invention
[0005] The embodiment of the present application provides a facial expression recognition method for driving scenarios, so as to solve the problem that the facial expression recognition technology in the prior art has poor robustness and adaptability in scenarios with rapidly changing lighting conditions.
[0006] The embodiment of the present application provides a facial expression recognition method for driving scenarios, including: Establish an illumination-invariant dynamic expression recognition model for driving scenarios. The illumination-invariant dynamic expression recognition model includes a low-level illumination-invariant expression feature learning module, a high-level dynamic saliency expression feature learning module, and a classifier module. Use the training set to train the illumination-invariant dynamic expression recognition model; Acquire a face video and extract a face image sequence from the face video; Inputting the facial image sequence into the trained illumination-invariant dynamic expression recognition model, and performing instance normalization and batch normalization on the facial image sequence using the low-level illumination-invariant expression feature learning module to obtain an illumination-invariant expression feature sequence; The illumination-invariant expression feature sequence is position-encoded according to the time information to obtain a position feature sequence; The position feature sequence is input into the advanced dynamic salient expression feature learning module, which includes a first-stage attention mechanism and a second-stage attention mechanism. The first-stage attention mechanism scales the values in the position feature sequence according to the relationship between the key and the query to obtain a scaled feature sequence. The second-stage attention mechanism aggregates the scaled feature sequence to obtain a global dynamic salient expression feature. The global dynamic significant expression features are input into the classifier module, and the classifier module classifies the global dynamic significant expression features to obtain the facial expression recognition result.
[0007] A facial expression recognition method for driving scenarios in this application has the following advantages: Through the illumination-invariant dynamic expression recognition model, several face images extracted from the face video are feature extracted and classified to obtain the facial expression recognition results. In the present application, the low-level illumination-invariant feature learning module removes instance-specific contrast information by integrating instance normalization in the shallow layer of the deep convolutional network, which helps to extract illumination-invariant expression features, and can enhance the modeling ability of the model in one domain (such as visible light images captured during the day) and the generalization ability in another domain (such as near-infrared images captured at night), which is suitable for facial expression recognition under driving scenes with complex lighting conditions. At the same time, the high-level dynamic saliency expression feature learning module in the present application uses a two-stage attention mechanism to blend and converge the information of the extracted multi-frame illumination-invariant expression features, which can identify the important frames in the feature sequence corresponding to the input image sequence and extract the key dynamic features in the facial expression. At the same time, the introduction of additive angular interval loss in the model training process helps to reduce the feature distance within the same expression category and expand the feature distance between different expression categories. In this way, the model can learn more discriminative features, thereby improving its recognition ability and generalization of unseen facial expression data. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0009] Figure 1 A flowchart of a facial expression recognition method for driving scenarios provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of the illumination-invariant dynamic expression recognition model provided in an embodiment of the present application. DETAILED DESCRIPTION
[0010] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0011] Figure 1 A flowchart of a facial expression recognition method for driving scenarios provided in an embodiment of the present application. An embodiment of the present application provides a facial expression recognition method for driving scenarios, including: S100, establishing an illumination-invariant dynamic expression recognition model for driving scenarios, wherein the illumination-invariant dynamic expression recognition model includes a low-level illumination-invariant expression feature learning module, a high-level dynamic saliency expression feature learning module, and a classifier module.
[0012] S110, training an illumination-invariant dynamic expression recognition model using the training set.
[0013] Exemplarily, the method for obtaining a training set includes: Obtain a dynamic expression database under visible light and near-infrared lighting conditions, the dynamic expression database containing sample face images and corresponding expression annotations; The sample face images and expression annotations are divided into a test set and a training set, and the sample face images in the test set and the training set are preprocessed.
[0014] Specifically, the dynamic expression database stores a large number of face videos, which are obtained by using a camera to record the facial states of multiple non-specific people in different expression states. Sample face images can be obtained by extracting the face videos frame by frame, and the corresponding expression annotations can be obtained by manually annotating each sample face image. For the training set, the expression annotations in the training set are the training set annotations, and for the test set, the expression annotations in the test set are the test set annotations.
[0015] Furthermore, the preprocessing of the sample face images includes random erasing, random cropping, random rotation and random horizontal flipping.
[0016] Specifically, before the sample face images are preprocessed, the sample face images extracted from the face video are cropped and resized to a size of 224×224 pixels.
[0017] Furthermore, when training the illumination-invariant dynamic expression recognition model, the additive angular margin loss function and the Adam optimization algorithm are used to optimize the illumination-invariant dynamic expression recognition model.
[0018] Specifically, the additive angle margin loss function is expressed as:
[0019] in, is the additive angle margin loss function, N is the number of samples in the training set, j is the label of the correct expression category, e is a natural constant, a hyperparameter s ≥1 is a scaling factor that controls the compactness of the class center description vector in the angular space, is the correct expression category in the sample The class center description vector and n Samples Corresponding global dynamic salient expression features The angle between n is the sample number, C is the number of expression categories, c is the label of the expression category, For the expression category other than the correct one The class center description vector of expression categories and n Samples Corresponding global dynamic salient expression features The angle between the hyperparameters m ≥0 is a and The penalty factor of the angle interval is used to improve the intra-class similarity and inter-class distinction.
[0020] After establishing the additive angular interval loss function, the Adam optimization algorithm is used to optimize the parameters of the illumination-invariant dynamic expression recognition model with the goal of minimizing the value of the additive angular interval loss function.
[0021] After completing the training of the illumination-invariant dynamic expression recognition model, the performance of the trained illumination-invariant dynamic expression recognition model is tested using the test set. In the test, the classification results output by the trained illumination-invariant dynamic expression recognition model are compared with the expression annotations of the sample face images in the test set, and the performance indicators of the trained illumination-invariant dynamic expression recognition model are calculated. After all performance indicators meet the requirements, the trained illumination-invariant dynamic expression recognition model is applied to actual work. If one or several performance indicators do not meet the requirements, the illumination-invariant dynamic expression recognition model is trained again.
[0022] S120, obtaining a face video, and extracting a face image sequence from the face video.
[0023] For example, starting from this step and applying it to actual work, a camera facing the driver can be installed inside the vehicle, and the camera can be used to capture the driver's facial video while driving. By extracting the facial video frame by frame, a facial image sequence containing multiple facial images can be obtained.
[0024] S130, inputting the facial image sequence into the trained illumination-invariant dynamic expression recognition model, and performing instance normalization processing and batch normalization processing on the facial image sequence using a low-level illumination-invariant expression feature learning module to obtain an illumination-invariant expression feature sequence.
[0025] For example, Figure 2 As shown in Figure 1, the low-level illumination invariant expression feature learning module contains three residual blocks connected in sequence. There is also an intermediate convolutional layer between two adjacent residual blocks. The facial feature map output by each residual block is expressed as:
[0026] in, is the face feature map, is the residual calculation, IN For instance normalization, BN For batch normalization, It is a face image in a face image sequence. The residual block of the previous level processes the face image to obtain the corresponding face feature map, which is input into the middle convolution layer. After the middle convolution layer extracts the features, the extracted features are input into the residual block of the next level until all residual blocks have completed the processing of the output features of the middle convolution layer of the previous level.
[0027] Instance Normalization IN The calculation of is as follows:
[0028] in, and Respectively face images The mean and variance of the pixel values in , is the scaling parameter, The scaling parameters and the shift parameters are learnable parameters determined during the training process. is a value that maintains numerical stability.
[0029] S140, position encoding the illumination-invariant expression feature sequence according to time information to obtain a position feature sequence.
[0030] Exemplarily, the illumination-invariant expression feature sequence is expressed as:
[0031] in, is the illumination invariant expression feature sequence The elements in .
[0032] After position encoding, the elements in the position feature sequence are represented as:
[0033]
[0034] in, and are the elements in the position feature sequence No. Dimensions and The eigenvalues of the dimensions, and The elements No. Dimensions and The value of the dimension, t is the label of the face image, For elements The dimension of .
[0035] S150, inputting the position feature sequence into the advanced dynamic significant expression feature learning module, the advanced dynamic significant expression feature learning module includes a first-stage attention mechanism and a second-stage attention mechanism, the first-stage attention mechanism scales the values in the position feature sequence according to the relationship between the key and the query to obtain a scaled feature sequence, and the second-stage attention mechanism aggregates the scaled feature sequence to obtain a global dynamic significant expression feature.
[0036] Exemplarily, the high-level dynamic salient expression feature learning module includes a two-layer transformer encoder and a feature aggregation layer, wherein the two-layer transformer encoder is the first-stage attention mechanism, and the feature aggregation layer is the second-stage attention mechanism.
[0037] Furthermore, the first-stage attention mechanism scales the values in the position feature sequence, expressed as:
[0038] in, is the attention mechanism, For query, , is the key, , For the value, , , and are the elements in the query, key, and value, respectively. , , , , and The query elements , key element Sum value elements The weight matrix of is the element in the position feature sequence, yes The dimension of , For elements in the scaled feature sequence, the superscript T Indicates transpose.
[0039] The global dynamic salient expression feature is expressed as:
[0040] in, a is the global dynamic salient expression feature, T is the number of face images in the face image sequence, t is the label of the face image, for The elements in is the self-attention weight;
[0041] in, MLP is a feedforward neural network with sigmoid function as activation function. u is the scaled feature sequence.
[0042] S160, inputting the global dynamic significant expression features into a classifier module, and the classifier module classifies the global dynamic significant expression features to obtain a facial expression recognition result.
[0043] For example, the classifier module includes a fully connected layer, and the probability of the expression category to which the face image belongs can be obtained through classification processing of the softmax function. p , which is the facial expression recognition result.
[0044] Experimental Description In order to demonstrate the effectiveness of the method in this application, verification experiments were carried out on the Oulu-CASIA (Institute of Automation Chinese Academy of Sciences) dataset and the DFEW (Dynamic Facial Expressions in the Wild) dataset.
[0045] First, a five-fold cross-validation experiment was conducted on the visible light subset and near-infrared subset of the Oulu-CASIA dataset, divided by test individuals. As a comparison, the basic model uses ResNet-18 as the backbone network to extract features, obtains expression categories through feature aggregation and classifiers, and trains the model with cross-entropy loss. The basic model trained on visible light data has an accuracy of 81.60% on the visible light test set and 74.38% on the near-infrared test set, and the recognition accuracy has dropped by 8.85%. When the method of the present application is trained on visible light data, the accuracy on the visible light test set is 86.11%, and the accuracy on the near-infrared test set is 82.22%, and the performance has only dropped by 4.52%, which reflects the adaptability of the method of the present application to changes in illumination.
[0046] In order to compare with other advanced methods, a ten-fold cross-validation experiment was conducted on the Oulu-CASIA dataset. The accuracy of the established model reached 90.21% in the visible light-normal lighting condition subset, 90.63% in the near-infrared-normal light lighting subset, and 88.96% in the near-infrared-dark light lighting subset. The above results have reached or are ahead of the advanced level in the industry. In order to further prove the effectiveness of the method of this application in facial expression recognition in natural scenes, after a five-fold cross-validation experiment on the DFEW dataset, the established model UAR (Unweighted Average Recallrate) index can reach 60.67%, and WAR (Weighted Average Recall rate) index can reach 71.48%, which has reached the advanced level in the industry.
[0047] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0048] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A facial expression recognition method for driving scenes, characterized in that: include: Establishing an illumination-invariant dynamic expression recognition model for driving scenes, wherein the illumination-invariant dynamic expression recognition model comprises a low-level illumination-invariant expression feature learning module, a high-level dynamic saliency expression feature learning module and a classifier module; Using the training set to train the illumination-invariant dynamic expression recognition model; Acquire a face video, and extract a face image sequence from the face video; Inputting the facial image sequence into the trained illumination-invariant dynamic expression recognition model, and performing instance normalization processing and batch normalization processing on the facial image sequence using the low-level illumination-invariant expression feature learning module to obtain an illumination-invariant expression feature sequence; Position encoding the illumination-invariant expression feature sequence according to time information to obtain a position feature sequence; Inputting the position feature sequence into the high-level dynamic significant expression feature learning module, the high-level dynamic significant expression feature learning module comprises a first-stage attention mechanism and a second-stage attention mechanism, the first-stage attention mechanism scales the values in the position feature sequence according to the relationship between the key and the query to obtain a scaled feature sequence, and the second-stage attention mechanism aggregates the scaled feature sequence to obtain a global dynamic significant expression feature; The global dynamic significant expression features are input into the classifier module, and the classifier module classifies the global dynamic significant expression features to obtain a facial expression recognition result.
2. A facial expression recognition method for driving scenes according to claim 1, characterized in that: The method for obtaining the training set includes: Acquire a dynamic expression database under visible light and near-infrared lighting conditions, wherein the dynamic expression database includes sample face images and corresponding expression annotations; The sample face images and the expression annotations are divided into a test set and a training set, and the sample face images in the test set and the training set are preprocessed.
3. The facial expression recognition method for driving scenes according to claim 2, characterized in that: The preprocessing of the sample face image includes random erasing, random cropping, random rotation and random horizontal flipping.
4. The facial expression recognition method for driving scenes according to claim 1, characterized in that: When training the illumination-invariant dynamic expression recognition model, the illumination-invariant dynamic expression recognition model is optimized using an additive angle margin loss function and an Adam optimization algorithm.
5. The method for facial expression recognition in a driving scenario according to claim 4, characterized in that: The additive angle interval loss function is expressed as: in, is the additive angle spacing loss function, N is the number of samples in the training set, j is the label of the correct expression category, e is a natural constant, a hyperparameter s ≥1 is a scaling factor that controls the compactness of the class center description vector in the angular space, is the correct expression category in the sample The class center description vector and n Samples Corresponding global dynamic salient expression features The angle between n is the sample number, C is the number of expression categories, c is the label of the expression category, For the expression category other than the correct one The class center description vector of expression categories and n Samples Corresponding global dynamic salient expression features The angle between the hyperparameters m ≥0 is a and The penalty factor for the angle interval between them.
6. The method for facial expression recognition in a driving scenario according to claim 1, characterized in that: The low-level illumination invariant expression feature learning module comprises three residual blocks connected in sequence, and an intermediate convolution layer is connected between two adjacent residual blocks. The facial feature map output by each residual block is expressed as: in, is the facial feature map, is the residual calculation, IN For instance normalization, BN For batch normalization, is a face image in the face image sequence; The calculation of instance normalization IN is as follows: in, and Respectively face images The mean and variance of the pixel values in , is the scaling parameter, is the movement parameter, is a value that maintains numerical stability.
7. The method for facial expression recognition in a driving scenario according to claim 1, characterized in that: The first-stage attention mechanism scales the values in the position feature sequence, expressed as: in, is the attention mechanism, For the query, , for the key, , is the value, , , and are the elements of the query, the key, and the value, respectively, , , , , and The query elements , key element Sum value elements The weight matrix of is an element in the position feature sequence, yes The dimension of , is the element in the scaling feature sequence, with the superscript T Indicates transpose.
8. The method for facial expression recognition in a driving scenario according to claim 1, characterized in that: The global dynamic salient expression feature is expressed as: in, a is the global dynamic significant expression feature, T is the number of face images in the face image sequence, t is the label of the face image, for The elements in is the self-attention weight; in, MLP is a feedforward neural network with sigmoid function as activation function. u is the scaling feature sequence, for u The elements in .
Citation Information
Patent Citations
Pedestrian re-identification method based on domain-independent foreground feature learning
CN112464730A
Controllable face generation method based on facial micro-expression
CN114973375A
Training method and device of driver expression recognition model, medium and electronic equipment
CN116844204A
Residual normalization for improved neural network classifications
US20220405547A1
Cited By
Face feature extraction model training method, face feature extraction method and device
CN119810894A
