Lightweight driver facial expression recognition method based on multi-scale convolutional neural network

Abstract

The application discloses a kind of light driver facial expression recognition methods based on multi-scale convolutional neural network, which realizes face positioning and alignment by multi-task convolutional neural network;Then the enhanced facial image is transmitted into the improved multi-scale convolutional neural network to recognize human face expression;The present application is different from the traditional expression recognition method based on convolutional neural network, a novel multi-scale residual attention module is designed, and a light and efficient network is constructed, the driver's facial image is non-contact to identify the driver's expression state and take corresponding measures, realize safe and intelligent driving, and the method extracts facial image features from multiple scales on one hand, effectively improves the accuracy of facial expression recognition;On the other hand, light network is conducive to deployment in car intelligent cockpit domain controller.

Description

Technical Field

[0001] This invention relates to a driver facial expression recognition method, specifically a lightweight driver facial expression recognition method based on a multi-scale convolutional neural network, belonging to the field of machine vision technology. Background Technology

[0002] Facial expressions are an important form of human emotion and one of the bridges for human-computer interaction. In recent years, with the rapid development of computer vision technology and the popularization of smart cockpits in automobiles, the application of facial expression recognition to intelligent driving has enormous potential.

[0003] In existing technologies, such as the facial expression recognition method based on ROI-KNN convolutional neural network disclosed in CN114944005A, the deep convolutional neural network model constructed by combining regions of interest and the K nearest neighbor algorithm can improve the training effect of facial expression data in the recognition model, thereby reducing the problem of insufficient generalization ability of deep convolutional neural networks due to insufficient facial expression training data, and improving the efficiency of facial expression recognition. Although existing deep learning networks have laid a good foundation for facial expression recognition and can be applied to some situations to a certain extent, for scenarios such as intelligent driving of automobiles, on the one hand, network performance is related to human life safety, and the accuracy of expression recognition still needs to be further improved; on the other hand, the expression recognition model is deployed on the intelligent cockpit domain controller, which also requires a certain degree of lightweighting of the model. Summary of the Invention

[0004] The purpose of this invention is to provide a lightweight driver facial expression recognition method based on a multi-scale convolutional neural network in order to solve at least one of the above-mentioned technical problems.

[0005] This invention achieves the above objective through the following technical solution: a lightweight driver facial expression recognition method based on a multi-scale convolutional neural network, comprising the following steps:

[0006] S1: Build and train a multi-task convolutional neural network (MTCNN) for detecting facial regions;

[0007] S2: Use MTCNN to crop and remove the background from the collected facial expression dataset to obtain a new facial expression dataset, which is then divided into a training set and a test set in a ratio of 8:2.

[0008] S3: Replace the basic convolutional module of the VGG-16 network with the designed multi-scale residual attention module to construct a driver expression recognition network based on a multi-scale convolutional neural network;

[0009] S4: Use the training set obtained in step S2 to train the driver facial expression recognition network constructed in step S3. After each training, use the test set to test the model performance.

[0010] S5: The network trained in S4 is fine-tuned on a self-made driver dataset and deployed on the vehicle smart cockpit domain controller.

[0011] As a further aspect of the present invention: In step S1, training the MTCNN network specifically includes the following steps:

[0012] S11: Select a certain number of images from the public dataset WIDER_FACE as the training set for MTCNN, and label the four points that make up the face bounding box;

[0013] S12: Obtain training sets of images with three different resolutions, namely 12×12, 24×24 and 48×48, from the WIDER_FACE training set;

[0014] S13: Train three-layer networks P-Net, R-Net, and O-Net using training sets of three different resolutions.

[0015] As a further aspect of the present invention: in S13, a loss function is used for training, and the loss function includes the following parts:

[0016] 1) Face detection uses the cross-entropy loss function:

[0017] ;

[0018] in, The probability of a face appearing. This is the true label for the area, indicating whether a face exists; 1 for yes and 0 for no.

[0019] 2) Face bounding box regression uses the sum of squares loss function:

[0020] ;

[0021] in, These are the bounding box coordinates predicted by the network. These are the actual bounding box coordinates. A face bounding box is represented by 4 coordinate points.

[0022] 3) The two loss functions above are weighted and summed to form the final training objective function:

[0023] min ;

[0024] Where N is the number of training samples. This indicates the importance of the task. For sample labels, The loss function is as described above. In P-Net and R-Net, In O-Net, .

[0025] As a further aspect of the present invention: In step S2, the collected facial expression dataset refers to the publicly available facial expression dataset, namely the FER2013 dataset. Expressions irrelevant to the driving scenario, including disgust and contempt, are removed from the original dataset, while expressions relevant to the driving scenario, including anger, fear, happiness, joy, and calmness, are retained, resulting in a transitional dataset.

[0026] As a further aspect of the present invention: In step S3, the driver expression recognition network based on multi-scale convolutional neural networks mainly includes: a traditional convolutional module, a multi-scale residual attention module, a max pooling layer, a global average pooling layer, a Dropout layer, and a Softmax layer.

[0027] As a further aspect of the present invention: the multi-scale residual attention module comprises three residual attention units, each of which operates as follows:

[0028] First, the input feature map is processed through a 1×1 convolution to adjust the number of channels. Then, it passes through the CBAM attention mechanism to output a feature map with channel and spatial dimension attention weights. This output map is then added element-wise to the feature map output from the feature extraction branch. The feature map F' obtained after passing through the multi-scale residual attention unit can be represented as:

[0029] ;

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] in, Given the input feature map, F1, F2, F3, F4, and F5 are the feature maps obtained after passing through five branches, respectively. for Convolution kernel size; For bias, , , , , This is the bias for the corresponding convolutional layer.

[0036] As a further aspect of the present invention: in step S4, the network parameters set during the training process are as follows:

[0037] The data augmentation method is random horizontal flipping, the batch size is 64, the activation function is ReLU, the optimization method is stochastic gradient descent, the loss function is cross-entropy, the total number of training epochs is 150, and the learning rate decays every 5 epochs starting from epoch 50. The learning rate 'a' is calculated using the following formula:

[0038] ;

[0039] In the formula, a0 is the initial learning rate; β is the decay exponent; n is the current training round number; N is the starting decay round number; c is the decay interval; take a0=0.01, β=0.9, N=50, c=5.

[0040] As a further aspect of the present invention: in step S5, the fine-tuning strategy for the trained network model is to use a transfer training method that freezes the convolutional layers and retrains the fully connected layers, and to fine-tune the model on a self-made driver expression dataset.

[0041] As a further aspect of the present invention: the self-made driver expression dataset is a small dataset formed by collecting images of different facial expressions of car drivers.

[0042] The beneficial effects of this invention are as follows: The multi-scale neural network with an attention mechanism constructed in this method achieves face localization and alignment through a multi-task convolutional neural network (MTCNN); then, the enhanced facial image is fed into the improved multi-scale convolutional neural network for facial expression recognition, which can achieve higher recognition accuracy than traditional convolutional networks, thus safeguarding the personal safety of drivers; on the one hand, facial image features are extracted from multiple scales, effectively improving the accuracy of facial expression recognition; on the other hand, the lightweight network is conducive to deployment on automotive smart cockpit domain controllers. While achieving high accuracy, the model parameters are only 28M, which can be deployed on automotive smart cockpit domain controllers; by collecting the driver's personal facial images and fine-tuning the model parameters, the driver's facial expression status can be identified non-contactly based on the driver's facial images and corresponding measures can be taken, which can effectively improve the accuracy of facial expression recognition for drivers and bring a better user driving experience. Attached Figure Description

[0043] Figure 1 This is a system flowchart of the present invention;

[0044] Figure 2 This is a flowchart of the MTCNN algorithm of the present invention;

[0045] Figure 3This is a flowchart of the multi-scale neural network model training process of the present invention;

[0046] Figure 4 This is a schematic diagram of the multi-scale residual attention unit of the present invention;

[0047] Figure 5 This is a schematic diagram of the multi-scale neural network model structure of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0049] Example 1, as Figures 1 to 5 As shown, a lightweight driver facial expression recognition method based on a multi-scale convolutional neural network includes the following steps:

[0050] S1: Build and train a multi-task convolutional neural network (MTCNN) for detecting facial regions;

[0051] S2: Use MTCNN to crop and remove the background from the collected facial expression dataset to obtain a new facial expression dataset, which is then divided into a training set and a test set in a ratio of 8:2.

[0052] S3: Replace the basic convolutional module of the VGG-16 network with the designed multi-scale residual attention module to construct a driver expression recognition network based on a multi-scale convolutional neural network;

[0053] S4: Use the training set obtained in step S2 to train the driver facial expression recognition network constructed in step S3. After each training, use the test set to test the model performance.

[0054] S5: The network trained in S4 is fine-tuned on a self-made driver dataset and deployed on the vehicle smart cockpit domain controller.

[0055] Example 2, in addition to all the technical features included in Example 1, also includes:

[0056] Step S1, training the MTCNN network specifically includes the following steps:

[0057] S11: Select a certain number of images from the public dataset WIDER_FACE as the training set for MTCNN, and label the four points that make up the face bounding box;

[0058] S12: Obtain training sets of images with three different resolutions, namely 12×12, 24×24 and 48×48, from the WIDER_FACE training set;

[0059] S13: Train three-layer networks P-Net, R-Net, and O-Net using training sets of three different resolutions.

[0060] In S13, a loss function is used for training, and the loss function includes the following parts:

[0061] 1) Face detection uses the cross-entropy loss function:

[0062] ;

[0063] in, The probability of a face appearing. This is the true label for the area, indicating whether a face exists; 1 for yes and 0 for no.

[0064] 2) Face bounding box regression uses the sum of squares loss function:

[0065] ;

[0066] in, These are the bounding box coordinates predicted by the network. These are the actual bounding box coordinates. A face bounding box is represented by 4 coordinate points.

[0067] 3) The two loss functions above are weighted and summed to form the final training objective function:

[0068] min ;

[0069] Where N is the number of training samples. This indicates the importance of the task. For sample labels, The loss function is as described above. In P-Net and R-Net, In O-Net, .

[0070] In step S2, the collected facial expression dataset refers to the publicly available facial expression dataset, namely the FER2013 dataset. Expressions irrelevant to the driving scenario, including disgust and contempt, are removed from the original dataset, while expressions relevant to the driving scenario, including anger, fear, happiness, joy, and calm, are retained, resulting in a transitional dataset.

[0071] Example 3, in addition to all the technical features included in Example 1, also includes:

[0072] In step S3, the driver expression recognition network based on multi-scale convolutional neural networks mainly includes: a traditional convolutional module, a multi-scale residual attention module, a max pooling layer, a global average pooling layer, a Dropout layer, and a Softmax layer.

[0073] The multi-scale residual attention module contains three residual attention units, and the working principle of each unit is as follows:

[0074] First, the input feature map is processed through a 1×1 convolution to adjust the number of channels. Then, it passes through the CBAM attention mechanism to output a feature map with channel and spatial dimension attention weights. This output map is then added element-wise to the feature map output from the feature extraction branch. The feature map F' obtained after passing through the multi-scale residual attention unit can be represented as:

[0075] ;

[0076] ;

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] in, Given the input feature map, F1, F2, F3, F4, and F5 are the feature maps obtained after passing through five branches, respectively. for Convolution kernel size; For bias, , , , , This is the bias for the corresponding convolutional layer.

[0082] Example 4, in addition to all the technical features included in Example 1, also includes:

[0083] In step S4, the network parameters set during training are as follows:

[0084] The data augmentation method is random horizontal flipping, the batch size is 64, the activation function is ReLU, the optimization method is stochastic gradient descent, the loss function is cross-entropy, the total number of training epochs is 150, and the learning rate decays every 5 epochs starting from epoch 50. The learning rate 'a' is calculated using the following formula:

[0085] ;

[0086] In the formula, a0 is the initial learning rate; β is the decay exponent; n is the current training round number; N is the starting decay round number; c is the decay interval; take a0=0.01, β=0.9, N=50, c=5.

[0087] In step S5, the fine-tuning strategy for the trained network model is to use a transfer training method that freezes the convolutional layers and retrains the fully connected layers, and then fine-tunes the model on a self-made driver expression dataset.

[0088] A self-made driver facial expression dataset is a small dataset created by collecting images of different facial expressions from car drivers.

[0089] Example 5: A lightweight driver facial expression recognition method based on a multi-scale convolutional neural network, the method comprising the following steps:

[0090] Step A: First, turn on the vehicle camera to obtain real-time footage of the driver driving;

[0091] Step B: Preprocess the image by using histogram equalization to enhance the image.

[0092] Step C: Input the processed image into the trained multi-task convolutional neural network (MTCNN). If MTCNN does not detect a face, continue to acquire images; otherwise, extract the face region from the image and input it into the multi-scale convolutional neural network for expression recognition.

[0093] Step D: If a happy or sad expression is detected, the in-vehicle voice assistant will perform human-computer interaction based on the driver's current state. For example, if the driver is detected to be sad, it will play soothing music.

[0094] Step E: If an angry or frightened expression is detected, and considering road rage and the possibility of an emergency ahead, automatic measures will be taken to slow down or reduce speed.

[0095] Working principle: This system non-contactly identifies the driver's facial expressions and takes appropriate measures based on facial images to achieve safe and intelligent driving. First, face localization and alignment are achieved using a multi-task convolutional neural network (MTCNN). Then, the enhanced facial image is fed into an improved multi-scale convolutional neural network for facial expression recognition.

[0096] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0097] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A lightweight driver facial expression recognition method based on multi-scale convolutional neural networks, characterized in that, The lightweight driver facial expression recognition method includes: S1: Build and train a multi-task convolutional neural network for detecting facial regions; S2: Use a multi-task convolutional neural network to crop and remove the background from the collected facial expression dataset to obtain a new facial expression dataset, which is then divided into a training set and a test set in a ratio of 8:

2. S3: Replace the basic convolutional module of the VGG-16 network with the designed multi-scale residual attention module to construct a driver expression recognition network based on a multi-scale convolutional neural network; In step S3, the driver expression recognition network based on multi-scale convolutional neural network mainly includes: a traditional convolutional module, a multi-scale residual attention module, a max pooling layer, a global average pooling layer, a Dropout layer, and a Softmax layer. The multi-scale residual attention module contains three residual attention units, and the working principle of each unit is as follows: First, the input feature map is processed through a 1×1 convolution to adjust the number of channels. Then, it passes through the CBAM attention mechanism to output a feature map with channel and spatial dimension attention weights. This output map is then added element-wise to the feature map output from the feature extraction branch. The feature map F' obtained after passing through the multi-scale residual attention unit can be represented as: ； ； ； ； ； ； in, Given the input feature map, F1, F2, F3, F4, and F5 are the feature maps obtained after passing through five branches, respectively. for Convolution kernel size; For bias, , , , , This is the bias for the corresponding convolutional layer; S4: Use the training set obtained in step S2 to train the driver facial expression recognition network constructed in step S3. After each training, use the test set to test the model performance. S5: The network trained in S4 is fine-tuned on a self-made driver dataset and deployed on the vehicle smart cockpit domain controller.

2. The lightweight driver facial expression recognition method according to claim 1, characterized in that: In step S1, training the MTCNN network specifically includes the following steps: S11: Select a certain number of images from the public dataset WIDER_FACE as the training set for MTCNN, and label the four points that make up the face bounding box; S12: Obtain training sets of images with three different resolutions, namely 12×12, 24×24 and 48×48, from the WIDER_FACE training set; S13: Train three-layer networks P-Net, R-Net, and O-Net using training sets of three different resolutions.

3. The lightweight driver facial expression recognition method according to claim 2, characterized in that: In step S13, a loss function is used for training, and the loss function includes the following parts: 1) Face detection uses the cross-entropy loss function: ； in, The probability of a face appearing. The region is the true label, i.e. whether a face exists; 1 for yes and 0 for no. 2) Face bounding box regression uses the sum of squares loss function: ； in, These are the bounding box coordinates predicted by the network. These are the actual bounding box coordinates; a face bounding box is represented by 4 coordinate points. 3) The two loss functions above are weighted and summed to form the final training objective function: min ; Where N is the training sample size. This indicates the importance of the task. For sample labels, For the loss function above, in P-Net and R-Net, In O-Net, .

4. The lightweight driver facial expression recognition method according to claim 1, characterized in that: In step S2, the collected facial expression dataset refers to the publicly available facial expression dataset, namely the FER2013 dataset. Expressions irrelevant to the driving scenario, including disgust and contempt, are removed from the original dataset, while expressions relevant to the driving scenario, including anger, fear, happiness, and calmness, are retained to obtain a transitional dataset.

5. The lightweight driver facial expression recognition method according to claim 1, characterized in that: In step S4, the network parameters set during training are as follows: The data augmentation method is random horizontal flipping, the batch size is 64, the activation function is ReLU, the optimization method is stochastic gradient descent, the cross-entropy loss function is used, the total number of training epochs is 150, and the learning rate decays every 5 epochs starting from epoch 50. The learning rate 'a' is calculated using the following formula: ； In the formula, a0 is the initial learning rate; β is the decay exponent; n is the current training round number; N is the starting decay round number; c is the decay interval; take a0=0.01, β=0.9, N=50, c=5.

6. The lightweight driver facial expression recognition method according to claim 1, characterized in that: In step S5, the fine-tuning strategy for the trained network model is to use a transfer training method that freezes the convolutional layers and retrains the fully connected layers, and then fine-tunes the model on a self-made driver expression dataset.

7. The lightweight driver facial expression recognition method according to claim 6, characterized in that: The self-made driver expression dataset is a small dataset formed by collecting images of different facial expressions of car drivers.

Images