A driver expression recognition method based on beluga whale optimization algorithm and transfer learning

Abstract

The application discloses a driver expression recognition method based on a beluga whale optimization algorithm and migration learning, and comprises the following steps: acquiring a driver expression dataset; performing image preprocessing on the driver expression dataset; constructing a driver expression recognition model; pre-training the expression recognition model; fine-tuning the driver expression recognition network after optimization of hyperparameters; and optimizing the hyperparameters of the expression recognition model by using the beluga whale optimization algorithm. The application can obtain the hyperparameter values suitable for driver expression recognition in a relatively short time, effectively overcomes the shortcomings of the convolutional neural network, such as the decrease of accuracy, slow training, and even non-convergence of the model, caused by improper selection of hyperparameters, and further improves the performance of the driver expression recognition model, thereby providing protection for safe and intelligent driving.

Description

TECHNICAL FIELD

[0001] The application relates to a driver expression recognition method, in particular to a driver expression recognition method based on a beluga whale optimization algorithm and transfer learning, and belongs to the technical field of deep learning of driver expression recognition of an automobile intelligent cabin. BACKGROUND

[0002] Facial expression is an important expression form of human emotion and is one of the bridges for human-computer interaction. In recent years, with the rapid development of computer vision technology and the popularity of automobile intelligent cabins, there is great potential for applying face expression recognition to intelligent driving. A driver facial expression recognition system of an automobile intelligent cabin can observe the negative emotions of a driver, timely issue a reminder, and automatically plan a route and adjust a speed, so that the purpose of safe and intelligent driving is achieved.

[0003] In the prior art, a driver expression recognition method based on a ViT and StarGAN is disclosed in CN114005154A. The method is a non-contact driver emotion recognition method based on a face image. Driver face image size is reduced and features are enhanced through face positioning and cutting based on a haar characteristic, and then a new data enhancement strategy is used to generate a driver facial expression image, that is, a StarGAN network is built and trained to obtain a driver facial expression expansion dataset. A ViT expression recognition network based on a multi-head self-attention mechanism is constructed. On one hand, the network training attention can gradually focus on important expression features, and on the other hand, more long-distance information of the expression image can be focused on. At the same time, the problem of insufficient data in the dataset is solved, and a higher recognition accuracy is achieved. The existing deep learning network lays a good foundation for facial expression recognition and can be applied to some occasions to a certain extent. However, for automobile intelligent driving and other scenarios, network performance is related to personnel safety, and the accuracy of the expression recognition model still needs to be further improved. The setting of the hyperparameters (learning rate, decay rate, regularization term coefficient and other parameters) of the deep learning model has a profound influence on the performance of the model. Improper setting of the hyperparameters will cause a series of problems in model training, such as slow learning speed, easy to fall into local optimal solution, time-consuming for manual setting of experience value and the like. SUMMARY

[0004] The purpose of the present application is to provide a driver expression recognition method based on a beluga whale optimization algorithm and transfer learning to solve at least one of the above technical problems. The hyperparameters of the driver expression recognition network are optimized by using the beluga whale optimization algorithm, the hyperparameter values suitable for driver expression recognition can be obtained in a shorter time, and the accuracy of driver expression recognition is improved, thereby ensuring safe and intelligent driving.

[0005] The application achieves the above-mentioned purpose through the following technical scheme: a driver expression recognition method based on a white whale optimization algorithm and a migration learning, comprising the following steps:

[0006] S1: obtaining a driver expression dataset;

[0007] S2: performing image preprocessing and data enhancement on the driver expression dataset to obtain processed driver expression image data;

[0008] S3: constructing a driver expression recognition network based on a multi-scale convolutional neural network;

[0009] S4: pre-training an expression recognition model by using the dataset 1;

[0010] S5: optimizing the hyperparameters of the driver expression recognition network by using the BWO algorithm, and performing model fine-tuning on the dataset 2.

[0011] As a further scheme of the application: in step S1, the obtained driver expression dataset comprises a natural scene-based facial expression dataset and a self-made small driver expression dataset.

[0012] As a further scheme of the application: in step S2, the steps of image preprocessing and data enhancement of the expression dataset are as follows:

[0013] S31: performing histogram equalization on dark light images in the dataset;

[0014] S32: performing normalization processing on the driver expression data in the dataset;

[0015] S33: performing random flipping and cropping operations on the images in the dataset to expand the dataset.

[0016] As a further scheme of the application: in step S3, the driver expression recognition network based on the multi-scale convolutional neural network mainly comprises a traditional convolution module, a multi-scale residual attention module, a max-pooling layer, a global average pooling layer, a Dropout layer and a Softmax layer.

[0017] As a further scheme of the application: the multi-scale residual attention module comprises three residual attention units, and the working principle of each unit is as follows:

[0018] First, the input feature map is adjusted in channel number through 1x1 convolution, and the feature map with channel and spatial dimension attention weights is output through the CBAM attention mechanism, and then the corresponding elements of the output feature map of the feature extraction branch are added, and the feature map F' obtained through the multi-scale residual attention unit can be expressed as:

[0019] F1 = ReLU(conv(F, k 1×1 )+b1)

[0020] F2 = ReLU(Dw(ReLU(conv(F, k 1×1 )+b2), k 3×3 ))

[0021] F3 = ReLU(Dw(ReLU(Dw(ReLU(conv(F, k 1×1 )+b3), k 3×3 )), k 3×3 ))

[0022] F4 = ReLU(conv(MaxPool(F, k 3×3 ), k 1×1 )+b4)

[0023] F5 = CBAM(ReLU(conv(F, k 1×1 )+b5)

[0024] F' = Add(Concat(F1, F2, F3, F4), F5)

[0025] Wherein, F1, F2, F3, F4 and F5 are feature maps obtained after five branches respectively; k i×i is an i x i size convolution kernel; b i is a bias.

[0026] As a further scheme of the application: in step S4, the network parameters set in the model pre-training process are respectively:

[0027] The batch size is set to 32, the activation function is ReLu function, the optimization method of stochastic gradient descent (SGD) is selected, the cross entropy loss function is used, the total number of training rounds is 300 times, the learning rate is decayed every 10 rounds from the 80th round, and the learning rate a is calculated as follows:

[0028]

[0029] In the formula, a0 is the initial learning rate, β is the decay index, n is the current training round, N is the start decay round, c is the decay interval, a0 = 0.01, β = 0.85, N = 80, and c = 10.

[0030] As a further scheme of the application: in step S5, a set of feasible solutions of the hyperparameters to be optimized in the expression recognition network is an individual, and the method for optimizing the expression recognition model hyperparameters by using the beluga whale optimization algorithm is:

[0031] S71: initialize the population number N and the maximum iteration number T, and set the hyperparameter optimization range;

[0032] S72: input the training data set into the expression recognition network, use the current hyperparameters for model training, and obtain the predicted driver expression classification result;

[0033] S73: calculate the fitness value corresponding to the current individual according to the predicted driver expression classification result;

[0034] S74: the algorithm executes three stages of exploration, development and whale falling, and searches for new individuals;

[0035] S75: according to the updated individual, the driver expression classification result is predicted again, the fitness value corresponding to the updated individual is calculated, and the optimal solution is updated;

[0036] S76: when the optimal solution is unchanged for 5 consecutive iterations or meets the iteration termination condition, the optimal hyperparameters are output;

[0037] S77: obtain the expression recognition network optimized by the optimal hyperparameters;

[0038] Among them, the model hyperparameters include the initial learning rate, the decay rate, the regularization term coefficient and the inactivation rate of the Dropout layer in the driver expression recognition network.

[0039] As a further scheme of the present application: the fitness function of the white whale optimization algorithm is the accuracy of expression classification, and the calculation formula is as follows:

[0040]

[0041] Among them, Accuracy represents the classification accuracy of the driver expression image, TP represents the number of positive samples predicted by the expression recognition network and actually positive, FP represents the number of positive samples predicted by the network but actually negative, TN represents the number of negative samples predicted by the network and actually negative, and FN represents the number of negative samples predicted by the network but actually positive.

[0042] As a further scheme of the present application: in step S71, the population number N is set to 10, the maximum iteration number T is set to 30, the initial learning rate range is set to [0.001, 0.1], the decay exponent is set to [0.85, 0.99], the regularization term coefficient is set to [0.0005, 0.01], and the inactivation rate is set to [0.3, 0.6].

[0043] As a further scheme of the present application: in step S74, in the exploration stage, the individual update formula is:

[0044]

[0045] wherein t is the current iteration number, is the new position of the i-th beluga whale in the j-th dimension, P j (j=1, 2, …, d) is randomly selected from the d dimensions, is the current position of the i-th beluga whale in the j-th dimension, and is the current position of the i-th beluga whale and the r-th beluga whale (r represents a randomly selected beluga whale), rand1, rand2 and p are random numbers ranging from (0.1) to enhance the random operator in the exploration phase, sin(2πrand2) and cos(2πrand2) are used to balance the random numbers between the fins.

[0046] The beneficial effects of the present application are: using the beluga whale optimization algorithm to optimize the hyperparameters of the driver expression recognition network can further improve the accuracy and reliability of the driver expression recognition network.

[0047] The method of the present application adopts a transfer learning strategy to optimize hyperparameters when fine-tuning the expression recognition pre-trained model, and the method can efficiently improve the accuracy of the final model in a shorter time. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is the step flowchart of the present application;

[0049] Figure 2 is the logic block diagram of the present application;

[0050] Figure 3 is the structure diagram of the driver expression recognition network in the embodiment of the present application;

[0051] Figure 4 is the structure diagram of the multi-scale residual attention unit in the embodiment of the present application;

[0052] Figure 5 is the flowchart of optimizing the hyperparameters of the driver expression recognition network using the BWO algorithm in the embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0054] Embodiment one

[0055] As Figures 1 to 5As shown, a driver expression recognition method based on white whale optimization algorithm and transfer learning includes the following steps:

[0056] S1: Obtain a driver expression dataset;

[0057] S2: Image preprocessing and data enhancement are performed on the driver expression dataset to obtain processed driver expression image data;

[0058] S3: Construct a driver expression recognition network based on a multi-scale convolutional neural network;

[0059] S4: Use dataset 1 to pre-train the expression recognition model;

[0060] S5: Use the BWO algorithm to optimize the hyperparameters of the driver expression recognition network, and fine-tune the model on dataset 2.

[0061] Embodiment two

[0062] In addition to including all the technical features in Embodiment One, this embodiment also includes:

[0063] In step S1, the obtained driver expression dataset includes a natural scene face expression dataset FER2013 (dataset 1) and a self-made small driver expression dataset (dataset 2);

[0064] In step S2, the steps of image preprocessing and data enhancement of the expression dataset are:

[0065] S31: Histogram equalization is performed on the dark light images in the dataset;

[0066] S32: The driver expression data in the dataset is normalized;

[0067] S33: Randomly flip and crop the images in the dataset to expand the dataset.

[0068] Embodiment three

[0069] In addition to including all the technical features in Embodiment One, this embodiment also includes:

[0070] In step S3, the driver expression recognition network based on the multi-scale convolutional neural network mainly includes: a traditional convolution module, a multi-scale residual attention module, a max-pooling layer, a global average pooling layer, a Dropout layer, and a Softmax layer.

[0071] The multi-scale residual attention module contains 3 residual attention units, and each unit works as follows:

[0072] First, the input feature map is adjusted in channel number by 1x1 convolution, passes through the CBAM attention mechanism, and outputs a feature map with channel and spatial dimension attention weights. Then, the output feature map of the feature extraction branch is added to the corresponding elements, and the feature map F' obtained by the multi-scale residual attention unit is represented as:

[0073] F1 = ReLU(conv(F, k 1×1 )+b1)

[0074] F2 = ReLU(Dw(ReLU(conv(F, k 1×1 )+b2), k 3×3 ))

[0075] F3 = ReLU(Dw(ReLU(Dw(ReLU(conv(F, k 1×1 )+b3), k 3×3 )), k 3×3 ))

[0076] F4 = ReLU(conv(MaxPool(F, k 3×3 ), k 1×1 )+b4)

[0077] F5 = CBAM(ReLU(conv(F, k 1×1 )+b5))

[0078] F' = Add(Concat(F1, F2, F3, F4), F5)

[0079] Where F1, F2, F3, F4 and F5 are feature maps obtained after five branches; k i×i is an i x i size convolution kernel; b i is the bias.

[0080] Example Four

[0081] In addition to including all the technical features in Example One, this example also includes:

[0082] In step S4, the network parameters set during the model pre-training process are as follows:

[0083] The batch size is set to 32, the activation function is ReLu function, the optimization method is selected as stochastic gradient descent (SGD), the cross-entropy loss function is used, the total number of training rounds is 300, the learning rate is decayed every 10 rounds from the 80th round, and the learning rate a is calculated as follows:

[0084]

[0085] In the formula, a0 is the initial learning rate; β is the decay index; n is the current training round; N is the starting decay round; c is the decay interval; take a0=0.01, β=0.85, N=80, c=10.

[0086] Embodiment five

[0087] In addition to including all the technical features in Embodiment One, this embodiment also includes:

[0088] In step S5, a set of feasible solutions of the hyperparameters to be optimized in the expression recognition network is regarded as an individual, and the method for optimizing the hyperparameters of the expression recognition model by using the beluga whale optimization algorithm is:

[0089] S71: initialize the population number N and the maximum iteration round T, and set the hyperparameter optimization range;

[0090] S72: input the training data set into the expression recognition network, use the current hyperparameters for model training, and obtain the predicted driver expression classification result;

[0091] S73: calculate the fitness value corresponding to the current individual according to the predicted driver expression classification result;

[0092] S74: the algorithm executes three stages of exploration, development and whale falling, and searches for new individuals;

[0093] S75: according to the updated individual, the driver expression classification result is predicted again, the fitness value corresponding to the updated individual is calculated, and the optimal solution is updated;

[0094] S76: when the optimal solution is unchanged for 5 consecutive iterations or meets the iteration termination condition, output the optimal hyperparameters;

[0095] S77: obtain the expression recognition network after hyperparameter optimization according to the optimal hyperparameters;

[0096] Among them, the model hyperparameters include the initial learning rate, the decay rate, the regularization term coefficient and the dropout layer inactivation rate in the driver expression recognition network.

[0097] The fitness function of the beluga whale optimization algorithm is the accuracy of expression classification, and the calculation formula is as follows:

[0098]

[0099] Among them, Accuracy represents the classification accuracy of the driver expression image, TP represents the number of positive samples predicted by the expression recognition network and actually positive samples, FP represents the number of positive samples predicted by the network but actually negative samples, TN represents the number of negative samples predicted by the network and actually negative samples, and FN represents the number of negative samples predicted by the network but actually positive samples.

[0100] In step S71, the population number N is set to 10, the maximum iteration number T is set to 30, the initial learning rate range is set to [0.001, 0.1], the decay exponent is set to [0.85, 0.99], the regularization coefficient is set to [0.0005, 0.01], and the inactivation rate is set to [0.3, 0.6].

[0101] In step S74, in the exploration phase, the individual update formula is:

[0102]

[0103] where t is the current iteration number, is the new position of the ith white whale in the jth dimension, P j (j = 1, 2, …, d) is randomly selected from d dimensions, is the current position of the ith white whale in the jth dimension, and is the current position of the ith white whale and the rth white whale (r represents a randomly selected white whale), rand1, rand2 and p are random numbers ranging from (0.1) to enhance the random operator in the exploration phase. sin(2πrand2) and cos(2πrand2) are used to balance the random numbers between the fins.

[0104] First, a driver expression dataset for network training is obtained, including a natural environment expression dataset and a self-made small driver expression dataset; then the pictures in the dataset are preprocessed, image enhancement is realized through histogram equalization, and the dataset is expanded through flipping and random cropping operations; then a driver expression recognition network model is constructed; then the processed dataset 1 is used to pre-train the driver expression recognition network; then the white whale optimization algorithm is used to optimize the hyperparameters of the driver expression recognition network, and a transfer learning strategy is adopted to fine-tune the expression recognition network after parameter optimization using the self-made small driver expression dataset; finally, a trained driver expression recognition network model is obtained, which is deployed in the car intelligent cockpit domain controller.

[0105] It will be obvious to a person skilled in the art that the application is not limited to the details of the above-described exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the application. Therefore, the embodiments should be considered as exemplary and non-limiting, and the scope of the application is defined by the appended claims rather than the above description, and all changes falling within the meaning and range of equivalents of the essential elements of the claims are intended to be encompassed by the application. Any reference signs in the claims should not be considered as limiting the claims involved.

[0106] Furthermore, it should be understood that although the description is made according to the embodiments, not every embodiment contains only one independent technical solution, and the description is made in this way only for the sake of clarity, and the person skilled in the art should consider the description as a whole, and the technical solutions in each embodiment can also be combined appropriately to form other embodiments that the person skilled in the art can understand.

Claims

1. A driver expression recognition method based on beluga whale optimization algorithm and transfer learning, characterized in that: The driver expression recognition method comprises the following steps: S1: obtaining a driver expression dataset; S2: performing image preprocessing and data enhancement on the driver expression dataset to obtain processed driver expression image data; S3: constructing a driver expression recognition network based on a multi-scale convolutional neural network; S4: pre-training the expression recognition model using dataset 1; S5: optimizing the hyperparameters of the driver expression recognition network using the white whale optimization algorithm and fine-tuning the model on dataset 2; In step S5, a set of feasible solutions of the hyperparameters to be optimized in the expression recognition network is an individual, and the method for optimizing the hyperparameters of the expression recognition model using the white whale optimization algorithm is as follows: S71: initialize the population size N and the maximum number of iterations T, and set the hyperparameter optimization range; S72: input the training dataset into the expression recognition network, use the current hyperparameters to train the model, and obtain the predicted driver expression classification result; S73: calculate the fitness value corresponding to the current individual according to the predicted driver expression classification result; S74: the algorithm executes three stages of exploration, development and whale falling, and searches for new individuals; S75: according to the updated individual, the driver expression classification result is predicted again, the fitness value corresponding to the updated individual is calculated, and the optimal solution is updated; S76: when the optimal solution is unchanged for 5 consecutive iterations or meets the iteration termination condition, the optimal hyperparameters are output; S77: obtain the expression recognition network optimized by the optimal hyperparameters; The model hyperparameters include the initial learning rate, the decay rate, the regularization term coefficient and the dropout layer inactivation rate in the driver expression recognition network. In step S74, in the exploration stage, the individual update formula is: ； where t is the current iteration number, is the new position of the ith white whale in the jth dimension, P j (j = 1, 2, …, d) is randomly selected from d dimensions, is the current position of the ith white whale in the jth dimension, and is the current position of the ith white whale and the rth white whale, rand1, rand2 and p are random numbers ranging in (0.1) to enhance the random operator in the exploration phase, and is the random number between the fins, r represents the randomly selected white whale.

2. The driver expression recognition method according to claim 1, characterized by: In step S1, the obtained driver expression dataset includes a natural scene facial expression dataset and a self-made small driver expression dataset.

3. The driver expression recognition method of claim 1, characterized by: In step S2, the steps of image preprocessing and data enhancement of the expression dataset are as follows: S31: histogram equalization is performed on the dark light images in the dataset; S32: normalize the driver expression data in the dataset; S33: perform random flipping and cropping operations on the images in the dataset to expand the dataset.

4. The driver expression recognition method of claim 1, characterized by: In step S3, the driver expression recognition network based on the multi-scale convolutional neural network mainly includes a traditional convolution module, a multi-scale residual attention module, a max pooling layer, a global average pooling layer, a Dropout layer and a Softmax layer.

5. The driver expression recognition method according to claim 4, characterized in that: The multi-scale residual attention module contains 3 residual attention units, and the working principle of each unit is as follows: First, adjust the channel number of the input feature map through 1x1 convolution, pass it through the CBAM attention mechanism, output a feature map with channel and spatial dimension attention weights, and then add the corresponding elements with the output feature map of the feature extraction branch. The feature map F' obtained through the multi-scale residual attention unit can be expressed as: ； ； ； ； ； ； Wherein, F1, F2, F3, F4 and F5 are feature maps obtained after passing through five branches respectively; For A convolution kernel of a size; is a bias.

6. The driver expression recognition method of claim 1, characterized in that: In step S4, the network parameters set during model pre-training are as follows: The batch size is set to 32, the activation function is a ReLu function, the optimization method is a random gradient descent, the cross-entropy loss function is used, the total number of training rounds is 300, the learning rate is decayed every 10 rounds starting from the 80th round, and the learning rate a is calculated according to the following formula: ； In the formula, a0 is the initial learning rate, β is the decay index, n is the current training round, N is the start decay round, c is the decay interval, a0=0.01, β=0.85, N=80, and c=10.

7. The driver expression recognition method of claim 1, characterized in that: The fitness function of the white whale optimization algorithm is the accuracy of expression classification, and the calculation formula is as follows: ； Where Accuracy represents the classification accuracy of the driver's expression image, TP represents the number of positive samples predicted by the expression recognition network and actually positive, FP represents the number of positive samples predicted by the network but actually negative, TN represents the number of negative samples predicted by the network and actually negative, and FN represents the number of negative samples predicted by the network but actually positive.

8. The driver expression recognition method of claim 1, characterized by: In step S71, the population size N is set to 10, the maximum iteration round T is set to 30, the initial learning rate range is set to [0.001, 0.1], the decay index is set to [0.85, 0.99], the regularization term coefficient is set to [0.0005, 0.01], and the inactivation rate is set to [0.3, 0.6].

Images