Facial expression recognition method and device based on YOLOv8 deep learning

Abstract

The invention provides a face expression recognition method and device based on YOLOv8 deep learning, and the method comprises the steps: S1, collecting a public data set of various basic expressions, carrying out the marking of the public data set, and carrying out the standardization processing and noise filtering of the marked data, and obtaining a data set; s2, the network structure of the YOLOv8 is improved, and an improved YOLOv8 expression recognition model is obtained; s3, training and reasoning the improved YOLOv8 expression recognition model by adopting a training set and a test set; and S4, using the trained improved YOLOv8 expression recognition model to recognize the face surface in the to-be-recognized image or video. According to the method, multi-scale features from overall expressions to muscle micro-motion can be captured, the problem that weak features of micro-expressions are difficult to capture is solved, meanwhile, angle correction, size normalization and shielding compensation are automatically completed in combination with an adaptive geometric correction module, side face and shielding interference is eliminated, and the recognition efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of face recognition, in particular to a face expression recognition method and device. BACKGROUND

[0002] Face expression recognition technology is an important research direction in the field of computer vision, and is widely used in human-computer interaction, emotion analysis, intelligent education and intelligent security fields. Early technologies mainly rely on hand-designed features, such as ASM (Active Shape Models) and AAM (Active Appearance Model) algorithms, which extract facial geometric features (such as mouth corner position and eye shape) for expression classification. However, such methods are sensitive to changes in light, posture and occlusion, and cannot capture subtle expression changes. With the development of deep learning, models based on convolutional neural networks (CNN) such as VGG and ResNet have gradually become mainstream, significantly improving recognition accuracy through end-to-end learning.

[0003] In recent years, the target detection framework YOLO series (such as YOLOv5 and YOLOv7) has been introduced into the expression recognition task due to its high efficiency, but it still has deficiencies in expression feature extraction and adaptability to complex scenes. For example: 1) Difficulty in capturing subtle features of micro-expressions (such as brief frowns and slight mouth movements); 2) Limited ability to distinguish rotated, scaled or partially occluded expression images, multi-scale expressions, low-contrast or non-uniform lighting conditions. SUMMARY

[0004] To overcome the above deficiencies, the present application provides a face expression recognition method and device based on YOLOv8 deep learning, which can capture multi-scale features from overall expressions to muscle micro-movements, solve the problem of difficulty in capturing subtle features of micro-expressions, and automatically complete angle correction, size normalization and occlusion compensation with an adaptive geometric correction module, eliminating side face and occlusion interference and improving recognition efficiency.

[0005] The technical scheme provided by the present application is as follows:

[0006] On the one hand, the present application provides a face expression recognition method based on YOLOv8 deep learning, comprising:

[0007] S1 collects various basic expression public datasets and labels them, and standardizes and filters noise to obtain a dataset, which includes a training set and a test set;

[0008] S2 improves the network structure of YOLOv8 to obtain an improved YOLOv8 expression recognition model;

[0009] The improved YOLOv8 expression recognition model comprises:

[0010] An adaptive geometric correction module is configured to automatically correct a face in an image to a front view angle.

[0011] A multi-scale dynamic SPPF module is configured to extract multi-scale features of overall expressions to muscle micro-motions of a data set layer by layer through multi-level pooling.

[0012] A feature fusion module is configured to perform channel weighted fusion on the multi-scale features based on an attention mechanism.

[0013] A feature enhancement module is configured to dynamically enhance details of the multi-scale features through depth separable convolution.

[0014] A time-series dynamic analysis module replaces a classification head in the YOLOv8 network, captures expression change trends, and smooths output results.

[0015] S3 trains and reasons the improved YOLOv8 expression recognition model using the training set and the test set obtained in S1.

[0016] S4 identifies a face surface in an image or a video to be recognized using the trained improved YOLOv8 expression recognition model.

[0017] In another aspect, the present application provides a face expression recognition device based on YOLOv8 deep learning, comprising:

[0018] A data acquisition module is configured to acquire public data sets of various basic expressions, label the data sets, and perform standardization processing and noise filtering on the labeled data to obtain data sets, wherein the data sets comprise a training set and a test set.

[0019] A network structure improvement module is configured to improve the network structure of YOLOv8 to obtain an improved YOLOv8 expression recognition model.

[0020] A model training module is configured to train and reason the improved YOLOv8 expression recognition model improved by the network structure improvement module using the training set and the test set obtained by the data acquisition module.

[0021] A face recognition module is configured to identify a face surface in an image or a video to be recognized using the improved YOLOv8 expression recognition model trained by the model training module.

[0022] The improved YOLOv8 expression recognition model comprises:

[0023] An adaptive geometric correction module is configured to automatically correct a face in an image to a front view angle.

[0024] a multi-scale dynamic SPPF module for extracting multi-scale features of overall expression to muscle micro-motion of the data set layer by layer through multi-level pooling;

[0025] a feature fusion module for channel weighted fusion of the multi-scale features based on an attention mechanism;

[0026] a feature enhancement module for dynamically enhancing details of the multi-scale features through depth separable convolution;

[0027] a time series dynamic analysis module replacing a classification head in the YOLOv8 network to capture expression change trends and smooth output results.

[0028] The YOLOv8 deep learning-based face facial expression recognition method and device provided by the application can at least bring the following beneficial effects:

[0029] 1) Through the multi-level pooling and depth separable convolution design of the multi-scale dynamic SPPF module, multi-scale features from overall expression to muscle micro-motion can be effectively captured, and the problem of difficult capture of micro-expression weak features is solved.

[0030] 2) By incorporating the attention mechanism and the multi-scale feature fusion module, the expression recognition accuracy under complex lighting conditions and head posture changes is significantly improved while maintaining real-time detection speed.

[0031] 3) The adaptive geometric correction module automatically completes angle correction, size normalization and occlusion compensation, eliminates side face and occlusion interference, and does not require manual adjustment of the camera angle or re-shooting, simplifying the operation process and reducing deployment costs. Through data enhancement technology, various scenes such as rotation, flipping, brightness adjustment, occlusion, and noise pollution are simulated, significantly improving the accuracy of the model in low light, occlusion and other complex environments, and adapting to complex environments without the need for additional hardware adjustments.

[0032] 4) The time series dynamic analysis technology uses an improved LSTM algorithm to cache continuous frame data and combines historical information to correct single-frame misjudgment, enhancing the stability of expression recognition.

[0033] 5) The feature equalization technology (regularization weight balance) dynamically balances the weights of basic features such as facial contours and semantic features such as emotional meaning during training, avoiding misjudgment of the model due to single training data and reducing the false positive rate in complex scenarios. Especially in night monitoring scenarios, the false positive rate is reduced.

[0034] 6) The combination of end-to-end training and transfer learning reduces development costs while ensuring high-precision real-time output, enabling the system to meet the real-time feedback needs of multiple scenarios such as classrooms and security, and overall achieving the technical effects of accurate micro-expression capture, strong adaptability to complex scenarios, high recognition stability, and low deployment cost.

[0035] Compared with the original YOLOv8 algorithm and traditional convolutional neural network models, the present application achieves an average accuracy improvement of 3.2% on the FER2013 and CK+ datasets, and an 18.7% improvement in the robustness of occluded expression recognition, providing a new technical solution for human-computer interaction systems, psychological state assessment, and intelligent security fields. Its lightweight design is particularly suitable for edge computing device deployment and has important value for promoting the practical application of affective computing technology. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A flowchart of a face facial expression recognition method based on YOLOv8 deep learning in the present application;

[0037] Figure 2 The SPPF algorithm based on the YOLOv8 spatial pyramid pooling layer in the multi-scale dynamic SPPF module of the present application;

[0038] Figure 3 A coordinate diagram of the chip layer, the fluorescent layer, and the driving layer in the present application;

[0039] Figure 4 A structure diagram of a Micro-LED thermal simulation device in the present application.

[0040] REFERENCE NUMERALS:

[0041] 10 - data acquisition module, 20 - network structure improvement module, 30 - model training module, 40 - face recognition module. DETAILED DESCRIPTION

[0042] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, specific embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings and other embodiments from these drawings without creative labor.

[0043] As shown in one embodiment of the present application, Figure 1 a face facial expression recognition method based on YOLOv8 deep learning, comprising:

[0044] S1 collects and labels the public dataset of various basic expressions, and standardizes and filters the noise of the labeled data to obtain a dataset, which includes a training set and a test set.

[0045] The basic expressions generally include 7 types of joy, anger, sadness, surprise, disgust, fear and neutrality. The public dataset can be any database including basic expression images, such as FER2013, CK+, etc. New self-built scene data (such as classroom, security monitoring, etc.) can also be added. The face video containing different light, posture and occlusion is collected by a camera, and the image is extracted by frame and the face area and expression category are manually labeled using a labeling tool. Then, standardization processing is performed.

[0046] During the standardization processing, all images are resized to 256x256 pixels to ensure the consistency of the input model size. At the same time, pixel normalization is performed to normalize the pixel value of each point to the interval [0, 1], and the formula is as follows:

[0047]

[0048] where x represents the pixel value of the current pixel point, Norm(x) represents the normalized value of the current pixel point, x min and x max represent the minimum and maximum values of the image pixel value, respectively.

[0049] After standardization processing, noise filtering is performed: taking 3x3 or 5x5 neighborhood as the center, the neighborhood is analyzed. The central pixel value is replaced by the middle value, and the salt and pepper noise is eliminated. The formula is as follows:

[0050] x clean = Median(Neighborhood(x))

[0051] where x clean represents the noise filtered value of the current pixel point, Neighborhood(x) represents the neighborhood set of the current pixel point, and Median represents the median function.

[0052] At the same time of noise filtering, the key contour such as eyebrow and mouth corner can also be identified by edge detection (such as Canny operator), and the noise in non-contour area is suppressed during filtering process, only the isolated noise points are removed, and the expression details are preserved.

[0053] It should be understood that in other embodiments, other ways can also be used to process the collected dataset, and the processing process can also be adjusted according to actual needs, which is not limited here.

[0054] S2 improves the network structure of YOLOv8 to obtain an improved YOLOv8 expression recognition model.

[0055] YOLOv8 is a target detection framework, and its standard structure mainly includes three parts: a backbone network, a neck network, and a detection head. In this step, based on the original YOLOv8 network, the following modules are improved to meet the requirements of micro-expression capture, pose robustness, and occlusion adaptability for expression recognition:

[0056] The improved YOLOv8 expression recognition model includes: an adaptive geometric correction module for automatically correcting the face in the image to a front view; a multi-scale dynamic SPPF module for extracting multi-scale features of the overall expression to muscle micro-movement of the dataset through multi-level pooling; a feature fusion module for channel weighting fusion of multi-scale features based on an attention mechanism; a feature enhancement module for dynamically enhancing the details of multi-scale features through depth separable convolution; and a time series dynamic analysis module for replacing the classification head in the YOLOv8 network to capture the trend of expression changes and smooth the output results. Specifically:

[0057] The adaptive geometric correction module (STN) automatically corrects the face tilt angle and occlusion area through a spatial transformation network and generates rotation and scaling parameters. In the improved YOLOv8 expression recognition model, this adaptive geometric correction module is located between the input image and the backbone network, corresponding to the preprocessing stage. The input image is corrected at the pixel level, and the process is as follows:

[0058] 1) Positioning problem (positioning network): the system first analyzes the input image to determine whether the face is tilted. For example, it detects that the face is tilted 15 degrees to the left or the chin is raised too high. This step is automatically completed by a small neural network without human intervention.

[0059] 2) Designing a "correction plan" (generating transformation parameters): based on the tilt angle, position offset, and other problems detected by the positioning network, the system automatically generates "correction parameters" such as "rotate 15 degrees to the left", "translate 10 pixels downward", "scale to 80% size", etc.

[0060] 3) Execute retouching (sampler deformation): the system adjusts the image at the pixel level according to the correction parameters. Intelligent adjustment includes:

[0061] 311) Rotation: straighten the tilted face.

[0062] 312) Scaling: if the face in the image is too small, it is enlarged to the standard size; if the face in the image is too large, it is appropriately reduced to the standard size.

[0063] 313) Stretch compensation: Rotation or scaling can cause face deformation (e.g. flattening), the system automatically fine-tunes it to keep the facial proportions natural.

[0064] The system has built-in intelligent calibration module to automatically complete the above adjustments, including:

[0065] 321) Angle correction: If the face is turned or tilted, the system will rotate it to the front view.

[0066] 322) Size normalization: Regardless of the distance of the face, it is uniformly scaled to the standard proportion.

[0067] 323) Occlusion compensation: The expression of the occluded area is inferred by surrounding features, such as when the forehead is covered by a hat, the eyes and mouth are combined to determine it.

[0068] In the implementation process, a lightweight CNN (3 convolution layers + fully connected layer) is used. After the input image is convolved to extract features, the output rotation angle θ, scaling factor s and translation parameters (t x ,t y ) are obtained:

[0069] (θ, s, t x , t y ) = FC (Conv3 (Conv2 (Conv1 (I))))

[0070] Where I is the input image, t x is the horizontal translation, t y is the vertical translation; Conv1, Conv2 and Conv3 are three convolution layers in the lightweight CNN, and FC is the fully connected layer in the lightweight CNN.

[0071] Then generate the transformation parameters. According to the output of the positioning network, generate the affine transformation matrix T, the formula is:

[0072]

[0073] Then apply the sampler to deform the image at the pixel level: based on the bilinear interpolation, the input image is deformed at the pixel level to correct the tilt, scaling and displacement, the formula is:

[0074] I warped (x, y) = ∑ i，j I(i, j)·max(0, 1-|x-i|)·max(0, 1-|y-j|)

[0075] Where I warped(x,y) represents the pixel value of the new image at coordinate (x,y) after deformation; I(i,j) represents the pixel value of the input image at coordinate (i,j); |xi| represents the horizontal distance between the target position x and the original position i, and |yj| represents the vertical distance between the target position y and the original position j.

[0076] Next, occlusion compensation is performed: using a context-aware network (CNN), the facial features of the occluded area (such as the forehead) are inferred from the features of the unoccluded area (such as the eyes, mouth, etc.), with the following formula:

[0077] F occluded =CAN(F visible Mask)

[0078] Among them, F visible For the characteristics of the unobstructed area, F occluded The occluded region features are represented by Mask, which is the occlusion mask, and CAN is the context-aware network.

[0079] The multi-scale dynamic SPPF module replaces the traditional SPPF module in the original YOLOv8 backbone network. It uses pooling scales to extract overall facial expressions, local muscle and microexpression features, and generates multi-scale feature maps by concatenation. The working principle is as follows:

[0080] 1) Multi-level pooling: Extract features layer by layer from large scale (overall expression) to small scale (micro-muscle movement) to ensure that weak signals of micro-expressions such as brief frowning and twitching of the corner of the mouth are captured.

[0081] 2) Depthwise Separable Convolution: After the system fuses multi-scale feature maps of different scales, it further enhances the extracted multi-scale feature maps with "detail enhancement" through depthwise separable convolution. For example, it makes blurred corners of the mouth clearer or highlights subtle muscle changes when frowning. Unlike simple magnification, this detail enhancement dynamically adjusts based on the complex relationships within the data itself, allowing the system to more accurately understand subtle differences in facial expressions. Specifically,

[0082] like Figure 2 As shown, the multi-scale dynamic SPPF module employs max pooling layers of three scales: 5×5, 9×9, and 13×13, to extract overall facial features (such as the global shape of a smile), local muscle features (such as wrinkles around the eyes), and micro-expression details (such as subtle movements of the corners of the mouth), respectively. The pooling results are then concatenated to generate a multi-scale feature map F. pool The formula is as follows:

[0083] F pool =Concat(P5,P9,P 13 )

[0084] where Concat is a concatenation function, P5 is a 5x5 pooling feature, P9 is a 9x9 pooling feature, P 13 is a 13x13 pooling feature.

[0085] The feature enhancement module is concatenated after the multi-scale dynamic SPPF module, close to the output end of the backbone network, and enhances feature details (such as muscle micro-movement) through deep separable convolution (DSConv), reduces the amount of calculation, and highlights key expression features, including:

[0086] 1) Basic deformation: Similar to filters and other operations to process images.

[0087] 2) Rotation / Flip: Slightly rotate the picture (±10 degrees), or mirror it left and right to simulate various angles of the face.

[0088] 3) Crop / Scale: Randomly crop part of the face (such as showing only half the face), or zoom in or out to train the system to adapt to different distances of shooting.

[0089] 4) Light adjustment: Add shadows, highlights, or dim to the picture to simulate complex lighting such as night, backlight, etc.

[0090] 5) Advanced interference: Simulate "unexpected situations".

[0091] 51) Add occlusion: Randomly paste "mosaic blocks" on the face (simulate masks, hats, hair occlusion), and let the system infer the expression through the visible part.

[0092] 52) Noise pollution: Add snowflakes or blur effects to the image to simulate scenarios with poor camera quality or motion blur.

[0093] 53) Color distortion: Randomly adjust skin color (such as reddish or bluish) to prevent the system from relying too much on color to judge expressions.

[0094] 6) Combination: Mix multiple interferences.

[0095] During the process, the multi-scale feature map F pool Apply deep separable convolution to separate spatial convolution and channel convolution, reduce the amount of calculation, and enhance detailed features, as follows:

[0096] F enhanced = DSConv(ReLU(Norm(F pool )))

[0097] where F enhanced is the dynamically enhanced feature map, F pool is the multi-scale feature map, Norm is the normalization operation, ReLU is the activation function, and DSconv is the deep separable convolution.

[0098] The feature fusion module is embedded in the neck network (FPN+PAN), and the multi-scale features are channel-weighted fused based on the cross-channel attention mechanism (CCA) to enhance the expression of expression-related features (such as higher feature weights of the corners of the mouth and the corners of the eyes). The spatial dimension is compressed by global average pooling (GAP), and the channel attention weight is generated by the fully connected layer (FC) to realize the channel-weighted fusion of multi-scale features, as follows:

[0099] A c = Sigmoid(FC(GAP(F pool )))

[0100] F fused = A c · F pool

[0101] Where A c is the channel attention weight, Sigmoid is the activation function, FC is the fully connected layer, GAP is the global average pooling, and F fused is the fused feature map.

[0102] The above gives the formula, and in the inference process, since the feature enhancement module is directly skipped, the object fused by the feature fusion module is the multi-scale feature map F pool . In the training process, the object fused by the feature fusion module is the dynamically enhanced feature map, and the corresponding formula is:

[0103] A c = Sigmoid(FC(GAP(F enhanced )))

[0104] F fused = A c · F enhanced

[0105] The time series dynamic analysis module (replaces the original classification head) adopts a transfer learning strategy to reuse the pre-trained weights of YOLOv8 on the COCO dataset to accelerate model convergence. In the process, first, short-term memory caching, such as tracking the expression trend within 3-5 seconds (such as the transition from calm to anger); then dynamic correction to combine historical data to correct single-frame misjudgment (such as misidentification caused by sudden changes in light):

[0106] In the cache, for video stream input, the feature vectors of continuous multiple frames (such as 5 frames) are cached, and the expression change trend is captured through an improved LSTM network, as follows:

[0107] h t = LSTM(h t-1 , F t )

[0108] wherein h t is the current frame hidden state, F t is the current frame feature, h t-1 denotes the hidden state of the previous frame;

[0109] In the dynamic correction, the single-frame prediction is weighted and averaged in combination with the historical frame results to smooth the output fluctuation, and the formula is as follows:

[0110]

[0111] wherein a is the current frame weight, is the corrected expression category probability of the current frame, y t is the expression category probability of the current frame before correction, is the corrected expression category probability of the previous frame.

[0112] S3 uses the training set and test set obtained in step S1 to train and infer the improved YOLOv8 expression recognition model, including:

[0113] S31 inputs the training set into the improved YOLOv8 expression recognition model, sets the initial learning rate to 0.001, the batch size to 32, and uses the Adam optimizer for training.

[0114] S32 loads the model weight after training to infer the test set, and outputs the expression category, confidence and key point coordinates.

[0115] In the training process, after initialization, the pre-trained weight of YOLOv8 on the COCO dataset is reused, the first 5 layers of the backbone network are frozen (in the improved YOLOv8 expression recognition model: the first 5 layers of the backbone network correspond to the first several C2f modules and Conv layers in the original YOLOv8 backbone, which are frozen after pre-training), and only the expression recognition related layers (corresponding to the multi-scale dynamic SPPF module, the feature enhancement module, the feature fusion module and the time sequence dynamic analysis module) are fine-tuned, which are located after the multi-scale dynamic SPPF module and before the feature fusion module, forming the order of SPPF→feature enhancement→feature fusion, to ensure that the enhanced features are effectively utilized. Then, the parameters are optimized, the optimizer is Adam, the initial learning rate is 1e -3 , the cosine annealing decay is decayed to 1e -5 ; the batch size is 32, and the training round is 200 epochs. The applicable loss function L is: cross-entropy loss L cls +ciou loss L ciou (for face positioning), and the formula is:

[0116] L=L cls +λ reg Lciou

[0117] wherein, λ reg = 0.5 is the regression loss weight.

[0118] In the training inference process, a feature equalization technology is also used to dynamically adjust the weights of the basic features and semantic features of the face, the basic features including facial contours, and the semantic features including emotional implications. As shown in Figure 3 The regularization weight balancing mechanism structure diagram is shown in FIG. 6. After the features are extracted, the weights of the basic features and the semantic features are further dynamically adjusted by the regularization weight balancing technology, and then the balanced results are generated based on the loss function.

[0119] In the step S32 of inferring the test set, the following steps are included:

[0120] S321. The adaptive geometric correction module performs geometric correction on the obtained image, and performs standardization processing on the corrected data.

[0121] In this process, after the image is captured by the camera, the adaptive geometric correction module is further applied to perform geometric correction to the whole surface view angle (angle deviation ≤ 5°), and then the data is standardized (mean 0.5, standard deviation 0.5). In order to ensure the real-time performance of the data, the feature enhancement module is not used for data enhancement processing in this process.

[0122] S322. The multi-scale dynamic SPPF module generates multi-scale features.

[0123] S323. The feature fusion module performs channel weighted fusion on the multi-scale features.

[0124] The multi-scale features are input into the neck network (FPN+PAN) for feature fusion.

[0125] S324. The detection head outputs the probabilities of each type of expression, and outputs the expression category, confidence and key point coordinates after Softmax activation.

[0126] In order to ensure real-time performance, compared with the training process, the inference process directly skips the feature enhancement module (in the inference process, the forward propagation of the feature enhancement module is controlled to be skipped through parameters), i.e. SPPF→feature fusion, the reasons are as follows:

[0127] 1) Real-time performance is given priority: DSConv can enhance details, but will increase the calculation by about 15%, and the frame rate can be increased from 30fps to more than 40fps by omitting it in inference.

[0128] 2) Training-inference consistency: the model has learned the rules of feature enhancement through DSConv during training, and even if the original Fpool is input during inference, the feature fusion module can still extract key expression features based on the attention mechanism.

[0129] The Softmax classifier outputs the probabilities of 7 expression categories. The threshold is set to 0.7 to filter out low-confidence results. The output includes the expression category, confidence score, and coordinates of facial landmarks (such as the corners of the eyes and mouth).

[0130] The S325 time-series dynamic analysis module caches results from multiple consecutive frames, performs a timestamp-weighted average, and outputs the results smoothly.

[0131] During this process, for the video stream, five consecutive frames are cached, and the output is smoothed by a timestamp-weighted average (with higher weight for closer frames) to reduce misjudgments caused by sudden changes in lighting or motion blur.

[0132] S4 uses a trained, improved YOLOv8 facial expression recognition model to identify the facial surfaces of people in the images or videos to be recognized.

[0133] Another embodiment of the present invention provides a facial expression recognition device based on YOLOv8 deep learning, corresponding to the aforementioned facial expression recognition method, such as... Figure 4 As shown, the facial expression recognition device includes: a data acquisition module 10, used to collect and label publicly available datasets of various basic expressions, and to standardize and filter the labeled data to obtain a dataset, which includes a training set and a test set; a network structure improvement module 20, used to improve the network structure of YOLOv8 to obtain an improved YOLOv8 expression recognition model; a model training module 30, used to train and infer the improved YOLOv8 expression recognition model obtained by the network structure improvement module using the training set and test set obtained by the data acquisition module; and a face recognition module 40, used to train the improved YOLOv8 expression recognition model trained by the model training module. The 8-expression recognition model identifies the facial surface in the image or video to be recognized. The improved YOLOv8 expression recognition model includes: an adaptive geometric correction module for automatically correcting faces in images to a frontal view; a multi-scale dynamic SPPF module for extracting multi-scale features from overall expression to muscle micro-movements in the dataset through multi-level pooling; a feature fusion module for channel-weighted fusion of multi-scale features based on an attention mechanism; a feature enhancement module for dynamically enhancing the details of multi-scale features through depthwise separable convolution; and a temporal dynamic analysis module, which replaces the classification head in the YOLOv8 network to capture expression change trends and smooth the output results. In this embodiment, the working principle and flow of each module correspond to the facial expression recognition method, and will not be elaborated here.

[0134] It should be noted that the above embodiments can be freely combined as needed. The above are only preferred embodiments of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A YOLOv8 deep learning-based face facial expression recognition method, characterized in that, The method comprises the following steps: S1: collecting and labeling public data sets of various basic expressions, and standardizing and filtering noise to obtain a data set, wherein the data set comprises a training set and a test set; S2: improving the network structure of YOLOv8 to obtain an improved YOLOv8 expression recognition model; The improved YOLOv8 expression recognition model comprises: An adaptive geometric correction module for automatically correcting the face in the image to a front view; A multi-scale dynamic SPPF module for extracting multi-scale features of the overall expression to muscle micro-movement of the data set through multi-level pooling; A feature enhancement module for dynamically enhancing the details of the multi-scale features through depth separable convolution; A feature fusion module for channel weighted fusion of the multi-scale features based on an attention mechanism; A time series dynamic analysis module replaces the classification head in the YOLOv8 network to capture the expression change trend and smooth the output result; S3: training and reasoning the improved YOLOv8 expression recognition model using the training set and test set obtained in step S1; S4: using the trained improved YOLOv8 expression recognition model to recognize the face surface in the image or video to be recognized.

2. The method of claim 1, wherein the face part expression recognition method is characterized by, The adaptive geometric correction module comprises: Lightweight CNN is adopted. After the input image is convolved to extract features, the output is the rotation angle θ, the scaling factor s and the translation parameters (t x ,t y ) (θ, s, t x , t y ) = FC(Conv3(Conv2(Conv1(I)))) wherein I is an input image, t x is a translation in the horizontal direction, t y is a translation in the vertical direction; Conv1, Conv2 and Conv3 are three convolutional layers in the lightweight CNN, and FC is a fully connected layer in the lightweight CNN; An affine transformation matrix T is generated based on the positioning network output, and the formula is as follows: The input image is deformed at the pixel level based on bilinear interpolation to correct the tilt, scaling and displacement, and the formula is as follows: I warped (x,y) = ∑ i，j I(i,j) · max(0, 1 - |x - i|) · max(0, 1 - |y - j|) wherein I warped (x,y) is the pixel value of the new image at the coordinate (x, y) after deformation; I(i,j) is the pixel value of the input image at the coordinate (i,j); |x-i| represents the distance between the target position x and the original position i in the horizontal direction, and |y-j| represents the distance between the target position y and the original position j in the vertical direction. The expression features of the occluded area are inferred using the features of the unoccluded area through a context perception network, and the formula is as follows: F occluded = CAN(F visible , Mask) wherein F visible is an unoccluded region feature, F occluded is an occluded region feature, Mask is an occlusion mask, and CAN is a context-aware network.

3. The method of claim 1, wherein the face part expression recognition method is characterized by, In the multi-scale dynamic SPPF module, three scales of maximum pooling layers of 5*5, 9*9 and 13*13 are adopted to extract overall expression features, local muscle features and micro-expression details respectively, and the pooling results are spliced to generate a multi-scale feature map F pool , and the formula is as follows: F pool = Concat(P5, P9, P 13 ) wherein Concat is a concatenation function, P5 is a 5x5 pooling feature, P9 is a 9x9 pooling feature, and P13 is a 13x13 pooling feature. 13 is a 13x13 pooling feature.

4. The face facial expression recognition method of claim 1, wherein the formula of the feature enhancement module is as follows:

5. The face facial expression recognition method of claim 1, wherein the formula of the feature fusion module is as follows: F enhanced = DSConv(ReLU(Norm(F pool ))) where F enhanced is the dynamic enhanced feature map, F pool is the multi-scale feature map, Norm is the normalization operation, ReLU is the activation function, and DSconv is the depth separable convolution. The time series dynamic analysis module comprises: For video stream input, the feature vectors of continuous multiple frames are cached, and the expression change trend is captured through an improved LSTM network, and the formula is as follows: A c = Sigmoid(FC(GAP(F pool ))) F fused = A c • F pool wherein A c is the channel attention weight, Sigmoid is the activation function, FC is the fully connected layer, GAP is the global average pooling, F fused is the fused feature map.

6. The method of claim 1, wherein the face part expression recognition method is characterized by, The single-frame prediction is weighted and averaged based on the historical frame results to smooth the output fluctuation, and the formula is as follows: Step S3 comprises: h t = LSTM(h t-1 ,F t ) where h t is the hidden state of the current frame, F t is the feature of the current frame, h t-1 denotes the hidden state of the previous frame; S31: inputting the training set into the improved YOLOv8 expression recognition model and training using an Adam optimizer; wherein a is a current frame weight, is a corrected expression class probability of a current frame, t is an uncorrected expression class probability of a current frame, is a corrected expression class probability of a previous frame.

7. The method according to any one of claims 1 to 6, wherein the face part of the human face is recognized by using a face part recognition method. S32: loading the model weight after training to reason the test set and output the expression category, confidence and key point coordinates. In the training and reasoning process, a feature equalization technology is used to dynamically adjust the weight of the basic features and semantic features of the face, wherein the basic features include the facial contour, and the semantic features include the emotional meaning. In step S32, the test set is reasoned, comprising:

8. The method of claim 7, wherein the face part expression recognition method is characterized by, S321: the adaptive geometric correction module geometrically corrects the obtained image and standardizes the corrected data; 9. The method of claim 8, wherein the face part expression recognition method is characterized by, S322: the multi-scale dynamic SPPF module generates multi-scale features; S323: the feature fusion module performs channel weighted fusion on the multi-scale features; ​ ​ S324 detects the head output of various expression probabilities, and outputs expression categories, confidence and key point coordinates after Softmax activation; S325 The time sequence dynamic analysis module caches continuous multiple frames of results, and outputs the results by time stamp weighted average and smoothing.

10. A YOLOv8 deep learning-based face facial expression recognition device, characterized in that, It includes: A data acquisition module for acquiring various basic expression public data sets and labeling, and standardizing and filtering noise to obtain a data set, the data set including a training set and a test set; A network structure improvement module for improving the network structure of YOLOv8 to obtain an improved YOLOv8 expression recognition model; A model training module for training and reasoning the improved YOLOv8 expression recognition model improved by the network structure improvement module using the training set and the test set obtained by the data acquisition module; A face recognition module for recognizing the face surface of a person in an image or video to be recognized using the improved YOLOv8 expression recognition model trained by the model training module; The improved YOLOv8 expression recognition model includes: An adaptive geometric correction module for automatically correcting the face in the image to a front view; A multi-scale dynamic SPPF module for extracting multi-scale features of the overall expression to muscle micro-movement of the data set layer by layer through multi-level pooling; A feature fusion module for channel weighted fusion of the multi-scale features based on an attention mechanism; A feature enhancement module for dynamically enhancing the details of the multi-scale features through depth separable convolution; A time sequence dynamic analysis module replaces the classification head in the YOLOv8 network, captures the expression change trend, and outputs the results smoothly.

Images