Real-time micro-expression recognition method based on yov5

Abstract

The invention relates to a real-time micro-expression recognition method based on yov5, and belongs to the field of computer vision. According to the method, the network model structure of the yov5 is improved, and the GUI operation interface is introduced, so that the micro-expressions represented by each face at different angles are accurately recognized in real time. A deformable convolution DCN module is introduced into a backbone part, an offset mask is calculated through convolution operation, deformable convolution operation is carried out, and convolution kernel space adaptability is improved; an AIFI internal scale feature interaction module is introduced into a Transform self-attention mechanism, and 2D sine wave position embedding is constructed to enhance important features and suppress unimportant features; a trained loss function is changed into an OTA optimization loss function, so that the detection accuracy and the recall rate are improved; and finally, software interface design is carried out by utilizing PyQt5, so that a user can conveniently carry out adjustment according to requirements. According to the method, the problems of processing delay and recognition precision in a traditional micro-expression recognition method can be solved, and the method has wide application prospects in real-time video monitoring, man-machine interaction and other applications.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to a real-time micro-expression recognition method based on yolov5. Background Art

[0002] Facial expression is a natural way of expressing emotions and inner states through the movement of facial muscles. It plays an important role in emotional transmission, communication and interpersonal relationships. Compared with easily recognizable overt expressions, micro-expressions are short and uncontrolled facial expressions, usually lasting between 0.04 and 0.2 seconds, reflecting the true emotions of individuals when they suppress or hide their emotions. Studies have shown that micro-expressions have a high density of emotional information and can reveal human emotions such as anger, fear, sadness, and surprise. This feature makes micro-expression recognition have unique application value in psychology, criminal investigation, crime prevention, emotional computing, human-computer interaction and other fields.

[0003] Existing research on micro-expression recognition is mostly based on the displacement of facial feature points or traditional machine learning methods such as optical flow. However, these methods usually require a lot of manual feature engineering and preprocessing steps, and need to manually annotate a large number of feature points and perform complex preprocessing, resulting in low recognition efficiency and poor real-time performance. Since micro-expressions last very short and vary slightly, traditional methods also face challenges in accuracy. The main difficulty lies in the complexity of data collection and annotation, and micro-expression datasets are usually obtained through induced experiments, which limits the number and diversity of samples. In addition, micro-expressions vary significantly in expression intensity, duration, and frequency of occurrence, resulting in the need for high-precision data annotation and optimization strategies during model training to ensure recognition results. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a real-time micro-expression recognition method based on yolov5. Through the improved yolov5 network model structure, richer and more diverse features can be extracted, and the accuracy, recall rate and robustness of the improved yolov5 network model can be improved, so that the improved yolov5 network model can run well on the designed PyQt5 and perform real-time recognition of micro-expressions.

[0005] In recent years, the introduction of deep learning technology, especially convolutional neural network (CNN), has brought new possibilities for micro-expression recognition. CNN can automatically extract features from raw image data, greatly simplifying feature engineering and improving recognition accuracy. YOLO is a real-time target detection framework that is widely used in target detection tasks due to its high efficiency and accuracy. As an improved version of YOLO, yolov5 has higher detection accuracy and speed, and is very suitable for real-time processing scenarios. By optimizing and training the model based on yolov5, micro-expression recognition can be effectively detected and classified in real-time video streams, thereby improving the intelligence level of emotion analysis and human-computer interaction.

[0006] The technical solution adopted by the present invention is: a real-time micro-expression recognition method based on yolov5, comprising the following specific steps:

[0007] Step 1: Build a micro-expression original dataset, pre-process the original dataset, and divide the new dataset obtained after pre-processing into a training set and a test set;

[0008] Step 2: Classify the new data set in Step 1 into five types of expressions: anger, happiness, normal, sadness, and disgust in the Conda virtual environment;

[0009] Step 3. Improve the structure of the yolov5 network model: replace the C3 module with the C3_DCN module in the backbone part, introduce the AIFI internal scale feature interaction module in the Transformer self-attention mechanism, and change the training loss function to the OTA optimization loss function;

[0010] Step 4: Put the training set into the improved yolov5 network model for training, put the test set into the improved yolov5 network model for feature analysis and recognition, obtain the classification and recognition of micro-expressions, and evaluate the performance of the improved yolov5 network model;

[0011] Step 5. Use PyQt5 to design the software interface, combine the improved yolov5 network model with PyQt5, and present real-time detection results;

[0012] Specifically, in the step Step 1, a deep learning motion magnification algorithm is used to pre-process the micro-expression video to capture useful facial motion information, and the micro-expression video set is intercepted into pictures to form a new data set with the remaining micro-expression pictures.

[0013] Specifically, in the step Step2, Labelimg is used to manually annotate the new data set, a training folder is created to annotate the facial expression anchor box and category information and store them as original photos, and then a new label folder is created to store the txt annotation file in the YOLO format of the specified folder, that is, the weight file.

[0014] Specifically, in the step 3:

[0015] (1) C3_DCN module, i.e. deformable convolution DCN module:

[0016] In the backbone part, the C3 module is replaced by the C3_DCN module. The offset mask is calculated through convolution operation, and the offset mask is decomposed into two parts: offset and mask. The offset and mask information are then used for deformable convolution operation. Finally, the final output is obtained through batch normalization and activation function processing. The specific expression is as follows:

[0017]

[0018] In the formula, y(p 0 ) indicates that the output feature map is at position p 0 The value of p 0 Indicates the position of the current output feature point; p n represents the fixed convolution kernel sampling position; Δp n represents the offset obtained by dynamic learning; R represents the sampling position set of the convolution kernel; w(p n ) represents the convolution kernel weight; x represents the input feature map;

[0019] (2) AIFI internal scale feature interaction module:

[0020] The AIFI internal scale feature interaction module is introduced into the Transformer self-attention mechanism to interactively model different positions of the feature map, capture long-distance dependencies when processing long sequences, and construct two-dimensional sine-cosine position encoding, so that the global dependency modeling capability of long sequences can perceive the spatial information of each position in the feature map;

[0021] (4) OTA optimization loss function

[0022] The training loss function is changed to the OTA optimization loss function, and the ComputeLossOTA loss function is used as the OTA optimization loss function. The specific representation of its two core parts is as follows:

[0023] L is the total loss of the OTA optimization loss function, and its expression is:

[0024]

[0025] Where M represents the set of all assigned positive sample pairs (i, j); ij Represents the allocation weight between the prediction box i and the target box j; represents the bounding box regression loss automatically obtained during the training process; Represents the classification loss automatically obtained during the training process; Represents the target confidence loss automatically obtained during the training process;

[0026] In the OTA optimization loss function, the cost matrix C ij It is the core calculation part, and its expression is:

[0027]

[0028] Cost matrix C ij is an N p ×N t A matrix where N p is the number of prediction boxes, N t is the number of target boxes. Where, IoU ij Represents the IoU between the i-th prediction box and the j-th target box; represents the classification loss; represents the target confidence loss; α represents the hyperparameter that controls the weight of the bounding box regression loss; β represents the hyperparameter that controls the weight of the classification loss; γ represents the hyperparameter that controls the weight of the target confidence loss.

[0029] Specifically, the indicators for evaluating the performance of the improved yolov5 network model include: intersection over union (IoU), recall rate (R), precision rate (P), average accuracy rate (AP) and average accuracy rate (mAP). The specific expressions are as follows:

[0030]

[0031]

[0032] In the formula, A∩B represents the intersection area of ​​the predicted box and the true box; A∪B represents the union area of ​​the predicted box and the true box; TP represents the number of correctly predicted positive samples; FN represents the number of true positive samples that are incorrectly predicted as negative samples; FP represents the number of incorrectly predicted positive samples; N represents the total number of target categories; AP c represents the average precision of category c; P(R) represents the precision of the model at the recall rate when the recall rate R reaches a certain value.

[0033] Specifically, in the step 5, PyQt5 is used for design and real-time micro-expression recognition, providing a software interface for users to select whether to detect images or videos, and using the file dialog function of PyQt5 to browse and select files. After the user clicks the load button on the software interface, the selected file path will be presented in this software interface. The software interface integrates a navigation menu, a display area, and an information bar. The navigation menu allows users to select, set, and adjust files. The display area presents real-time detection results, and the information bar reflects the current recognition status and related prompts. The core requirements of the software interface are as follows:

[0034] (1) In terms of weight, users are allowed to change weight files to adapt to different detection tasks;

[0035] (2) In terms of image and video detection, upload images or video files to detect expressions in them;

[0036] (3) In terms of real-time detection, the default camera is used to perform real-time expression recognition;

[0037] (4) In terms of exporting recognition text boxes, when exporting video files, you can export a txt file that records what expression appeared at what minute and second in the video; when exporting real-time monitoring videos, you can export a txt file that records what expression appeared at what hour and minute in Beijing time;

[0038] (5) In terms of exporting recognition results, you can export the files after detection.

[0039] The beneficial effects of the present invention are as follows: the method has efficient micro-expression detection and recognition capabilities through the optimization of the yolov5 model, and realizes micro-expression analysis with strong real-time performance and high precision. The method can adapt to a variety of complex scenarios and show good robustness and stability. Its lightweight design makes it easy to deploy in embedded or edge computing devices, reducing hardware costs, and has the advantage of strong scalability, and the recognition effect can be further improved by customizing data sets. It can be widely used in fields such as mental health monitoring, emotion analysis and public safety, and has significant social value and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of a real-time micro-expression recognition method based on yolov5 of the present invention;

[0041] Figure 2 This is the effect diagram of the deep learning motion amplification algorithm when amp=10;

[0042] Figure 3 This is the flow chart of the C3_DCN module;

[0043] Figure 4It is the AIFI internal scale feature interaction network model;

[0044] Figure 5 This is the structure diagram of the ComputeLossOTA loss function;

[0045] Figure 6 Flowchart designed for PyQt5 software;

[0046] Figure 7 This is the interface diagram of the PyQt5 software;

[0047] Figure 8 This is the effect diagram of real-time micro-expression recognition. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the embodiments and drawings, but the protection content of the present invention is not limited to the scope described. The specific steps are as follows:

[0049] Step 1: Build a micro-expression original dataset and pre-process the constructed original dataset;

[0050] Step 1.1, obtain the original data set of micro-expressions, and use the deep learning motion magnification algorithm to pre-process the micro-expression video. The specific steps are as follows:

[0051] Get the original micro-expression video from the micro-expression datasets published by the SAMM and CASME open source platforms, create a micro-expression video folder, use the FFmpeg tool to convert the video file into a series of image frames, and save them in the newly created folder. Use a deep convolutional neural network (CNN) to implement the deep learning motion amplification algorithm, and use two frames of input for training. Add the trained weights and select the amp value of 10 for processing according to the effect. Finally, add the processed images to the frame list to generate a video and save it. You can select the required micro-expression video and image in the frame list. The effect is as follows: Figure 2 shown.

[0052] Step 1.2, select pictures from the processed micro-expression video set, and then select the last three pictures of each volunteer for each expression. After the videos in the original dataset are processed into pictures, they are combined with the pictures in the original dataset to form a new dataset with a total of 981 pictures;

[0053] Step 2: Classify the new data set into five types of expressions: anger, happiness, normal, sadness, and disgust in the Conda virtual environment;

[0054] Step 2 The specific operations are as follows:

[0055] Step 2.1. Install yolov5 related environment in Windows environment without compilation, use Conda to create a virtual environment. Then open Pycharm, configure the virtual environment, install Pytorch dependencies that support GPU, download yolov5 source code, use pip to download requirements, and complete the environment installation;

[0056] Step 2.2, use Labelimg to manually annotate the new dataset, which is divided into five expressions, namely anger, happiness, normal, sad, and disgust. Create a training folder to annotate the facial expression anchor box and category information and store them as original photos. Then create a new label folder to store the YOLO format txt annotation file in the specified folder, that is, the weight file;

[0057] Step 3: Improve the network model structure of yolov5 and add modules. The specific process is as follows:

[0058] Step 3.1. Replace the C3 module with the C3_DCN module in the backbone part. The flow chart is as follows: Figure 3 As shown in the figure, the traditional residual structure is combined with the dynamic convolution, the offset mask is calculated through the convolution operation, the offset mask is decomposed into two parts: the offset and the mask, and then the deformable convolution operation is performed using this information. Finally, the final output is obtained through batch normalization and activation function processing. The specific expression is as follows:

[0059]

[0060] In the formula, y(p 0 ) indicates that the output micro-expression image is at position p 0 The value of p 0 Indicates the position of the current output feature point; p n represents the fixed convolution kernel sampling position; Δp n represents the offset obtained by dynamic learning; R represents the sampling position set of the convolution kernel; w(p n ) represents the weight of the convolution kernel obtained in automatic training; x represents the input micro-expression picture.

[0061] Table 1 Performance comparison between the original yolov5 and the OTA module and DCN module

[0062]

[0063] After its introduction, the precision and recall rates increased by 0.8% and 0.2%, which are not high, but the amount of computation decreased by 10% compared to YOLOv5, and the inference time increased by 7%.

[0064] Step 3.2: Introduce the AIFI internal scale feature interaction module into the Transformer self-attention mechanism. The model is as follows: Figure 4 As shown in the figure, the interaction modeling of different positions of the feature map can better capture long-distance dependencies when processing long sequences. The construction of two-dimensional sine and cosine position encoding enables the global dependency modeling capability of long sequences to perceive the spatial information of each position in the feature map. The combination of multi-branch convolution and attention mechanism enables the AIFI internal scale feature interaction module to extract richer and more diverse features, thereby improving the detection accuracy and robustness of the model.

[0065] Step 3.3, change the training loss function to OTA optimization loss function, the structure diagram is as follows Figure 5 As shown in the figure, based on the traditional binary cross entropy loss, the function of smoothing the impact of missing labels is added, and a focusing factor is introduced to weight the loss, reducing the focus on easy-to-classify samples and emphasizing the impact of difficult-to-classify samples. This is suitable for class imbalance problems. The ComputeLossOTA loss function is used as the OTA optimization loss function, and the specific representation of its two core parts is as follows:

[0066] L is the total loss of the OTA optimization loss function, and its expression is:

[0067]

[0068] Where M represents the set of all assigned positive sample pairs (i, j); ij Indicates that the distribution weight between the prediction box i and the target box j is automatically calculated during the training process; represents the bounding box regression loss automatically calculated during training; Represents the classification loss automatically calculated during training; Represents the target confidence loss automatically calculated during training.

[0069] In the OTA optimization loss function, the cost matrix C ij It is the core calculation part, and its expression is:

[0070]

[0071] Where, IoU ij Represents the IoU between the i-th prediction box and the j-th target box; Represents the classification loss automatically calculated during training; Represents the target confidence loss automatically calculated during training; α represents the hyperparameter that controls the weight of the bounding box regression loss; β represents the hyperparameter that controls the weight of the classification loss; γ represents the hyperparameter that controls the weight of the target confidence loss.

[0072] Table 2 Comparison of P, R, mAP and inference speed of different loss functions

[0073]

[0074] By comparison, it is found that when the difference in other performances is very small, the YOLOv5-OTA model inference speed is 2.46ms faster than the YOLOv5 model, which improves the performance by 13%.

[0075] Step 4: Put the training set into the improved yolov5 network model for training, put the test set into the improved yolov5 network model for feature analysis and recognition, obtain the classification and recognition of micro-expressions, and evaluate the performance of the improved yolov5 network model. The specific steps are as follows:

[0076] Step 4.1, put the micro-expression pictures into the improved yolov5 network model for training;

[0077] Step 4.2, for the evaluation of model performance, the indicators used in this invention are intersection over union (IoU), recall rate (R), precision rate (P), average accuracy rate (AP) and mean average accuracy rate (mAP), and the specific expressions are as follows:

[0078]

[0079] Where A∩B represents the intersection area of ​​the predicted box and the true box; A∪B represents the union area of ​​the predicted box and the true box; TP represents the number of correctly predicted true micro-expression categories; FN represents the number of true micro-expression categories that are incorrectly predicted as other categories; FP represents the number of incorrectly predicted true micro-expression categories; N represents the number of classified micro-expression categories; AP c represents the average precision of micro-expression category c; P(R) represents the precision of the model at the recall rate when the recall rate R reaches a certain value.

[0080] Step 5: Use PyQt5 to design the software interface. Figure 7 As shown, the improved yolov5 network model is combined with PyQt5 to present real-time detection results. The information bar reflects the current recognition status and related prompts. The specific steps are as follows:

[0081] Step 5.1. Provide a panel for users to choose whether to detect images or videos.

[0082] Step 5.2, use the file dialog function of PyQt5 to browse and select files. After the user clicks the button, the selected file path will be displayed in this software interface. The specific process is as follows:

[0083] After running the source code, the software interface appears. In the weight file column of the software interface, click to select the trained micro-expression weight file, and then click the load picture column to select the micro-expression picture you want to recognize. At this time, after the calculation of the improved yolov5 network model, the selected micro-expression picture and the recognized micro-expression category will be displayed in the display area.

[0084] The core requirements of the software interface are as follows:

[0085] (1) In terms of weights, users are allowed to replace the trained weight files according to different recognition projects to adapt to detection tasks with different requirements;

[0086] (2) In terms of image and video detection, upload images or video files to detect expressions in them;

[0087] (3) In terms of real-time detection, the default camera is used to perform real-time expression recognition;

[0088] (4) In terms of exporting recognition text boxes, when exporting video files, you can export a txt file that records what expression appeared at what minute and second in the video; when exporting real-time monitoring videos, you can export a txt file that records what expression appeared at what hour and minute in Beijing time;

[0089] (5) In terms of exporting recognition results, you can export the files after detection.

[0090] By optimizing and training the model based on yolov5, the present invention can effectively detect and classify micro-expression recognition in real-time video streams, thereby improving the intelligent level of emotion analysis and human-computer interaction.

[0091] The present invention is explained through specific implementation processes. Various changes and equivalent substitutions may be made to the present invention without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific implementation processes disclosed, but should include all implementation plans that fall within the scope of the claims of the present invention.

Claims

1. A real-time micro-expression recognition method based on yolov5, characterized in that: The specific steps include: Step 1: Build a micro-expression original dataset, pre-process the original dataset, and divide the new dataset obtained after pre-processing into a training set and a test set; Step 2: Classify the new data set in Step 1 into five types of expressions: anger, happiness, normal, sadness, and disgust in the Conda virtual environment; Step 3. Improve the structure of the yolov5 network model: replace the C3 module with the C3_DCN module in the backbone part, introduce the AIFI internal scale feature interaction module in the Transformer self-attention mechanism, and change the training loss function to the OTA optimization loss function; Step 4: Put the training set into the improved yolov5 network model for training, put the test set into the improved yolov5 network model for feature analysis and recognition, obtain the classification and recognition of micro-expressions, and evaluate the performance of the improved yolov5 network model; Step 5. Use PyQt5 to design the software interface, combine the improved yolov5 network model with PyQt5, and present real-time detection results.

2. A real-time micro-expression recognition method based on yolov5 according to claim 1, characterized in that: In the step 1, the micro-expression video is pre-processed using a deep learning motion magnification algorithm to capture useful facial motion information, and the micro-expression video set is intercepted into images to form a new data set with the remaining micro-expression images.

3. A real-time micro-expression recognition method based on yolov5 according to claim 1, characterized in that: In the step Step2, Labelimg is used to manually annotate the new data set, a training folder is created to annotate the facial expression anchor box and category information and store them as original photos, and then a new label folder is created to store the txt annotation file in the YOLO format of the specified folder, that is, the weight file.

4. A real-time micro-expression recognition method based on yolov5 according to claim 1, characterized in that: In the step 3: (1) C3_DCN module, i.e. deformable convolution DCN module: In the backbone part, the C3 module is replaced by the C3_DCN module. The offset mask is calculated through convolution operation, and the offset mask is decomposed into two parts: offset and mask. The offset and mask information are then used for deformable convolution operation. Finally, the final output is obtained through batch normalization and activation function processing. The specific expression is as follows: In the formula, y(p0) represents the value of the output feature map at position p0; p0 represents the position of the current output feature point; p n represents the fixed convolution kernel sampling position; Δp n represents the offset obtained by dynamic learning; R represents the sampling position set of the convolution kernel; w(p n ) represents the convolution kernel weight; x represents the input feature map; (2) AIFI internal scale feature interaction module: The AIFI internal scale feature interaction module is introduced into the Transformer self-attention mechanism to interactively model different positions of the feature map, capture long-distance dependencies when processing long sequences, and construct two-dimensional sine-cosine position encoding, so that the global dependency modeling capability of long sequences can perceive the spatial information of each position in the feature map; (3) OTA optimization loss function The training loss function is changed to the OTA optimization loss function, and the ComputeLossOTA loss function is used as the OTA optimization loss function. The specific representation of its two core parts is as follows: L is the total loss of the OTA optimization loss function, and its expression is: Where M represents the set of all assigned positive sample pairs (i, j); ij Represents the allocation weight between the prediction box i and the target box j; represents the bounding box regression loss automatically obtained during the training process; Represents the classification loss automatically obtained during the training process; Represents the target confidence loss automatically obtained during the training process; In the OTA optimization loss function, the cost matrix C ij It is the core calculation part, and its expression is: Cost matrix C ij is an N p ×N t A matrix where N p is the number of prediction boxes, N t is the number of target boxes, where IoU ij Represents the IoU between the i-th prediction box and the j-th target box; represents the classification loss; represents the target confidence loss; α represents the hyperparameter that controls the weight of the bounding box regression loss; β represents the hyperparameter that controls the weight of the classification loss; γ represents the hyperparameter that controls the weight of the target confidence loss.

5. A real-time micro-expression recognition method based on yolov5 according to claim 1, characterized in that: The indicators for evaluating the performance of the improved yolov5 network model include: intersection over union (IoU), recall rate (R), precision rate (P), average accuracy rate (AP) and average accuracy rate (mAP). The specific expressions are as follows: In the formula, A∩B represents the intersection area of ​​the predicted box and the true box; A∪B represents the union area of ​​the predicted box and the true box; TP represents the number of correctly predicted positive samples; FN represents the number of true positive samples that are incorrectly predicted as negative samples; FP represents the number of incorrectly predicted positive samples; N represents the total number of target categories; AP c represents the average precision of category c; P(R) represents the precision of the model at the recall rate when the recall rate R reaches a certain value.

6. A real-time micro-expression recognition method based on yolov5 according to claim 1, characterized in that: In the step 5, PyQt5 is used for design and real-time micro-expression recognition, providing a software interface for users to select whether to detect images or videos, and using the file dialog function of PyQt5 to browse and select files. After the user clicks the load button on the software interface, the selected file path will be presented in this software interface. The software interface integrates a navigation menu, a display area, and an information bar. The navigation menu allows users to select, set, and adjust files. The display area presents real-time detection results, and the information bar reflects the current recognition status and related prompts. The core requirements of the software interface are as follows: (1) In terms of weight, users are allowed to change weight files to adapt to different detection tasks; (2) In terms of image and video detection, upload images or video files to detect expressions in them; (3) In terms of real-time detection, the default camera is used to perform real-time expression recognition; (4) In terms of exporting recognized text boxes, when exporting a video file, you can export a txt file that contains the expressions that appeared at the exact minute and second of the video. When exporting real-time monitoring video, you can export a txt file showing what expression appeared at what hour, minute, and second in Beijing time; (5) In terms of exporting recognition results, you can export the files after detection.

Images