A real-time expression recognition method based on YOLOv5l and attention mechanism

Abstract

The application discloses a real-time expression recognition method based on YOLOv5l and an attention mechanism, and the method comprises the following steps: acquiring training images and verification images; pre-training an original YOLOv5l model by using the training images to obtain pre-training weights of the original YOLOv5l model; adding an attention mechanism to a backbone network structure of the original YOLOv5l model to obtain an improved YOLOv5l model; re-training the improved YOLOv5l model by using the training images and the pre-training weights to obtain new training weights of the improved YOLOv5l model; and detecting the expression of a target object by using the improved YOLOv5l model with the new training weights. The application combines channel and spatial attention mechanisms, not only realizes the function of real-time recognition of a target expression, improves the accuracy of expression recognition, but also improves the speed of expression recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of expression recognition in computer vision, and particularly relates to a real-time expression recognition method based on YOLOv5 and an attention mechanism. BACKGROUND

[0002] Expression recognition has always been a hot topic in pattern recognition. In recent years, facial expression recognition (FER) has attracted great interest from computer scientists and psychologists. At the same time, since the emotion of teachers is closely related to the quality of teaching, many researchers try to introduce FER into the field of education and teaching in order to play an important role in classroom teaching activities. How to improve the precision and real-time performance of FER is a key problem to be solved when this technology is put into practical application.

[0003] Expression recognition algorithms are divided into unsupervised learning and supervised learning, and most of the algorithms are based on supervised learning. In supervised learning, it is generally divided into two categories: one-stage algorithm and two-stage algorithm. Among them, the more classic two-stage algorithms are R-CNN, Faster-RCNN, VGGNet, GoogLeNet. The advantage of two-stage algorithm is high precision, but the efficiency is relatively low, and the real-time performance is not high. One-stage algorithm mainly includes YOLOv1, YOLOv2, YOLOv3, YOLOv4, YOLOv5, etc. One-stage algorithm is based on two-stage algorithm, which can effectively solve the problem of low efficiency of two-stage algorithm, but the disadvantage is that the detection accuracy of the algorithm is sacrificed,

[0004] The existing problems of facial expression recognition are that it is difficult to maintain the timeliness of the algorithm while maintaining high accuracy of the algorithm, or the accuracy of the algorithm is difficult to guarantee when the timeliness of the algorithm is maintained. SUMMARY

[0005] In order to solve the technical problem of real-time expression recognition, the present application provides a real-time expression recognition method based on YOLOv5l and an attention mechanism, which can not only complete the target expression recognition task in real time, but also enhance the ability of the target detection model to extract facial expression features.

[0006] The real-time expression recognition method based on YOLOv5l and an attention mechanism provided by the present application comprises the following steps:

[0007] Step S1: obtaining training images and verification images;

[0008] Step S2: pre-training the original YOLOv5l model using the training images to obtain the pre-training weight of the original YOLOv5l model;

[0009] Step S3: adding an attention mechanism to the backbone network structure of the original YOLOv5l model to obtain an improved YOLOv5l model;

[0010] Step S4: retraining the improved YOLOv5l model using the training images and the pre-training weights to obtain new training weights of the improved YOLOv5l model;

[0011] Step S5: detecting the expression of the target object by using the improved YOLOv5l model with the new training weights.

[0012] Further, the step S1 comprises the following steps:

[0013] Step S1-1: screening the images in the public data set according to a preset rule;

[0014] Step S1-2: proofreading and labeling the screened images to obtain expression label data corresponding to the images;

[0015] Step S1-3: dividing the labeled images into training images and verification images.

[0016] Further, in step S3, an attention mechanism is added after each standard convolution CBS module of each convolutional residual network CSP1_X module in the backbone network structure of the original YOLOv5l model to obtain an improved YOLOv5l model.

[0017] Further, the improved YOLOv5l model comprises, in sequence, a Focus module, a first CBS module, a CSP1_1A module, a second CBS module, a first CSP1_3A module, a third CBS module, a second CSP1_3A module, a fourth CBS module, a spatial pyramid pooling SPP module, a first CSP2_1 module, a fifth CBS module, a first up-sampling Unsample module, a first concatenation Concat module, a second CSP2_1 module, a sixth CBS module, a second up-sampling Unsample module, a second concatenation Concat module, a third CSP2_1 module, a first convolution module Conv, a seventh CBS module, a third concatenation Concat module, a fourth CSP2_1 module, a second convolution module Conv, an eighth CBS module, a fourth concatenation Concat module, a fifth CSP2_1 module, and a third convolution module Conv, wherein the output of the first up-sampling Unsample module is concatenated with the output of the second CSP1_3A module and input into the first concatenation Concat module; the output of the second up-sampling Unsample module is concatenated with the output of the first CSP1_3A module and input into the second concatenation Concat module; the output of the third CSP2_1 module is input into the input end of the first convolution module Conv, and the output of the first convolution module Conv is taken as the first output result of the improved YOLOv5l model; the output of the fourth CSP2_1 module is input into the input end of the second convolution module Conv, and the output of the second convolution module Conv is taken as the second output result of the improved YOLOv5l model; the output of the fifth CSP2_1 module is input into the input end of the third convolution module Conv, and the output of the third convolution module Conv is taken as the third output result of the improved YOLOv5l model.

[0018] Further, in step S3, attention mechanisms are added after the Focus module in the original YOLOv5l model backbone network structure, or attention mechanisms are added after each CBS module in the original YOLOv5l model backbone network structure, or attention mechanisms are added after the SPP module in the original YOLOv5l model backbone network structure, or attention mechanisms are added after the Focus module and the SPP module in the original YOLOv5l model backbone network structure respectively, or attention mechanisms are added after each CBS module and the SPP module in the original YOLOv5l model backbone network structure respectively, or attention mechanisms are added after the Focus module and each CBS module in the original YOLOv5l model backbone network structure respectively, or attention mechanisms are added after each CBS module and each standard convolution CBS module of each CSP1_X module in the original YOLOv5l model backbone network structure respectively, or attention mechanisms are added after the Focus module, each CBS module, each standard convolution CBS module of each CSP1_X module and the SPP module in the original YOLOv5l model backbone network structure respectively.

[0019] Further, in step S4, when retraining the improved YOLOv5l model, the training image is input into the improved YOLOv5l model, and the retraining of the improved YOLOv5l model is realized by adjusting the learning rate, momentum and iteration number of the model.

[0020] Further, the step S5 comprises the following steps:

[0021] Step S5-1: loading the improved YOLOv5l model with new training weights;

[0022] Step S5-2: selecting a recognition data source through an interface, wherein the data source comprises a camera or a local video;

[0023] Step S5-3: if the camera is selected, the camera is opened, the expression recognition result storage position is input, if the target object expression is detected, the target object face is automatically framed out using a rectangular frame, the interface displays the expression recognition result, and the expression recognition result log can be printed; if the local video is selected, the local video is opened, the recognition frame number and the expression recognition result storage position are input, if the target object expression is detected, the target object face is automatically framed out using a rectangular frame, the interface displays the expression recognition result, and the expression recognition result log can be printed.

[0024] Compared with the prior art, the present application has the following technical effects:

[0025] The application combines the channel and space attention mechanism, realizes the function of real-time target expression recognition, improves the accuracy of expression recognition, and improves the speed of expression recognition.

[0026] It should be noted that the application is supported by the National Natural Science Foundation Project No. 11404240 and the Tianjin Municipal Natural Science Foundation General Project No. 20JCYBJC00300. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A flowchart of a real-time expression recognition method based on YOLOv5l and an attention mechanism according to an embodiment of the application;

[0028] Figure 2A A structural diagram of a YOLOv5l-CSP1_XA model and a CSP1_XA module according to an embodiment of the application;

[0029] Figure 2B A structural diagram of a YOLOv5l-FA model and a FA module according to an embodiment of the application;

[0030] Figure 2C A structural diagram of a YOLOv5l-CBSA model and a CBSA module according to an embodiment of the application;

[0031] Figure 2D A structural diagram of a YOLOv5l-SA model and a SA module according to an embodiment of the application;

[0032] Figure 2E A structural diagram of a YOLOv5l-FASA model according to an embodiment of the application;

[0033] Figure 2F A structural diagram of a YOLOv5l-(CBSA+SA) model according to an embodiment of the application;

[0034] Figure 2G A structural diagram of a YOLOv5l-(FA+CBSA) model according to an embodiment of the application;

[0035] Figure 2H A structural diagram of a YOLOv5l-(CBSA+CSP1_XA) model according to an embodiment of the application;

[0036] Figure 2I A structural diagram of a YOLOv5l-(FA+CBSA+CSP1_XA+SA) model according to an embodiment of the application. DETAILED DESCRIPTION

[0037] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be given below with reference to the embodiments and in conjunction with the drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present application. In addition, in the following description, the description of well-known structures and techniques is omitted to avoid unnecessary confusion of the concept of the present application.

[0038] The real-time expression recognition method based on YOLOv5l and attention mechanism provided by the present application can be applied to real-time recognition of teacher expressions, as shown in the following formula: Figure 1 The real-time expression recognition method based on YOLOv5l and attention mechanism includes the following steps:

[0039] Step S1: obtaining training images and verification images;

[0040] In an embodiment of the present application, the training images and verification images come from the facial expression RAF-DB dataset.

[0041] Further, the step S1 includes the following steps:

[0042] Step S1-1: screening the images in the public dataset according to a preset rule;

[0043] The preset rule can be set according to the needs of actual application, and the present application does not particularly limit its specific content. In an embodiment of the present application, the preset rule can be expressed as excluding pictures with expression disagreement.

[0044] Step S1-2: correcting and labeling the screened images to obtain expression label data corresponding to the images;

[0045] The labeling of images can be realized by means of labeling software such as LabelImg.

[0046] In an embodiment of the present application, about 2240 facial expression images and 2240 corresponding expression label data can be obtained.

[0047] Step S1-3: dividing the labeled images into training images and verification images.

[0048] In an embodiment of the present application, the labeled images can be divided in the ratio of 8:2 to obtain training images and verification images.

[0049] Step S2: pre-training the original YOLOv5l model using the training images to obtain the pre-training weight of the original YOLOv5 model;

[0050] The training of the YOLOv5l model can be performed according to the knowledge mastered by those skilled in the art, and the present application does not make too many descriptions thereof.

[0051] Step S3: adding an attention mechanism to the backbone network structure of the original YOLOv5l model to obtain an improved YOLOv5l model;

[0052] In an embodiment of the present application, when the attention mechanism is added to the backbone network structure of the original YOLOv5l model, an SE (Squeeze and Excitation attention module), an ECA (Efficient Channel Attention module), a CBAM (Convolutional Block Attention Module) or a Coordinate (Coordinate Attention module) attention mechanism can be added after all the standard convolution CBS modules of each CSP1_X module in the backbone network structure of the original YOLOv5l model. For example, the backbone network structure of the original YOLOv5l model includes 3 CSP1_X modules. Among them, X represents the number of residual modules in the CSP1_X module, which are one CSP1_1 module and two CSP1_3 modules, that is, one CSP1 module including 1 residual module and two CSP1 modules including 3 residual modules, then an attention mechanism is added after each CBS module of each CSP1_X module, so that the CSP1_1 module becomes a CSP1_1A module and the CSP1_3 module becomes a CSP1_3A module. The YOLOv5l model obtained at this time can be referred to as a YOLOv5l-CSP1_XA model, and the structure diagram of the YOLOv5l-CSP1_XA model and the CSP1_XA module is shown in Figure 2A After inserting the attention mechanism into the CBS module in the CSP1_X module, not only can the understanding ability of the feature extraction network to the channel and spatial information be effectively increased, but also the gradient value of the interlayer backpropagation of the residual module therein can be increased, avoiding the gradient vanishing caused by the deepening of the network structure, so as to extract more fine-grained features, that is, to make the model improve the clarity of the feature information, so that the network can more efficiently process the features that need to be focused in the feature map.

[0053] As Figure 2AAs shown, the YOLOv5l-CSP1_XA model includes, in sequence, a Focus module, a first CBS module, a CSP1_1A module, a second CBS module, a first CSP1_3A module, a third CBS module, a second CSP1_3A module, a fourth CBS module, a spatial pyramid pooling SPP module, a first CSP2_1 module, a fifth CBS module, a first up-sampling Unsample module, a first concatenation Concat module, a second CSP2_1 module, a sixth CBS module, a second up-sampling Unsample module, a second concatenation Concat module, a third CSP2_1 module, a first convolution module Conv, a seventh CBS module, a third concatenation Concat module, a fourth CSP2_1 module, a second convolution module Conv, an eighth CBS module, a fourth concatenation Concat module, a fifth CSP2_1 module, and a third convolution module Conv. The output of the first up-sampling Unsample module is concatenated with the output of the second CSP1_3A module and input to the first concatenation Concat module; the output of the second up-sampling Unsample module is concatenated with the output of the first CSP1_3A module and input to the second concatenation Concat module; the output of the third CSP2_1 module is input to the input end of the first convolution module Conv, and the output of the first convolution module Conv is taken as the first output result of the YOLOv5l-CSP1_XA model; the output of the fourth CSP2_1 module is input to the input end of the second convolution module Conv, and the output of the second convolution module Conv is taken as the second output result of the YOLOv5l-CSP1_XA model; the output of the fifth CSP2_1 module is input to the input end of the third convolution module Conv, and the output of the third convolution module Conv is taken as the third output result of the YOLOv5l-CSP1_XA model.

[0054] In another embodiment of the present application, when the attention mechanism is added to the backbone network structure of the original YOLOv5l model, the SE, ECA, CBAM or Coordinate attention mechanism can also be added after the Focus module in the backbone network structure of the original YOLOv5l model. The Focus module with the added attention mechanism can be referred to as an FA module. The YOLOv5l model obtained at this time can be referred to as a YOLOv5l-FA model. The structure diagram of the YOLOv5l-FA model and the FA module is as shown in Figure 2B

[0055] ​In another embodiment of the present application, when attention mechanism is added to the backbone network structure of the original YOLOv5l model, SE, ECA, CBAM or Coordinate attention mechanism can also be added after each CBS module in the backbone network structure of the original YOLOv5l model, the CBS module with added attention mechanism can be called CBSA module, and the obtained YOLOv5l model can be called YOLOv5l-CBSA model, and the structural diagram of the YOLOv5l-CBSA model and the CBSA module is as shown in Figure 2C .

[0056] In another embodiment of the present application, when attention mechanism is added to the backbone network structure of the original YOLOv5l model, SE, ECA, CBAM or Coordinate attention mechanism can also be added after the SPP module in the backbone network structure of the original YOLOv5l model, the SPP module with added attention mechanism can be called SA module, and the obtained YOLOv5l model can be called YOLOv5l-SA model, and the structural diagram of the YOLOv5l-SA model and the SA module is as shown in Figure 2D .

[0057] In another embodiment of the present application, the positions of the added attention mechanisms can also be combined to obtain YOLOv5l models with different structures, for example, attention mechanisms can be added after the Focus module and the SPP module in the backbone network structure of the original YOLOv5l model respectively, and the obtained YOLOv5l model can be called YOLOv5l-FASA model, and the structural diagram of the YOLOv5l-FASA model is as shown in Figure 2E ; attention mechanisms can be added after each CBS module and the SPP module in the backbone network structure of the original YOLOv5l model respectively, and the obtained YOLOv5l model can be called YOLOv5l-(CBSA+SA) model, and the structural diagram of the YOLOv5l-(CBSA+SA) model is as shown in Figure 2F ; attention mechanisms can be added after the Focus module and each CBS module in the backbone network structure of the original YOLOv5l model respectively, and the obtained YOLOv5l model can be called YOLOv5l-(FA+CBSA) model, and the structural diagram of the YOLOv5l-(FA+CBSA) model is as shown in Figure 2G ; attention mechanisms can be added after each CBS module and each CBS module of each CSP1_X module in the backbone network structure of the original YOLOv5l model respectively, and the obtained YOLOv5l model can be called YOLOv5l-(CBSA+CSP1_XA) model, and the structural diagram of the YOLOv5l-(CBSA+CSP1_XA) model is as shown in Figure 2HAs shown; also, attention mechanisms can be added respectively after the Focus module, each CBS module, each CBS module of each CSP1_X module, and the SPP module in the original YOLOv5l model backbone network structure, and the YOLOv5l model obtained at this time can be referred to as a YOLOv5l-(FA+CBSA+CSP1_XA+SA) model, and a structural diagram of the YOLOv5l-(FA+CBSA+CSP1_XA+SA) model is as shown in Figure 2I Of course, in addition to the above examples, other forms of combinations of the positions of the added attention mechanisms can also be used, and those skilled in the art can master other forms of combinations of the positions of the added attention mechanisms based on the above examples, and the present application will not be described again in this regard.

[0058] Step S4: retraining the improved YOLOv5l model using the training images and the pre-training weights to obtain new training weights of the improved YOLOv5l model;

[0059] During the retraining of the improved YOLOv5l model, the training images are input into the improved YOLOv5l model, and the retraining of the improved YOLOv5l model is realized by adjusting the learning rate, momentum, iteration number, and other parameters of the model, so as to obtain the new training weights of the improved YOLOv5l model. The cosine annealing algorithm can be used to dynamically adjust the learning rate, that is, the learning rate is constantly adjusted according to the learning rate and the callback function.

[0060] Further, for the process of inputting the training images into the improved YOLOv5l model, taking the improved YOLOv5l-CSP1_XA as an example. Assuming that the training image size is 640*640 and the channel number is 3, as shown in Figure 2A The training image is input into the YOLOv5l-CSP1_XA model, and the training image is down-sampled by the first layer, that is, the Focus module, and a first feature map with a size of 320*320*32 is output;

[0061] The first feature map is input into the second layer, that is, the first CBS module, to obtain a second feature map with a size of 160*160*64, wherein the first CBS includes a convolution layer with a convolution kernel size of 3*3 and a step of 2, a normalization BN layer, and an activation function SiLU layer;

[0062] input the second feature map to a third layer, a CSP1_1A module added with an attention mechanism, in the CSP1_XA module, the second feature map passes through two branches, one branch passes through a standard convolution module CBS with a convolution kernel size of 1*1 and a step size of 1, an attention module and another residual module once, and the other branch passes through a standard convolution module CBS with a convolution kernel size of 1*1 and a step size of 1 and an attention module, the outputs of the two branches are spliced, pass through a CBS module, and then pass through an attention module to obtain a third feature map with a size of 160*160*64;

[0063] input the third feature map to a fourth layer, a second CBS module, to obtain a fourth feature map with a size of 80*80*128;

[0064] input the fourth feature map to a fifth layer, a first CSP1_3A module added with an attention mechanism, to obtain a fifth feature map with a size of 80*80*128;

[0065] input the fifth feature map to a sixth layer, a third CBS module, to obtain a sixth feature map with a size of 40*40*256;

[0066] input the sixth feature map to a seventh layer, a second CSP1_3A module added with an attention mechanism, to obtain a seventh feature map with a size of 40*40*256;

[0067] input the seventh feature map to an eighth layer, a fourth CBS module, to obtain an eighth feature map with a size of 20*20*512;

[0068] input the eighth feature map to a ninth layer, an SPP module. In the SPP module, the eighth feature map first passes through a standard convolution block CBS with a convolution kernel size of 1*1 and a step size of 1, and then sequentially passes through three maximum pooling layers, the outputs of the convolution block and the three maximum pooling layers are subjected to a Concat splicing operation, and finally pass through a standard convolution block CBS with a convolution kernel size of 1*1 and a step size of 1 to obtain a ninth feature map with a size of 20*20*512;

[0069] input the ninth feature map to a tenth layer, a first CSP2_1 module, in the CSP2_1 module, the ninth feature map passes through two branches, one branch first passes through a standard convolution block CBS with a convolution kernel size of 1*1 and a step size of 1, and then passes through 2*1 standard convolution blocks CBS, and the other branch passes through a standard convolution block CBS with a convolution kernel size of 1*1 and a step size of 1, the outputs of the two branches are spliced, pass through a CBS module, to obtain a tenth feature map with a size of 20*20*256;

[0070] The tenth feature map is input into an eleventh layer, that is, a fifth CBS module, to obtain an eleventh feature map with a size of 20*20*256;

[0071] The eleventh feature map is input into a twelfth layer, that is, a first up-sampling Unsample module, without changing the channel number of the feature map but increasing the width and height of the feature map, to obtain a twelfth feature map with a size of 40*40*256;

[0072] The twelfth feature map is concatenated with the seventh feature Figure 1 and input into a thirteenth layer, that is, a first concatenation Concat module, without changing the size of the feature map but increasing the channel number of the feature map, to obtain a thirteenth feature map with a size of 40*40*512;

[0073] The thirteenth feature map is input into a fourteenth layer, that is, a second CSP2_1 module, to obtain a fourteenth feature map with a size of 40*40*256;

[0074] The fourteenth feature map is input into a fifteenth layer, that is, a sixth CBS module, with a convolution kernel size of 1*1 and a step of 1, to obtain a fifteenth feature map with a size of 40*40*128;

[0075] The fifteenth feature map is input into a sixteenth layer, that is, a second up-sampling Unsample module, without changing the channel number of the feature map but increasing the width and height of the feature map, to obtain a sixteenth feature map with a size of 80*80*128;

[0076] The sixteenth feature map is concatenated with the fifth feature Figure 1 and input into a seventeenth layer, that is, a second concatenation Concat module, without changing the size of the feature map but increasing the channel number of the feature map, to obtain a seventeenth feature map with a size of 80*80*256;

[0077] The seventeenth feature map is input into an eighteenth layer, that is, a third CSP2_1 module, to obtain an eighteenth feature map with a size of 80*80*128;

[0078] The eighteenth feature map is input into a nineteenth layer, that is, a seventh CBS module, with a convolution kernel size of 1*1 and a step of 1, to obtain a nineteenth feature map with a size of 40*40*128;

[0079] The nineteenth feature map is concatenated with the fifteenth feature Figure 1 and input into a twentieth layer, that is, a third concatenation Concat module, without changing the size of the feature map but increasing the channel number of the feature map, to obtain a twentieth feature map with a size of 40*40*256;

[0080] The twenty-first feature map is input into a twenty-second layer, that is, an eighth CBS module, to obtain a twenty-second feature map with a size of 20*20*256.

[0081] The twenty-second feature map is input into a twenty-third layer, that is, a fourth splicing Concat module, without changing the size of the feature map but increasing the number of channels of the feature map, to obtain a twenty-third feature map with a size of 20*20*512.

[0082] The twenty-second feature map is input into a twenty-third layer, that is, a fourth splicing Concat module, without changing the size of the feature map but increasing the number of channels of the feature map, to obtain a twenty-third feature map with a size of 20*20*512. Figure 1

[0083] The twenty-third feature map is input into a twenty-fourth layer, that is, a fifth CSP2_1 module, to obtain a twenty-fourth feature map with a size of 20*20*512.

[0084] The eighteenth feature map, the twenty-first feature map and the twenty-fourth feature map are respectively input into a first convolution module Conv, a second convolution module Conv and a third convolution module Conv to obtain a detection result and perform fusion, filter out a detection frame with a confidence lower than a preset threshold, such as 0.25, perform post-processing by using a non-maximum suppression method, extract a target detection frame with a high confidence, and suppress a false detection frame with a low confidence to obtain a final target detection frame.

[0085] In an embodiment of the present application, the performance of the improved YOLOv5l model can be evaluated by means of the verification image. For example, the verification image can be input into the improved YOLOv5l model, and relevant evaluation indexes of the model can be calculated according to the output result of the improved YOLOv5l model, so as to evaluate the performance of the model.

[0086] Step S5: detecting the expression of the target object by using the improved YOLOv5l model with new training weights.

[0087] The target object can be a teacher or other groups that need to be detected for expression. When detecting the expression of the target object by using the improved YOLOv5l model with new training weights, the target object can be constructed based on the improved YOLOv5l model with new training weights, such as developing an interface and an interface by means of PYQT5 and opencv functions, and building the target object expression real-time recognition system based on the improved YOLOv5l model with new training weights. When used, the step S5 includes:

[0088] Step S5-1: loading the improved YOLOv5l model with new training weights;​

[0089] Step S5-2: Selecting a data source such as a camera or a local video through an interface;

[0090] Step S5-3: If the camera is selected, the camera is opened, a facial expression recognition result storage location is input, if a target object facial expression is detected, the target object face is automatically framed using a rectangular frame, the facial expression recognition result is displayed on the interface, and the facial expression recognition result log can be printed; if the local video is selected, the local video is opened, a recognition frame number and a facial expression recognition result storage location are input, if a target object facial expression is detected, the target object face is automatically framed using a rectangular frame, the facial expression recognition result is displayed on the interface, and the facial expression recognition result log can be printed.

[0091] The above embodiments are implemented on the premise of the technical scheme of the present application, and detailed implementation manners and specific operation processes are given, but the protection scope of the present application is not limited to the above embodiments. The methods used in the above embodiments are all conventional methods unless otherwise specified.

[0092] It should be understood that the above specific embodiments of the present application are only used for illustrative or explanatory purposes of the principles of the present application, and do not constitute a limitation of the present application. Therefore, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the present application shall be included in the protection scope of the present application. In addition, the appended claims of the present application are intended to cover all variations and modifications falling within the scope and boundary of the appended claims, or the equivalent forms of such scope and boundary.

Claims

1. A real-time expression recognition method based on YOLOv5l and attention mechanism, characterized in that: step S1: obtaining training images and verification images; step S2: pre-training an original YOLOv5l model using the training images to obtain pre-training weights of the original YOLOv5l model; step S3: adding an attention mechanism to a backbone network structure of the original YOLOv5l model to obtain an improved YOLOv5l model; step S4: retraining the improved YOLOv5l model using the training images and the pre-training weights to obtain new training weights of the improved YOLOv5l model; step S5: detecting the expression of a target object using the improved YOLOv5l model with the new training weights. In step S3, an attention mechanism is added after each standard convolution CBS module of each convolutional residual network CSP1_X module in the backbone network structure of the original YOLOv5l model to obtain an improved YOLOv5l model, which includes a Focus module, a first CBS module, a CSP1_1A module, a second CBS module, a first CSP1_3A module, a third CBS module, a second CSP1_3A module, a fourth CBS module, a spatial pyramid pooling SPP module, a first CSP2_1 module, a fifth CBS module, a first upsampling Unsample module, a first concatenation Concat module, a second CSP2_1 module, a sixth CBS module, a second upsampling Unsample module, a second concatenation Concat module, a third CSP2_1 module, a first convolution module Conv, a seventh CBS module, a third concatenation Concat module, a fourth CSP2_1 module, a second convolution module Conv, an eighth CBS module, a fourth concatenation Concat module, a fifth CSP2_1 module, and a third convolution module Conv, which are connected in sequence, wherein the output of the first upsampling Unsample module is concatenated with the output of the second CSP1_3A module and input into the first concatenation Concat module; the output of the second upsampling Unsample module is concatenated with the output of the first CSP1_3A module and input into the second concatenation Concat module; the output of the third CSP2_1 module is input into the input end of the first convolution module Conv, and the output of the first convolution module Conv serves as the first output result of the improved YOLOv5l model; the output of the fourth CSP2_1 module is input into the input end of the second convolution module Conv, and the output of the second convolution module Conv serves as the second output result of the improved YOLOv5l model; the output of the fifth CSP2_1 module is input into the input end of the third convolution module Conv, and the output of the third convolution module Conv serves as the third output result of the improved YOLOv5l model. The step S1 includes the following steps: Step S1-1: screening images in a public data set according to a preset rule. ​ ​ ​ 2. The method of claim 1, wherein, ​ ​ Step S1-2, correcting and labeling the images screened to obtain expression label data corresponding to the images; Step S1-3, dividing the labeled images into training images and verification images.

3. The method of claim 1, wherein, In step S3, attention mechanism is added after the Focus module in the original YOLOv5l model backbone network structure, or attention mechanism is added after each CBS module in the original YOLOv5l model backbone network structure, or attention mechanism is added after the SPP module in the original YOLOv5l model backbone network structure, or attention mechanism is added after the Focus module and the SPP module in the original YOLOv5l model backbone network structure respectively, or attention mechanism is added after each CBS module and the SPP module in the original YOLOv5l model backbone network structure respectively, or attention mechanism is added after the Focus module and each CBS module in the original YOLOv5l model backbone network structure respectively, or attention mechanism is added after each CBS module and each standard convolution CBS module of each CSP1_X module in the original YOLOv5l model backbone network structure respectively, or attention mechanism is added after the Focus module, each CBS module, each standard convolution CBS module of each CSP1_X module and the SPP module in the original YOLOv5l model backbone network structure respectively.

4. The method of claim 1, wherein, In step S4, when retraining the improved YOLOv5l model, the training images are input into the improved YOLOv5l model, and the retraining of the improved YOLOv5l model is realized by adjusting the learning rate, momentum and iteration number of the model.

5. The method of claim 1, wherein, The step S5 includes the following steps: Step S5-1: loading the improved YOLOv5l model with new training weights; Step S5-2: selecting a data source for recognition through an interface, wherein the data source includes a camera or a local video; Step S5-3: if the camera is selected, the camera is opened, the expression recognition result saving position is input, if the target object expression is detected, the target object face is automatically framed out using a rectangular frame, the expression recognition result is displayed on the interface, and the expression recognition result log can be printed; if the local video is selected, the local video is opened, the recognition frame number and the expression recognition result saving position are input, if the target object expression is detected, the target object face is automatically framed out using a rectangular frame, the expression recognition result is displayed on the interface, and the expression recognition result log can be printed.

Images