A method and device for recognizing facial expression

Abstract

The present invention discloses a facial expression recognition method and device, which are used to solve the technical problem of poor recognition accuracy caused by existing facial expression recognition methods. The method includes: obtaining a facial image of a video frame to be detected, and generating a global facial image and an initial key point set based on the facial image of the video frame to be detected; using a local candidate module based on a regional relationship algorithm to generate multiple facial local region images based on the initial key point set; inputting each facial local region image into a preset local expression feature extraction network to perform local feature extraction and output multiple local expression feature data; performing global feature extraction on the global facial image using a global expression feature extraction network based on a sustained attention algorithm to generate global expression feature data corresponding to the global facial image; using a preset feature classifier to perform facial expression recognition based on the global expression feature data and each local expression feature data, and generating a facial expression recognition result.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for recognizing facial expressions. Background Art

[0002] Facial expression recognition (FER) technology has attracted widespread attention in recent years, primarily due to its potential to enhance human-computer interaction. The core task of FER technology is to enable computers to capture and distinguish facial expressions from static images or video sequences. Currently, FER work primarily relies on deep learning-based neural network models, which excel in FER tasks in video streams due to their excellent feature learning and acquisition capabilities.

[0003] The basic facial expressions are typically categorized into eight categories: anger, contempt, disgust, fear, happiness, neutrality, sadness, and surprise. Accurately recognizing facial expressions helps computers understand and respond to users' emotional states, significantly improving the human-computer interaction experience. This capability will contribute to the construction of more natural and effective interactive systems, and will promote the application of this technology in areas such as social robotics, affective computing, and intelligent surveillance.

[0004] Existing facial expression recognition methods propose a FER recognition model based on the fusion of local and global features. Its main model framework consists of two branches, namely the local detail branch and the global key point branch. For the global key point branch, key points are first detected on the entire image to generate relevant facial masks. Subsequently, in the feature extraction network, the key area data information is assisted in highlighting. For the local detail branch, fixed key areas such as the eyes and mouth are selected, and each cropped image is input into the feature extraction network for feature extraction, thereby completing facial expression recognition. However, due to the lack of a relationship mechanism between facial regions, this FER model can only perform fixed positioning and division of the global image and local images, resulting in poor recognition accuracy. Summary of the Invention

[0005] The present invention provides a facial expression recognition method and device, which are used to solve the technical problem of poor recognition accuracy caused by existing facial expression recognition methods.

[0006] A first aspect of the present invention provides a method for recognizing facial expressions, comprising:

[0007] Acquire a face image of a video frame to be detected, perform target detection on the face image of the video frame to be detected, and generate a global face image and an initial key point set corresponding to the global face image;

[0008] Generate multiple facial local area images based on the initial key point set using a local candidate module based on a region relationship algorithm;

[0009] Inputting each of the local face area images into a preset local expression feature extraction network to extract local features, and outputting local expression feature data corresponding to each of the local face area images;

[0010] Performing global feature extraction on the global facial image through a global expression feature extraction network based on a sustained attention algorithm to generate global expression feature data corresponding to the global facial image;

[0011] A preset feature classifier is used to perform facial expression recognition based on the global expression feature data and each of the local expression feature data to generate a facial expression recognition result.

[0012] Optionally, performing target detection on the face image of the video frame to be detected to generate a global face image and an initial key point set corresponding to the global face image includes:

[0013] Performing preliminary face detection and marking operations on the face image of the video frame to be detected to obtain an initial detected face image;

[0014] Performing image preprocessing on the initially detected face image to generate a global face image;

[0015] Perform facial key point detection on the global face image, and output an initial key point set corresponding to the global face image.

[0016] Optionally, the initial key point set includes a plurality of initial key points; and the local candidate module based on the region relationship algorithm generates a plurality of local face region images according to the initial key point set, including:

[0017] Performing local area coarse matching on each of the initial key points to generate multiple initial local area key point groups;

[0018] Calculating the correlation degree corresponding to each initial local area key point in each initial local area key point group based on the confidence, confidence threshold, and feature detection support corresponding to each initial local area key point in each initial local area key point group;

[0019] In each of the initial local area key point groups, any initial local area key point corresponding to a correlation degree less than a preset correlation degree threshold is eliminated to determine an intermediate local area key point group corresponding to each of the initial local area key point groups;

[0020] Based on the feature detection support corresponding to each intermediate local area key point in each intermediate local area key point group, the intermediate local area key point corresponding to the largest feature detection support is selected as the core local area key point in each intermediate local area key point group;

[0021] performing weighted Mahalanobis distance calculations on the core local area key point and the multiple intermediate local area key points in each of the intermediate local area key point groups, respectively, to determine the weighted Mahalanobis distance corresponding to each intermediate local area key point in each of the intermediate local area key point groups;

[0022] In each of the intermediate local area key point groups, any intermediate local area key point corresponding to a weighted Mahalanobis distance less than a preset Mahalanobis distance threshold is removed to determine a target local area key point group corresponding to each of the intermediate local area key point groups;

[0023] generating a two-dimensional facial mask matrix according to pixel position information corresponding to each target local area key point in each target local area key point group;

[0024] Determining a mask image based on the two-dimensional facial mask matrix;

[0025] A plurality of local face area images are generated according to the two-dimensional face mask matrix and the mask image.

[0026] Optionally, the preset local expression feature extraction network includes a convolutional pooling subnetwork, three cascaded fully connected modules and a Softmax activation function layer; the inputting each of the local facial area images into the preset local expression feature extraction network for local feature extraction, and outputting local expression feature data corresponding to each of the local facial area images, includes:

[0027] Adjusting the image format of each of the facial local area images to determine a target facial local area image corresponding to each of the facial local area images;

[0028] Using a convolutional pooling subnetwork to perform convolutional pooling on each of the target face local area images, respectively, to generate a local convolutional pooling feature map corresponding to each of the face local area images;

[0029] Using each of the convolutional pooling feature maps as the input of three cascaded fully connected modules, and outputting the fully connected feature maps corresponding to each of the convolutional pooling feature maps;

[0030] Nonlinear mapping is performed on each of the fully connected feature maps through a Softmax activation function layer to generate local expression feature data corresponding to each of the fully connected feature maps.

[0031] Optionally, the convolutional pooling subnetwork includes two serially connected 3×3 convolutional modules, a maximum pooling layer, and three cascaded 3×3 convolutional modules; the convolutional pooling subnetwork is used to perform convolutional pooling on each of the target face local area images to generate a local convolutional pooling feature map corresponding to each of the face local area images, including:

[0032] Using two serially connected 3×3 convolution modules to perform a convolution operation on each of the target face local area images, and outputting a first convolution feature map corresponding to each of the target face local area images;

[0033] Performing maximum pooling on each of the first convolutional feature maps through a maximum pooling layer to generate a first maximum pooling feature map corresponding to each of the first convolutional feature maps;

[0034] Input each of the first maximum pooling feature maps into two series-connected 3×3 convolution modules for convolution operation, and output a second convolution feature map corresponding to each of the first maximum pooling feature maps;

[0035] Performing maximum pooling on each of the second convolutional feature maps using a maximum pooling layer to generate a second maximum pooling feature map corresponding to each of the second convolutional feature maps;

[0036] Performing a convolution operation on each of the second maximum pooling feature maps through three cascaded 3×3 convolution modules, and outputting a third convolution feature map corresponding to each of the second maximum pooling feature maps;

[0037] Inputting each of the third convolutional feature maps into a maximum pooling layer for maximum pooling, and outputting a third maximum pooling feature map corresponding to each of the third convolutional feature maps;

[0038] Using three cascaded 3×3 convolution modules to perform a convolution operation on each of the third maximum pooling feature maps, and outputting a fourth convolution feature map corresponding to each of the third maximum pooling feature maps;

[0039] Inputting each of the fourth convolutional feature maps into a maximum pooling layer for maximum pooling, and outputting a fourth maximum pooling feature map corresponding to each of the fourth convolutional feature maps;

[0040] Using three cascaded 3×3 convolution modules to perform a convolution operation on each of the fourth maximum pooling feature maps, and outputting a fifth convolution feature map corresponding to each of the fourth maximum pooling feature maps;

[0041] Input each of the fifth convolutional feature maps into the maximum pooling layer for maximum pooling, and output the local convolutional pooling feature map corresponding to each of the fifth convolutional feature maps.

[0042] Optionally, the global expression feature extraction network based on the sustained attention algorithm includes a convolutional pooling subnetwork and a local enhancement module based on the sustained attention algorithm; performing global feature extraction on the global facial image through the global expression feature extraction network based on the sustained attention algorithm to generate global expression feature data corresponding to the global facial image includes:

[0043] Performing convolution pooling on the global face image using a convolution pooling subnetwork to generate a global convolution pooling feature map;

[0044] The global convolution pooling feature map is subjected to feature extraction through a local enhancement module based on a continuous attention algorithm to generate global expression feature data corresponding to the global convolution pooling feature map.

[0045] Optionally, the local enhancement module based on the sustained attention algorithm includes a maximum pooling layer, a convolution layer, an average pooling layer, and a Sigmoid activation function layer; the local enhancement module based on the sustained attention algorithm performs feature extraction on the global convolution pooling feature map to generate global expression feature data corresponding to the global convolution pooling feature map, including:

[0046] Performing maximum pooling on the global convolutional pooling feature map through a maximum pooling layer, and outputting a global maximum pooling feature map;

[0047] Performing average pooling on the global convolutional pooling feature map using an average pooling layer to generate a global average pooling feature map;

[0048] Splicing the global maximum pooling feature map and the global average pooling feature map to generate a spliced ​​feature map;

[0049] Inputting the spliced ​​feature map into the convolution layer to perform a convolution operation to generate a global convolution feature map;

[0050] A Sigmoid activation function layer is used to perform nonlinear mapping on the global convolution feature map to generate an initial attention map;

[0051] Using a continuous attention mechanism to generate a target attention map based on the initial attention map;

[0052] Perform matrix element-by-element multiplication on the target attention map and the global convolution pooling feature map, and output the global expression feature data corresponding to the global convolution pooling feature map.

[0053] Optionally, the adopting a preset feature classifier to perform facial expression recognition based on the global expression feature data and each of the local expression feature data to generate a facial expression recognition result includes:

[0054] Performing early linear fusion processing on the global expression feature data and each of the local expression feature data to generate multi-scale features;

[0055] A preset feature classifier is used to perform recognition based on the multi-scale features to generate a facial expression recognition result.

[0056] Optionally, the training process of the local enhancement module based on the sustained attention algorithm includes:

[0057] Obtaining a face image of a video frame to be trained, and generating a two-dimensional facial mask matrix to be trained and a global maximum pooling feature map to be trained based on the face image of the video frame to be trained;

[0058] An initial local enhancement module based on a continuous attention algorithm is used to generate an initial attention map to be trained according to the global maximum pooling feature map to be trained;

[0059] Calculating a target loss value based on the initial attention map to be trained and the two-dimensional facial mask matrix to be trained using a preset regression loss function;

[0060] Based on the target loss value, the network parameters of the initial local enhancement module based on the continuous attention algorithm are updated, the intermediate local enhancement module based on the continuous attention algorithm is determined, and the number of iterations is counted in real time;

[0061] Determine whether the number of iterations reaches a preset number of updates;

[0062] If achieved, the intermediate local enhancement module based on the continuous attention algorithm is used as the trained local enhancement module based on the continuous attention algorithm.

[0063] A second aspect of the present invention provides a facial expression recognition device, comprising:

[0064] An acquisition module is used to acquire a face image of a video frame to be detected, and perform target detection on the face image of the video frame to be detected, to generate a global face image and an initial key point set corresponding to the global face image;

[0065] A generating module, configured to generate a plurality of facial local area images according to the initial key point set by using a local candidate module based on a region relationship algorithm;

[0066] A local feature extraction module is used to input each of the local face area images into a preset local expression feature extraction network to extract local features and output local expression feature data corresponding to each of the local face area images;

[0067] A global feature extraction module is used to extract global features from the global facial image using a global expression feature extraction network based on a sustained attention algorithm, and generate global expression feature data corresponding to the global facial image;

[0068] The recognition module is used to use a preset feature classifier to perform facial expression recognition based on the global expression feature data and each local expression feature data to generate a facial expression recognition result.

[0069] It can be seen from the above technical solutions that the present invention has the following advantages:

[0070] The above technical solution of the present invention provides a facial expression recognition method, which first obtains a face image of a video frame to be detected, performs target detection on the face image of the video frame to be detected, and generates a global face image and an initial key point set corresponding to the global face image; then, a local candidate module based on a regional relationship algorithm is used to generate multiple face local area images according to the initial key point set; each face local area image is input into a preset local expression feature extraction network for local feature extraction, and local expression feature data corresponding to each face local area image is output; global feature extraction is performed on the global face image through a global expression feature extraction network based on a sustained attention algorithm, and global expression feature data corresponding to the global face image is generated; finally, a preset The feature classifier performs facial expression recognition based on global expression feature data and each local expression feature data to generate a facial expression recognition result; based on the above scheme, the generated global face image and the initial key point set are processed by a local candidate module based on a regional relationship algorithm, a preset local expression feature extraction network, and a global expression feature extraction network based on a sustained attention algorithm to obtain global expression feature data and each local expression feature data, and then facial expression recognition is performed by a preset feature classifier to generate a facial expression recognition result. The present invention combines the regional relationship algorithm with the sustained attention algorithm, and has the ability to learn the relationship between features at different levels, thereby achieving granularity alignment of global and local expressions, thereby improving the accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 A flowchart of a facial expression recognition method provided in accordance with the first embodiment of the present invention;

[0073] Figure 2 A conventional arrangement diagram of facial key points provided in Example 1 of the present invention;

[0074] Figure 3 This is an example diagram of a two-dimensional face mask provided in Example 1 of the present invention;

[0075] Figure 4 A schematic diagram of the structure of a preset local expression feature extraction network provided in the first embodiment of the present invention;

[0076] Figure 5 A schematic diagram of the structure of a local enhancement module based on a continuous attention mechanism provided in the first embodiment of the present invention;

[0077] Figure 6 A workflow diagram of a preset local expression feature extraction network and a global expression feature extraction network based on a sustained attention algorithm provided in the first embodiment of the present invention;

[0078] Figure 7 A flowchart of the steps of the training process of the local enhancement module based on the continuous attention algorithm provided in the second embodiment of the present invention;

[0079] Figure 8 This is a structural block diagram of a facial expression recognition device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0080] The embodiments of the present invention provide a facial expression recognition method and apparatus, which are used to solve the technical problem of poor recognition accuracy caused by existing facial expression recognition methods.

[0081] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0082] See also Figure 1 , Figure 1 This is a flowchart of the steps of a facial expression recognition method provided in Example 1 of the present invention.

[0083] The present invention provides a method for recognizing facial expressions, comprising:

[0084] Step 101: Acquire a face image of a video frame to be detected, perform target detection on the face image of the video frame to be detected, and generate a global face image and an initial key point set corresponding to the global face image.

[0085] It should be noted that the complete video data is processed according to the fixed interval sampling method, with a fixed frame number interval a f To select frames from the video stream to form a sampling sequence frame I=(i0,i1,...,i n-1 ,i n ,...,i N ), that is, the face image of the video frame to be detected is obtained, thereby completing the conversion of the video into the original data in the form of frame image, which can simplify the data processing process and reduce the calculation cost; where i0 is the face image of the video frame to be detected in the first frame, i N is the face image of the video frame to be detected in the N+1th frame.

[0086] Furthermore, the process of performing target detection on the face image of the video frame to be detected and generating a global face image and an initial key point set corresponding to the global face image can be achieved by executing the following steps S21 to S23:

[0087] Step S21: Perform preliminary face detection and marking operations on the face image of the video frame to be detected to obtain an initial detected face image;

[0088] It should be noted that based on the OpenCV and Dlib library related algorithms and code calls, the face detection module is constructed to detect the current frame image i n (The face image of the video frame to be detected) performs preliminary face detection and marking operations to obtain an initial detected face image.

[0089] Step S22: performing image preprocessing on the initial detected face image to generate a global face image;

[0090] It should be noted that the initial detection face image is preprocessed. Specifically, the image preprocessing includes fixing the size of the initial detection face image, cropping, and aligning it, and finally normalizing it to a 3×244×244 RGB global face image F n .

[0091] Step S23: perform facial key point detection on the global face image, and output an initial key point set corresponding to the global face image.

[0092] The initial key point set includes multiple initial key points.

[0093] Please note that Figure 2 , for F n Execute as Figure 2, 68 key points are arranged as shown in the example diagram of the face key point detection, and F n The initial key point set X n , where each initial key point is represented as p k , k∈(0,…,67). For each initial key point p k , each has its corresponding key point data, the key point data includes: the key point in F n Two-dimensional pixel position information in 、 ; The confidence level during the key point detection process and confidence threshold , ∈[0,1], ∈(0,1);The feature detection support of this key point ∈[0,1].

[0094] It is worth mentioning that It indicates the degree of credibility or accuracy of the prediction result of the key point in the key point detection process. It is expressed in the form of probability. A confidence level of 1 means that the model is very sure that the prediction result is correct, and vice versa.

[0095] Furthermore, It is data set based on prior knowledge. The data setting is based on the local area where the key point is located, the marginal degree of the arrangement position, the masking tolerance, etc. Specifically, the more central the key point is in the key point group, the lower the confidence requirement and the lower the confidence threshold value is set; conversely, the more marginal the key point position is, the higher the masking tolerance is. In order to ensure the accuracy and simplicity of the local area, the confidence threshold value needs to be set higher.

[0096] Furthermore, It is data set based on prior knowledge. The data setting is based on whether the key point is the core key point of the local area. The greater the influence of the key point on the positioning and division of the local area, the more support it provides for subsequent feature acquisition, and the higher its support value is set, and vice versa.

[0097] In an embodiment of the present invention, a face image of a video frame to be detected is obtained, and target detection is performed on the face image of the video frame to be detected to generate a global face image and an initial key point set corresponding to the global face image.

[0098] Step 102: Generate multiple facial local area images based on the initial key point set using a local candidate module based on a region relationship algorithm.

[0099] It should be noted that, based on the aforementioned facial image and key point data specifications, a key point set of 68 and a mask image with a size of 224×224 were selected as quantification and visualization tools, and the conventional facial area division method, facial key point arrangement sequence, facial key texture information area, etc. were integrated as adjustable prior data.

[0100] Specifically, step 102 may include the following sub-steps S21-S29:

[0101] Step S21: performing local region coarse matching on each initial key point to generate multiple initial local region key point groups;

[0102] The initial local region key point group includes multiple initial local region key points.

[0103] The multiple initial local area key point groups include the left eye local area key point group, the right eye local area key point group, the left eyebrow local area key point group, the right eyebrow local area key point group, the nose local area key point group, the mouth local area key point group, and the facial contour local area key point group.

[0104] It should be noted that when the facial expression changes, the key texture areas where the features are mainly distributed are four areas: left eye, right eye, nose, and mouth. In order to focus on multiple local facial areas in a more specific and detailed manner and to amplify the differences between different local features, the present invention divides the face into seven local areas based on the above: left eye, right eye, left eyebrow, right eyebrow, nose, mouth, and face outline. Based on the X obtained in the above steps, n The key point arrangement sequence, position information, etc., all key points are roughly matched to the local area, such as p 0 ~p 16 Divide into face contour area, p 17 ~p 21 Divide into the left eyebrow area, etc.

[0105] It is worth mentioning that after completing the local area coarse matching of all initial key points, each initial key point is used as the corresponding initial local area key point, and according to the corresponding local area, the initial local area key point group corresponding to each local area is constructed based on the initial local area key point.

[0106] Step S22: calculating the correlation degree corresponding to each initial local area key point in each initial local area key point group based on the confidence degree, confidence threshold, and feature detection support degree corresponding to each initial local area key point in each initial local area key point group;

[0107] It should be noted that, since the local regions of each face image have relative position differences and regional size fluctuations, in order to effectively solve the problem of local region spatial position matching and segmentation size adaptation, the present invention also needs to be based on X n The key point arrangement sequence, location information, confidence and other data are used to perform more detailed and scientific quantitative calculations to improve the positioning and division of local areas.

[0108] Specifically, for X n All key points in the initial local area key point group are filtered for correlation, so as to remove facial key points with low confidence and low support, and reduce the impact of poor sampling and complex recognition. That is, according to the confidence, confidence threshold, and feature detection support corresponding to each initial local area key point in each initial local area key point group, the correlation corresponding to each initial local area key point is calculated. The calculation process of the correlation is:

[0109] ;

[0110] in, is the correlation degree corresponding to the kth initial local area key point; is the confidence corresponding to the kth initial local area key point; is the confidence threshold corresponding to the kth initial local area key point; is the feature detection support corresponding to the kth initial local area key point.

[0111] Step S23: In each initial local area key point group, any initial local area key point corresponding to a correlation degree less than a preset correlation degree threshold is removed to determine an intermediate local area key point group corresponding to each initial local area key point group;

[0112] It should be noted that setting a preset correlation threshold , to filter out the correlation less than The key points of the initial local area key points are obtained, thereby obtaining new key point groups of each area (intermediate local area key point groups), and obtaining a new key point data set Y composed of each intermediate local area key point group. n , where each intermediate local area key point is represented by q k , k∈(0,…,67), the process can be expressed as:

[0113] ;

[0114] in, is the kth middle local area key point; is the kth initial local area key point; is the correlation degree corresponding to the kth initial local area key point; The preset correlation threshold.

[0115] It is worth mentioning that the two-dimensional position information of the key points in each intermediate local area is inherited from 、 In , the feature detection support is inherited from middle.

[0116] Step S24: Based on the feature detection support corresponding to each intermediate local area key point in each intermediate local area key point group, the intermediate local area key point corresponding to the largest feature detection support is selected as the core local area key point in each intermediate local area key point group;

[0117] It should be noted that by going deep into each local area in turn and performing clustering tests on its key points, the key points that are too free are eliminated by measuring the distance between points and point groups, so as to make the key point group information in each local area more meaningful and significant.

[0118] Specifically, first obtain each area in Y n middle The highest key point As the core point of the region , that is, in each intermediate local area key point group, the intermediate local area key point corresponding to the largest feature detection support in each intermediate local area key point group is selected as the core local area key point; if there are multiple core key points, the pixel position mean point of these key points is taken as .

[0119] Step S25: performing weighted Mahalanobis distance calculations on the core local region key point and the multiple intermediate local region key points in each intermediate local region key point group, and determining the weighted Mahalanobis distance corresponding to each intermediate local region key point in each intermediate local region key point group;

[0120] It should be noted that the core key points selected based on the key point group of each intermediate local area , and calculate the distance and distribution characteristics of each key point in the area. Since the traditional Euclidean distance measures the direct distance between two points, it does not take into account the distribution characteristics of the data and cannot truly reflect the actual distance between a point and the overall distribution. Therefore, based on the calculation idea of ​​Mahalanobis distance, the present invention normalizes the covariance in the distance calculation and uses the prior correlation value between points and point groups as weighting. This can effectively avoid the risks caused by the different variances of data features due to the Euclidean distance.

[0121] Furthermore, all key points in each region are sequentially subjected to weighted Mahalanobis distance calculations to obtain the weighted Mahalanobis distance of each point, which is as follows:

[0122] ;

[0123] in, is the weighted Mahalanobis distance corresponding to the key point of the kth middle local area; is the kth middle local area key point; is the u-th core local area key point; T is the transpose; for same The covariance matrix of .

[0124] Step S26: In each intermediate local region key point group, any intermediate local region key point corresponding to a weighted Mahalanobis distance less than a preset Mahalanobis distance threshold is removed to determine a target local region key point group corresponding to each intermediate local region key point group;

[0125] It should be noted that a preset Mahalanobis distance threshold is set , filter out Less than The key points of the local area are clustered and filtered. The key point group in the local area is then updated, and the out-of-class key points are removed. The retained middle local area key points are used as the target local area key points. According to the corresponding local area, the target local area key point group corresponding to each local area is constructed based on the target local area key points. Thus, the key point groups C = (c0, ..., c6) of the seven areas are respectively collected, where c0 is the first target local area key point group.

[0126] Step S27: generating a two-dimensional facial mask matrix based on the pixel position information corresponding to each target local area key point in each target local area key point group;

[0127] Step S28: determining a mask image based on the two-dimensional facial mask matrix;

[0128] Step S29: Generate multiple facial local area images based on the two-dimensional facial mask matrix and the mask image.

[0129] Please note that Figure 3 , using the pixel position information of each point in the key point group C to obtain a two-dimensional facial mask matrix f M , and then by f M Generate a mask image F of size 224×224 M , that is, the mask matrix is ​​1, indicating that F M There is an activated key point at the corresponding pixel position. If it is 0, there is no key point activated here. Finally, F M The ROI surrounded by the key point group is closed and filled according to the local area clustering, and the visualization is as follows Figure 3Example diagram. In which, the mapping gets f M The formula is:

[0130] ;

[0131] in, is an element in the two-dimensional face mask matrix; is the kth target local area key point; is the key point group of the i-th target local area.

[0132] Furthermore, relying on the facial mask matrix f M and mask image F M etc., determine the location and size of each local area, and n Complete 7 face local area images F L The correction and cropping work is performed to obtain multiple local face area images, which include the left eye image, the right eye image, the left eyebrow image, the right eyebrow image, the nose image, the mouth image, and the facial contour image.

[0133] In an embodiment of the present invention, the local candidate module based on the regional relationship algorithm can flexibly select and divide key local areas of expression features, solving problems such as inaccurate local area positioning, poor robustness, and bloated calculations, thereby effectively improving the efficiency of capturing local area expression features and capturing feature information accurately and comprehensively.

[0134] Step 103: Input each facial local area image into a preset local expression feature extraction network to extract local features, and output local expression feature data corresponding to each facial local area image.

[0135] The preset local expression feature extraction network includes a convolutional pooling sub-network, three cascaded fully connected modules and a Softmax activation function layer.

[0136] It should be noted that, an expression feature acquisition network based on deep learning (pre-set local expression feature extraction network) is used to perform feature extraction on the multiple local facial area images obtained based on the above steps.

[0137] Further, see Figure 4 The preset local expression feature extraction network proposed in the present invention adopts an improved VGG16 network (Visual Geometry Group 16-layer network) architecture, such as Figure 4As shown. Compared with the traditional network architecture, the improved VGG16 uses three 3×3 convolution kernels to replace the large 7×7 or 5×5 convolution kernels. This ensures that the network has the same perceptual field, not only improving the network depth and the neural network feature expression effect, but also effectively reducing the number of convolution calculation parameters. Preset local expression feature extraction network. The improved VGG16 network is mainly composed of three basic structural layers: input layer, convolution layer, and pooling layer to realize the global expression feature extraction of face images. The specific configuration and structure are as follows: Figure 4 shown.

[0138] Specifically, the preset local expression feature extraction network consists of a convolutional pooling subnetwork, three cascaded fully connected modules and a Softmax activation function layer. The convolutional pooling subnetwork consists of five groups of convolutions and five maximum pooling layers. The five groups of convolutions include two groups of two serially connected 3×3 convolutional modules and three groups of three cascaded 3×3 convolutional modules. The convolutional module includes a convolutional layer and a ReLU activation function layer, and the fully connected module includes a fully connected layer and a ReLU activation function layer.

[0139] Specifically, step 103 may include the following sub-steps S31-S34:

[0140] Step S31: Adjust the image format of each facial local area image to determine the target facial local area image corresponding to each facial local area image;

[0141] Step S32: Using the convolutional pooling sub-network to perform convolutional pooling on each target face local area image, generating a local convolutional pooling feature map corresponding to each face local area image;

[0142] The convolutional pooling subnetwork consists of two serially connected 3×3 convolutional modules, a maximum pooling layer, and three cascaded 3×3 convolutional modules.

[0143] Furthermore, step S32 may include the following sub-steps: step S321 to step S3210:

[0144] Step S321: Use two serially connected 3×3 convolution modules to perform a convolution operation on each target face local area image, and output a first convolution feature map corresponding to each target face local area image;

[0145] Step S322: performing maximum pooling on each first convolutional feature map through a maximum pooling layer to generate a first maximum pooling feature map corresponding to each first convolutional feature map;

[0146] Step S323: Input each first maximum pooling feature map into two series-connected 3×3 convolution modules for convolution operation, and output a second convolution feature map corresponding to each first maximum pooling feature map;

[0147] Step S324: Perform maximum pooling on each second convolutional feature map using a maximum pooling layer to generate a second maximum pooling feature map corresponding to each second convolutional feature map;

[0148] Step S325: Perform a convolution operation on each second maximum pooling feature map through three cascaded 3×3 convolution modules, and output a third convolution feature map corresponding to each second maximum pooling feature map;

[0149] Step S326: Input each third convolutional feature map into the maximum pooling layer for maximum pooling, and output the third maximum pooling feature map corresponding to each third convolutional feature map;

[0150] Step S327: Use three cascaded 3×3 convolution modules to perform a convolution operation on each third maximum pooling feature map, and output a fourth convolution feature map corresponding to each third maximum pooling feature map;

[0151] Step S328: Input each fourth convolutional feature map into the maximum pooling layer for maximum pooling, and output the fourth maximum pooling feature map corresponding to each fourth convolutional feature map;

[0152] Step S329: Use three cascaded 3×3 convolution modules to perform a convolution operation on each fourth maximum pooling feature map, and output a fifth convolution feature map corresponding to each fourth maximum pooling feature map;

[0153] Step S3210: Input each fifth convolutional feature map into the maximum pooling layer for maximum pooling, and output the local convolutional pooling feature map corresponding to each fifth convolutional feature map.

[0154] Step S33: using each convolutional pooling feature map as the input of three cascaded fully connected modules, and outputting the fully connected feature map corresponding to each convolutional pooling feature map;

[0155] Step S34: Perform nonlinear mapping on each fully connected feature map through a Softmax activation function layer to generate local expression feature data corresponding to each fully connected feature map.

[0156] It should be noted that for the input local face area image , m∈(0,…,6), first needs to be adjusted to a 3×224×224 RGB image format to meet the requirements of the output layer. The improved VGG16 has 5 groups of convolutions, a total of 13 convolution layers. Each convolution module consists of a 3×3 convolution layer and a ReLU activation function layer. After the first convolution of each group, a batchnormalization (BN) layer is used to solve the problem of data distribution in the middle layer to prevent the gradient from exploding or disappearing, speed up the convergence of the network, and then input it to the ReLU activation function layer for processing. The input image is effectively extracted and filtered through each group of convolution operations and BN adjustments from shallow to deep. In addition, after each group of convolutions, the feature data will undergo a 2×2 maximum pooling operation to reduce the size of the feature map. After completing all the convolution and pooling operations in sequence, the extracted local expression feature data is obtained. , m∈(0,…,6).

[0157] In an embodiment of the present invention, each local face region image is input into a preset local expression feature extraction network for local feature extraction, and local expression feature data corresponding to each local face region image is output.

[0158] Step 104: extract global features from the global facial image using a global expression feature extraction network based on a sustained attention algorithm to generate global expression feature data corresponding to the global facial image.

[0159] The global expression feature extraction network based on the continuous attention algorithm includes a convolutional pooling sub-network and a local enhancement module based on the continuous attention algorithm.

[0160] Please note that Figure 5 Based on the improved VGG16 network proposed in the present invention, a local enhancement module based on the sustained attention algorithm is introduced to constitute a global expression feature extraction network with adaptive properties (global expression feature extraction network based on the sustained attention algorithm), wherein the local enhancement module based on the sustained attention algorithm includes a maximum pooling layer, a convolutional layer, an average pooling layer, and a Sigmoid activation function layer.

[0161] Specifically, step 104 may include the following sub-steps S41-S42:

[0162] Step S41: Use the convolution pooling sub-network to perform convolution pooling on the global face image to generate a global convolution pooling feature map;

[0163] It should be noted that the convolutional pooling sub-network's processing principle for the global face image is consistent with the above-mentioned convolutional pooling sub-network's processing principle for the target face local area image, and the present invention will not elaborate on it.

[0164] Step S42: extract features from the global convolution pooling feature map through a local enhancement module based on a sustained attention algorithm to generate global expression feature data corresponding to the global convolution pooling feature map.

[0165] Furthermore, step S42 may include the following sub-steps S421 to S427:

[0166] Step S421: performing maximum pooling on the global convolutional pooling feature map through a maximum pooling layer, and outputting a global maximum pooling feature map;

[0167] Step S422: Use the average pooling layer to perform average pooling on the global convolutional pooling feature map to generate a global average pooling feature map;

[0168] Step S423: splicing the global maximum pooling feature map and the global average pooling feature map to generate a spliced ​​feature map;

[0169] Step S424: input the spliced ​​feature map into the convolution layer to perform a convolution operation to generate a global convolution feature map;

[0170] Step S425: Use the Sigmoid activation function layer to perform nonlinear mapping on the global convolution feature map to generate an initial attention map;

[0171] It should be noted that the feature F (global convolution pooling feature map) of the local enhancement module based on the continuous attention algorithm is subjected to maximum pooling and average pooling operations respectively to obtain two single-channel feature maps. 、 , H is the height, W is the width, then the two feature maps are spliced ​​together, and then convolution is used to reduce them to one channel, and then the Sigmoid function is used for activation to generate the initial attention map , where the processing of the initial attention map can be expressed as:

[0172]

[0173] in, is the initial attention map; is the global maximum pooling feature map; is the global average pooling feature map; is average pooling; is the maximum pooling; is the Sigmoid activation function; To perform a convolution operation with a kernel size of 1×1.

[0174] Step S426: Generate a target attention map based on the initial attention map using a continuous attention mechanism;

[0175] Step S427: perform matrix element-by-element multiplication on the target attention map and the global convolution pooling feature map, and output the global expression feature data corresponding to the global convolution pooling feature map.

[0176] It should be noted that there is often correlation between feature data. By modeling such correlation, a more accurate attention map can be obtained without introducing additional supervision. The present invention adjusts the generation of the attention map of the current frame image by introducing the attention map of the previous sequence frame image as prior knowledge, thereby establishing an attention mechanism (continuous attention mechanism) that associates adjacent frames. By associating the attention mechanism between adjacent sequence frames, the local enhancement module can make the global image gradually and robustly focus on more critical and subtle expression change areas during the feature extraction process, rather than focusing on different areas of different sequence frames. In the local enhancement module of the present invention, the formula for the attention mechanism between adjacent frames is, that is, the processing process of the target attention map can be expressed as:

[0177]

[0178] in, is the final attention map of the nth sequence frame, that is, the target attention map corresponding to the face image of the nth video frame to be detected; is the initial attention map of the nth sequence frame, that is, the initial attention map corresponding to the face image of the nth video frame to be detected; is the maximum pooling; is the target attention map corresponding to the face image of the n-1th video frame to be detected; Element-wise multiplication of matrices.

[0179] Furthermore, after the last set of convolutional and pooling layer operations, the feature F is obtained, and then F is passed through the local enhancement module to finally obtain the output feature, that is, the processing process of the global expression feature data, which can be expressed as:

[0180]

[0181] in, is the global expression feature data; is the final attention map of the nth sequence frame, that is, the target attention map corresponding to the face image of the nth video frame to be detected; F is the global convolution pooling feature map.

[0182] In an embodiment of the present invention, global features of a global facial image are extracted through a global expression feature extraction network based on a sustained attention algorithm to generate global expression feature data corresponding to the global facial image.

[0183] Step 105: Using a preset feature classifier to perform facial expression recognition based on the global expression feature data and each local expression feature data, and generating a facial expression recognition result.

[0184] Specifically, step 105 may include the following sub-steps S51-S52:

[0185] Step S51: performing early linear fusion processing on the global expression feature data and each local expression feature data to generate multi-scale features;

[0186] Step S52: using a preset feature classifier to perform recognition based on multi-scale features to generate a facial expression recognition result.

[0187] It should be noted that all local and global features captured by the two feature extraction branches are fed into the facial feature database. Subsequently, the global and local features undergo early linear fusion processing to achieve multi-scale feature concatenation. These features are then fed into the feature classifier for feature recognition, resulting in facial expression recognition results.

[0188] For example, see Figure 6 First, the video data is framed to obtain the original image data of the FER model (pre-set local expression feature extraction network, global expression feature extraction network based on continuous attention algorithm), that is, the face image of the video frame to be detected. Then, face detection and key point detection are performed on the current frame image (the face image of the video frame to be detected), and the local candidate module is used to divide the local expression area. At the same time, local expression feature extraction is performed based on deep learning, that is, the image of each local face area is input into the preset local expression feature extraction network for local feature extraction, as well as global expression feature extraction based on the attention algorithm, and the various extracted feature information is integrated to obtain the facial expression recognition result.

[0189] In the embodiment of the present invention, a preset feature classifier is used to perform facial expression recognition based on global expression feature data and each local expression feature data to generate a facial expression recognition result.

[0190] For comparison purposes, existing technologies can be used as a reference. To better analyze FER model features and achieve optimal FER performance, comprehensive and accurate FER model features must be captured. Currently, most FER feature extraction methods directly extract global features from complete video frames, which contain extensive facial information. To balance the capture of both global and detailed local features, some methods employ modules and algorithms for optimization. For example, attention mechanisms are added to focus on key facial regions, shifting from the entire face to key expression areas, extracting recognition features in these key facial regions. Furthermore, facial anchor points are used to enhance the face before inputting it into a network to obtain multi-layered facial features. Existing technologies have proposed an FER recognition model based on the fusion of local and global features. Its main model framework consists of two branches: a local detail branch and a global keypoint branch. The global keypoint branch first detects keypoints across the entire image and generates a corresponding facial mask. This is then used in the feature extraction network to help highlight key region data. The local detail branch selects fixed key regions, such as the eyes and mouth, and then feeds each cropped image into the feature extraction network for feature extraction.

[0191] Furthermore, the FER model used in the aforementioned techniques lacks a mechanism for determining the relationship between facial regions, and can only perform fixed localization and segmentation of the global and local images. This results in low regional localization accuracy, an inability to handle complex situations, and lengthy computational times. This in turn leads to inefficient, information-lacking, and inaccurate feature data extraction from each region. This is the primary technical issue addressed by this patent. Furthermore, the FER model used in the aforementioned techniques lacks a mechanism for sustained attention to key local regions of facial expression, making it unable to adaptively adjust the relocation and weighting of key regions during the feature extraction phase. Consequently, the feature extraction process suffers from issues such as ignoring local details, failing to suppress irrelevant feature information, and lacking the ability to continuously optimize the overall extracted features.

[0192] The present invention addresses the problems of existing FER models in the feature capture process, such as an inability to effectively utilize the relationship between local and global regions, large computational parameters, and a lack of a mechanism for sustained attention to key areas of expression. The present invention proposes a facial expression recognition method based on a hybrid intelligent algorithm combining a regional relationship algorithm and a sustained attention algorithm, and applies this algorithm to the FER model for expression feature capture. The present invention leverages the regional relationship algorithm's ability to learn relationships between features at different levels, using facial detection key points as a bridge to achieve granular alignment of global and local expressions, facial ROI region positioning, and local expression region cropping. Furthermore, during the global expression feature acquisition phase, the present invention uses a sustained attention algorithm and attention map design to correlate key area information of facial images between adjacent frames. This allows the FER model to obtain an accurate attention map for the current frame without introducing additional supervision, thereby robustly extracting key feature data with the most significant characteristics.

[0193] In summary, the present invention's preset local expression feature extraction network and the global expression feature extraction network based on the sustained attention algorithm can capture facial expression features continuously and quickly locate key expression areas, reduce model computational complexity, and adaptively extract more discriminative expression feature data. Furthermore, the present invention combines the regional relationship algorithm with the sustained attention algorithm, leveraging the advantages of both regional relationship integration and sustained attention calibration, enabling the FER model to more effectively and accurately capture expression features.

[0194] In an embodiment of the present invention, the present invention provides a facial expression recognition method, which first obtains a face image of a video frame to be detected, performs target detection on the face image of the video frame to be detected, and generates a global face image and an initial key point set corresponding to the global face image; then, a local candidate module based on a regional relationship algorithm is used to generate multiple face local area images according to the initial key point set; each face local area image is input into a preset local expression feature extraction network for local feature extraction, and local expression feature data corresponding to each face local area image is output; global feature extraction is performed on the global face image through a global expression feature extraction network based on a sustained attention algorithm, and global expression feature data corresponding to the global face image is generated; finally, a preset local expression feature extraction network is used to extract the local features of the global face image, and global expression feature data corresponding to the global face image is generated; A feature classifier is set to perform facial expression recognition based on global expression feature data and each local expression feature data to generate a facial expression recognition result; based on the above scheme, the generated global face image and the initial key point set are processed by a local candidate module based on a regional relationship algorithm, a preset local expression feature extraction network, and a global expression feature extraction network based on a sustained attention algorithm to obtain global expression feature data and each local expression feature data, and then facial expression recognition is performed by the preset feature classifier to generate a facial expression recognition result. The present invention combines the regional relationship algorithm with the sustained attention algorithm, and is capable of learning the relationship between features at different levels, thereby achieving granularity alignment of global and local expressions, thereby improving the accuracy of recognition.

[0195] For better explanation, refer to Figure 7 , shows a flowchart of the steps of the training process of the local enhancement module based on the continuous attention algorithm provided by the second embodiment of the present invention, which may include the following steps:

[0196] Step 701: Obtain a face image of a video frame to be trained, and generate a two-dimensional facial mask matrix to be trained and a global maximum pooling feature map to be trained based on the face image of the video frame to be trained;

[0197] Step 702: using an initial local enhancement module based on a continuous attention algorithm to generate an initial attention map to be trained according to the global maximum pooling feature map to be trained;

[0198] Step 703: Calculate a target loss value using a preset regression loss function based on the initial attention map to be trained and the two-dimensional facial mask matrix to be trained;

[0199] Step 704: Update the network parameters of the initial local enhancement module based on the sustained attention algorithm based on the target loss value, determine the intermediate local enhancement module based on the sustained attention algorithm, and count the number of iterations in real time;

[0200] Step 705: Determine whether the number of iterations reaches the preset number of updates;

[0201] Step 706: If the target is reached, the intermediate local enhancement module based on the sustained attention algorithm is used as the trained local enhancement module based on the sustained attention algorithm.

[0202] It should be noted that the specific configuration of the convolutional pooling subnetwork in the global expression feature extraction network based on the sustained attention algorithm proposed in the present invention has been completed, so the present invention only performs model training on the local enhancement module based on the sustained attention algorithm. Among them, the two-dimensional facial mask matrix to be trained is obtained by processing the initial key point set corresponding to the facial image of the training video frame through the local candidate module based on the region relationship algorithm, and the global maximum pooling feature map to be trained is obtained by processing the global facial image corresponding to the facial image of the training video frame through the convolutional pooling subnetwork.

[0203] Furthermore, in order to transform the initial attention map Focusing on the unobstructed and key local expression areas of the face, the two-dimensional mask matrix f is generated by the local candidate module based on the regional relationship algorithm. M , and define a regression loss function such that More suitable for f M , thereby achieving adaptive enhancement adjustment of the attention map. Among them, the preset regression loss function formula is:

[0204] ;

[0205] in, is the target loss value; is the initial attention map to be trained; is the two-dimensional facial mask matrix to be trained; N is the number of samples; Element-wise multiplication of matrices.

[0206] It is worth mentioning that due to the M In the , the parameters of the masked and non-critical areas are 0, and the regression loss function can be used to adjust the weight differences of such areas to suppress the extraction and collection of non-critical features.

[0207] Furthermore, if the number of iterations does not reach the preset number of updates, the intermediate local enhancement module based on the continuous attention algorithm will be used as a new initial local enhancement module based on the continuous attention algorithm, and the process will jump to step 702 until the number of iterations reaches the preset number of updates, and the intermediate local enhancement module based on the continuous attention algorithm determined when the number of iterations reaches the preset number of updates will be used as a trained local enhancement module based on the continuous attention algorithm.

[0208] In an embodiment of the present invention, the local enhancement module based on the continuous attention algorithm can realize continuous updating and weight distribution of key areas in the feature extraction stage of the global feature area of ​​facial expressions, effectively solving the problems of missing information in key areas and unable to suppress information in non-key areas, and realizing the continuous optimization function of feature extraction work under unsupervised conditions.

[0209] See also Figure 8 , Figure 8 This is a structural block diagram of a facial expression recognition device provided in Example 3 of the present invention.

[0210] The present invention provides a facial expression recognition device, comprising:

[0211] An acquisition module 801 is configured to acquire a face image of a video frame to be detected, perform target detection on the face image of the video frame to be detected, and generate a global face image and an initial key point set corresponding to the global face image;

[0212] A generating module 802 is configured to generate a plurality of facial local region images according to an initial key point set using a local candidate module based on a region relationship algorithm;

[0213] The local feature extraction module 803 is used to input each facial local area image into a preset local expression feature extraction network to extract local features and output local expression feature data corresponding to each facial local area image;

[0214] A global feature extraction module 804 is configured to extract global features from a global facial image using a global expression feature extraction network based on a sustained attention algorithm, thereby generating global expression feature data corresponding to the global facial image.

[0215] The recognition module 805 is used to use a preset feature classifier to perform facial expression recognition based on the global expression feature data and each local expression feature data to generate a facial expression recognition result.

[0216] Furthermore, the acquisition module 801 is specifically configured to:

[0217] Perform preliminary face detection and marking operations on the face image of the video frame to be detected to obtain an initial detected face image;

[0218] Perform image preprocessing on the initial detected face image to generate a global face image;

[0219] Perform facial key point detection on the global face image and output the initial key point set corresponding to the global face image.

[0220] Furthermore, the initial key point set includes a plurality of initial key points; the generating module 802 is specifically configured to:

[0221] Perform local area coarse matching on each initial key point to generate multiple initial local area key point groups;

[0222] Calculate the correlation degree corresponding to each initial local area key point in each initial local area key point group based on the confidence, confidence threshold, and feature detection support corresponding to each initial local area key point in each initial local area key point group;

[0223] In each initial local area key point group, any initial local area key point corresponding to a correlation degree less than a preset correlation degree threshold is eliminated to determine an intermediate local area key point group corresponding to each initial local area key point group;

[0224] Based on the feature detection support corresponding to each intermediate local area key point in each intermediate local area key point group, the intermediate local area key point corresponding to the largest feature detection support is selected as the core local area key point in each intermediate local area key point group;

[0225] Performing weighted Mahalanobis distance calculations on the core local region key point and multiple intermediate local region key points in each intermediate local region key point group, respectively, to determine the weighted Mahalanobis distance corresponding to each intermediate local region key point in each intermediate local region key point group;

[0226] In each intermediate local area key point group, any intermediate local area key point corresponding to a weighted Mahalanobis distance less than a preset Mahalanobis distance threshold is removed to determine the target local area key point group corresponding to each intermediate local area key point group;

[0227] Generate a two-dimensional facial mask matrix according to the pixel position information corresponding to each target local area key point in each target local area key point group;

[0228] Determine a mask image based on a two-dimensional facial mask matrix;

[0229] According to the two-dimensional facial mask matrix and the mask image, multiple facial local area images are generated.

[0230] Furthermore, the preset local expression feature extraction network includes a convolutional pooling subnetwork, three cascaded fully connected modules and a Softmax activation function layer; the local feature extraction module 803 includes:

[0231] The first submodule is used to adjust the image format of each facial local area image and determine the target facial local area image corresponding to each facial local area image;

[0232] The second submodule is used to use the convolution pooling subnetwork to perform convolution pooling on each target face local area image respectively, and generate a local convolution pooling feature map corresponding to each face local area image;

[0233] The third submodule is used to use each convolutional pooling feature map as the input of three cascaded fully connected modules, and output the fully connected feature map corresponding to each convolutional pooling feature map;

[0234] The fourth submodule is used to perform nonlinear mapping on each fully connected feature map through the Softmax activation function layer to generate local expression feature data corresponding to each fully connected feature map.

[0235] Furthermore, the convolutional pooling subnetwork includes two serially connected 3×3 convolutional modules, a maximum pooling layer, and three cascaded 3×3 convolutional modules; the second submodule is specifically used to:

[0236] Use two series-connected 3×3 convolution modules to perform convolution operations on each target face local area image, and output the first convolution feature map corresponding to each target face local area image;

[0237] Performing maximum pooling on each first convolutional feature map through a maximum pooling layer to generate a first maximum pooling feature map corresponding to each first convolutional feature map;

[0238] Input each first maximum pooling feature map into two series-connected 3×3 convolution modules for convolution operation, and output the second convolution feature map corresponding to each first maximum pooling feature map;

[0239] Performing maximum pooling on each second convolutional feature map using a maximum pooling layer to generate a second maximum pooling feature map corresponding to each second convolutional feature map;

[0240] Perform convolution operations on each second maximum pooling feature map through three cascaded 3×3 convolution modules, and output the third convolution feature map corresponding to each second maximum pooling feature map;

[0241] Input each third convolutional feature map into the maximum pooling layer for maximum pooling, and output the third maximum pooling feature map corresponding to each third convolutional feature map;

[0242] Use three cascaded 3×3 convolution modules to perform convolution operations on each third maximum pooling feature map, and output the fourth convolution feature map corresponding to each third maximum pooling feature map;

[0243] Input each fourth convolutional feature map into the maximum pooling layer for maximum pooling, and output the fourth maximum pooling feature map corresponding to each fourth convolutional feature map;

[0244] Use three cascaded 3×3 convolution modules to perform convolution operations on each fourth maximum pooling feature map, and output the fifth convolution feature map corresponding to each fourth maximum pooling feature map;

[0245] Each fifth convolutional feature map is input into the maximum pooling layer for maximum pooling, and the local convolutional pooling feature map corresponding to each fifth convolutional feature map is output.

[0246] Furthermore, the global expression feature extraction network based on the sustained attention algorithm includes a convolutional pooling subnetwork and a local enhancement module based on the sustained attention algorithm; the global feature extraction module 804 includes:

[0247] The fifth submodule is used to perform convolution pooling on the global face image using the convolution pooling subnetwork to generate a global convolution pooling feature map;

[0248] The sixth submodule is used to extract features from the global convolution pooling feature map through a local enhancement module based on the continuous attention algorithm to generate global expression feature data corresponding to the global convolution pooling feature map.

[0249] Furthermore, the local enhancement module based on the continuous attention algorithm includes a maximum pooling layer, a convolutional layer, an average pooling layer, and a Sigmoid activation function layer; the sixth submodule is specifically used to:

[0250] Perform maximum pooling on the global convolution pooling feature map through the maximum pooling layer, and output the global maximum pooling feature map;

[0251] Use the average pooling layer to average pool the global convolution pooling feature map to generate a global average pooling feature map;

[0252] Splicing the global maximum pooling feature map and the global average pooling feature map to generate a spliced ​​feature map;

[0253] Input the concatenated feature map into the convolution layer to perform convolution operation and generate a global convolution feature map;

[0254] The Sigmoid activation function layer is used to perform nonlinear mapping on the global convolution feature map to generate the initial attention map;

[0255] Use the continuous attention mechanism to generate the target attention map based on the initial attention map;

[0256] Perform matrix element-by-element multiplication on the target attention map and the global convolution pooling feature map, and output the global expression feature data corresponding to the global convolution pooling feature map.

[0257] Furthermore, the identification module 805 is specifically configured to:

[0258] Perform early linear fusion processing on global expression feature data and local expression feature data to generate multi-scale features;

[0259] The preset feature classifier is used to perform recognition based on multi-scale features to generate facial expression recognition results.

[0260] In an optional embodiment of the device, the device further comprises:

[0261] The first module is used to obtain the face image of the video frame to be trained, and generate the two-dimensional facial mask matrix to be trained and the global maximum pooling feature map to be trained based on the face image of the video frame to be trained;

[0262] The second module is used to generate an initial attention map to be trained according to the global maximum pooling feature map to be trained using an initial local enhancement module based on a continuous attention algorithm;

[0263] The third module is used to calculate the target loss value according to the initial attention map to be trained and the two-dimensional facial mask matrix to be trained using a preset regression loss function;

[0264] The fourth module is used to update the network parameters of the initial local enhancement module based on the continuous attention algorithm based on the target loss value, determine the intermediate local enhancement module based on the continuous attention algorithm, and count the number of iterations in real time;

[0265] The fifth module is used to determine whether the number of iterations reaches the preset number of updates;

[0266] The sixth module is used to use the intermediate local enhancement module based on the continuous attention algorithm as the trained local enhancement module based on the continuous attention algorithm if it is achieved.

[0267] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and sub-modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0268] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A facial expression recognition method, characterized in that: include: Acquire a face image of a video frame to be detected, perform target detection on the face image of the video frame to be detected, and generate a global face image and an initial key point set corresponding to the global face image; Generate multiple facial local area images based on the initial key point set using a local candidate module based on a region relationship algorithm; Inputting each of the local face area images into a preset local expression feature extraction network to extract local features, and outputting local expression feature data corresponding to each of the local face area images; Performing global feature extraction on the global facial image through a global expression feature extraction network based on a sustained attention algorithm to generate global expression feature data corresponding to the global facial image; Using a preset feature classifier to perform facial expression recognition based on the global expression feature data and each of the local expression feature data to generate a facial expression recognition result; The initial key point set includes a plurality of initial key points; The local candidate module based on the region relationship algorithm generates a plurality of local face region images according to the initial key point set, including: Performing local area coarse matching on each of the initial key points to generate multiple initial local area key point groups; Calculating the correlation degree corresponding to each initial local area key point in each initial local area key point group based on the confidence, confidence threshold, and feature detection support corresponding to each initial local area key point in each initial local area key point group; In each of the initial local area key point groups, any initial local area key point corresponding to a correlation degree less than a preset correlation degree threshold is eliminated to determine an intermediate local area key point group corresponding to each of the initial local area key point groups; Based on the feature detection support corresponding to each intermediate local area key point in each intermediate local area key point group, the intermediate local area key point corresponding to the largest feature detection support is selected as the core local area key point in each intermediate local area key point group; performing weighted Mahalanobis distance calculations on the core local area key point and the multiple intermediate local area key points in each of the intermediate local area key point groups, respectively, to determine the weighted Mahalanobis distance corresponding to each intermediate local area key point in each of the intermediate local area key point groups; In each of the intermediate local area key point groups, any intermediate local area key point corresponding to a weighted Mahalanobis distance less than a preset Mahalanobis distance threshold is removed to determine a target local area key point group corresponding to each of the intermediate local area key point groups; generating a two-dimensional facial mask matrix according to pixel position information corresponding to each target local area key point in each target local area key point group; Determining a mask image based on the two-dimensional facial mask matrix; generating a plurality of local face area images according to the two-dimensional face mask matrix and the mask image; The global expression feature extraction network based on the sustained attention algorithm includes a convolutional pooling subnetwork and a local enhancement module based on the sustained attention algorithm; the global expression feature extraction network based on the sustained attention algorithm performs global feature extraction on the global face image to generate global expression feature data corresponding to the global face image, including: Performing convolution pooling on the global face image using a convolution pooling subnetwork to generate a global convolution pooling feature map; The global convolution pooling feature map is subjected to feature extraction through a local enhancement module based on a continuous attention algorithm to generate global expression feature data corresponding to the global convolution pooling feature map.

2. The facial expression recognition method according to claim 1, wherein The performing target detection on the face image of the video frame to be detected to generate a global face image and an initial key point set corresponding to the global face image includes: Performing preliminary face detection and marking operations on the face image of the video frame to be detected to obtain an initial detected face image; Performing image preprocessing on the initially detected face image to generate a global face image; Perform facial key point detection on the global face image, and output an initial key point set corresponding to the global face image.

3. The facial expression recognition method according to claim 1, wherein The preset local expression feature extraction network includes a convolutional pooling subnetwork, three cascaded fully connected modules and a Softmax activation function layer; the local facial region images are input into the preset local expression feature extraction network for local feature extraction, and the local expression feature data corresponding to the local facial region images are output, including: Adjusting the image format of each of the facial local area images to determine a target facial local area image corresponding to each of the facial local area images; Using a convolutional pooling subnetwork to perform convolutional pooling on each of the target face local area images, respectively, to generate a local convolutional pooling feature map corresponding to each of the face local area images; Using each of the convolutional pooling feature maps as the input of three cascaded fully connected modules, and outputting the fully connected feature maps corresponding to each of the convolutional pooling feature maps; Nonlinear mapping is performed on each of the fully connected feature maps through a Softmax activation function layer to generate local expression feature data corresponding to each of the fully connected feature maps.

4. The facial expression recognition method according to claim 3, wherein: The convolutional pooling subnetwork includes two serially connected 3×3 convolutional modules, a maximum pooling layer, and three cascaded 3×3 convolutional modules; the convolutional pooling subnetwork is used to perform convolutional pooling on each of the target face local area images to generate a local convolutional pooling feature map corresponding to each of the face local area images, including: Using two serially connected 3×3 convolution modules to perform a convolution operation on each of the target face local area images, and outputting a first convolution feature map corresponding to each of the target face local area images; Performing maximum pooling on each of the first convolutional feature maps through a maximum pooling layer to generate a first maximum pooling feature map corresponding to each of the first convolutional feature maps; Input each of the first maximum pooling feature maps into two series-connected 3×3 convolution modules for convolution operation, and output a second convolution feature map corresponding to each of the first maximum pooling feature maps; Performing maximum pooling on each of the second convolutional feature maps using a maximum pooling layer to generate a second maximum pooling feature map corresponding to each of the second convolutional feature maps; Performing a convolution operation on each of the second maximum pooling feature maps through three cascaded 3×3 convolution modules, and outputting a third convolution feature map corresponding to each of the second maximum pooling feature maps; Inputting each of the third convolutional feature maps into a maximum pooling layer for maximum pooling, and outputting a third maximum pooling feature map corresponding to each of the third convolutional feature maps; Using three cascaded 3×3 convolution modules to perform a convolution operation on each of the third maximum pooling feature maps, and outputting a fourth convolution feature map corresponding to each of the third maximum pooling feature maps; Inputting each of the fourth convolutional feature maps into a maximum pooling layer for maximum pooling, and outputting a fourth maximum pooling feature map corresponding to each of the fourth convolutional feature maps; Using three cascaded 3×3 convolution modules to perform a convolution operation on each of the fourth maximum pooling feature maps, and outputting a fifth convolution feature map corresponding to each of the fourth maximum pooling feature maps; Input each of the fifth convolutional feature maps into the maximum pooling layer for maximum pooling, and output the local convolutional pooling feature map corresponding to each of the fifth convolutional feature maps.

5. The facial expression recognition method according to claim 1, wherein The local enhancement module based on the sustained attention algorithm includes a maximum pooling layer, a convolution layer, an average pooling layer, and a Sigmoid activation function layer; the local enhancement module based on the sustained attention algorithm performs feature extraction on the global convolution pooling feature map to generate global expression feature data corresponding to the global convolution pooling feature map, including: Performing maximum pooling on the global convolutional pooling feature map through a maximum pooling layer, and outputting a global maximum pooling feature map; Performing average pooling on the global convolutional pooling feature map using an average pooling layer to generate a global average pooling feature map; Splicing the global maximum pooling feature map and the global average pooling feature map to generate a spliced ​​feature map; Inputting the spliced ​​feature map into the convolution layer to perform a convolution operation to generate a global convolution feature map; A Sigmoid activation function layer is used to perform nonlinear mapping on the global convolution feature map to generate an initial attention map; Using a continuous attention mechanism to generate a target attention map based on the initial attention map; Perform matrix element-by-element multiplication on the target attention map and the global convolution pooling feature map, and output the global expression feature data corresponding to the global convolution pooling feature map.

6. The facial expression recognition method according to claim 1, wherein The method of using a preset feature classifier to perform facial expression recognition based on the global expression feature data and each of the local expression feature data to generate a facial expression recognition result includes: Performing early linear fusion processing on the global expression feature data and each of the local expression feature data to generate multi-scale features; A preset feature classifier is used to perform recognition based on the multi-scale features to generate a facial expression recognition result.

7. The facial expression recognition method according to claim 5, wherein: The training process of the local enhancement module based on the continuous attention algorithm includes: Obtaining a face image of a video frame to be trained, and generating a two-dimensional facial mask matrix to be trained and a global maximum pooling feature map to be trained based on the face image of the video frame to be trained; An initial local enhancement module based on a continuous attention algorithm is used to generate an initial attention map to be trained according to the global maximum pooling feature map to be trained; Calculating a target loss value based on the initial attention map to be trained and the two-dimensional facial mask matrix to be trained using a preset regression loss function; Based on the target loss value, the network parameters of the initial local enhancement module based on the continuous attention algorithm are updated, the intermediate local enhancement module based on the continuous attention algorithm is determined, and the number of iterations is counted in real time; Determine whether the number of iterations reaches a preset number of updates; If achieved, the intermediate local enhancement module based on the continuous attention algorithm is used as the trained local enhancement module based on the continuous attention algorithm.

8. A facial expression recognition device, applied to the facial expression recognition method according to claim 1, characterized in that: include: An acquisition module is used to acquire a face image of a video frame to be detected, and perform target detection on the face image of the video frame to be detected, to generate a global face image and an initial key point set corresponding to the global face image; A generating module, configured to generate a plurality of facial local area images according to the initial key point set by using a local candidate module based on a region relationship algorithm; A local feature extraction module is used to input each of the local face area images into a preset local expression feature extraction network to extract local features and output local expression feature data corresponding to each of the local face area images; A global feature extraction module is used to extract global features from the global facial image using a global expression feature extraction network based on a sustained attention algorithm, and generate global expression feature data corresponding to the global facial image; The recognition module is used to use a preset feature classifier to perform facial expression recognition based on the global expression feature data and each local expression feature data to generate a facial expression recognition result.

Images