Facial expression recognition method and device

Abstract

The invention discloses a facial expression recognition method and a facial expression recognition device, which are used for solving the technical problem of poor recognition accuracy caused by the existing facial expression recognition method. The method comprises the following steps: acquiring a to-be-detected video frame face image, and generating a global face image and an initial key point set based on the to-be-detected video frame face image; generating a plurality of face local area images according to the initial key point set by adopting a local candidate module based on an area relation algorithm; inputting each face local area image into a preset local expression feature extraction network for local feature extraction, and outputting multiple pieces of local expression feature data; performing global feature extraction on the global face image through a global expression feature extraction network based on a continuous attention algorithm, and generating global expression feature data corresponding to the global face image; and a preset feature classifier is adopted to perform facial expression recognition according to the global expression feature data and the local expression feature data, and a facial expression recognition result is generated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for recognizing facial expressions. Background Art

[0002] Facial Expression Recognition (FER) technology has attracted widespread attention in related fields in recent years, mainly due to its potential in enhancing human-computer interaction. The core task of FER technology is to enable computers to obtain and identify facial expression categories from static images or video sequences. Currently, FER work mainly relies on deep learning-based neural network models, which perform well in video stream FER tasks due to their excellent feature learning and acquisition capabilities.

[0003] The basic facial expressions of a person are usually divided into eight categories: anger, contempt, disgust, fear, happy, neutral, sadness, and surprise. Accurate recognition of facial expressions helps computers understand and respond to the user's emotional state, thereby significantly improving the human-computer interaction experience. This capability will help build more natural and effective interaction systems and promote the application of technology in areas such as social robots, emotional computing, and intelligent surveillance.

[0004] The existing facial expression recognition methods propose a FER recognition model based on the fusion of local and global features. Its main model framework includes two branches, namely the local detail branch and the global key point branch. For the global key point branch, the key points of the entire image are first detected to generate relevant facial masks, and then the key area data information is assisted in highlighting in the feature extraction network; for the local detail branch, the eyes and mouth are selected as fixed key areas, and each cropped image is input into the feature extraction network for feature extraction, thereby completing facial expression recognition. However, due to the lack of a relationship mechanism between facial regions, the FER model can only perform fixed positioning and division of global images and local images, resulting in poor recognition accuracy. Summary of the invention

[0005] The present invention provides a facial expression recognition method and device, which are used to solve the technical problem that the existing facial expression recognition method leads to poor recognition accuracy.

[0006] A first aspect of the present invention provides a method for recognizing facial expressions, comprising:

[0007] Acquire a face image of a video frame to be detected, and perform target detection on the face image of the video frame to be detected to generate a global face image and an initial key point set corresponding to the global face image;

[0008] Generate multiple local face region images according to the initial key point set using a local candidate module based on a region relationship algorithm;

[0009] Inputting each of the local face area images into a preset local expression feature extraction network to extract local features, and outputting local expression feature data corresponding to each of the local face area images;

[0010] Performing global feature extraction on the global face image through a global expression feature extraction network based on a sustained attention algorithm to generate global expression feature data corresponding to the global face image;

[0011] A preset feature classifier is used to perform facial expression recognition based on the global expression feature data and each of the local expression feature data to generate a facial expression recognition result.

[0012] Optionally, the performing target detection on the face image of the to-be-detected video frame to generate a global face image and an initial key point set corresponding to the global face image includes:

[0013] Performing preliminary face detection and marking operations on the face image of the video frame to be detected to obtain an initial detected face image;

[0014] Performing image preprocessing on the initial detected face image to generate a global face image;

[0015] Perform facial key point detection on the global face image, and output an initial key point set corresponding to the global face image.

[0016] Optionally, the initial key point set includes a plurality of initial key points; the local candidate module based on the region relationship algorithm is used to generate a plurality of local face region images according to the initial key point set, including:

[0017] Performing local area rough matching on each of the initial key points to generate multiple initial local area key point groups;

[0018] Calculate the correlation degree corresponding to each initial local area key point in each initial local area key point group based on the confidence, confidence threshold, and feature detection support corresponding to each initial local area key point in each initial local area key point group;

[0019] In each of the initial local area key point groups, any initial local area key point corresponding to a correlation degree less than a preset correlation degree threshold is eliminated to determine an intermediate local area key point group corresponding to each of the initial local area key point groups;

[0020] Based on the feature detection support corresponding to each intermediate local area key point in each intermediate local area key point group, in each intermediate local area key point group, the intermediate local area key point corresponding to the largest feature detection support is selected as the core local area key point;

[0021] Performing weighted Mahalanobis distance calculations on the core local area key points and the multiple intermediate local area key points in each of the intermediate local area key point groups, respectively, to determine the weighted Mahalanobis distances corresponding to each of the intermediate local area key points in each of the intermediate local area key point groups;

[0022] In each of the intermediate local area key point groups, any intermediate local area key point corresponding to a weighted Mahalanobis distance less than a preset Mahalanobis distance threshold is eliminated to determine a target local area key point group corresponding to each of the intermediate local area key point groups;

[0023] Generate a two-dimensional facial mask matrix according to pixel position information corresponding to each target local area key point in each target local area key point group;

[0024] Determining a mask image based on the two-dimensional facial mask matrix;

[0025] A plurality of local face area images are generated according to the two-dimensional face mask matrix and the mask image.

[0026] Optionally, the preset local expression feature extraction network includes a convolutional pooling subnetwork, three cascaded fully connected modules and a Softmax activation function layer; the inputting each of the local face area images into the preset local expression feature extraction network for local feature extraction, and outputting local expression feature data corresponding to each of the local face area images, includes:

[0027] Adjusting the image format of each of the local face area images to determine a target local face area image corresponding to each of the local face area images;

[0028] Using a convolutional pooling subnetwork to perform convolutional pooling on each of the target face local area images, respectively, to generate a local convolutional pooling feature map corresponding to each of the face local area images;

[0029] Using each of the convolutional pooling feature maps as the input of three cascaded fully connected modules, and outputting the fully connected feature maps corresponding to each of the convolutional pooling feature maps;

[0030] Nonlinear mapping is performed on each of the fully connected feature maps through a Softmax activation function layer to generate local expression feature data corresponding to each of the fully connected feature maps.

[0031] Optionally, the convolutional pooling subnetwork includes two serially connected 3×3 convolutional modules, a maximum pooling layer, and three cascaded 3×3 convolutional modules; the convolutional pooling subnetwork is used to perform convolutional pooling on each of the target face local area images to generate a local convolutional pooling feature map corresponding to each of the face local area images, including:

[0032] Using two serially connected 3×3 convolution modules to perform convolution operations on each of the target face local area images, and outputting a first convolution feature map corresponding to each of the target face local area images;

[0033] Performing maximum pooling on each of the first convolutional feature maps through a maximum pooling layer to generate a first maximum pooling feature map corresponding to each of the first convolutional feature maps;

[0034] Input each of the first maximum pooling feature maps into two series-connected 3×3 convolution modules for convolution operation, and output a second convolution feature map corresponding to each of the first maximum pooling feature maps;

[0035] Performing maximum pooling on each of the second convolutional feature maps using a maximum pooling layer to generate a second maximum pooling feature map corresponding to each of the second convolutional feature maps;

[0036] Performing a convolution operation on each of the second maximum pooling feature maps through three cascaded 3×3 convolution modules, and outputting a third convolution feature map corresponding to each of the second maximum pooling feature maps;

[0037] Inputting each of the third convolutional feature maps into a maximum pooling layer for maximum pooling, and outputting a third maximum pooling feature map corresponding to each of the third convolutional feature maps;

[0038] Using three cascaded 3×3 convolution modules to perform convolution operations on each of the third maximum pooling feature maps, and outputting fourth convolution feature maps corresponding to each of the third maximum pooling feature maps;

[0039] Inputting each of the fourth convolutional feature maps into a maximum pooling layer for maximum pooling, and outputting a fourth maximum pooling feature map corresponding to each of the fourth convolutional feature maps;

[0040] Using three cascaded 3×3 convolution modules to perform a convolution operation on each of the fourth maximum pooling feature maps, and outputting a fifth convolution feature map corresponding to each of the fourth maximum pooling feature maps;

[0041] Input each of the fifth convolutional feature maps into the maximum pooling layer for maximum pooling, and output the local convolutional pooling feature map corresponding to each of the fifth convolutional feature maps.

[0042] Optionally, the global expression feature extraction network based on the sustained attention algorithm includes a convolutional pooling subnetwork and a local enhancement module based on the sustained attention algorithm; the global expression feature extraction network based on the sustained attention algorithm performs global feature extraction on the global face image to generate global expression feature data corresponding to the global face image, including:

[0043] Using a convolutional pooling subnetwork to perform convolutional pooling on the global face image to generate a global convolutional pooling feature map;

[0044] The global convolutional pooling feature map is subjected to feature extraction through a local enhancement module based on a continuous attention algorithm to generate global expression feature data corresponding to the global convolutional pooling feature map.

[0045] Optionally, the local enhancement module based on the sustained attention algorithm includes a maximum pooling layer, a convolution layer, an average pooling layer, and a Sigmoid activation function layer; the local enhancement module based on the sustained attention algorithm performs feature extraction on the global convolution pooling feature map to generate global expression feature data corresponding to the global convolution pooling feature map, including:

[0046] Performing maximum pooling on the global convolutional pooling feature map through a maximum pooling layer, and outputting a global maximum pooling feature map;

[0047] Performing average pooling on the global convolutional pooling feature map using an average pooling layer to generate a global average pooling feature map;

[0048] Splicing the global maximum pooling feature map and the global average pooling feature map to generate a spliced ​​feature map;

[0049] Inputting the concatenated feature map into the convolution layer to perform a convolution operation to generate a global convolution feature map;

[0050] A Sigmoid activation function layer is used to perform nonlinear mapping on the global convolution feature map to generate an initial attention map;

[0051] Using a continuous attention mechanism to generate a target attention map according to the initial attention map;

[0052] Perform matrix element-by-element multiplication on the target attention map and the global convolutional pooling feature map, and output the global expression feature data corresponding to the global convolutional pooling feature map.

[0053] Optionally, the adopting a preset feature classifier to perform facial expression recognition according to the global expression feature data and each of the local expression feature data to generate a facial expression recognition result includes:

[0054] Performing early linear fusion processing on the global expression feature data and each of the local expression feature data to generate multi-scale features;

[0055] A preset feature classifier is used to perform recognition based on the multi-scale features to generate a facial expression recognition result.

[0056] Optionally, the training process of the local enhancement module based on the continuous attention algorithm includes:

[0057] Acquire a face image of a video frame to be trained, and generate a two-dimensional facial mask matrix to be trained and a global maximum pooling feature map to be trained according to the face image of the video frame to be trained;

[0058] Using an initial local enhancement module based on a continuous attention algorithm to generate an initial attention map to be trained according to the global maximum pooling feature map to be trained;

[0059] Using a preset regression loss function to calculate a target loss value according to the initial attention map to be trained and the two-dimensional facial mask matrix to be trained;

[0060] Based on the target loss value, the network parameters of the initial local enhancement module based on the continuous attention algorithm are updated, the intermediate local enhancement module based on the continuous attention algorithm is determined, and the number of iterations is counted in real time;

[0061] Determine whether the number of iterations reaches a preset number of updates;

[0062] If achieved, the intermediate local enhancement module based on the continuous attention algorithm is used as the trained local enhancement module based on the continuous attention algorithm.

[0063] A second aspect of the present invention provides a facial expression recognition device, comprising:

[0064] An acquisition module is used to acquire a face image of a video frame to be detected, and perform target detection on the face image of the video frame to be detected, to generate a global face image and an initial key point set corresponding to the global face image;

[0065] A generating module, configured to generate a plurality of local face region images according to the initial key point set by using a local candidate module based on a region relationship algorithm;

[0066] A local feature extraction module, used for inputting each of the local face area images into a preset local expression feature extraction network for local feature extraction, and outputting local expression feature data corresponding to each of the local face area images;

[0067] A global feature extraction module, used to extract global features from the global face image through a global expression feature extraction network based on a continuous attention algorithm, and generate global expression feature data corresponding to the global face image;

[0068] The recognition module is used to use a preset feature classifier to perform facial expression recognition according to the global expression feature data and each of the local expression feature data to generate a facial expression recognition result.

[0069] It can be seen from the above technical solutions that the present invention has the following advantages:

[0070] The above technical scheme of the present invention provides a method for facial expression recognition, which first obtains a face image of a video frame to be detected, performs target detection on the face image of the video frame to be detected, and generates a global face image and an initial key point set corresponding to the global face image; then, a local candidate module based on a regional relationship algorithm is used to generate multiple face local area images according to the initial key point set; each face local area image is input into a preset local expression feature extraction network for local feature extraction, and local expression feature data corresponding to each face local area image is output; global feature extraction is performed on the global face image through a global expression feature extraction network based on a continuous attention algorithm, and global expression feature data corresponding to the global face image is generated; finally, a preset The feature classifier performs facial expression recognition according to global expression feature data and each local expression feature data to generate a facial expression recognition result; based on the above scheme, the generated global face image and the initial key point set are processed by a local candidate module based on a regional relationship algorithm, a preset local expression feature extraction network, and a global expression feature extraction network based on a continuous attention algorithm to obtain global expression feature data and each local expression feature data, and then facial expression recognition is performed by a preset feature classifier to generate a facial expression recognition result. The present invention combines the regional relationship algorithm with the continuous attention algorithm, and has the ability to learn the relationship between features at different levels, thereby achieving granularity alignment of global and local expressions, thereby improving the accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0072] Figure 1 A flowchart of a method for recognizing facial expressions provided in accordance with the first embodiment of the present invention;

[0073] Figure 2 A conventional arrangement diagram of key points of a human face provided in the first embodiment of the present invention;

[0074] Figure 3 This is an example diagram of a two-dimensional face mask provided in the first embodiment of the present invention;

[0075] Figure 4 A schematic diagram of the structure of a preset local expression feature extraction network provided in the first embodiment of the present invention;

[0076] Figure 5 A schematic diagram of the structure of a local enhancement module based on a continuous attention mechanism provided in the first embodiment of the present invention;

[0077] Figure 6 A workflow diagram of a preset local expression feature extraction network and a global expression feature extraction network based on a continuous attention algorithm provided in the first embodiment of the present invention;

[0078] Figure 7 A flowchart of the steps of the training process of the local enhancement module based on the continuous attention algorithm provided in the second embodiment of the present invention;

[0079] Figure 8 This is a structural block diagram of a facial expression recognition device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0080] The embodiments of the present invention provide a facial expression recognition method and device, which are used to solve the technical problem that the existing facial expression recognition method leads to poor recognition accuracy.

[0081] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] See also Figure 1 , Figure 1 A flowchart of the steps of a facial expression recognition method provided in Embodiment 1 of the present invention.

[0083] The present invention provides a method for recognizing facial expressions, comprising:

[0084] Step 101: obtain a face image of a video frame to be detected, perform target detection on the face image of the video frame to be detected, and generate a global face image and an initial key point set corresponding to the global face image.

[0085] It should be noted that the complete video data is processed according to the fixed interval sampling method, with a fixed frame number interval a f To select frames from the video stream to form a sampling sequence frame I=(i0,i1,...,i n-1 ,i n ,...,i N ), that is, the face image of the video frame to be detected is obtained, thereby completing the conversion of the video into raw data in the form of frame images, which can simplify the data processing process and reduce the calculation cost; where i0 is the face image of the first frame of the video frame to be detected, i N is the face image of the video frame to be detected in the N+1th frame.

[0086] Furthermore, the process of performing target detection on the face image of the video frame to be detected and generating a global face image and an initial key point set corresponding to the global face image can be achieved by executing the following steps S21 to S23:

[0087] Step S21, performing preliminary face detection and marking operations on the face image of the video frame to be detected to obtain an initial detected face image;

[0088] It should be noted that based on the relevant algorithms and code calls of OpenCV and Dlib libraries, a face detection module is constructed to detect the current frame image i n (The face image of the video frame to be detected) performs preliminary face detection and marking operations to obtain an initial detected face image.

[0089] Step S22, performing image preprocessing on the initial detected face image to generate a global face image;

[0090] It should be noted that the initial detected face image is subjected to image preprocessing, which specifically includes fixing, cropping, and aligning the size of the initial detected face image, and finally normalizing it to a 3×244×244 RGB global face image F n .

[0091] Step S23: perform facial key point detection on the global face image, and output an initial key point set corresponding to the global face image.

[0092] The initial key point set includes multiple initial key points.

[0093] Please note that Figure 2 , for F n Execute as Figure 2, 68 key points are arranged as shown in the example figure of face key points detection, and F n The initial key point set X n , where each initial key point is represented by p k , k∈(0,…,67). For each initial key point p k , each has its corresponding key point data, the key point data includes: the key point in F n The two-dimensional pixel position information in , ; The confidence level in the key point detection process and confidence threshold , ∈[0,1], ∈(0,1);The feature detection support of this key point ∈[0,1].

[0094] It is worth mentioning that It indicates the degree of credibility or accuracy of the prediction result of the point in the key point detection process. It is expressed in the form of probability. A confidence level of 1 means that the model is very sure that the prediction result is correct, and vice versa.

[0095] Further, It is data set based on prior knowledge. The data setting is based on the local area where the key point is located, the marginal degree of the arrangement position, the masking tolerance, etc. Specifically, the more core the position of the key point in the key point group, the lower the confidence requirement, and the lower the confidence threshold value is set; conversely, the more marginal the key point position is, the higher the masking tolerance is. In order to ensure the accuracy and simplicity of the local area, the confidence threshold value needs to be set higher.

[0096] Further, It is data set based on prior knowledge. The basis for setting the data is whether the key point is the core key point of the local area. The greater the influence of the key point on the positioning and division of the local area, the more support it has for subsequent feature acquisition, and the higher its support value is set, and vice versa.

[0097] In the embodiment of the present invention, a face image of a video frame to be detected is obtained, and target detection is performed on the face image of the video frame to be detected to generate a global face image and an initial key point set corresponding to the global face image.

[0098] Step 102: Generate multiple facial local area images according to the initial key point set using a local candidate module based on a regional relationship algorithm.

[0099] It should be noted that, based on the aforementioned facial images and key point data specifications, a key point set of 68 and a mask image with a size of 224×224 were selected as quantification and visualization tools, and the conventional face area division method, face key point arrangement sequence, face key texture information area, etc. were integrated as adjustable prior data.

[0100] Specifically, step 102 may include the following sub-steps S21-S29:

[0101] Step S21, performing local area rough matching on each initial key point to generate multiple initial local area key point groups;

[0102] The initial local area key point group includes multiple initial local area key points.

[0103] The multiple initial local area key point groups include a left eye local area key point group, a right eye local area key point group, a left eyebrow local area key point group, a right eyebrow local area key point group, a nose local area key point group, a mouth local area key point group, and a facial contour local area key point group.

[0104] It should be noted that when the expression changes, the key texture areas where the features are mainly distributed are four areas: left eye, right eye, nose, and mouth. In order to pay more attention to multiple local facial areas in a more specific and detailed manner, and to amplify the differences between different local features, the present invention divides the face into seven local areas on the basis of the above: left eye, right eye, left eyebrow, right eyebrow, nose, mouth, and face contour. Based on the X obtained in the above steps n The key point arrangement sequence, location information, etc., roughly match all key points to local areas, such as p 0 ~p 16 Divide into facial contour area, p 17 ~p 21 Divide into the left eyebrow area, etc.

[0105] It is worth mentioning that after completing the local area rough matching for all initial key points, each initial key point is used as the corresponding initial local area key point, and according to the corresponding local area, an initial local area key point group corresponding to each local area is constructed based on the initial local area key point.

[0106] Step S22, calculating the correlation degree corresponding to each initial local area key point in each initial local area key point group based on the confidence degree, confidence threshold, and feature detection support degree corresponding to each initial local area key point in each initial local area key point group;

[0107] It should be noted that, since the local regions of each face image have relative position differences and regional size fluctuations, in order to effectively solve the problem of matching the spatial position of the local regions and adaptive segmentation size, the present invention also needs to be based on X n The key point arrangement sequence, location information, confidence and other data are used to perform more detailed and scientific quantitative calculations to improve the positioning and division of local areas.

[0108] Specifically, for X n All key points in the initial local area key point group are filtered for correlation to remove low confidence and low support facial key points and reduce the impact of poor sampling and complex recognition. That is, according to the confidence, confidence threshold, and feature detection support corresponding to each initial local area key point in each initial local area key point group, the correlation corresponding to each initial local area key point is calculated. The calculation process of the correlation is:

[0109] ;

[0110] in, is the correlation degree corresponding to the kth initial local area key point; is the confidence corresponding to the kth initial local area key point; is the confidence threshold corresponding to the kth initial local area key point; is the feature detection support corresponding to the kth initial local area key point.

[0111] Step S23: in each initial local area key point group, remove any initial local area key point corresponding to a correlation degree less than a preset correlation degree threshold, and determine an intermediate local area key point group corresponding to each initial local area key point group;

[0112] It should be noted that setting a preset correlation threshold , to filter out the correlation less than The key points of the initial local area are obtained by using the new key point groups of each area (intermediate local area key point groups) and obtaining a new key point data set Y composed of each intermediate local area key point group. n , where each intermediate local area key point is represented by q k , k∈(0,…,67), the process can be expressed as:

[0113] ;

[0114] in, is the kth middle local area key point; is the kth initial local area key point; is the correlation degree corresponding to the kth initial local area key point; The preset correlation threshold.

[0115] It is worth mentioning that the two-dimensional position information of the key points in each intermediate local area is inherited from , In , the feature detection support is inherited from middle.

[0116] Step S24, based on the feature detection support corresponding to each intermediate local area key point in each intermediate local area key point group, select the intermediate local area key point corresponding to the largest feature detection support in each intermediate local area key point group as the core local area key point;

[0117] It should be noted that we go deep into each local area in turn, perform clustering tests on its key points, and remove key points that are too free by measuring the distance between points and point groups, so as to make the key point group information in each local area more meaningful and significant.

[0118] Specifically, first obtain each area in Y n middle The highest key point As the core point of the region , that is, in each intermediate local area key point group, the intermediate local area key point corresponding to the largest feature detection support in each intermediate local area key point group is selected as the core local area key point; if there are multiple core key points, the pixel position mean point of these key points is taken as .

[0119] Step S25, respectively performing weighted Mahalanobis distance calculations on the core local area key points and the multiple intermediate local area key points in each intermediate local area key point group, and determining the weighted Mahalanobis distance corresponding to each intermediate local area key point in each intermediate local area key point group;

[0120] It should be noted that the core key points selected based on each intermediate local area key point group , and let each key point in the area calculate the distance and distribution characteristics with it. Since the traditional Euclidean distance measures the direct distance between two points, but does not take into account the distribution characteristics of the data, it cannot truly reflect the actual distance between a point and the overall distribution. Therefore, based on the calculation idea of ​​the Mahalanobis distance, the present invention normalizes the covariance in the distance calculation, and uses the prior correlation value between points and point groups as weighting, which can effectively avoid the risks caused by the different variances of data features due to the Euclidean distance.

[0121] Furthermore, all key points in each region are sequentially calculated with weighted Mahalanobis distance to obtain the weighted Mahalanobis distance of each point, and the formula is:

[0122] ;

[0123] in, is the weighted Mahalanobis distance corresponding to the key point of the kth intermediate local area; is the kth middle local area key point; is the u-th core local area key point; T is the transpose; for same The covariance matrix of .

[0124] Step S26: In each intermediate local area key point group, any intermediate local area key point corresponding to a weighted Mahalanobis distance less than a preset Mahalanobis distance threshold is removed to determine a target local area key point group corresponding to each intermediate local area key point group;

[0125] It should be noted that a preset Mahalanobis distance threshold is set , filter out Less than The key points of the local area are then updated, and the out-of-class key points are removed, that is, the reserved middle local area key points are used as the target local area key points, and the target local area key point groups corresponding to each local area are constructed based on the target local area key points according to the corresponding local area, so that the key point groups C=(c0,...,c6) of the seven areas are respectively obtained, where c0 is the first target local area key point group.

[0126] Step S27, generating a two-dimensional facial mask matrix according to the pixel position information corresponding to each target local area key point in each target local area key point group;

[0127] Step S28, determining a mask image based on the two-dimensional facial mask matrix;

[0128] Step S29: Generate multiple local face area images based on the two-dimensional facial mask matrix and the mask image.

[0129] Please note that Figure 3 , using the pixel position information of each point in the key point group C to obtain a two-dimensional facial mask matrix f M , and then by f M Generate a mask image F of size 224×224 M , that is, the mask matrix is ​​1, indicating that F M There is an activated key point at the corresponding pixel position. If it is 0, there is no key point activated here. Finally, F M The ROI surrounded by the key point group is closed and filled according to the local area clustering, and the visualization is as follows Figure 3Example diagram. Among them, the mapping obtains f M The formula is:

[0130] ;

[0131] in, is the element in the two-dimensional face mask matrix; is the kth target local area key point; is the key point group of the i-th target local area.

[0132] Furthermore, relying on the facial mask matrix f M and mask image F M and other information to determine the location and size of each local area. n Complete 7 face local area images F L The correction and cropping work is performed to obtain multiple local face area images, which include a left eye image, a right eye image, a left eyebrow image, a right eyebrow image, a nose image, a mouth image, and a facial contour image.

[0133] In an embodiment of the present invention, the local candidate module based on the regional relationship algorithm can flexibly select and divide key local areas of expression features, thereby solving problems such as inaccurate local area positioning, poor robustness, and bloated calculations, thereby effectively improving the efficiency of capturing local area expression features and capturing feature information accurately and comprehensively.

[0134] Step 103: Input each face local area image into a preset local expression feature extraction network to extract local features, and output local expression feature data corresponding to each face local area image.

[0135] The preset local expression feature extraction network includes a convolutional pooling sub-network, three cascaded fully connected modules and a Softmax activation function layer.

[0136] It should be noted that, an expression feature acquisition network based on deep learning (preset local expression feature extraction network) is used to extract features from multiple local face area images obtained based on the above steps.

[0137] For further information, see Figure 4 The preset local expression feature extraction network proposed in the present invention adopts an improved VGG16 network (Visual Geometry Group 16-layer network) architecture, such as Figure 4As shown. Compared with the traditional network architecture, the improved VGG16 uses three 3×3 convolution kernels to replace the large 7×7 or 5×5 convolution kernels. This not only improves the network depth and the neural network feature expression effect while ensuring the same receptive field, but also effectively reduces the number of convolution calculation parameters. Preset local expression feature extraction network. The improved VGG16 network is mainly composed of three basic structural layers: input layer, convolution layer, and pooling layer to realize the global expression feature extraction of face images. The specific configuration and structure are as follows Figure 4 shown.

[0138] Specifically, the preset local expression feature extraction network is composed of a convolutional pooling sub-network, three cascaded fully connected modules and a Softmax activation function layer, wherein the convolutional pooling sub-network is composed of five groups of convolutions and five maximum pooling layers. The five groups of convolutions include two groups of two 3×3 convolutional modules in series and three groups of three cascaded 3×3 convolutional modules. The convolutional module includes a convolutional layer and a ReLU activation function layer, and the fully connected module includes a fully connected layer and a ReLU activation function layer.

[0139] Specifically, step 103 may include the following sub-steps S31-S34:

[0140] Step S31, adjusting the image format of each face local area image to determine the target face local area image corresponding to each face local area image;

[0141] Step S32: using a convolutional pooling subnetwork to perform convolutional pooling on each target face local area image, respectively, to generate a local convolutional pooling feature map corresponding to each face local area image;

[0142] The convolutional pooling subnetwork consists of two serially connected 3×3 convolutional modules, a maximum pooling layer, and three cascaded 3×3 convolutional modules.

[0143] Further, step S32 may include the following sub-steps: step S321 to step S3210:

[0144] Step S321, using two serially connected 3×3 convolution modules to perform convolution operations on each target face local area image, and outputting a first convolution feature map corresponding to each target face local area image;

[0145] Step S322: performing maximum pooling on each first convolutional feature map through a maximum pooling layer to generate a first maximum pooling feature map corresponding to each first convolutional feature map;

[0146] Step S323: input each first maximum pooling feature map into two series-connected 3×3 convolution modules for convolution operation, and output a second convolution feature map corresponding to each first maximum pooling feature map;

[0147] Step S324: perform maximum pooling on each second convolutional feature map using a maximum pooling layer to generate a second maximum pooling feature map corresponding to each second convolutional feature map;

[0148] Step S325: performing a convolution operation on each second maximum pooling feature map through three cascaded 3×3 convolution modules, and outputting a third convolution feature map corresponding to each second maximum pooling feature map;

[0149] Step S326: input each third convolutional feature map to the maximum pooling layer for maximum pooling, and output the third maximum pooling feature map corresponding to each third convolutional feature map;

[0150] Step S327: Use three cascaded 3×3 convolution modules to perform convolution operations on each third maximum pooling feature map, and output a fourth convolution feature map corresponding to each third maximum pooling feature map;

[0151] Step S328: input each fourth convolution feature map to the maximum pooling layer for maximum pooling, and output the fourth maximum pooling feature map corresponding to each fourth convolution feature map;

[0152] Step S329: Use three cascaded 3×3 convolution modules to perform convolution operations on each fourth maximum pooling feature map, and output a fifth convolution feature map corresponding to each fourth maximum pooling feature map;

[0153] Step S3210: input each fifth convolutional feature map into the maximum pooling layer for maximum pooling, and output the local convolutional pooling feature map corresponding to each fifth convolutional feature map.

[0154] Step S33: using each convolutional pooling feature map as the input of three cascaded fully connected modules, and outputting the fully connected feature map corresponding to each convolutional pooling feature map;

[0155] Step S34: Perform nonlinear mapping on each fully connected feature map through a Softmax activation function layer to generate local expression feature data corresponding to each fully connected feature map.

[0156] It should be noted that for the input local face area image , m∈(0,…,6), first needs to be adjusted to a 3×224×224 RGB image format to meet the requirements of the output layer. The improved VGG16 has 5 groups of convolutions and a total of 13 convolutional layers. Each group of convolution modules consists of a 3×3 convolutional layer and a ReLU activation function layer. After the first convolution of each group, a batchnormalization (BN) layer is used to solve the problem of data distribution in the intermediate layer to prevent gradient explosion or disappearance, speed up the convergence of the network, and then input to the ReLU activation function layer for processing. The input image is effectively extracted and filtered through each group of convolution operations and BN adjustments from shallow to deep. In addition, after each group of convolutions, the feature data will undergo a 2×2 maximum pooling operation to reduce the size of the feature map. After completing all convolution and pooling operations in sequence, the extracted local expression feature data is obtained. , m∈(0,…,6).

[0157] In the embodiment of the present invention, each local area image of a human face is input into a preset local expression feature extraction network for local feature extraction, and local expression feature data corresponding to each local area image of a human face is output.

[0158] Step 104: extract global features from the global facial image using a global expression feature extraction network based on a sustained attention algorithm to generate global expression feature data corresponding to the global facial image.

[0159] The global expression feature extraction network based on the continuous attention algorithm includes a convolutional pooling sub-network and a local enhancement module based on the continuous attention algorithm.

[0160] Please note that Figure 5 Based on the improved VGG16 network proposed in the present invention, a local enhancement module based on the continuous attention algorithm is introduced to constitute a global expression feature extraction network with adaptive properties (global expression feature extraction network based on the continuous attention algorithm), wherein the local enhancement module based on the continuous attention algorithm includes a maximum pooling layer, a convolutional layer, an average pooling layer, and a Sigmoid activation function layer.

[0161] Specifically, step 104 may include the following sub-steps S41-S42:

[0162] Step S41, using the convolutional pooling subnetwork to perform convolutional pooling on the global face image to generate a global convolutional pooling feature map;

[0163] It should be noted that the processing principle of the convolutional pooling sub-network for the global face image is consistent with the processing principle of the convolutional pooling sub-network for the local area image of the target face, and the present invention will not go into details.

[0164] Step S42: extract features from the global convolutional pooling feature map through a local enhancement module based on a continuous attention algorithm to generate global expression feature data corresponding to the global convolutional pooling feature map.

[0165] Further, step S42 may include the following sub-steps S421 to S427:

[0166] Step S421, performing maximum pooling on the global convolutional pooling feature map through a maximum pooling layer, and outputting a global maximum pooling feature map;

[0167] Step S422: using an average pooling layer to perform average pooling on the global convolutional pooling feature map to generate a global average pooling feature map;

[0168] Step S423, splicing the global maximum pooling feature map and the global average pooling feature map to generate a spliced ​​feature map;

[0169] Step S424: input the concatenated feature map to the convolution layer to perform a convolution operation to generate a global convolution feature map;

[0170] Step S425: Use the Sigmoid activation function layer to perform nonlinear mapping on the global convolution feature map to generate an initial attention map;

[0171] It should be noted that the feature F (global convolution pooling feature map) of the local enhancement module based on the continuous attention algorithm is subjected to maximum pooling and average pooling operations respectively to obtain two single-channel feature maps. , , H is the height, W is the width, then the two feature maps are concatenated, and then convolution is used to reduce them to one channel, and then the Sigmoid function is used to activate and generate the initial attention map , where the processing of the initial attention map can be expressed as:

[0172]

[0173] in, is the initial attention map; is the global maximum pooling feature map; is the global average pooling feature map; is average pooling; is the maximum pooling; is the Sigmoid activation function; To perform a convolution operation with a kernel size of 1×1.

[0174] Step S426: Generate a target attention map based on the initial attention map using a continuous attention mechanism;

[0175] Step S427, perform matrix element-by-element multiplication on the target attention map and the global convolution pooling feature map, and output the global expression feature data corresponding to the global convolution pooling feature map.

[0176] It should be noted that there is often correlation between feature data. By modeling such correlation, a more accurate attention map can be obtained without introducing additional supervision. The present invention introduces the attention map of the previous sequence frame image as prior knowledge to adjust the generation of the attention map of the current frame image, thereby establishing an attention mechanism (continuous attention mechanism) that associates adjacent frames. By associating the attention mechanism between adjacent sequence frames, the local enhancement module can make the global image gradually and robustly focus on more critical and subtle expression change areas during the feature extraction process, rather than focusing on different areas of different sequence frames. In the local enhancement module of the present invention, the formula for the attention mechanism between adjacent frames is, that is, the processing process of the target attention map, which can be expressed as:

[0177]

[0178] in, is the final attention map of the nth sequence frame, that is, the target attention map corresponding to the face image of the nth video frame to be detected; is the initial attention map of the nth sequence frame, that is, the initial attention map corresponding to the face image of the nth video frame to be detected; is the maximum pooling; is the target attention map corresponding to the face image of the n-1th video frame to be detected; Element-wise multiplication of matrices.

[0179] Furthermore, after the last set of convolutional layers and pooling layers, feature F is obtained, and then F is passed through the local enhancement module to finally obtain the output feature, that is, the processing process of the global expression feature data, which can be expressed as:

[0180]

[0181] in, is the global expression feature data; is the final attention map of the nth sequence frame, that is, the target attention map corresponding to the face image of the nth video frame to be detected; F is the global convolution pooling feature map.

[0182] In an embodiment of the present invention, global features are extracted from a global facial image using a global expression feature extraction network based on a sustained attention algorithm to generate global expression feature data corresponding to the global facial image.

[0183] Step 105: Use a preset feature classifier to perform facial expression recognition based on the global expression feature data and each local expression feature data to generate a facial expression recognition result.

[0184] Specifically, step 105 may include the following sub-steps S51-S52:

[0185] Step S51, performing early linear fusion processing on the global expression feature data and each local expression feature data to generate multi-scale features;

[0186] Step S52: using a preset feature classifier to perform recognition based on multi-scale features to generate a facial expression recognition result.

[0187] It should be noted that all local features and global features captured by the two feature extraction branches are sent to the facial feature database together. Subsequently, the global features and local features are subjected to early linear fusion processing to achieve the connection of multi-scale features, and then sent to the feature classifier for feature recognition, thereby obtaining the facial expression recognition result.

[0188] For example, see Figure 6 First, the video data is framed to obtain the original image data of the FER model (preset local expression feature extraction network, global expression feature extraction network based on continuous attention algorithm), that is, the face image of the video frame to be detected. Then, face detection and key point detection are performed on the current frame image (the face image of the video frame to be detected), and the local candidate module is used to divide the local expression area. At the same time, local expression feature extraction is performed based on deep learning, that is, the image of each local face area is input into the preset local expression feature extraction network for local feature extraction, as well as global expression feature extraction based on the attention algorithm, and the various feature information extracted are integrated to obtain the facial expression recognition result.

[0189] In the embodiment of the present invention, a preset feature classifier is used to perform facial expression recognition based on global expression feature data and each local expression feature data to generate a facial expression recognition result.

[0190] As a comparison of technical effects, it can be combined with existing technologies for reference. In order to better analyze the FER model features and achieve the best FER effect, the FER model features must first be captured comprehensively and accurately. At present, most FER feature acquisition methods directly extract global features from complete video frames containing a large amount of facial information. In order to compensate for the capture of global and local detailed features, some methods will use modules and algorithms for optimization, such as: adding an attention mechanism, turning from the whole face to the key expression area, extracting recognition features in the key facial area; according to the facial anchor point, the face is enhanced and input into the network to obtain multi-level facial features. The existing technology proposes a FER recognition model based on the fusion of local and global features. Its main model framework includes two branches, namely the local detail branch and the global key point branch. For the global key point branch, the key point detection is first performed on the entire image to generate the relevant facial mask, and then in the feature extraction network, the key area data information is assisted to be highlighted; for the local detail branch, the eyes and mouth are selected as fixed key areas, and each cropped image is input into the feature extraction network for feature extraction.

[0191] Furthermore, the FER model used in the above-mentioned technology can only perform fixed positioning and division of global images and local images due to the lack of a relationship mechanism between facial regions. It has problems such as low regional positioning accuracy, inability to cope with complex situations, and long calculation time, which leads to low efficiency, missing information and inaccuracy when extracting feature data from each region. This is the main technical problem solved by this patent. At the same time, the FER model used in the above-mentioned technology lacks a continuous attention mechanism for the key local areas of expression, and cannot adaptively adjust the re-positioning and weighting of key areas in the feature extraction stage. Therefore, the feature extraction process has problems and deficiencies such as ignoring local detail information, being unable to suppress irrelevant feature information, and lacking the ability to continuously optimize the comprehensive extraction of features.

[0192] The present invention aims at the problems that the existing FER model cannot effectively utilize the relationship between local and global regions, has large calculation parameters, lacks a mechanism for continuous attention to key areas of expression, and the like in the process of feature capture. The present invention proposes a method for facial expression recognition, which is based on a hybrid intelligent algorithm of a regional relationship algorithm and a continuous attention algorithm, and applies it to the FER model for expression feature capture. The present invention relies on the ability of the regional relationship algorithm to learn the relationship between features at different levels, and uses the key points of face detection as a bridge to achieve granularity alignment of global and local expressions, facial ROI area positioning, and local expression area cropping; at the same time, the present invention associates the key area information of facial images between adjacent frames based on the continuous attention algorithm and attention map design in the stage of acquiring global expression features, so that the FER model can obtain the accurate attention map of the current frame without introducing additional supervision, and then robustly extract the key feature data with the most significant characteristics.

[0193] In summary, when the preset local expression feature extraction network of the present invention and the global expression feature extraction network based on the continuous attention algorithm are used to capture facial expression features, the expression key areas can be continuously and quickly located, the amount of model calculation can be reduced, and expression feature data with more discriminative characteristics can be adaptively extracted. At the same time, the present invention can combine the regional relationship algorithm with the continuous attention algorithm, and has the advantages of regional relationship combination and continuous attention calibration, which can make the FER model more effective and accurate in acquiring expression features.

[0194] In an embodiment of the present invention, the present invention provides a method for recognizing facial expressions. First, a facial image of a video frame to be detected is obtained, and target detection is performed on the facial image of the video frame to be detected, so as to generate a global facial image and an initial key point set corresponding to the global facial image; then, a local candidate module based on a regional relationship algorithm is used to generate a plurality of local facial area images according to the initial key point set; each local facial area image is input into a preset local expression feature extraction network for local feature extraction, and local expression feature data corresponding to each local facial area image is output; global feature extraction is performed on the global facial image through a global expression feature extraction network based on a continuous attention algorithm, and global expression feature data corresponding to the global facial image is generated; finally, the preset local expression feature extraction network is used to extract the local expression feature data corresponding to the global facial image. A feature classifier is set to perform facial expression recognition according to global expression feature data and each local expression feature data to generate a facial expression recognition result; based on the above scheme, the generated global face image and the initial key point set are processed by a local candidate module based on a regional relationship algorithm, a preset local expression feature extraction network, and a global expression feature extraction network based on a continuous attention algorithm to obtain global expression feature data and each local expression feature data, and then facial expression recognition is performed by a preset feature classifier to generate a facial expression recognition result. The present invention combines the regional relationship algorithm with the continuous attention algorithm, and is capable of learning the relationship between features at different levels, thereby achieving granularity alignment of global and local expressions, thereby improving the accuracy of recognition.

[0195] For better explanation, refer to Figure 7 , shows a flowchart of the steps of the training process of the local enhancement module based on the continuous attention algorithm provided in the second embodiment of the present invention, and the process may include the following steps:

[0196] Step 701: obtaining a face image of a video frame to be trained, and generating a two-dimensional facial mask matrix to be trained and a global maximum pooling feature map to be trained according to the face image of the video frame to be trained;

[0197] Step 702: using an initial local enhancement module based on a continuous attention algorithm to generate an initial attention map to be trained according to the global maximum pooling feature map to be trained;

[0198] Step 703: Calculate the target loss value according to the initial attention map to be trained and the two-dimensional facial mask matrix to be trained using a preset regression loss function;

[0199] Step 704: Update the network parameters of the initial local enhancement module based on the continuous attention algorithm based on the target loss value, determine the intermediate local enhancement module based on the continuous attention algorithm, and count the number of iterations in real time;

[0200] Step 705: determine whether the number of iterations reaches the preset number of updates;

[0201] Step 706: If reached, the intermediate local enhancement module based on the sustained attention algorithm is used as the trained local enhancement module based on the sustained attention algorithm.

[0202] It should be noted that the specific configuration of the convolutional pooling subnetwork in the global expression feature extraction network based on the continuous attention algorithm proposed in the present invention has been set, so the present invention only performs model training for the local enhancement module based on the continuous attention algorithm. Among them, the two-dimensional facial mask matrix to be trained is obtained by processing the initial key point set corresponding to the face image of the video frame to be trained through the local candidate module based on the regional relationship algorithm, and the global maximum pooling feature map to be trained is obtained by processing the global face image corresponding to the face image of the video frame to be trained through the convolutional pooling subnetwork.

[0203] Furthermore, in order to transform the initial attention map Focusing on the unobstructed and key local expression areas of the face, the two-dimensional mask matrix f is generated by the local candidate module based on the regional relationship algorithm M , and define a regression loss function such that More suitable for f M , thereby achieving adaptive enhancement adjustment of the attention map. Among them, the preset regression loss function formula is:

[0204] ;

[0205] in, is the target loss value; is the initial attention map to be trained; is the two-dimensional facial mask matrix to be trained; N is the number of samples; Element-wise multiplication of matrices.

[0206] It is worth mentioning that due to f M In the above example, if the parameters of masked and non-critical areas are 0, the weight difference of such areas can be adjusted through the regression loss function to suppress the extraction and collection of non-critical features.

[0207] Furthermore, if the number of iterations does not reach the preset update number, the intermediate local enhancement module based on the continuous attention algorithm is used as a new initial local enhancement module based on the continuous attention algorithm, and the process jumps to step 702 until the number of iterations reaches the preset update number, and the intermediate local enhancement module based on the continuous attention algorithm determined when the number of iterations reaches the preset update number is used as a trained local enhancement module based on the continuous attention algorithm.

[0208] In an embodiment of the present invention, a local enhancement module based on a continuous attention algorithm can achieve continuous updating and weight allocation of key areas during the feature extraction stage of the global feature area of ​​facial expressions, effectively solving the problems of missing information in key areas and unsuppressed information in non-key areas, and realizing the continuous optimization function of feature extraction work under unsupervised operation.

[0209] See also Figure 8 , Figure 8 This is a structural block diagram of a facial expression recognition device provided in Example 3 of the present invention.

[0210] The present invention provides a facial expression recognition device, comprising:

[0211] The acquisition module 801 is used to acquire a face image of a video frame to be detected, and perform target detection on the face image of the video frame to be detected, and generate a global face image and an initial key point set corresponding to the global face image;

[0212] A generating module 802, configured to generate a plurality of local face region images according to an initial key point set by using a local candidate module based on a region relationship algorithm;

[0213] The local feature extraction module 803 is used to input each face local area image into a preset local expression feature extraction network for local feature extraction, and output local expression feature data corresponding to each face local area image;

[0214] A global feature extraction module 804 is used to extract global features from a global face image through a global expression feature extraction network based on a sustained attention algorithm, and generate global expression feature data corresponding to the global face image;

[0215] The recognition module 805 is used to use a preset feature classifier to perform facial expression recognition based on the global expression feature data and each local expression feature data to generate a facial expression recognition result.

[0216] Furthermore, the acquisition module 801 is specifically used for:

[0217] Perform preliminary face detection and marking operations on the face image of the video frame to be detected to obtain an initial detected face image;

[0218] Perform image preprocessing on the initial detected face image to generate a global face image;

[0219] Perform facial key point detection on the global face image and output the initial key point set corresponding to the global face image.

[0220] Further, the initial key point set includes a plurality of initial key points; the generating module 802 is specifically used for:

[0221] Perform local area rough matching on each initial key point to generate multiple initial local area key point groups;

[0222] Based on the confidence, confidence threshold, and feature detection support corresponding to each initial local area key point in each initial local area key point group, the correlation corresponding to each initial local area key point in each initial local area key point group is calculated;

[0223] In each initial local area key point group, any initial local area key point corresponding to a correlation degree less than a preset correlation degree threshold is eliminated to determine an intermediate local area key point group corresponding to each initial local area key point group;

[0224] Based on the feature detection support corresponding to each intermediate local area key point in each intermediate local area key point group, the intermediate local area key point corresponding to the largest feature detection support is selected as the core local area key point in each intermediate local area key point group;

[0225] Performing weighted Mahalanobis distance calculations on the core local area key points and multiple intermediate local area key points in each intermediate local area key point group, respectively, to determine the weighted Mahalanobis distance corresponding to each intermediate local area key point in each intermediate local area key point group;

[0226] In each intermediate local area key point group, any intermediate local area key point corresponding to a weighted Mahalanobis distance less than a preset Mahalanobis distance threshold is removed to determine the target local area key point group corresponding to each intermediate local area key point group;

[0227] Generate a two-dimensional facial mask matrix according to pixel position information corresponding to each target local area key point in each target local area key point group;

[0228] Based on the two-dimensional facial mask matrix, a mask image is determined;

[0229] A plurality of local face area images are generated according to the two-dimensional face mask matrix and the mask image.

[0230] Furthermore, the preset local expression feature extraction network includes a convolutional pooling subnetwork, three cascaded fully connected modules and a Softmax activation function layer; the local feature extraction module 803 includes:

[0231] The first submodule is used to adjust the image format of each local face area image and determine the target local face area image corresponding to each local face area image;

[0232] The second submodule is used to perform convolution pooling on each target face local area image respectively using a convolution pooling subnetwork to generate a local convolution pooling feature map corresponding to each face local area image;

[0233] The third submodule is used to use each convolutional pooling feature map as the input of three cascaded fully connected modules, and output the fully connected feature map corresponding to each convolutional pooling feature map;

[0234] The fourth submodule is used to perform nonlinear mapping on each fully connected feature map through a Softmax activation function layer to generate local expression feature data corresponding to each fully connected feature map.

[0235] Furthermore, the convolutional pooling subnetwork includes two serially connected 3×3 convolutional modules, a maximum pooling layer, and three cascaded 3×3 convolutional modules; the second submodule is specifically used for:

[0236] Two 3×3 convolution modules connected in series are used to perform convolution operations on each target face local area image, and the first convolution feature map corresponding to each target face local area image is output;

[0237] Performing maximum pooling on each first convolutional feature map through a maximum pooling layer to generate a first maximum pooling feature map corresponding to each first convolutional feature map;

[0238] Input each first maximum pooling feature map into two series-connected 3×3 convolution modules for convolution operation, and output the second convolution feature map corresponding to each first maximum pooling feature map;

[0239] Performing maximum pooling on each second convolutional feature map using a maximum pooling layer to generate a second maximum pooling feature map corresponding to each second convolutional feature map;

[0240] Perform convolution operations on each second maximum pooling feature map through three cascaded 3×3 convolution modules, and output third convolution feature maps corresponding to each second maximum pooling feature map;

[0241] Input each third convolution feature map into the maximum pooling layer for maximum pooling, and output the third maximum pooling feature map corresponding to each third convolution feature map;

[0242] Using three cascaded 3×3 convolution modules to perform convolution operations on each third maximum pooling feature map, and outputting a fourth convolution feature map corresponding to each third maximum pooling feature map;

[0243] Input each fourth convolutional feature map into the maximum pooling layer for maximum pooling, and output the fourth maximum pooling feature map corresponding to each fourth convolutional feature map;

[0244] Using three cascaded 3×3 convolution modules to perform convolution operations on each fourth maximum pooling feature map, and outputting a fifth convolution feature map corresponding to each fourth maximum pooling feature map;

[0245] Each fifth convolutional feature map is input into the maximum pooling layer for maximum pooling, and the local convolutional pooling feature map corresponding to each fifth convolutional feature map is output.

[0246] Furthermore, the global expression feature extraction network based on the continuous attention algorithm includes a convolutional pooling subnetwork and a local enhancement module based on the continuous attention algorithm; the global feature extraction module 804 includes:

[0247] The fifth submodule is used to perform convolution pooling on the global face image using a convolution pooling subnetwork to generate a global convolution pooling feature map;

[0248] The sixth sub-module is used to extract features from the global convolutional pooling feature map through a local enhancement module based on a continuous attention algorithm, and generate global expression feature data corresponding to the global convolutional pooling feature map.

[0249] Furthermore, the local enhancement module based on the continuous attention algorithm includes a maximum pooling layer, a convolution layer, an average pooling layer, and a Sigmoid activation function layer; the sixth submodule is specifically used for:

[0250] Perform maximum pooling on the global convolutional pooling feature map through the maximum pooling layer, and output the global maximum pooling feature map;

[0251] The average pooling layer is used to average pool the global convolutional pooling feature map to generate a global average pooling feature map;

[0252] The global maximum pooling feature map and the global average pooling feature map are spliced ​​to generate a spliced ​​feature map;

[0253] Input the concatenated feature map into the convolution layer to perform convolution operation and generate a global convolution feature map;

[0254] The Sigmoid activation function layer is used to perform nonlinear mapping on the global convolution feature map to generate the initial attention map;

[0255] Use continuous attention mechanism to generate target attention map based on initial attention map;

[0256] Perform matrix element-by-element multiplication on the target attention map and the global convolutional pooling feature map, and output the global expression feature data corresponding to the global convolutional pooling feature map.

[0257] Furthermore, the identification module 805 is specifically used for:

[0258] Perform early linear fusion processing on global expression feature data and local expression feature data to generate multi-scale features;

[0259] The preset feature classifier is used to perform recognition based on multi-scale features to generate facial expression recognition results.

[0260] In an optional device embodiment, it also includes:

[0261] The first module is used to obtain a face image of a video frame to be trained, and generate a two-dimensional facial mask matrix to be trained and a global maximum pooling feature map to be trained according to the face image of the video frame to be trained;

[0262] The second module is used to generate an initial attention map to be trained according to the global maximum pooling feature map to be trained by adopting an initial local enhancement module based on a continuous attention algorithm;

[0263] The third module is used to calculate the target loss value according to the initial attention map to be trained and the two-dimensional facial mask matrix to be trained by using a preset regression loss function;

[0264] The fourth module is used to update the network parameters of the initial local enhancement module based on the continuous attention algorithm based on the target loss value, determine the intermediate local enhancement module based on the continuous attention algorithm, and count the number of iterations in real time;

[0265] The fifth module is used to determine whether the number of iterations reaches the preset number of updates;

[0266] The sixth module is used to use the intermediate local enhancement module based on the continuous attention algorithm as the trained local enhancement module based on the continuous attention algorithm if achieved.

[0267] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and sub-modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0268] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing facial expressions, characterized in that: include: Acquire a face image of a video frame to be detected, and perform target detection on the face image of the video frame to be detected to generate a global face image and an initial key point set corresponding to the global face image; Generate multiple local face region images according to the initial key point set using a local candidate module based on a region relationship algorithm; Inputting each of the local face area images into a preset local expression feature extraction network to extract local features, and outputting local expression feature data corresponding to each of the local face area images; Performing global feature extraction on the global face image through a global expression feature extraction network based on a sustained attention algorithm to generate global expression feature data corresponding to the global face image; A preset feature classifier is used to perform facial expression recognition based on the global expression feature data and each of the local expression feature data to generate a facial expression recognition result.

2. The facial expression recognition method according to claim 1, characterized in that: The performing target detection on the face image of the video frame to be detected to generate a global face image and an initial key point set corresponding to the global face image includes: Performing preliminary face detection and marking operations on the face image of the video frame to be detected to obtain an initial detected face image; Performing image preprocessing on the initial detected face image to generate a global face image; Perform facial key point detection on the global face image, and output an initial key point set corresponding to the global face image.

3. The facial expression recognition method according to claim 1, characterized in that: The initial key point set includes a plurality of initial key points; The local candidate module based on the regional relationship algorithm is used to generate multiple local face area images according to the initial key point set, including: Performing local area rough matching on each of the initial key points to generate multiple initial local area key point groups; Calculating the correlation degree corresponding to each initial local area key point in each initial local area key point group based on the confidence, confidence threshold, and feature detection support corresponding to each initial local area key point in each initial local area key point group; In each of the initial local area key point groups, any initial local area key point corresponding to a correlation degree less than a preset correlation degree threshold is eliminated to determine an intermediate local area key point group corresponding to each of the initial local area key point groups; Based on the feature detection support corresponding to each intermediate local area key point in each intermediate local area key point group, in each intermediate local area key point group, the intermediate local area key point corresponding to the largest feature detection support is selected as the core local area key point; Performing weighted Mahalanobis distance calculations on the core local area key points and the multiple intermediate local area key points in each of the intermediate local area key point groups, respectively, to determine the weighted Mahalanobis distances corresponding to each of the intermediate local area key points in each of the intermediate local area key point groups; In each of the intermediate local area key point groups, any intermediate local area key point corresponding to a weighted Mahalanobis distance less than a preset Mahalanobis distance threshold is eliminated to determine a target local area key point group corresponding to each of the intermediate local area key point groups; Generate a two-dimensional facial mask matrix according to pixel position information corresponding to each target local area key point in each target local area key point group; Determining a mask image based on the two-dimensional facial mask matrix; A plurality of local face area images are generated according to the two-dimensional face mask matrix and the mask image.

4. The facial expression recognition method according to claim 1, characterized in that: The preset local expression feature extraction network includes a convolutional pooling sub-network, three cascaded fully connected modules and a Softmax activation function layer; the local facial region images are input into the preset local expression feature extraction network for local feature extraction, and the local expression feature data corresponding to the local facial region images are output, including: Adjusting the image format of each of the local face area images to determine a target local face area image corresponding to each of the local face area images; Using a convolutional pooling subnetwork to perform convolutional pooling on each of the target face local area images, respectively, to generate a local convolutional pooling feature map corresponding to each of the face local area images; Using each of the convolutional pooling feature maps as the input of three cascaded fully connected modules, and outputting the fully connected feature maps corresponding to each of the convolutional pooling feature maps; Nonlinear mapping is performed on each of the fully connected feature maps through a Softmax activation function layer to generate local expression feature data corresponding to each of the fully connected feature maps.

5. The facial expression recognition method according to claim 4, characterized in that: The convolutional pooling subnetwork includes two serially connected 3×3 convolutional modules, a maximum pooling layer, and three cascaded 3×3 convolutional modules; the convolutional pooling subnetwork is used to perform convolutional pooling on each of the target face local area images to generate a local convolutional pooling feature map corresponding to each of the face local area images, including: Using two serially connected 3×3 convolution modules to perform convolution operations on each of the target face local area images, and outputting a first convolution feature map corresponding to each of the target face local area images; Performing maximum pooling on each of the first convolutional feature maps through a maximum pooling layer to generate a first maximum pooling feature map corresponding to each of the first convolutional feature maps; Input each of the first maximum pooling feature maps into two series-connected 3×3 convolution modules for convolution operation, and output a second convolution feature map corresponding to each of the first maximum pooling feature maps; Performing maximum pooling on each of the second convolutional feature maps using a maximum pooling layer to generate a second maximum pooling feature map corresponding to each of the second convolutional feature maps; Performing a convolution operation on each of the second maximum pooling feature maps through three cascaded 3×3 convolution modules, and outputting a third convolution feature map corresponding to each of the second maximum pooling feature maps; Inputting each of the third convolutional feature maps into a maximum pooling layer for maximum pooling, and outputting a third maximum pooling feature map corresponding to each of the third convolutional feature maps; Using three cascaded 3×3 convolution modules to perform convolution operations on each of the third maximum pooling feature maps, and outputting fourth convolution feature maps corresponding to each of the third maximum pooling feature maps; Inputting each of the fourth convolutional feature maps into a maximum pooling layer for maximum pooling, and outputting a fourth maximum pooling feature map corresponding to each of the fourth convolutional feature maps; Using three cascaded 3×3 convolution modules to perform a convolution operation on each of the fourth maximum pooling feature maps, and outputting a fifth convolution feature map corresponding to each of the fourth maximum pooling feature maps; Input each of the fifth convolutional feature maps into the maximum pooling layer for maximum pooling, and output the local convolutional pooling feature map corresponding to each of the fifth convolutional feature maps.

6. The facial expression recognition method according to claim 1, characterized in that: The global expression feature extraction network based on the sustained attention algorithm includes a convolutional pooling subnetwork and a local enhancement module based on the sustained attention algorithm; the global expression feature extraction network based on the sustained attention algorithm performs global feature extraction on the global face image to generate global expression feature data corresponding to the global face image, including: Using a convolutional pooling subnetwork to perform convolutional pooling on the global face image to generate a global convolutional pooling feature map; The global convolutional pooling feature map is subjected to feature extraction through a local enhancement module based on a continuous attention algorithm to generate global expression feature data corresponding to the global convolutional pooling feature map.

7. The facial expression recognition method according to claim 6, characterized in that: The local enhancement module based on the continuous attention algorithm includes a maximum pooling layer, a convolution layer, an average pooling layer, and a Sigmoid activation function layer; the local enhancement module based on the continuous attention algorithm performs feature extraction on the global convolution pooling feature map to generate global expression feature data corresponding to the global convolution pooling feature map, including: Performing maximum pooling on the global convolutional pooling feature map through a maximum pooling layer, and outputting a global maximum pooling feature map; Performing average pooling on the global convolutional pooling feature map using an average pooling layer to generate a global average pooling feature map; Splicing the global maximum pooling feature map and the global average pooling feature map to generate a spliced ​​feature map; Inputting the concatenated feature map into the convolution layer to perform a convolution operation to generate a global convolution feature map; A Sigmoid activation function layer is used to perform nonlinear mapping on the global convolution feature map to generate an initial attention map; Using a continuous attention mechanism to generate a target attention map according to the initial attention map; Perform matrix element-by-element multiplication on the target attention map and the global convolutional pooling feature map, and output the global expression feature data corresponding to the global convolutional pooling feature map.

8. The facial expression recognition method according to claim 1, characterized in that: The method of using a preset feature classifier to perform facial expression recognition according to the global expression feature data and each of the local expression feature data to generate a facial expression recognition result includes: Performing early linear fusion processing on the global expression feature data and each of the local expression feature data to generate multi-scale features; A preset feature classifier is used to perform recognition based on the multi-scale features to generate a facial expression recognition result.

9. The facial expression recognition method according to claim 7, characterized in that: The training process of the local enhancement module based on the continuous attention algorithm includes: Acquire a face image of a video frame to be trained, and generate a two-dimensional facial mask matrix to be trained and a global maximum pooling feature map to be trained according to the face image of the video frame to be trained; Using an initial local enhancement module based on a continuous attention algorithm to generate an initial attention map to be trained according to the global maximum pooling feature map to be trained; Using a preset regression loss function to calculate a target loss value according to the initial attention map to be trained and the two-dimensional facial mask matrix to be trained; Based on the target loss value, the network parameters of the initial local enhancement module based on the continuous attention algorithm are updated, the intermediate local enhancement module based on the continuous attention algorithm is determined, and the number of iterations is counted in real time; Determine whether the number of iterations reaches a preset number of updates; If achieved, the intermediate local enhancement module based on the continuous attention algorithm is used as the trained local enhancement module based on the continuous attention algorithm.

10. A facial expression recognition device, characterized in that: include: An acquisition module is used to acquire a face image of a video frame to be detected, and perform target detection on the face image of the video frame to be detected, to generate a global face image and an initial key point set corresponding to the global face image; A generating module, configured to generate a plurality of local face region images according to the initial key point set by using a local candidate module based on a region relationship algorithm; A local feature extraction module, used for inputting each of the local face area images into a preset local expression feature extraction network for local feature extraction, and outputting local expression feature data corresponding to each of the local face area images; A global feature extraction module, used to extract global features from the global face image through a global expression feature extraction network based on a continuous attention algorithm, and generate global expression feature data corresponding to the global face image; The recognition module is used to use a preset feature classifier to perform facial expression recognition according to the global expression feature data and each of the local expression feature data to generate a facial expression recognition result.

Images