Facial expression recognition method and system based on deep spatiotemporal network decision fusion

Abstract

The present invention provides a facial expression recognition method based on deep spatiotemporal network decision fusion, comprising: preprocessing each facial expression image of an original facial expression data set, extracting a facial landmark vector for each preprocessed facial expression image, and selecting a peak expression image; obtaining a global temporal feature of the facial expression according to the facial landmark vector, classifying the facial expression based on the global temporal feature of the facial expression, and obtaining a first facial expression classification result; obtaining a spatial feature of the facial expression according to the selected peak expression image, classifying the facial expression based on the spatial feature of the facial expression, and obtaining a second facial expression classification result; performing decision-level fusion on the first facial expression classification result and the second facial expression classification result to obtain a final facial expression classification result, and obtaining a final classification result by utilizing the global temporal feature and spatial feature of the facial expression image, thereby achieving a better facial expression recognition effect.

Description

Technical Field

[0001] The present invention relates to the field of facial expression recognition in the field of emotion recognition technology, and in particular to a facial expression recognition method and system based on deep spatiotemporal network decision fusion. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Facial expression is the most common way for humans to express their inner emotions and is a key factor in machines' perception of human emotions. With the growth of computer computing power, facial expression recognition has gradually become a research hotspot in the field of human-computer interaction. Facial expression recognition (FER) refers to the use of computers to capture facial expressions and their changes to analyze the emotional changes in humans. It has been widely used in many fields, such as distance education, public safety, and commercial marketing. However, FER is a rather complex facial analysis task, and even humans find it difficult to accurately identify the emotions of others through a single facial feature.

[0004] Current FER methods can be roughly divided into two categories: FER methods based on a single static image and FER methods based on dynamic image sequences. The FER method based on a single static image can effectively extract spatial features, but it cannot utilize the dynamic information in the process of facial expression changes. The FER method based on a dynamic image sequence can capture the dynamic facial changes of several consecutive frames in the image sequence, and then extract temporal features, but often ignores the spatial information of the image. Whether it is a FER method based on a single static image or a FER method based on a dynamic image sequence, they only focus on one-sided facial expression features and it is difficult to achieve a high recognition accuracy. Therefore, how to extract more efficient temporal features and spatial features and apply them to the FER task has become a key challenge.

[0005] The prior art has the following technical problems:

[0006] Convolutional Neural Network (CNN) has become an effective model for extracting facial expression features. However, the small amount of data in facial expression databases is often a major obstacle to directly applying these CNN models to recognize facial expressions. Some researchers have proposed that reducing the depth of the hidden layer of the neural network is a feasible way to overcome the overfitting problem. However, a network that is too shallow is difficult to achieve the desired effect and is not conducive to extracting deep features. How to retain spatial features in the hidden layer of the convolutional neural network in a targeted manner has become a major technical difficulty.

[0007] The generation of facial expressions can be considered as the dynamic changes of key parts of the face, which are expressed as a whole through these dynamic changes to form the change process of the entire facial expression. Traditional facial expression recognition methods usually use hand-crafted descriptors to extract temporal features hidden in facial images. With the widespread application of deep learning technology, some researchers have directly input facial image sequences into recurrent neural networks (RNNs) to extract temporal features of facial expressions, and achieved good results. However, this method cannot effectively utilize prior knowledge, nor can it extract contextual information between facial expression video frames. How to capture the dynamic information of key parts of the face in facial expression videos is a technical problem that needs to be solved at present. Summary of the invention

[0008] In order to solve the technical problems existing in the above-mentioned background technology, the present disclosure provides a facial expression recognition method and system based on deep spatiotemporal network decision fusion, which divides the facial expression image sequence into four subsequences according to the facial area, and constructs a BiLSTM model for each of the four subsequences to extract local temporal features, which can capture the local morphological features of facial expressions in more detail and make maximum use of the temporal features of dynamic facial expressions; uses a VGG network to extract a shallow spatial feature map of the peak expression image and uses SENet to assign channel weights to it, which can retain effective spatial features in a targeted manner and reduce the risk of model overfitting.

[0009] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0010] A first aspect of the present disclosure provides a facial expression recognition method based on deep spatiotemporal network decision fusion, comprising:

[0011] Preprocess each facial expression image of the original facial expression data set, extract the facial landmark point vector of each preprocessed facial expression image, and select the peak expression image;

[0012] Obtaining global temporal features of facial expressions according to facial landmark vectors, performing facial expression classification on the global temporal features of facial expressions, and obtaining a first facial expression classification result;

[0013] Obtaining spatial features of facial expressions according to the selected peak expression image, performing facial expression classification on the spatial features of facial expressions, and obtaining a second facial expression classification result;

[0014] The first facial expression classification result and the second facial expression classification result are fused at the decision level to obtain the final facial expression classification result.

[0015] A second aspect of the present disclosure provides a facial expression recognition system based on deep spatiotemporal network decision fusion, comprising:

[0016] The data preprocessing module is configured to preprocess each facial expression image of the original facial expression data set, extract the facial landmark vector of each facial expression image after preprocessing, and select the peak expression image;

[0017] A temporal feature extraction module is configured to obtain a global temporal feature of a facial expression according to a facial landmark vector, and classify the facial expression based on the global temporal feature of the facial expression to obtain a first facial expression classification result;

[0018] A spatial feature extraction module is configured to obtain spatial features of facial expressions according to the selected peak expression image, perform facial expression classification on the spatial features of facial expressions, and obtain a second facial expression classification result;

[0019] The decision fusion module is configured to perform decision-level fusion on the first facial expression classification result and the second facial expression classification result to obtain a final facial expression classification result.

[0020] A third aspect of the present disclosure provides a computer-readable storage medium.

[0021] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a facial expression recognition method based on deep spatiotemporal network decision fusion as described in the first aspect above.

[0022] A fourth aspect of the present disclosure provides a computer device.

[0023] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a method for facial expression recognition based on deep spatiotemporal network decision fusion as described in the first aspect above are implemented.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] The present invention firstly preprocesses the original facial expression image sequence, and the preprocessing steps include face cropping, grayscale processing, data enhancement, facial landmark extraction and peak image frame selection; secondly, a temporal feature extraction module based on a bidirectional long short-term memory network (Bi-directional Long Short-Term Memory, BiLSTM) is proposed, and the facial landmarks are respectively input into the BiLSTM according to different face regions to extract local temporal features, and the local temporal features are fused to obtain the global temporal features of the facial expression, and the global temporal features are input into the softmax to calculate the facial expression classification result; thirdly, a temporal feature extraction module based on VGG (Visual Geometry Group, VGG) and SENet (Squeeze and Excitation The spatial feature extraction module of the temporal feature extraction module and the spatial feature extraction module is integrated to obtain the final facial expression classification result. Finally, the public datasets CK+ (The Extended Cohn-KanadeDataset, CK+) and Oulu-CASIA (Oulu-CASIA NIR&VIS Facial Expression Database, Oulu-CASIA) are used to evaluate the performance of the method.

[0026] The present disclosure consists of 4 parts: data preprocessing module, temporal feature extraction module, spatial feature extraction module and decision fusion module. Through analysis, it is found that the generation of facial expressions can often be regarded as the dynamic changes of key parts of the face (eyebrows, eyes, nose and mouth), and the overall expression of these dynamic changes forms the change process of the entire facial expression. The present disclosure proposes a temporal feature extraction module, which divides the facial expression image sequence into four subsequences according to the facial area, and constructs BiLSTM models for each of the four subsequences to extract local temporal features, which can capture the local morphological features of facial expressions in more detail and maximize the use of the temporal features of dynamic facial expressions.

[0027] In addition, in order to retain spatial features in the hidden layer of the convolutional neural network in a targeted manner, the present disclosure proposes a spatial feature extraction module, which uses the VGG network to extract the shallow spatial feature map of the peak expression image and uses SENet to assign channel weights to it, which can retain effective spatial features in a targeted manner and reduce the risk of model overfitting.

[0028] Finally, in order to fully integrate the information of temporal features and spatial features, the present invention adopts a weighted average method to fuse the facial expression classification results of the temporal feature extraction module and the spatial feature extraction module to obtain and output the final facial expression classification result.

[0029] Advantages of additional aspects of the present disclosure will be given in part in the following description and in part will become apparent from the following description or will be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0031] Figure 1 It is a flow chart of the facial expression recognition method based on deep spatiotemporal network decision fusion in the first embodiment of the present disclosure;

[0032] Figure 2 This is a flowchart of a facial expression recognition method based on deep spatiotemporal network decision fusion in the first embodiment of the present disclosure;

[0033] Figure 3 It is a design diagram of a temporal feature extraction module in the second embodiment of the present disclosure;

[0034] Figure 4 It is a design diagram of the spatial feature extraction module in the second embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0038] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code may include one or more executable instructions for implementing the logical functions specified in each embodiment. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram can be implemented using a dedicated hardware-based system that performs a specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0039] Embodiment 1

[0040] like Figure 1 As shown, this embodiment provides a facial expression recognition method based on deep spatiotemporal network decision fusion. This embodiment uses the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0041] Step S1: For the original facial expression dataset D, perform face cropping, grayscale processing and data enhancement on each image in the dataset D to obtain the dataset D * ;

[0042] Step S2: For data set D * For each grayscale face image in the dataset D, extract its facial landmark vector and * Select the peak expression image from each facial expression image sequence;

[0043] Step S3: Divide the facial landmark vector into four sub-vectors according to the key facial parts of eyebrows, eyes, nose and mouth; input them into four BiLSTMs respectively to extract the local temporal features of facial expressions. The global temporal features F of facial expressions are obtained by fusing the local temporal features. benm , F benm Send to the softmax classifier to calculate the facial expression classification result P T ;

[0044] Step S4: Use the VGG network to extract the shallow spatial features of the peak expression image, use SENet to assign channel weights to the shallow spatial feature map to obtain a weighted feature map, and use the weighted feature map as the spatial feature F of the facial expression s , F s Send to the softmax classifier to calculate the facial expression classification result P S ;

[0045] Step S5: T and P S Perform weighted fusion, calculate and output the final facial expression classification results.

[0046] In step S1 of the embodiment, face cropping processing is performed on the image sequence in the original facial expression recognition data set to remove images unrelated to facial expressions to obtain a facial expression image sequence; grayscale processing is performed on the facial expression image sequence to avoid the influence of factors such as lighting and color on the classification results and only retain facial expression features; data enhancement is performed on the image sequence after grayscale processing to expand the data set by 14 times by rotating and flipping; for each facial expression image in the expanded data set, its facial landmark points are extracted, and its peak expression image is selected.

[0047] Initialize the input original facial expression data set D = [S1, S2, ..., S N ], where S1, S2, …, S N is the facial expression image sequence of the subject, where n∈N, N represents the number of subjects, and n is the number of images in each subject’s facial expression image sequence. For each image S in the facial expression image sequence of each subject in the dataset D ni Perform face cropping and grayscale processing to obtain a grayscale face image S ni * , where i∈I, I is the number of images contained in the facial expression image sequence. The present disclosure uses the Dlib toolkit to perform face cropping and grayscale processing operations on the image, and the size of the cropped face image is 64×64 pixels;

[0048] Then, an offline data augmentation method was used to rotate each training image by an angle of {-15°, -10°, -5°, 0°, 15°, 10°, 15°}, flip the rotated image on the X axis and resize the image S ni * After flipping and rotating it 14 times, we get the dataset D * .

[0049] Most video datasets are saved in the form of image sequences by segmenting the video into frames. The dataset D here contains the facial expression image sequences of N people (that is, the expression change video data of N people is segmented into the form of image sequences), S1, S2, …, S N It represents the expression image sequence of these N subjects (one for each person). Each expression image sequence of each subject contains several images, denoted by S ni Indicates any one of the images.

[0050] In step S2 of this embodiment, the present disclosure uses the Dlib toolkit to analyze the dataset D * 68 facial landmarks are extracted from each grayscale face image in D, and the last image in each facial expression image sequence is selected as the peak expression image; * Each grayscale face image S in ni * , extract its facial landmarks Where M represents the number of facial landmarks, and is obtained from the dataset D * Select the peak expression image S from each facial expression image sequence npf * .

[0051] In step S3 of the present embodiment, the facial landmark vector is divided into four sub-vectors according to the key parts of the face (eyebrows, eyes, nose and mouth); the number of facial landmarks in the four sub-vectors is 10, 12, 9 and 22 respectively; first, the four sub-vectors are respectively input into four BiLSTMs to extract the local temporal features of the facial expression; then, the local temporal features are fused to obtain the global temporal features F of the facial expression. benm ; Finally, F benm Send it to the softmax classifier to calculate the first facial expression classification result P T .

[0052] For the design of temporal feature extraction module, the facial landmark vector P ni * According to the key parts of the face (eyebrows, eyes, nose and mouth), it is divided into four sub-vectors, and the vector matrices of eyebrows, eyes, nose and mouth are obtained as V eb 、Vey 、V no and V mo ; Input each of the four vector matrices into a BiLSTM, and obtain the local temporal features of eyebrows, eyes, nose and mouth at the output layer of BiLSTM, which correspond to F eb 、F ey 、F no and F mo , the local temporal features are fused to obtain the global temporal features F of facial expressions benm , and send it to softmax for facial expression classification, and save the first facial expression classification result P T (0≤P T (k)≤1)=[P T (1),P T (2),…,P T (K)], where k∈K, K is the number of facial expression classification categories.

[0053] The local temporal features of eyebrows F eb For example, the calculation formula of the BiLSTM hidden layer of the t-th input image is as follows:

[0054] f bt =σ[w bf (h bt-1 ,x bt )+b bf ] (1)

[0055] f bt is the forget gate, which uses the sigmoid activation function to determine how much information in the previous state needs to be discarded, w bf is the calculation weight of the forget gate, b bf represents the calculation bias of the forget gate, x bt is the input vector, h bt-1 Represents the output result of the BiLSTM hidden layer with the t-1th image as input;

[0056] i bt =σ[w bi (h bt-1 ,x bt )+b bi ] (2)

[0057] i bt is the input gate, which determines the information that needs to be retained at the current time node, where σ is the sigmoid activation function, tanh is the hyperbolic tangent activation function, and w bi is the calculation weight of the input gate, b bi represents the calculation bias of the input gate, x bt is the input vector, hbt-1 Represents the output result of the BiLSTM hidden layer with the t-1th image as input;

[0058]

[0059] is the current candidate update unit, which contains all the update information of the current time node. The specific amount of information to be retained is determined by the current update unit c bt Decision, among which is the calculation weight of the current candidate update unit, represents the calculation bias of the current candidate update unit, x bt is the input vector, h bt-1 Represents the output result of the BiLSTM hidden layer with the t-1th image as input;

[0060]

[0061] o bt =σ[w bo (h bt-1 ,x bt )+b bo ] (5)

[0062]

[0063] c bt is the current update unit, which not only obtains the available information of the candidate update unit, but also passes the forget gate f bt Get the previous image c bt-1 The available information is used and the sigmoid activation function is used to determine the output of the current update unit.

[0064] o bt Represents the output gate, and o bt The output information of the control is the same as the c after tanh processing bt Multiply them together to get the forward LSTM hidden layer output of the t-th input image where w bo is the calculation weight of the output gate, b bo represents the calculation bias of the output gate, x bt is the input vector, h bt-1 Represents the output result of the BiLSTM hidden layer with the t-1th image as input.

[0065] Then output to the LSTM hidden layer The calculation formula is the same as the forward one;

[0066] Merge the LSTM hidden layer outputs of the BiLSTM forward and backward passes and Get the BiLSTM hidden layer output h covering forward and backward information bt , which is used as the local temporal feature F eb .

[0067]

[0068]

[0069]

[0070] Splice the local temporal features F corresponding to eyebrows, eyes, nose and mouth respectively eb 、F ey 、F no and F mo Get the global time series feature matrix F benm :

[0071] F benm =[F eb ; F ey ; F no ; F mo ] (10)

[0072] The feature matrix F benm Input into the softmax function to classify facial expressions, P T (k) represents the probability that the current subject's expression prediction belongs to the kth category, k∈K, K is the number of facial expression classification categories, z represents the input vector of the softmax function, and its formula is defined as:

[0073]

[0074] Among them, z j represents the calculated output value of the jth category in the output vector, z k Represents the category output value that needs to be calculated currently, and its loss function can be defined as:

[0075]

[0076] Among them, k∈K, K is the number of facial expression classification categories, T k Indicates the real expression label value of the current subject;

[0077] In step S4 of this embodiment, the input is a peak expression image pixel matrix with a dimension of 64×64×1. First, after two 3×3×64, two 3×3×128, four 3×3×256 and one 3×3×512 convolution operations in the VGG network, a shallow spatial feature map U with a dimension of 8×8×512 is obtained. Then, SENet is used to assign weights to the 512 channels of the shallow spatial feature map and perform weighted operations to obtain a weighted feature map. Finally, the weighted feature map is used as the spatial feature F of the facial expression. s , and sent to softmax to calculate the second facial expression classification result P S .

[0078] For the design of the spatial feature extraction module, first, the peak expression image S npf * Input VGG network to extract shallow spatial feature map U A×B×G , where A×B×G is the spatial feature dimension and G represents the number of feature map channels. Then, SENet is used to explicitly model the interdependence between feature map channels, automatically obtain the weight of each feature map channel, and assign the weights of feature map channels to obtain the spatial features F of facial expressions. s Finally, the spatial feature F s Send it to softmax for facial expression classification, and save the second facial expression classification result as P S (0≤P S (k)≤1)=[P S (1),P S (2),…,P S (K)].

[0079] The calculation process of assigning feature map channel weights using SENet is as follows:

[0080] (1) For feature map U A×B×G For each feature map channel g, calculate an aggregate statistic V g , where g∈G, then V g The calculation formula is:

[0081]

[0082] Where A and B represent the dimension length and width of the two-dimensional feature map in each feature channel g, u g (a, b) represents the g-th two-dimensional feature matrix in the shallow feature map U.

[0083] (2) By using V g The information training parameter w aggregated in the feature map is used to assign weights to feature map channels. The weight calculation amount S for each feature map channel g is gThe calculation process is:

[0084] S g =σ(w2δ(w1V g )) (14)

[0085] Among them, δ represents the relu activation function, r is a hyperparameter.

[0086] (3) The weight calculation amount S of the feature map channel g obtained in step (2) g and the original spatial feature map U g Multiplying together gives the weighted feature map f s (g), the formula is:

[0087] f s (g) = S g U g (15)

[0088] (4) At this time, the spatial feature F of facial expression S =[f s (1),f s (2),…,f s (G)], where g∈G, f s (g) represents the two-dimensional feature map of the g-th feature channel. At this time, F S Represents a weighted spatial feature with dimension A×B×G.

[0089] The spatial feature F S Input into the softmax function to classify facial expressions, P S (k) represents the probability that the current subject's expression prediction belongs to the kth category, k∈K, K is the number of facial expression classification categories, z represents the input vector of the softmax function, and its formula is defined as:

[0090]

[0091] Among them, z j represents the calculated output value of the jth category in the output vector, z k Represents the category output value that needs to be calculated currently, and its loss function can be defined as:

[0092]

[0093] Among them, k∈K, K is the number of facial expression classification categories, T k Indicates the real expression label value of the current subject;

[0094] In step S5 of this embodiment, for the design of the decision fusion algorithm, P S (k) and PT (k) performs weighted fusion, calculates and outputs the final facial expression classification result. The calculation formula of the current facial expression classification result Prediction(k) is:

[0095] Prediction(k)=argmax(αP T (k)+(1-α)P S (k)) (18)

[0096] Where k∈K, K is the number of facial expression classification categories, α=0.5, P T (k) represents the first facial expression classification result, P S (k) represents the second facial expression classification result.

[0097] Embodiment 2

[0098] This embodiment provides a facial expression recognition system based on deep spatiotemporal network decision fusion, including:

[0099] The data preprocessing module is configured to preprocess each facial expression image of the original facial expression data set, extract the facial landmark vector of each facial expression image after preprocessing, and select the peak expression image;

[0100] A temporal feature extraction module is configured to obtain a global temporal feature of a facial expression according to a facial landmark vector, and classify the facial expression based on the global temporal feature of the facial expression to obtain a first facial expression classification result;

[0101] A spatial feature extraction module is configured to obtain spatial features of facial expressions according to the selected peak expression image, perform facial expression classification on the spatial features of facial expressions, and obtain a second facial expression classification result;

[0102] The decision fusion module is configured to perform decision-level fusion on the first facial expression classification result and the second facial expression classification result to obtain a final facial expression classification result.

[0103] It should be noted that the above data preprocessing module, time series feature extraction module, spatial feature extraction module, and decision fusion module correspond to steps S1 to S5 in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Embodiment 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0104] Embodiment 3

[0105] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps in the facial expression recognition method based on deep spatiotemporal network decision fusion as described in the first embodiment above are implemented.

[0106] Embodiment 4

[0107] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for facial expression recognition based on deep spatiotemporal network decision fusion as described in the first embodiment are implemented.

[0108] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0109] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0112] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0113] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A facial expression recognition method based on deep spatiotemporal network decision fusion, characterized in that: include: Preprocess each facial expression image of the original facial expression data set, extract the facial landmark point vector of each preprocessed facial expression image, and select the peak expression image; The global temporal features of facial expressions are obtained according to the facial landmark vectors, and facial expressions are classified based on the global temporal features of facial expressions to obtain the first facial expression classification result, which is specifically: Divide the facial landmark vector into four sub-vectors according to eyebrows, eyes, nose and mouth; The four sub-vectors are input into four bidirectional long short-term memory networks BiLSTM to extract the local temporal features of facial expressions; The local temporal features of facial expressions are integrated to obtain the global temporal features of facial expressions; The global features of facial expressions are classified using a softmax classifier to obtain the first facial expression classification result; The spatial features of facial expressions are obtained according to the selected peak expression image, and facial expressions are classified based on the spatial features of facial expressions to obtain the second facial expression classification result, which is specifically: The shallow spatial feature map of the sewn expression images is extracted using super-resolution test sequences; SENet is used to assign channel weights to shallow spatial feature maps, and the weighted feature maps are used as spatial features of facial expressions; The spatial features of facial expressions are classified using a softmax classifier to obtain a second facial expression classification result; The first facial expression classification result and the second facial expression classification result are fused at the decision level to obtain the final facial expression classification result.

2. The facial expression recognition method based on deep spatiotemporal network decision fusion as claimed in claim 1, characterized in that: The preprocessing of each facial expression image of the original facial expression data set is specifically as follows: Perform face cropping on each facial expression image in the original facial expression data set, remove images irrelevant to facial expressions, and obtain a facial expression image sequence; Perform grayscale processing on facial expression image sequences to retain only facial expression features; The facial expression image sequences after grayscale processing are enhanced by rotating and flipping the dataset to expand it 14 times.

3. The facial expression recognition method based on deep spatiotemporal network decision fusion as claimed in claim 1, characterized in that: The four sub-vectors are respectively input into four bidirectional long short-term memory networks BiLSTM to extract the local temporal features of facial expressions, specifically: The four sub-vectors are input into a BiLSTM respectively, and the local temporal features of eyebrows, eyes, nose and mouth are obtained at the output layer of BiLSTM, corresponding to F eb 、F ey 、F no and F mo ; The local temporal features of eyebrows F eb For example, the calculation formula of the BiLSTM hidden layer of the t-th input image is as follows: f bt =σ[w bf (h bt-1 ,x bt )+b bf ] (1) f bt It is the forget gate, which uses the sigmoid activation function to determine how much information in the previous state needs to be lost. bf is the calculation weight of the forget gate, b bf represents the calculation bias of the forget gate, x bt is the input vector, h bt-1 Represents the output result of the BiLSTM hidden layer with the t-1th image as input; i bt =σ[w bi (h bt-1 ,x bt )+b bi ] (2) i bt is the input gate, which determines the information that needs to be retained at the current time node, where σ is the sigmoid activation function, tanh is the hyperbolic tangent activation function, and w bi is the computational weight of the input gate, b bi represents the calculation bias of the input gate, x bt is the input vector, h bt-1 Represents the output result of the BiLSTM hidden layer with the t-1th image as input; is the current candidate update unit, which contains all the update information of the current time node. The specific amount of information to be retained is determined by the current update unit c bt Decision, among which is the calculation weight of the current candidate update unit, represents the calculation bias of the current candidate update unit, x bt is the input vector, h bt-1 Represents the output result of the BiLSTM hidden layer with the t-1th image as input; the bt =σ[w bo (h bt-1 ,x bt )+b bo ] (5) c bt is the current update unit, which obtains the available information of the candidate update unit through the forget gate f bt Get the previous image c bt-1 The available information is used, and the sigmoid activation function is used to determine the output of the current update unit, o bt Represents the output gate, and o bt The output information of the control is the same as the c after tanh processing bt Multiply them together to get the forward LSTM hidden layer output of the t-th input image where w bo is the calculation weight of the output gate, b bo represents the calculation bias of the output gate, x bt is the input vector, h bt-1 Represents the output of the BiLSTM hidden layer with the t-1th image as input, and then outputs to the hidden layer The calculation method of is the same as the forward direction; Merge the hidden layer outputs of the BiLSTM forward and backward passes and Get the BiLSTM hidden layer output h covering forward and backward information bt , which is used as the local temporal feature F eb ; 4. The facial expression recognition method based on deep spatiotemporal network decision fusion as claimed in claim 1, characterized in that: SENet is used to assign channel weights to shallow spatial feature maps. The specific process is as follows: Step (1): For feature map U A×B×G For each feature map channel g, calculate an aggregate statistic V g , where g∈G, then V g The calculation formula is: Where A and B represent the dimension length and width of the two-dimensional feature map in each feature channel g, u g (a, b) represents the g-th two-dimensional feature matrix in the shallow feature map U; Step (2): By using V g The information training parameter w aggregated in the feature map is used to assign weights to feature map channels. The weight calculation amount S for each feature map channel g is g The calculation process is: S g =σ(w2δ(w1V g )) (14) Among them, δ represents the relu activation function, r is a hyperparameter; Step (3): Substitute the weight calculation amount S of the feature map channel g obtained in step (2) g And the original spatial feature map U g Multiplying together gives the weighted feature map f s (g), whose formula is: f s (g)=S g U g (15); Step (4): Spatial features of facial expressions F S =[f s (1),f s (2),…,f s (G)], where g∈G, f s (g) represents the two-dimensional feature map of the g-th feature channel; At this time, F S Represents a weighted spatial feature with dimension A×B×G.

5. The facial expression recognition method based on deep spatiotemporal network decision fusion as claimed in claim 1, characterized in that: The calculation formula for the decision-level fusion of the first facial expression classification result and the second facial expression classification result is: Prediction(k)=argmax(αP T (k)+(1-α)P S (k)) (18).

6. A facial expression recognition system based on deep spatiotemporal network decision fusion, characterized in that: include: The data preprocessing module is configured to preprocess each facial expression image of the original facial expression data set, extract the facial landmark vector of each facial expression image after preprocessing, and select the peak expression image; The temporal feature extraction module is configured to obtain the global temporal features of the facial expression according to the facial landmark vector, perform facial expression classification on the global temporal features of the facial expression, and obtain the first facial expression classification result, which is specifically: Divide the facial landmark vector into four sub-vectors according to eyebrows, eyes, nose and mouth; The four sub-vectors are input into four bidirectional long short-term memory networks BiLSTM to extract the local temporal features of facial expressions; The local temporal features of facial expressions are integrated to obtain the global temporal features of facial expressions; The global features of facial expressions are classified using a softmax classifier to obtain the first facial expression classification result; The spatial feature extraction module is configured to obtain the spatial features of the facial expression according to the selected peak expression image, perform facial expression classification on the spatial features of the facial expression, and obtain a second facial expression classification result, specifically: The shallow spatial feature map of the sewn expression images is extracted using super-resolution test sequences; SENet is used to assign channel weights to shallow spatial feature maps, and the weighted feature maps are used as spatial features of facial expressions; The spatial features of facial expressions are classified using a softmax classifier to obtain a second facial expression classification result; The decision fusion module is configured to perform decision-level fusion on the first facial expression classification result and the second facial expression classification result to obtain a final facial expression classification result.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a facial expression recognition method based on deep spatiotemporal network decision fusion as described in any one of claims 1 to 5 are implemented.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the facial expression recognition method based on deep spatiotemporal network decision fusion as described in any one of claims 1-5 are implemented.

Images