A multi-task expression recognition method and system based on a deep neural network

Abstract

The application discloses a kind of multi-task expression recognition method and system based on deep neural network, comprising: obtaining face image;Face image is transported to joint network model;Feature extraction network is used to carry out key point prediction to face image to produce the heat map prediction of face feature point, and heat map prediction is used as the key point feature information of face;FPN pyramid model is used to carry out up-sampling and down-sampling to the key point feature information of face, and the down-sampling feature obtained by down-sampling is transported to encoder;Encoder is used to encode the input down-sampling feature, and the down-sampling feature after encoding is transported to FPN pyramid model and is output with the feature information of different levels after up-sampling fusion;First multilayer perceptron layer is used to perceive classification to the input feature information of different levels, to output the recognition prediction result of face facial expression;The expression recognition method and system improve the prediction performance of network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of expression recognition, and particularly relates to a multi-task expression recognition method and system based on a deep neural network. BACKGROUND

[0002] Compared with other types of intelligent recognition technologies, facial expression recognition technology has not yet broken through the technical barrier and is insufficient in life application. Especially in the technical field related to emotional computing and emotional analysis, the current expression recognition research is still very immature. At present, most of the existing expression recognition technologies are for discrete expression recognition tasks, and the recognition environment is relatively simple. When problems such as poor light conditions, image blur, posture deviation, and occlusion are processed, the results are not satisfactory, and practical application cannot be achieved. At the same time, since these simple expression classifications cannot reflect the relatively complex and rich emotions exhibited by humans in natural daily situations, they are also insufficient in emotional analysis.

[0003] Expression recognition based on a continuous emotional dimension model is extremely difficult for computer-based systems. In recent years, more and more research on expression recognition based on a continuous emotional dimension has been conducted. However, the above work is still insufficient for processing emotional analysis expression recognition tasks. SUMMARY

[0004] Based on the technical problems existing in the background art, the present application proposes a multi-task expression recognition method and system based on a deep neural network, which optimizes the facial expression recognition network and improves the prediction performance of the network.

[0005] The multi-task expression recognition method based on a deep neural network proposed by the present application comprises the following steps:

[0006] Obtaining a facial image;

[0007] Transporting the facial image to a joint network model, wherein the joint network model comprises a feature extraction network, an FPN pyramid model, an encoder, and a first multi-layer perception layer;

[0008] The feature extraction network is used for key point prediction of the facial image to generate a heat map prediction of facial feature points, and the heat map prediction is used as key point feature information of the face;

[0009] The FPN pyramid model is used for up-sampling and down-sampling of the key point feature information of the face, and down-sampling features obtained by down-sampling are transported to the encoder;

[0010] The encoder is used for encoding processing of the input down-sampling features, and the encoded down-sampling features are transported to the FPN pyramid model to be fused with up-sampling and then output feature information of different levels;

[0011] The first multi-layer perception layer is used for perceiving and classifying different levels of feature information of the input to output a recognition prediction result of a facial expression in the face image.

[0012] Further, the feature extraction network comprises a fourth-order hourglass model formed by stacking hourglass models, and the hourglass model adopts a ConvBlock bottleneck block connection.

[0013] The FPN pyramid model is a three-layer pyramid model, and up-sampling and down-sampling form up-sampling features and down-sampling features respectively, and the output of the down-sampling features of each layer is connected to the input of the up-sampling features through an encoder set in the corresponding layer.

[0014] Further, the encoder comprises a normalization layer, a multi-head self-attention layer and a second multi-layer perception layer connected in sequence.

[0015] The input of the first normalization layer is connected to the down-sampling features of the same layer, the feature sequence obtained by adding the input down-sampling features and the output of the multi-head self-attention layer is taken as the input of the second normalization layer, and the feature sequence is added to the output of the multi-layer perception layer to output three different levels of feature information, and the three different levels of feature information are taken as the input of the first multi-layer perception layer.

[0016] Further, the multi-head self-attention layer comprises a linear transformation layer, a scaled dot-product attention layer, a concatenation layer and a linear output layer connected in sequence, wherein the linear transformation layer and the scaled dot-product attention layer are arranged in each head of the multi-head self-attention layer, the concatenation layer and the linear output layer are arranged in sequence in the multi-head self-attention layer as a whole, and the linear connection layer is provided with three transformation mapping layers, and the three transformation mapping layers are respectively used to map the normalized down-sampling features to three matrices through three linear transformations, respectively, and the three matrices comprise a query matrix Q, a key matrix K and a value matrix V.

[0017] Further, the scaled dot-product attention layer comprises a first matrix multiplication layer, an array standard layer, a softmax regression layer and a second matrix multiplication layer.

[0018] The input of the first matrix multiplication layer is connected to the query matrix Q and the key matrix K, the first matrix multiplication layer, the array standard layer and the softmax regression layer are connected in sequence, and the input of the second matrix multiplication layer is connected to the output of the softmax regression layer and the value matrix V.

[0019] Further, the linear transformation layer scales the input sequence after normalization, specifically as follows:

[0020] Q=XW Q

[0021] K=XW K

[0022] V = XW V

[0023] wherein Q represents a query matrix, K represents a key matrix, V represents a value matrix, and are matrix coefficients, and D is a feature dimension.

[0024] Further, the processing formula of the down-sampling feature in the encoder is as follows:

[0025] MSA(Q,K,V) = Concat(head1,...,head h )W i O

[0026]

[0027] X' = MSA(Q,K,V) + X

[0028] X out = MLP + X'

[0029] wherein d = D / h, i = 1, 2,... h, h is the number of multi-head self-attention heads, D is a feature dimension, MSA represents a multi-head self-attention layer, Norm is a normalization operator, MLP represents a multi-layer perceptron, Concat represents concatenation, X represents a down-sampling feature input to the multi-head self-attention layer, X out represents an output of the encoder,

[0030] Further, the key point feature information R of the face is:

[0031]

[0032] wherein u and v are pixel point coordinates, k i and k j represent predicted key point coordinates, and sigma is a set radiation radius with the key point as the center.

[0033] A multi-task expression recognition system based on a deep neural network comprises an acquisition module, an input module, a feature extraction network module, an FPN pyramid module, an encoder module, and a multi-layer perceptron module.

[0034] The acquisition module is configured to acquire a face image.

[0035] The input module is used for conveying a face image into a joint network model, and the joint network model comprises a feature extraction network, an FPN pyramid model, an encoder and a first multi-layer perception layer;

[0036] The feature extraction network module is used for key point prediction on the face image to generate a heat map prediction of face feature points, and the heat map prediction is taken as key point feature information of the face;

[0037] The FPN pyramid module is used for up-sampling and down-sampling on the key point feature information of the face, and down-sampled features obtained through the down-sampling are conveyed into the encoder;

[0038] The encoder module is used for encoding processing on the input down-sampled features, and the encoded down-sampled features are conveyed into the FPN pyramid model to be fused with up-sampled features and then output feature information of different levels;

[0039] The multi-layer perception module is used for perception classification on the input feature information of different levels to output a recognition prediction result of a face expression in the face image.

[0040] The multi-task expression recognition method and system based on the deep neural network have the advantages that the multi-task expression recognition method and system based on the deep neural network provided in the structure ensure the recognition performance under natural conditions, can simultaneously output prediction results of 68 face feature points, classification results of discrete expressions and regression results of continuous expressions, are better in distinguishing the similarity of image features, are mainly aimed at the intra-class difference and the inter-class similarity, further optimize the joint network model, use the attention mechanism of the Vision Transformer as a basis to build a feature pyramid structure to improve the expression recognition method, optimize the face expression recognition network and improve the prediction performance of the network, and the construction of the joint network model enables the joint network model to obtain more useful feature information in the emotion recognition process, and the face alignment result and the expression recognition result are more accurate and persuasive. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The figure is a flowchart of the present application;

[0042] Figure 2 The figure is a structure diagram of the joint network model;

[0043] Figure 3 The figure is a diagram of the FPN pyramid model;

[0044] Figure 4 The figure is a structure diagram of the encoder;

[0045] Figure 5 Structure diagram of a multi-head self-attention layer;

[0046] Figure 6 Structure diagram of one of the scaled dot-product attention layers;

[0047] Figure 7 Flowchart of use of the present application;

[0048] Figure 8 The recognition prediction result output after the face image is processed by the joint network model. DETAILED DESCRIPTION

[0049] Hereinafter, the technical solutions of the present application will be described in detail through specific embodiments. In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the scope of the present application. Therefore, the present application is not limited by the specific implementation disclosed below.

[0050] In the process of coping with multi-modal ubiquitous psychological computing tasks, it is found that a single result output network model is very difficult in joint tasks, and building multiple network models will cause a great burden to the entire multi-modal task. The present patent intends to use the existing face alignment framework, and on this basis, fuse the Transformer network model to build an end-to-end multi-output deep neural network architecture. This architecture can jointly perform face alignment and simultaneously predict the classification of discrete emotions and the regression estimation of continuous emotions. At the same time, the Transformer network structure model is used to make corresponding improvements to the regression and classification task parts, add self-attention mechanism, MLP, and use FPN network structure to build this part of the network, etc., to more effectively utilize facial landmark information, so as to maximize attention to the significant features related to expressions in the facial region, in order to solve the intra-class difference and inter-class similarity problems in sample images. On this basis, an expression recognition system with real-time detection function is designed to test the performance of the algorithm model in real-time application. This system needs to have data import function to obtain the detection target, and needs to have multi-task expression recognition function to output the prediction results of 68 facial feature points, the classification results of discrete expressions, and the regression results of continuous dimensional emotions, etc. The system backend is deployed using the flask framework, and the front end is built using Vue3, as follows.

[0051] As Figures 1 to 8 shown, the multi-task expression recognition method based on a deep neural network proposed by the present application includes S1 to S6:

[0052] S1: obtaining a face image;

[0053] A camera is used to acquire target information of a person's expression under natural conditions.

[0054] S2: delivering the face image into a joint network model, the joint network model comprising a feature extraction network, an FPN pyramid model, an encoder and a first multi-layer perception layer;

[0055] S3: the feature extraction network is used for key point prediction of the face image to generate a heat map prediction of face feature points, and the heat map prediction is taken as key point feature information of the face;

[0056] The feature extraction network of the embodiment is stacked by hourglass models, all of which are connected by ConvBlock bottleneck blocks. The preprocessed picture is used for face expression feature extraction by a fourth-order hourglass model. The fourth-order hourglass model is used for feature extraction. The fourth-order hourglass model down-samples the input face image to a very small resolution, and then up-samples it to combine the features of the same size. Specifically, before each down-sampling, the upper half is separated to retain the original scale information; after each up-sampling, the data of the previous scale is added.

[0057] The key point feature information R of the face is:

[0058]

[0059] Where u and v are pixel coordinates, k i and k j represent the predicted key point coordinates, and sigma is the set radiation radius with the key point as the center.

[0060] It can be known from the formula of the face key point feature information R that the response value generated by the Gaussian formula for the pixel value closer to the key point is larger. By calculating the 68 key points respectively and concentrating them on a picture, a face heat map including the responses of the 68 key points can be obtained.

[0061] Because if only the heat map information extracted by the last hourglass model is selected during feature extraction, a lot of key information in the middle may be missing. The feature extraction network of the embodiment uses four fourth-order hourglass models in cascade. Each fourth-order hourglass model generates a heat map prediction. The heat map information generated by the previous fourth-order hourglass model can be used as auxiliary information for the prediction of the next fourth-order hourglass model. Through the cascade of the four networks, the deviation of the key point prediction can be reduced, the position of the face key point can be continuously refined, the accuracy of the finally formed heat map can be improved, and the last heat map prediction is taken as the key point feature information of the face.

[0062] After the 68 facial key point feature information is extracted, the feature extraction network extracts the facial landmark key point feature Wherein P is the number of patches, and D is the feature dimension.

[0063] S4: The FPN pyramid model is used for upsampling and downsampling of the key point feature information of the face, and the down-sampled feature obtained by the downsampling is transmitted to the encoder;

[0064] Figure 3 The network structure shown is an FPN network model. Since the pyramid network structure thereof can obtain multiple features of different scales, in solving the problem of the present application, the expression features of the part with relatively high similarity can be distinguished by combining the shallow features and the high-level features, so as to improve the performance of the joint network model.

[0065] The pyramid structure of the FPN includes downsampling from bottom to top and upsampling from top to bottom, and the upsampling and the downsampling are laterally connected through the encoder. The FPN pyramid model is a three-layer pyramid model. The upsampling and the downsampling form the up-sampled feature and the down-sampled feature, respectively. The output of the down-sampled feature of each layer is connected to the input of the up-sampled feature through the encoder arranged in the corresponding layer. The size between the next layer feature map and the previous layer feature map is generally a two-fold relationship. When the features are fused, the topmost layer feature map needs to be up-sampled by two times and then added to the feature map of the same size extracted by the down-sampling part through the encoder, so that the feature fusion of the shallow features and the high-level features is completed, the semantic information of the image features is enriched, and finally three different levels of feature information are obtained to perform expression classification and regression prediction.

[0066] In detail, the key point feature is input into the three-layer FPN pyramid model constructed. The feature information obtained by each layer is different, and then the up-sampling operation is performed from top to bottom for each layer. In order to better combine the high-level features and the shallow features, the encoder is adopted in the present embodiment to fuse the feature information after the up-sampling operation of the feature information layer on the right side and the encoding of the feature information layer on the left side and the attention pooling of the multi-head self-attention mechanism. Finally, three different levels of feature information are obtained to perform expression classification and regression prediction. Figure 1

[0067] S5: The encoder is used for encoding processing on the input down-sampled feature, and the encoded down-sampled feature is transmitted to the FPN pyramid model for fusion with the up-sampling and output of different levels of feature information;

[0068] The horizontal connection of each layer feature fusion in the three-layer FPN pyramid model is to directly apply the encoder to process the facial key point feature information ​The self-attention mechanism is implemented by a multi-head self-attention layer (MSA) in the encoder.

[0069] The encoder is constructed as follows:

[0070] The encoder comprises, in sequence, a normalization layer, a multi-head self-attention layer and a second multi-layer perceptron layer; the input of the first normalization layer is connected to the down-sampled features of the same layer, the feature sequence obtained by adding the input down-sampled features and the output of the multi-head self-attention layer is taken as the input of the second normalization layer, and the feature sequence is added to the output of the multi-layer perceptron layer to output three different levels of feature information, which is taken as the input of the first multi-layer perceptron layer.

[0071] The multi-head self-attention layer comprises, in sequence, a linear transformation layer, a scaled dot-product attention layer, a concatenation layer and a linear output layer, wherein the linear transformation layer and the scaled dot-product attention layer are arranged in each head of the multi-head self-attention layer, the concatenation layer and the linear output layer are arranged in the multi-head self-attention layer in sequence as a whole, and the linear connection layer is provided with three transformation mapping layers, which are respectively used to map the normalized down-sampled features to three matrices through three linear transformations, respectively, wherein the three matrices comprise a query matrix Q, a key matrix K and a value matrix V.

[0072] The scaled dot-product attention layer comprises a first matrix multiplication layer, an array standard layer, a softmax regression layer and a second matrix multiplication layer; the input of the first matrix multiplication layer is connected to the query matrix Q and the key matrix K, the first matrix multiplication layer, the array standard layer and the softmax regression layer are connected in sequence, and the input of the second matrix multiplication layer is connected to the output of the softmax regression layer and the value matrix V.

[0073] The encoding process of the encoder on the input down-sampled features is specifically as follows:

[0074] As shown in Figure 4 The encoder delivers the obtained down-sampled features to the first normalization layer Norm for normalization processing, which mainly adopts LN (LayerNorm) here. In this process, the dimension of the feature information does not change. Then it is sent to the multi-head attention layer Multi-Head Attention for processing. As shown in Figure 5The illustrated structure is a multi-head self-attention layer structure. Multi-head self-attention is transformed from a self-attention mechanism. In actual application, the self-attention mechanism pays too much attention to its own position, easily ignores the relationship with other vectors and the rationality of a position as a whole, and affects the performance of the entire network. The birth of multi-head attention is to make up for this defect and to make the attention mechanism pay more attention to feature information, so that more feature information can be obtained. Then, the second normalization layer Norm is used for LN normalization processing. Finally, a second multi-layer perceptron layer MLP is used for final classification. The structure of the second multi-layer perceptron layer MLP is relatively simple and is composed of two fully connected layers and a GELU activation function.

[0075] In detail, the face feature information X (X is the down-sampled feature) of the input encoder first needs to be scaled and normalized by similarity measurement to obtain normalized feature information X is mapped to three matrices: query matrix Q, key matrix K and value matrix V through three linear transformations, as shown in Figure 6 The MSA multi-head self-attention network result is shown:

[0076] Q=XW Q

[0077] K=XW K

[0078] V=XW V

[0079] Wherein, Q represents the query matrix, K represents the key matrix, and V represents the value matrix, and are matrix coefficients, and D is the feature dimension.

[0080] Next, the multi-head self-attention layer Multi-Head Attention is input to perform attention pooling operation. When the multi-attention head is n, that is, there are n self-attention heads, then the dimensions of the n groups of query matrix, key matrix and value matrix are 1 / n of the original, but finally the n parallel self-attention heads need to be spliced. The encoder output maintains the same size as the encoder input and is represented as follows:

[0081] MSA(Q,K,V)=Concat(head1,…,head h )W i O

[0082]

[0083] X' = MSA(Q, K, V) + X

[0084] X out = MLP + X'

[0085] wherein d = D / h, i = 1, 2, … h, h is the number of multi-head self-attention heads, D is the feature dimension, MSA represents a multi-head self-attention layer, Norm is a normalization operator, MLP represents a multi-layer perceptron, Concat represents concatenation, and X represents down-sampled features input to the multi-head self-attention layer, X out represents the output of the encoder,

[0086] Finally, the output X out is scaled and normalized and then sent to the first multi-layer perceptron layer MLP head to return the predicted emotion label wherein N is the number of final task output categories.

[0087] S6: The first multi-layer perceptron layer is configured to perceive and classify different levels of feature information of the input to output a recognition prediction result of a facial expression of the face in the face image.

[0088] The overall structure of the final joint network model is shown in Figure 2 The network combines a feature extraction network with an HourGlass model as the backbone, uses an encoder (a Transformer network model) and an FPN pyramid network structure, adds a multi-head self-attention mechanism, and builds a classification and regression prediction module with an encoder and a multi-layer perceptron. On this basis, an end-to-end multi-output deep neural network architecture is constructed, which can jointly perform face alignment and simultaneously predict the classification of discrete emotions and the regression estimation of continuous emotions.

[0089] According to the system framework required by the above functions, the main purpose of building the system in this embodiment is to test the real-time application performance of the multi-task expression recognition method. The backend is deployed using the flask framework, and the frontend is designed using Vue3. The advantage of using the flask framework based on python is that the backend is very simple to build, and there is no need to additionally arrange algorithm deployment. Only the successfully verified algorithm model needs to be called out. The front-end design mainly reflects the function use, and the function deployment is mainly divided into static image recognition and dynamic video sequence recognition (by calling the camera API interface of HTML5 to realize the function of real-time monitoring of human emotions). When designing the software, first determine the functional requirements of the visualization display part and the data format of the chart rendering, and then determine the function implementation of the backend interface according to this. Then, code is written according to this data request and response process.

[0090] The specific flow is as shown in the figure: the user performs the operation of the data import function through the designed interactive interface, and sends the user demand to the front-end browser interface. The browser sends the request and the data information to be detected from the front end to the back end according to the indication received by the user, and the back-end controller uses the facial expression recognition algorithm deployed in the embodiment to perform facial expression recognition prediction on the information transmitted to the back end by calling the corresponding method interface. Finally, the front-end browser needs to receive the result information of the expression recognition prediction and present the output result on the front-end interface through interface rendering and data processing. Figure 7 The final effect is as shown in the figure.

[0091] Figure 8 The system functions mainly include data import function and expression recognition function: (1) the data import function has two choices, including data file uploading and camera real-time monitoring. The data file generally has two formats, picture file and video file. The data import function also has withdrawal and deletion to operate the error data. (2) The expression recognition function is mainly to realize the expression recognition function by using the joint network model recorded in the embodiment; the specific operation is to call the imported data for functional recognition, among which the imported picture data is directly subjected to expression recognition function; and the imported video data is extracted from every second to 12 frames for expression recognition. The main recognition results output by the facial expression recognition system built in this paper include 68 facial key points, discrete expression recognition classification, and continuous emotion, i.e. valence and arousal estimation.

[0092] In summary, when processing the emotional analysis calculation task, the embodiment provides a multi-task expression recognition method based on deep neural network, which guarantees the recognition performance under natural conditions, and enables it to simultaneously output the prediction results of 68 facial feature points, the classification results of discrete expressions and the regression results of continuous expressions, and integrates all the above contents into an end-to-end joint network model.

[0093] In addition, in the expression recognition method of the embodiment, in order to better distinguish the similarity of image features, mainly aiming at the intra-class difference problem and the inter-class similarity problem, the joint network model is optimized again, the attention mechanism of VisionTransformer is used, and a feature pyramid structure is built on this basis to improve the expression recognition method, so as to optimize the facial expression recognition network and improve the prediction performance of the network.

[0094] ​The multi-task recognition researched in the embodiment includes 68 facial feature points, discrete expression classification results and continuous expression regression results; the focus is on multi-task combination, rather than dynamic or static expression recognition. Whether processing dynamic tasks or static tasks, whether single person or multiple persons, as long as it is above the level of naked eye distinguishable, emotion labeling can be performed; especially in the field of multi-modal emotion analysis and human-computer interaction, the multi-task method designed by the application can well perform many research works; therefore, the application has a relatively wide range of uses, and has a relatively wide range of applications in medical monitoring systems, intelligent conversation systems, smart campuses and auxiliary safe driving, etc.

[0095] The above merely describes a preferred specific embodiment of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes within the technical range disclosed by the application according to the technical solution and inventive concept of the application, which should be covered within the protection scope of the application.

Claims

1. A deep neural network based multi-task expression recognition method, characterized in that, The method comprises the following steps: obtaining a face image; feeding the face image into a joint network model, wherein the joint network model comprises a feature extraction network, an FPN pyramid model, an encoder and a first multi-layer perception layer; the feature extraction network is used for key point prediction on the face image to generate a heat map prediction of face feature points, and the heat map prediction is used as key point feature information of the face; the FPN pyramid model is used for up-sampling and down-sampling of the key point feature information of the face, and down-sampled features obtained by down-sampling are fed into the encoder; the encoder is used for encoding processing of the input down-sampled features, and the encoded down-sampled features are fed into the FPN pyramid model to output feature information of different levels after being fused with up-sampled features; the first multi-layer perception layer is used for perception classification of the input feature information of different levels to output a recognition prediction result of a facial expression in the face image; the feature extraction network comprises a fourth-order hourglass model formed by stacking hourglass models, and the hourglass models are connected by ConvBlock bottleneck blocks; the FPN pyramid model is a three-layer pyramid model, and up-sampling and down-sampling form up-sampled features and down-sampled features respectively, and the output of the down-sampled features of each layer is connected to the input of the up-sampled features through the encoder arranged in the corresponding layer; the encoder comprises a normalization layer, a multi-head self-attention layer and a second multi-layer perception layer connected in sequence; the input of the first normalization layer is connected to the down-sampled features of the same layer, a feature sequence obtained by adding the input down-sampled features and the output of the multi-head self-attention layer is used as the input of the second normalization layer, and the feature sequence is added with the output of the multi-layer perception layer to output three feature information of different levels, which are used as the input of the first multi-layer perception layer; the processing formula of the down-sampled features in the encoder is as follows: wherein Q represents a query matrix, K represents a key matrix, and V represents a value matrix, , , , , , , is the number of multi-head self-attention heads, is a feature dimension, MSA represents a multi-head self-attention layer, Norm is a normalization operator, and MLP represents a multi-layer perceptron, represents concatenation, represents down-sampled features input to the multi-head self-attention layer, , represents an output of the encoder, . 2.The deep neural network based multi-task expression recognition method of claim 1, wherein, the multi-head self-attention layer comprises a linear transformation layer, a scaled dot-product attention layer, a concatenation layer and a linear output layer connected in sequence, wherein the linear transformation layer and the scaled dot-product attention layer are arranged in each head of the multi-head self-attention layer, the concatenation layer and the linear output layer are arranged in sequence in the multi-head self-attention layer as a whole, and the linear connection layer is provided with three transformation mapping layers, which are respectively used for mapping the normalized down-sampled features into three matrices through three linear transformations, and the three matrices comprise a query matrix Q, a key matrix K and a value matrix V. 3.The deep neural network based multi-task expression recognition method of claim 2, wherein, The scaled dot-product attention layer includes a first matrix multiplication layer, an array standard layer, softmax a regression layer, and a second matrix multiplication layer. The input connections of the first matrix multiplication layer are the query matrix Q and the key matrix K. (First matrix multiplication layer, array standard layer, ...) softmax The regression layers are connected sequentially, and the input of the second matrix multiplication layer is connected sequentially. softmax The output and value matrix V of the regression layer. 4.The deep neural network based multi-task expression recognition method of claim 2, wherein, the linear transformation layer scales the input sequence after normalization, and the scaling is as follows: wherein, , and are matrix coefficients, D is the feature dimension. 5.The deep neural network based multi-task expression recognition method of claim 1, wherein, Key point feature information of the face : wherein, and are pixel point coordinates, and represent predicted key point coordinates, is a set radiation radius with the key point as the center.

6. A deep neural network based multi-task expression recognition system, characterized in that, The method comprises an acquisition module, an input module, a feature extraction network module, an FPN pyramid module, an encoder module and a multi-layer perception module; the acquisition module is used for acquiring a face image; the input module is used for feeding the face image into a joint network model, wherein the joint network model comprises a feature extraction network, an FPN pyramid model, an encoder and a first multi-layer perception layer; The feature extraction network module is configured to perform key point prediction on the face image to generate a heat map prediction of face feature points, and take the heat map prediction as key point feature information of the face; The FPN pyramid module is configured to perform up-sampling and down-sampling on the key point feature information of the face, and transmit down-sampled features to the encoder; The encoder module is configured to perform encoding processing on the input down-sampled features, and transmit the encoded down-sampled features to the FPN pyramid model for fusion with up-sampled features to output feature information of different levels; The multi-layer perception module is configured to perform perception classification on the input feature information of different levels to output a recognition prediction result of a facial expression in the face image; The feature extraction network includes a fourth-order hourglass model stacked by hourglass models, and the hourglass models are connected by ConvBlock bottleneck blocks; The FPN pyramid model is a three-layer pyramid model, and the up-sampling and down-sampling form up-sampled features and down-sampled features, respectively, and the output of the down-sampled features of each layer is connected to the input of the up-sampled features through the encoder arranged in the corresponding layer; The encoder includes a normalization layer, a multi-head self-attention layer, and a second multi-layer perception layer connected in sequence; The input of the first normalization layer is connected to the down-sampled features of the same layer, the feature sequence obtained by adding the input down-sampled features and the output of the multi-head self-attention layer is taken as the input of the second normalization layer, and the feature sequence is added with the output of the multi-layer perception layer to output three feature information of different levels, which are taken as the input of the first multi-layer perception layer; The processing formula of the down-sampled features in the encoder is as follows: wherein Q denotes a query matrix, K denotes a key matrix, and V denotes a value matrix, , , , , , , is the number of multi-head self-attention heads, is a feature dimension, MSA denotes a multi-head self-attention layer, Norm is a normalization operator, and MLP represents a multi-layer perceptron, denotes concatenation, denotes down-sampled features input to the multi-head self-attention layer, , denotes an output of the encoder, .

Images