An emotion recognition method and system using multi-cue joint learning

Abstract

The present invention discloses an emotion recognition method and system using multi-cue joint learning, comprising the following steps: extracting facial expression data and body posture data from acquired interactive videos, and processing the facial expression data and body posture data into facial key feature points and skeleton joint points; inputting the facial key feature points and skeleton joint points into a pre-trained multi-cue network model to output predicted emotion recognition categories, wherein the multi-cue network model includes a facial extraction module, a skeleton extraction module, a feature fusion module, a loss function calculation module, and a loss gradient optimization module; the emotion recognition method and system combine facial and body cues, utilize multi-label multi-loss features to optimize loss calculation, and improve the emotion recognition accuracy during human-computer interaction.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to an emotion recognition method and system using multi-cue joint learning. Background Art

[0002] Patent [CN110370275A] provides an emotional chatbot based on facial expression recognition. This invention simulates human chat scenes and incorporates a system for controlling and playing emotional animations, maximizing the effect of real-life chats. This ensures the authenticity, fun, and interactivity of the game scene, providing users with genuine care and a strong practical value.

[0003] Patent [CN110046580A] discloses a human-computer interaction method and system based on emotion recognition. The method comprises: acquiring a user image in real time, performing image processing on the user image based on iris recognition technology and facial image processing technology to obtain the user's facial recognition information; inputting the facial recognition information into a pre-trained deep neural network for facial recognition processing to obtain a facial recognition result of the user; the facial recognition result includes emotional information; determining the user's current emotional state based on the emotional information, and adjusting the corresponding emotional interaction mode so that the robot responds to the user accordingly. By combining iris recognition technology and facial image processing technology, the present invention enables the robot to efficiently identify the user's emotions and provide different responses based on different emotions, thereby meeting the user's emotional needs at a specific moment.

[0004] Patent [CN113741702B] discloses a human-computer interaction method and system for cognitive impairment based on emotion monitoring. The human-computer interaction method includes obtaining the user's cognitive ability level; generating a human-computer interaction plan based on the cognitive ability level; generating the Nth human-computer interaction task based on the human-computer interaction plan; emotionally monitoring the user's performance in the Nth human-computer interaction task, and returning to the previous step to generate the Nth human-computer interaction task based on the human-computer interaction plan until all tasks in the human-computer interaction plan are completed. The human-computer interaction method mainly includes cognitive assessment and human-computer interaction tasks. At the same time, it monitors the user's various basic emotional information in real time during the human-computer interaction process and obtains the user's comprehensive emotional index. The comprehensive emotional index is used to adjust the user's subsequent human-computer interaction tasks and status, thereby achieving the purpose of improving the cognitive human-computer interaction effect.

[0005] Patent [CN108074336A] discloses an intelligent voice-interactive facial recognition and emotion-based noodle-serving robot. The robot's hardware components include a body, a shelf mounted on the body, and a human-machine interface. The shelf displays noodles of various flavors. The robot also includes a software algorithm that allows users to automatically select desired noodles or, through facial recognition algorithms, identify facial emotions and recommend noodles. Users then set their desired noodles, pay, wait, pick up their noodles, and enjoy their meal in the dining area behind the robot. This invention enhances the passenger experience while exploring potential interests and enabling personalized service recommendations.

[0006] Based on the aforementioned existing patents, the current field of emotion recognition in human-computer interaction still primarily relies on single-modality methods, particularly facial expression recognition. However, facial expressions are easily disguised and often fail to effectively convey a user's true inner thoughts and emotions, limiting the potential for comprehensive user interaction in human-computer interaction. Furthermore, existing research primarily focuses on feature fusion, while feature extraction modules have not seen significant improvements. Furthermore, previous research has overlooked the critical issue of gradient conflicts between multiple losses, which directly impacts model performance. Summary of the Invention

[0007] Based on the technical problems existing in the background technology, the present invention proposes an emotion recognition method and system using multi-cue joint learning, which combines facial and body clues, uses multi-label and multi-loss features to optimize loss calculation, and improves the emotion recognition accuracy during human-computer interaction.

[0008] The present invention proposes an emotion recognition method and system using multi-cue joint learning, comprising the following steps:

[0009] Extracting facial expression data and body posture data from the acquired interactive video, and processing the facial expression data and body posture data into facial key feature points and skeleton joint points;

[0010] Inputting the facial key feature points and skeleton joint points into a pre-trained multi-cue network model to output a predicted emotion recognition category, the multi-cue network model comprising a face extraction module, a skeleton extraction module, a feature fusion module, a loss function calculation module, and a loss gradient optimization module;

[0011] Using the facial extraction module to identify key facial feature points to output facial features;

[0012] The skeleton extraction module is used to identify the skeleton joint points to output skeleton features;

[0013] The facial features and skeleton features are fused using fusion features to obtain body features;

[0014] Use the loss function calculation module to calculate the facial features and obtain the facial loss function L f , calculate the skeleton features to obtain the skeleton loss function L b , calculate the body features to get the body loss function L w , and the facial loss function L f , skeleton loss function L b , body loss function L w Add together to get the fusion loss function L m ;

[0015] Use the loss gradient optimization module to input the face loss function L f Or skeleton loss function L b Or the body loss function L w Or fusion loss function L m Perform gradient optimization and output the optimized features through the output layer to obtain the predicted emotion recognition category.

[0016] Furthermore, the face extraction module includes a convolutional layer, a RESBLOCK layer, an average pooling layer and a face temporal pooling layer;

[0017] The convolutional layer, RESBLOCK layer, and average pooling layer are connected in sequence to form ResNet50. ResNet50 inputs facial key feature points and its output is connected to the average pooling layer. The input of the facial time pooling layer is connected to the output of the average pooling layer, and the facial time pooling layer outputs facial features.

[0018] Furthermore, the skeleton extraction module includes an activation layer ReLU, a projection layer, an activation layer Softmax, a graph attention module GAB and a skeleton time pooling layer;

[0019] The activation layer ReLU inputs the skeleton joint points and the output is connected to the input of the graph attention module GAB. The output of the graph attention module GAB is connected to the input of the skeleton time pooling layer through the activation layer Softmax. The skeleton time pooling layer outputs the skeleton features.

[0020] The input skeleton joint points are mapped to the topological graph, and the topological graph is input into the graph attention module GAB. The graph attention module GAB uses the joint points in the topological graph as input nodes and the bones in the topological graph as the edges of the input nodes. Multiple joint points and edges in the topological graph are input into the graph attention module GAB, and after splicing or average pooling, they are input into the skeleton time pooling layer.

[0021] Furthermore, the loss function calculation module is specifically used as follows:

[0022] Output facial score s using facial features f , based on the facial score s fand the true face label y f Calculate the cross entropy loss and get the facial loss function L f ;

[0023] Use skeleton features to output skeleton score s b , based on the skeleton score s b and the true bone label y b Calculate the cross entropy loss and get the skeleton loss function L b ;

[0024] Use body features to output body score s w , based on body score s w and the real body tag y w Calculate the cross entropy loss and get the body loss function L w ;

[0025] Score the face f , skeleton score b and body scores w Add up to get the fusion score s m , based on the fusion score s m and the true fusion label y w Calculate the cross entropy loss and get the fusion loss function L m .

[0026] Furthermore, the loss gradient optimization module optimizes the facial loss function L f Or skeleton loss function L b Or the body loss function L w Or fusion loss function L m The optimization process is as follows:

[0027] When L f 、L b 、L w 、L m When there is a gradient conflict between the two loss functions in i The gradient g i Projection to another loss function L j The gradient g j On the normal plane, it is expressed as negative cosine similarity;

[0028] If the cosine similarity is negative, it means that there is a conflict between the gradients. The clues pull the multi-cue network model in different directions, and the gradient g i Replaced by gradient g j The value obtained by projection on the normal plane;

[0029] If the cosine similarity is not negative, the gradient g i remain unchanged;

[0030] The loss gradient optimization module is different from the gradient g in the same batch i All other clues iteratively perform the above process and update the gradient g i , and get the final gradient

[0031] Loss L i and L j The gradient g i and g j The calculation formula of cosine similarity between is as follows:

[0032]

[0033] Satisfies the following formula:

[0034]

[0035] Among them, similarity and cos(g i ,g j ) represent cosine similarity.

[0036] Furthermore, the facial expression data is processed into facial key feature points through OpenFace technology, and the body posture data is processed into skeleton joint points through OpenPose technology.

[0037] An emotion recognition system that uses multi-cue joint learning to extract facial expression data and body posture data from the acquired interactive video, and processes the facial expression data and body posture data into facial key feature points and skeleton joint points;

[0038] Inputting the facial key feature points and skeleton joint points into a pre-trained multi-cue network model to output a predicted emotion recognition category, the multi-cue network model comprising a face extraction module, a skeleton extraction module, a feature fusion module, a loss function calculation module, and a loss gradient optimization module;

[0039] Using the facial extraction module to identify key facial feature points to output facial features;

[0040] The skeleton extraction module is used to identify the skeleton joint points to output skeleton features;

[0041] The facial features and skeleton features are fused using fusion features to obtain body features;

[0042] Use the loss function calculation module to calculate the facial features and obtain the facial loss function L f , calculate the skeleton features to obtain the skeleton loss function L b , calculate the body features to get the body loss function Lw , and the facial loss function L f , skeleton loss function L b , body loss function L w Add together to get the fusion loss function L m ;

[0043] Use the loss gradient optimization module to input the face loss function L f Or skeleton loss function L b Or the body loss function L w Or fusion loss function L m Perform gradient optimization and output the optimized features through the output layer to obtain the predicted emotion recognition category.

[0044] A computer-readable storage medium stores a plurality of programs, wherein the plurality of programs are used to be called by a processor and execute the emotion recognition method described above.

[0045] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0046] The advantages of the emotion recognition method and system using multi-cue joint learning provided by the present invention are as follows: The present invention incorporates a graph attention module (GAB) within the skeleton extraction module. This module utilizes a graph network and an attention mechanism with skeletal joints to effectively capture body cues, thereby enhancing the multi-cue network model's ability to extract emotional information from body posture. The proposed loss gradient optimization module (LGO) uses a loss function to optimize gradients to minimize conflicts between multiple cues during the joint learning process, thereby improving the model's predictive capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a structural schematic diagram of the present invention;

[0048] Figure 2 Schematic diagram of the structure of the graph attention module (GAB);

[0049] Figure 3 This is the algorithm principle diagram of the loss gradient optimization module (LGO). DETAILED DESCRIPTION

[0050] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0051] like Figures 1 to 3 As shown, the present invention proposes an emotion recognition method using multi-cue joint learning, which includes the following steps:

[0052] Extracting facial expression data and body posture data from the acquired interactive video, and processing the facial expression data and body posture data into facial key feature points and skeleton joint points;

[0053] Inputting the facial key feature points and skeleton joint points into a pre-trained multi-cue network model to output a predicted emotion recognition category, the multi-cue network model comprising a face extraction module, a skeleton extraction module, a feature fusion module, a loss function calculation module, and a loss gradient optimization module;

[0054] Using the facial extraction module to identify key facial feature points to output facial features;

[0055] The skeleton extraction module is used to identify the skeleton joint points to output skeleton features;

[0056] The facial features and skeleton features are fused using fusion features to obtain body features;

[0057] Use the loss function calculation module to calculate the facial features and obtain the facial loss function L f , calculate the skeleton features to obtain the skeleton loss function L b , calculate the body features to get the body loss function L w , and the facial loss function L f , skeleton loss function L b , body loss function L w Add together to get the fusion loss function L m ;

[0058] Use the loss gradient optimization module to input the face loss function L f Or skeleton loss function L b Or the body loss function L w Or fusion loss function L m Perform gradient optimization and output the optimized features through the output layer to obtain the predicted emotion recognition category.

[0059] In this embodiment, facial and body cues are combined, and multi-label multi-loss features are used to optimize loss calculation, thereby improving the accuracy of emotion recognition during human-computer interaction.

[0060] A graph attention module (GAB) is designed within the skeleton extraction module. It leverages a graph network and an attention mechanism with skeleton joints to effectively capture body cues, thereby enhancing the ability of the multi-cue network model to extract emotional information from body posture. The proposed loss gradient optimization module (LGO) uses a loss function to optimize the gradient to minimize the conflict between multiple cues during joint learning, thereby improving the model's predictive ability.

[0061] Facial expression data and body posture data are extracted from human-computer interaction videos, and processed into facial key feature points (facial landmarks) and skeleton joint points through OpenFace and OpenPose technologies, and input into the multi-cue network model respectively.

[0062] For facial key feature points, the pre-trained ResNet50 model is used to extract the features, and finally the time information is extracted through the time pooling layer to output facial features.

[0063] For the skeleton joints, they are converted into a topological graph structure, which is used as the input of the graph attention module (GAB). Finally, the temporal information is extracted through the temporal pooling layer to output the skeleton features.

[0064] This method processes facial key feature points and skeleton joint points in parallel, then fuses the processed facial features and skeleton features to obtain body features. The facial features, skeleton features, and body features are then optimized again through the loss gradient optimization module. This avoids conflicts between multiple cues during the joint learning process and improves the accuracy of emotion recognition based on multi-cue joint learning. The details are explained below.

[0065] (A) The face extraction module in the multi-cue network model is based on a pre-trained ResNet50 model. The face extraction module includes a convolutional layer, a RESBLOCK layer, an average pooling layer, and a facial temporal pooling layer. The convolutional layer, the RESBLOCK layer, and the average pooling layer are sequentially connected to form a ResNet50. The ResNet50 inputs facial key feature points and the output is connected to the average pooling layer. The facial temporal pooling layer input is connected to the average pooling layer output, and the facial temporal pooling layer outputs facial features.

[0066] (B) The skeleton extraction module includes an activation layer ReLU, a projection layer, an activation layer Softmax, a graph attention module GAB, and a skeleton temporal pooling layer; the activation layer ReLU inputs skeleton joint points and the output is connected to the input of the graph attention module GAB. The output of the graph attention module GAB is connected to the input of the skeleton temporal pooling layer through the activation layer Softmax, and the skeleton temporal pooling layer outputs skeleton features.

[0067] The input skeleton joint points are mapped to the topological graph, and the topological graph is input into the graph attention module GAB. The graph attention module GAB uses the joint points in the topological graph as input nodes and the bones in the topological graph as the edges of the input nodes. Multiple joint points and edges in the topological graph are input into the graph attention module GAB, and after splicing or average pooling, they are input into the skeleton time pooling layer.

[0068] The skeleton extraction module stacks layers (the stacking of layers refers to the number of layers in the model. By stacking layers, the nodes can pay attention to more features). The nodes can pay attention to the features of their neighborhoods and allow different weights to be implicitly assigned according to their importance. The weight is the value of the edge of the skeleton as the input node. The weight size is initially preset and changes with model training. That is to say, in the subsequent model training process, the weight is dynamically adjusted until the model training is completed.

[0069] For the graph attention module GAB, the skeleton joint graph (a graph formed by bones and joints) is used to represent the bones and joints of the body and hands as nodes and edges of nodes, forming a topological graph structure. The topological graph and the relationship between bones and joints captured in the adjacency matrix are passed to the multi-threaded network model for further processing and feature extraction. For a topological graph, one image is a topological graph, and the input is a sequence of images, so several images are several topological graphs. For example Figure 2 middle, The corresponding nodes represent bones, The corresponding nodes represent the skeletal components after the bones and joints are fused. The corresponding nodes represent the skeleton features to be extracted temporally after being processed by the graph attention module GAB, and then the skeleton features to be extracted temporally are obtained after the skeleton features are extracted temporally through the skeleton temporal pooling layer.

[0070] It should be noted that the facial time pooling layer is set to extract the time of the current acquisition of key facial feature points. Because the interactive video is recognized and processed, the interactive video is represented as a sequence of pictures in the subsequent processing. The time features of each picture sequence are extracted through the time pooling layer, so that in the subsequent expression recognition process, they can be sorted in chronological order to obtain more accurate expression recognition results. Similarly, the skeleton time pooling layer is also used to extract the time of the current skeleton joint point, and the subsequent body features are obtained by fusing facial features and skeleton features. This fusion is based on the corresponding splicing and fusion of the times extracted by the facial time pooling layer and the skeleton time pooling layer, and is not an arbitrary fusion of the face and body.

[0071] (C) Loss function calculation module and loss gradient optimization module

[0072] (C1) The loss function calculation module uses different losses at different stages, specifically:

[0073] (C1-1) Output facial score s using facial features f , based on the facial score s f and the true face label y f Calculate the cross entropy loss and get the facial loss function L f ;

[0074] (C1-2) Use skeleton features to output skeleton score s b , based on the skeleton score s b and the true bone label y b Calculate the cross entropy loss and get the skeleton loss function L b ;

[0075] (C1-3) Use body features to output body score s w , based on body score s w and the real body tag y w Calculate the cross entropy loss and get the body loss function L w ;

[0076] (C1-4) The facial score s f , skeleton score b and body scores w Add up to get the fusion score s m , based on the fusion score s m and the true fusion label y w Calculate the cross entropy loss and get the fusion loss function L m .

[0077] The (C2) loss gradient optimization module uses cosine similarity to minimize the negative influence of mutual gradients on each other by projecting the gradient of each conflicting clue onto the normal plane of the gradient of the other clue as follows:

[0078] (C2-1) When L f 、L b 、L w 、L m When there is a gradient conflict between the two loss functions in , it is expressed as negative cosine similarity, and a loss function L i The gradient g i Projection to another loss function L j The gradient g j This projection can eliminate conflicts, reduce interference in loss calculation, and allow more effective joint learning of multiple cues.

[0079] Loss L i and L j The gradient g i and g j The calculation formula of cosine similarity between is as follows:

[0080]

[0081] Among them, similarity and cos(g i ,g j ) represents cosine similarity, L i Indicates L f 、L b 、L w 、L m A loss function in j Indicates L f 、L b 、L w 、L m Another loss function in g i 、g j For L i 、L j The corresponding gradient, L i and L j Represents different loss functions.

[0082] If the cosine similarity is negative, it means there is a conflict between the gradients, indicating that the clues are pulling the model in different directions, then g i Replace with g j The value obtained by projection on the normal plane. Conversely, if the cosine similarity is not negative, then g i Remain unchanged.

[0083] During training, the loss gradient optimization module optimizes the loss gradients of different g in the same batch. i All other clues iteratively perform the above calculations and update the gradient g i , and get the final gradient The formula is as follows:

[0084]

[0085] This iterative process effectively alleviates the cue conflict problem in joint multi-cue learning and improves the overall stability and performance of training.

[0086] As an example:

[0087] Step 1: Obtain a human-computer interaction video dataset and use OpenFace and OpenPose technologies to extract the key facial feature points and skeleton joint points of the subjects.

[0088] Step 2: Input the facial key feature point data into the face extraction module for feature extraction, and finally extract the time information through the face time pooling layer to obtain facial features.

[0089] Step 3: Convert the skeleton joint point data into a topological graph structure and input it into the skeleton extraction module for feature extraction. Finally, extract the time information through the skeleton time pooling layer to obtain the skeleton features.

[0090] Step 4: Fuse facial features and skeleton features to obtain body features;

[0091] Step 5: Based on the loss function calculation module, the loss function is calculated for facial features, skeleton features and body features to obtain the facial loss function L f , skeleton loss function L b , body loss function L w And the fusion loss function L m ;

[0092] Step 6: Input the multi-cue multi-loss function features in step 5 into the designed loss gradient optimization module, and use the loss function to optimize the gradient to minimize the conflict between multiple cues in the joint learning process, thereby improving the prediction ability of the model and finally outputting the predicted emotion recognition category.

[0093] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. An emotion recognition method using multi-cue joint learning, characterized in that: The steps include: Extracting facial expression data and body posture data from the acquired interactive video, and processing the facial expression data and body posture data into facial key feature points and skeleton joint points; Inputting the facial key feature points and skeleton joint points into a pre-trained multi-cue network model to output a predicted emotion recognition category, the multi-cue network model comprising a face extraction module, a skeleton extraction module, a feature fusion module, a loss function calculation module, and a loss gradient optimization module; Using the facial extraction module to identify key facial feature points to output facial features; The skeleton extraction module is used to identify the skeleton joint points to output skeleton features; The facial features and skeleton features are fused using fusion features to obtain body features; Use the loss function calculation module to calculate facial features and obtain the facial loss function , calculate the skeleton features to obtain the skeleton loss function , calculate the body features to get the body loss function , and the facial loss function , skeleton loss function , body loss function Add together to get the fusion loss function ; Use the loss gradient optimization module to optimize the facial loss function of the input Or skeleton loss function or body loss function Or fusion loss function Perform gradient optimization and output the optimized features through the output layer to obtain the predicted emotion recognition category; The optimization process of the loss gradient optimization module is as follows: when 、 、 、 When there is a gradient conflict between the two loss functions in Gradient Projection to another loss function Gradient On the normal plane, it is expressed as negative cosine similarity; If the cosine similarity is negative, it means there is a conflict between the gradients. The clues pull the multi-cue network model in different directions, which will cause the gradients to Replace with gradient The value obtained by projection on the normal plane; If the cosine similarity is not negative, the gradient remain unchanged; The loss gradient optimization module performs different gradient All other clues iterate the above process and update the gradient , and get the final gradient ; loss and Gradient and The calculation formula of cosine similarity between is as follows: Satisfies the following formula: in, and Both represent cosine similarity.

2. The emotion recognition method using multi-cue joint learning according to claim 1, characterized in that: The face extraction module includes a convolutional layer, a RESBLOCK layer, an average pooling layer and a face time pooling layer; The convolutional layer, RESBLOCK layer, and average pooling layer are connected in sequence to form ResNet50. ResNet50 inputs facial key feature points and its output is connected to the average pooling layer. The input of the facial time pooling layer is connected to the output of the average pooling layer, and the facial time pooling layer outputs facial features.

3. The emotion recognition method using multi-cue joint learning according to claim 1, characterized in that: The skeleton extraction module includes an activation layer ReLU, a graph attention module GAB, an activation layer Softmax and a skeleton time pooling layer; The activation layer ReLU inputs the skeleton joint points and the output is connected to the input of the graph attention module GAB. The output of the graph attention module GAB is connected to the input of the skeleton time pooling layer through the activation layer Softmax. The skeleton time pooling layer outputs the skeleton features. The input skeleton joint points are mapped to the topological graph, and the topological graph is input into the graph attention module GAB. The graph attention module GAB uses the joint nodes as the input nodes in the topological graph and the bone edges as the edges of the input nodes in the topological graph. The topological graph is then input into the graph attention module GAB and input into the skeleton temporal pooling layer after splicing or average pooling.

4. The emotion recognition method using multi-cue joint learning according to claim 1, characterized in that: The loss function calculation module is specifically used as follows: Output facial score using facial features , based on the face score and ground-truth face labels Calculate the cross entropy loss and get the facial loss function ; Use skeleton features to output skeleton scores , based on the skeleton score and the true bone labels Calculate the cross entropy loss and get the skeleton loss function ; Output body score using body features , based on body score and real body tags Calculate the cross entropy loss and get the body loss function ; Score the face , skeleton score and body score Add up to get the fusion score , based on the fusion score and real fusion tags Calculate the cross entropy loss and get the fusion loss function .

5. The emotion recognition method using multi-cue joint learning according to claim 1, characterized in that: The facial expression data is processed into facial key feature points using OpenFace technology, and the body posture data is processed into skeleton joint points using OpenPose technology.

6. An emotion recognition system using multi-cue joint learning, characterized in that: Extracting facial expression data and body posture data from the acquired interactive video, and processing the facial expression data and body posture data into facial key feature points and skeleton joint points; Inputting the facial key feature points and skeleton joint points into a pre-trained multi-cue network model to output a predicted emotion recognition category, the multi-cue network model comprising a face extraction module, a skeleton extraction module, a feature fusion module, a loss function calculation module, and a loss gradient optimization module; Using the facial extraction module to identify key facial feature points to output facial features; The skeleton extraction module is used to identify the skeleton joint points to output skeleton features; The facial features and skeleton features are fused using fusion features to obtain body features; Use the loss function calculation module to calculate facial features and obtain the facial loss function , calculate the skeleton features to obtain the skeleton loss function , calculate the body features to get the body loss function , and the facial loss function , skeleton loss function , body loss function Add together to get the fusion loss function ; Use the loss gradient optimization module to optimize the facial loss function of the input Or skeleton loss function or body loss function Or fusion loss function Perform gradient optimization and output the optimized features through the output layer to obtain the predicted emotion recognition category; The optimization process of the loss gradient optimization module is as follows: when 、 、 、 When there is a gradient conflict between the two loss functions in Gradient Projection to another loss function Gradient On the normal plane, it is expressed as negative cosine similarity; If the cosine similarity is negative, it means there is a conflict between the gradients. The clues pull the multi-cue network model in different directions, which will cause the gradients to Replace with gradient The value obtained by projection on the normal plane; If the cosine similarity is not negative, the gradient remain unchanged; The loss gradient optimization module performs different gradient All other clues iterate the above process and update the gradient , and get the final gradient ; loss and Gradient and The calculation formula of cosine similarity between is as follows: Satisfies the following formula: in, and Both represent cosine similarity.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of programs, which are used to be called by a processor and execute the emotion recognition method according to claim 1.

Images