Emotion recognition method and system based on multi-branch fusion attention mechanism

Abstract

Disclosed in the present invention are an emotion recognition method and system based on a multi-branch fusion attention mechanism. The method comprises: inputting a target body image into a target body branch network of an emotion recognition model, so as to obtain a target body facial feature and a target body behavior feature, and fusing the target facial feature and the target behavior feature, so as to obtain a target body emotion feature fO; inputting a region environment image into an environment influence branch network of the emotion recognition model, so as to obtain an environment influence feature fE; performing feature fusion on the environment influence feature fE and the target body emotion feature fO, so as to obtain a feature vector f; and using a target emotion classifier of the emotion recognition model to classify the feature vector f, and outputting an emotion recognition result. In the present invention, emotion clues in a target region environment are fully utilized for emotion recognition, and thus the accuracy of emotion recognition is greatly improved.

Description

An emotion recognition method and system based on multi-branch fusion attention mechanism Technical Field

[0001] The present invention belongs to the field of image recognition technology, and specifically relates to an emotion recognition method and system based on a multi-branch fusion attention mechanism. Background Art

[0002] Emotional recognition is one of the core tasks of intelligent perception technology. By accurately identifying the emotional state of a target object, computer systems and intelligent terminals can better understand its needs, thereby providing a more intelligent and personalized human-computer interaction experience.

[0003] Currently, there are two main common methods for emotion perception: the first is contact detection, which mainly detects the physiological signals of the target object (such as EEG and muscle signals) and obtains emotional representation through analysis. The advantages of this type of method are high detection reliability and meeting real-time requirements. However, the disadvantage is that the device needs to be in contact with the target object, which inevitably causes interference to the target object. At the same time, since sensors and other equipment are relatively expensive, it is not suitable for large-scale applications.

[0004] The second type is non-contact detection: it intelligently analyzes the detailed features of the target object's face, eyes, body shape, etc. to accurately identify the target object's emotional state. However, most existing emotion recognition networks classify emotions by analyzing the target object's facial features in real time, and lack analysis of the impact of the target area environment and other individuals in the spatiotemporal environment on the target object's emotions.

[0005] Summary of the Invention

[0006] The present invention provides an emotion recognition method and system based on a multi-branch fusion attention mechanism to solve the problem that existing emotion recognition methods do not fully utilize emotional clues in the target area environment and lack analysis of the impact of the target area environment on the target object's emotions.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] The first aspect of the present invention provides an emotion recognition method based on a multi-branch fusion attention mechanism, comprising:

[0009] Collecting target body data and regional environment data, performing data preprocessing and key frame extraction on the target body data and regional environment data to obtain target body images and regional environment images;

[0010] The target image and the regional environment image are input into the pre-trained emotion recognition model to obtain the emotion recognition results. The process includes:

[0011] The target ontology image is input into the target ontology branch network of the emotion recognition model to obtain the target ontology facial features and target ontology behavioral features, and the target facial features and target behavioral features are fused to obtain the target ontology emotional features f O ;

[0012] Input the regional environment image into the environmental impact branch network of the emotion recognition model to obtain the environmental impact feature f E ; The environmental impact characteristics f E and the target's emotional characteristics f O Perform feature fusion to obtain a feature vector f; and use the target emotion classifier of the emotion recognition model to classify the feature vector f and output an emotion recognition result.

[0013] Preferably, the method for preprocessing the target ontology data and the regional environment data includes:

[0014] The region of interest is calibrated for each frame in the target body data and each frame in the regional environment data, and noise reduction and data enhancement processing are performed on the calibrated target body data and target body data.

[0015] Preferably, the method of performing key frame extraction on the pre-processed target body data and regional environment data to obtain the target body image and the regional environment image includes:

[0016] Positioning, cropping and key point marking of the region of interest in each frame of the target body data and each frame of the regional environment data;

[0017] The Lucas-Kanade optical flow method is used to calculate the average optical flow of the region of interest in each frame;

[0018] Key frames are extracted from the frames where the average optical flow of the region of interest is greater than the set threshold T to obtain the target body image and the regional environment image; the target body image and the regional environment image are inserted into the candidate frame priority queue according to the optical flow intensity, which is used to limit the order in which the target body image and the regional environment image are input into the pre-trained emotion recognition model.

[0019] Preferably, the method of inputting the target ontology image into the target ontology branch network of the emotion recognition model to obtain the target ontology facial features and target ontology behavioral features includes:

[0020] Input the target ontology image into the target ontology facial feature extraction unit of the target ontology branch network to obtain the target ontology facial features; input the target ontology image into the target ontology behavior feature extraction unit of the target ontology branch network to obtain the target ontology behavior features;

[0021] The target ontology facial feature extraction unit and the target ontology behavioral feature extraction unit include a CNN network module and an attention mechanism module; the CNN network module is sequentially provided with a convolutional layer, a batch normalization layer and a GELU activation function layer; the attention mechanism module is set to an ECA attention mechanism;

[0022] The target image is sequentially input into the convolution layer, batch normalization layer and GELU activation function layer to obtain the feature map Y Oi , and input the target ontology image into the attention mechanism module to obtain the attention weight ω i , the feature map Y Oi and attention weight ω i Perform dot multiplication to output the target entity’s facial features or target entity’s behavioral features.

[0023] Preferably, the regional environment image is input into the environmental impact branch network of the emotion recognition model to obtain the environmental impact feature f E The methods include:

[0024] The regional environmental image is input into the FPN feature extraction module of the environmental impact branch network to obtain the feature pyramid P = {P2, P3, P4, P5}; the feature pyramid P = {P2, P3, P4, P5} is output to the global average pooling layer and two fully connected layers in sequence, and the environmental impact feature f is output. E .

[0025] Preferably, the method of inputting the regional environment image into the FPN feature extraction module of the environmental impact branch network to obtain the feature pyramid P={P2, P3, P4, P5} includes:

[0026] Inputting the regional environment image into the FPN feature extraction module of the environmental impact branch network, wherein the FPN feature extraction module includes a VGG network unit K1, a VGG network unit K2, a VGG network unit K3, a VGG network unit K4 and a VGG network unit K5;

[0027] The regional environment image is sequentially downsampled by the VGG network unit K1, the VGG network unit K2, the VGG network unit K3, the VGG network unit K4, and the VGG network unit K5, and feature maps C2, C3, C4, and C5 are output respectively by the VGG network unit K2, the VGG network unit K3, the VGG network unit K4, and the VGG network unit K5;

[0028] The bicubic interpolation method is used to upsample the feature map C5 to obtain the feature map P5; the bicubic interpolation method is used to upsample the feature map C4 and add it to the corresponding position of the feature map P5 to obtain the feature map P4; the bicubic interpolation method is used to upsample the feature map C3 and add it to the corresponding position of the feature map P4 to obtain the feature map P3; the bicubic interpolation method is used to upsample the feature map C2 and add it to the corresponding position of the feature map P3 to obtain the feature map P2;

[0029] The feature map P2, feature map P3, feature map P4 and feature map P5 constitute a feature pyramid P = {P2, P3, P4, P5}.

[0030] Preferably, the training process of the emotion recognition model includes:

[0031] The training samples are collected to form a training data set, and the training data set is used to train the emotion recognition model, and the emotion recognition training results are output by the emotion recognition model; the cross entropy loss function is used to calculate the loss value Loss(w) between the emotion recognition training results and the emotion labels of the training samples; based on the loss value Loss(w), the Adam optimizer is used to adaptively adjust the emotion recognition model parameters; the training process of the emotion recognition model is repeated until the set number of iterations is reached and the trained emotion recognition model is output.

[0032] Preferably, the cross entropy loss function is used to calculate the loss value Loss(w) between the emotion recognition training result and the emotion label of the training sample, and the expression formula is:

[0033] In the formula, m represents the number of training samples in the training data set, d represents the number of emotion categories, Indicates whether the emotion label of the i-th training sample belongs to category j, j∈A,h w (x (i) ) j represents the predicted probability that the emotion recognition model maps the i-th training sample to emotion category j.

[0034] In a second aspect, the present invention provides an emotion recognition system based on a multi-branch fusion attention mechanism, comprising a cloud server and an edge platform; the cloud server and the edge platform are connected in a bidirectional communication manner; and the edge platform is provided with a plurality of intelligent agents;

[0035] The intelligent body is provided with an Actor network prediction sub-platform and a Critic network evaluation sub-platform; the emotion recognition model is provided in the Actor network prediction sub-platform; the Critic network evaluation sub-platforms in each intelligent body are bidirectionally connected in sequence; the Actor network prediction sub-platform is used to extract the target entity image and the regional environment image from the collected target entity data and the regional environment data, input the target entity image and the regional environment image into the pre-trained emotion recognition model to obtain the emotion recognition result and output the emotion recognition result; the Critic network evaluation sub-platform is used to use the cross entropy loss function to calculate the loss value Los(w) between the emotion recognition training result and the emotion label of the training sample.

[0036] Preferably, the edge platform calculates the emotion recognition confidence of the emotion recognition model in each intelligent agent. When there is an emotion recognition model in the edge platform whose confidence is lower than a set confidence threshold, the edge platform performs local training and outputs the trained edge platform parameters; the edge platform parameters include the optimized joint strategy H and the emotion recognition model parameters;

[0037] The edge platform encodes the trained edge platform parameters and uploads them to the cloud server; the cloud server receives and decodes the edge platform parameters and sends the optimized edge platform parameters to other edge devices.

[0038] Preferably, the method for the edge platform to perform local training and output trained edge platform parameters includes:

[0039] The training samples are collected to form a training data set, and the emotion recognition model in each agent is trained using the training data set; during the training process, a Markov quadruple is established based on the Markov decision method. i γ>; where S represents the data set of feature vectors f extracted by each emotion recognition model, and A represents the emotional state of the target entity; r i represents the reward function of the emotion recognition model in the i-th agent; γ represents the discount factor;

[0040] According to the Markov quadruple i γ>Calculate the total training reward of the emotion recognition model in all agents, with the goal of obtaining the maximum total training reward. Optimize the parameters and weights of the emotion recognition model in each agent, and iterate the edge platform's optimization process of each agent's weight until the set number of iterations is reached to output the optimized joint strategy H and the parameters of the trained emotion recognition model.

[0041] Preferably, the reward function r of the emotion recognition model in the i-th agent of the edge platform is i , the expression formula is: ​​

[0042] In the formula, C k =0 indicates that the classification result of the emotion recognition model in the kth agent is wrong;

[0043] C k =1 indicates that the classification result of the emotion recognition model in the kth agent is wrong; N represents the number of all agents in the system; β i (t) is the emotion classification reward of the i-th agent at time slot t.

[0044] Preferably, the edge platform is based on the Markov quadruple i γ> calculates the total training reward of the emotion recognition model in all agents, expressed as:

[0045] In the formula, denoted as the total training reward of the emotion recognition model in all agents; H denotes the joint strategy of the edge platform; and E{·} denotes the expectation function.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention inputs the regional environment image into the environmental impact branch network of the emotion recognition model to obtain the environmental impact feature f E ; The environmental impact characteristics f E and the target's emotional characteristics f O Feature fusion is performed to obtain a feature vector f; the target emotion classifier of the emotion recognition model is used to classify the feature vector f and output an emotion recognition result; the emotion clues in the target area environment are fully utilized for emotion recognition, which greatly improves the accuracy of emotion recognition.

[0048] The present invention collects target body data and regional environment data, performs data preprocessing and key frame extraction on the target body data and regional environment data to obtain target body images and regional environment images; inputs the target body images and regional environment images into a pre-trained emotion recognition model to obtain emotion recognition results, which can reduce the amount of transmitted data without losing key image change information, thereby reducing the load of network transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] FIG1 is a flow chart of the emotion recognition method provided in Example 1;

[0050] FIG2 is a relationship diagram of the target ontology branch network and the environmental impact branch network provided in Example 1;

[0051] 3 is a structural diagram of a target ontology facial feature extraction unit or a target ontology behavioral feature extraction unit provided in Example 1;​

[0052] FIG4 is a structural diagram of an environmental impact branch network provided in Example 1;

[0053] FIG5 is a diagram showing the connection relationship between various intelligent entities in the edge platform provided in Example 2;

[0054] FIG6 is a structural diagram of the emotion recognition system provided in Example 2. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0056] Example 1

[0057] As shown in Figure 1, this implementation provides an emotion recognition method based on a multi-branch fusion attention mechanism, including:

[0058] The target body data and regional environment data are collected, the region of interest is calibrated for each frame in the target body data and each frame in the regional environment data, and the calibrated target body data and target body data are subjected to noise reduction and data enhancement processing.

[0059] The method of performing key frame extraction on the pre-processed target body data and regional environment data to obtain the target body image and the regional environment image includes:

[0060] Positioning, cropping and key point marking of the region of interest in each frame of the target body data and each frame of the regional environment data;

[0061] The Lucas-Kanade optical flow method is used to calculate the average optical flow of the region of interest in each frame;

[0062] Key frames are extracted from the frames containing the regions of interest whose average optical flow is greater than the set threshold T to obtain the target image and the regional environment image; the target image and the regional environment image are inserted into the candidate frame priority queue according to the optical flow intensity, which is used to limit the order in which the target image and the regional environment image are input into the pre-trained emotion recognition model; the key frame extraction method of the region of interest based on optical flow can reduce the amount of transmitted data without losing key image change information, thereby reducing the network transmission load.

[0063] The target image and the regional environment image are input into the pre-trained emotion recognition model to obtain the emotion recognition results. The process includes:

[0064] The method of inputting the target ontology image into the target ontology branch network of the emotion recognition model to obtain the target ontology facial features and target ontology behavioral features includes:

[0065] Input the target ontology image into the target ontology facial feature extraction unit of the target ontology branch network to obtain the target ontology facial features; input the target ontology image into the target ontology behavior feature extraction unit of the target ontology branch network to obtain the target ontology behavior features;

[0066] The target ontology facial feature extraction unit and the target ontology behavioral feature extraction unit include a CNN network module and an attention mechanism module; the CNN network module is sequentially provided with a convolutional layer, a batch normalization layer and a GELU activation function layer; the attention mechanism module is set to an ECA attention mechanism;

[0067] The activation function expression of the GELU activation function layer is: GELU(x)=xΦ(x)

[0068] In the formula, x is the input parameter, and Φ(x) is the probability function of the Gaussian normal distribution.

[0069] The target image is sequentially input into the convolution layer, batch normalization layer and GELU activation function layer to obtain the feature map Y Oi , and input the target ontology image into the attention mechanism module to obtain the attention weight ω i , the feature map Y Oi and attention weight ω i Perform dot multiplication to output the target entity facial features or target entity behavioral features; fuse the target facial features and target behavioral features to obtain the target entity emotional features f O .

[0070] Input the regional environment image into the environmental impact branch network of the emotion recognition model to obtain the environmental impact feature f E Specific methods include:

[0071] Inputting the regional environment image into the FPN feature extraction module of the environmental impact branch network, wherein the FPN feature extraction module includes a VGG network unit K1, a VGG network unit K2, a VGG network unit K3, a VGG network unit K4 and a VGG network unit K5;

[0072] The regional environment image is sequentially downsampled by the VGG network unit K1, the VGG network unit K2, the VGG network unit K3, the VGG network unit K4, and the VGG network unit K5, and feature maps C2, C3, C4, and C5 are output respectively by the VGG network unit K2, the VGG network unit K3, the VGG network unit K4, and the VGG network unit K5;

[0073] The bicubic interpolation method is used to upsample the feature map C5 to obtain the feature map P5; the bicubic interpolation method is used to upsample the feature map C4 and add it to the corresponding position of the feature map P5 to obtain the feature map P4; the bicubic interpolation method is used to upsample the feature map C3 and add it to the corresponding position of the feature map P4 to obtain the feature map P3; the bicubic interpolation method is used to upsample the feature map C2 and add it to the corresponding position of the feature map P3 to obtain the feature map P2;

[0074] The feature pyramid P = {P2, P3, P4, P5} is composed of feature maps P2, P3, P4, and P5. The feature pyramid P = {P2, P3, P4, P5} is sequentially output to the global average pooling layer and two fully connected layers to output the environmental impact feature f E .

[0075] The environmental impact characteristic f E and the target's emotional characteristics f O Perform feature fusion to obtain the feature vector f, which can be expressed as: f = concatnate[f O , f E ]; Use the target emotion classifier of the emotion recognition model to classify the feature vector f and output the emotion recognition result.

[0076] The training process of the emotion recognition model includes:

[0077] The training samples are collected to form a training data set, and the emotion recognition model is trained using the training data set. The emotion recognition training results output by the emotion recognition model are used to calculate the loss value Loss(w) between the emotion recognition training results and the emotion labels of the training samples using the cross entropy loss function; the expression formula is:

[0078] In the formula, m represents the number of training samples in the training data set, d represents the number of emotion categories, Indicates whether the emotion label of the i-th training sample belongs to category j, j∈A,h w (x (i) ) j represents the predicted probability that the emotion recognition model maps the i-th training sample to emotion category j.

[0079] Based on the loss value Loss(w), the Adam optimizer is used to adaptively adjust the emotion recognition model parameters; the training process of the emotion recognition model is repeated until the set number of iterations is reached and the trained emotion recognition model is output.

[0080] In this implementation, the emotion recognition model with a multi-branch fusion attention mechanism integrates the emotional features of the target entity and the target area environment, and takes into account the emotional mutual influence relationship between the target entities, which greatly improves the accuracy of emotion recognition. In this implementation, the cross-entropy loss function is used to calculate the loss value Loss(w) between the emotion recognition training results and the emotion labels of the training samples. Based on the loss value Loss(w), the Adam optimizer is used to adaptively adjust the emotion recognition model parameters to improve the robustness of the emotion recognition model.

[0081] Example 2

[0082] This embodiment provides an emotion recognition system based on a multi-branch fusion attention mechanism, comprising: a cloud server and an edge platform; a bidirectional communication connection between the cloud server and the edge platform; and a plurality of intelligent agents disposed in the edge platform.

[0083] The intelligent body is provided with an Actor network prediction sub-platform and a Critic network evaluation sub-platform; the emotion recognition model described in Example 1 is provided in the Actor network prediction sub-platform; the Critic network evaluation sub-platforms in each intelligent body are bidirectionally connected in sequence; the Actor network prediction sub-platform is used to extract a target entity image and a regional environment image from the collected target entity data and regional environment data, input the target entity image and the regional environment image into a pre-trained emotion recognition model to obtain an emotion recognition result and output the emotion recognition result; the Critic network evaluation sub-platform is used to calculate the loss value Los(w) between the emotion recognition training result and the emotion label of the training sample using a cross entropy loss function.

[0084] The edge platform calculates the emotion recognition confidence of the emotion recognition model in each intelligent agent. When there is an emotion recognition model in the edge platform whose confidence is lower than a set confidence threshold, the edge platform performs local training and outputs the trained edge platform parameters. The method includes:

[0085] The training samples are collected to form a training data set, and the emotion recognition model in each agent is trained using the training data set; during the training process, a Markov quadruple is established based on the Markov decision method. i γ>; where S represents the data set of feature vectors f extracted by each emotion recognition model, and A represents the emotional state of the target entity; r i represents the reward function of the emotion recognition model in the i-th agent; γ represents the discount factor;

[0086] ​The emotional state A of the target entity can be expressed as: A = {'angry', 'disgust', 'fear', 'happy', 'sad', 'surprise', 'neutral'}

[0087] The reward function r of the emotion recognition model in the i-th agent of the edge platform i , the expression formula is:

[0088] In the formula, C k = 0 indicates that the classification result of the emotion recognition model in the kth agent is wrong; C k =1 indicates that the classification result of the emotion recognition model in the kth agent is wrong; N represents the number of all agents in the system; β i (t) is the emotion classification reward of the i-th agent at time slot t.

[0089] The edge platform is based on the Markov quadruple i γ> calculates the total training reward of the emotion recognition model in all agents, expressed as:

[0090] In the formula, denoted as the total training reward of the emotion recognition model in all agents; H denotes the joint strategy of the edge platform; and E{·} denotes the expectation function.

[0091] With the goal of maximizing the total training reward, the parameters and weights of the emotion recognition model in each agent are optimized. The edge platform's optimization process of each agent's weight is iterated repeatedly until the set number of iterations is reached and the edge platform parameters are output. The edge platform parameters include the optimized joint strategy H and the parameters of the trained emotion recognition model. Multi-agent reinforcement learning emotion fusion training is used to improve the robustness of the emotion recognition model.

[0092] The edge platform encodes the trained edge platform parameters and uploads them to the cloud server. This process does not include any data containing user privacy. The transmission process uses quantization and sparsity methods to compress and encode the model weight matrix, reducing transmission bandwidth and storage space. The cloud server receives and decodes the edge platform parameters and distributes the optimized edge platform parameters to other edge devices. The emotion-aware edge-cloud collaborative framework based on federated learning can achieve collaborative emotion recognition model optimization between the edge platform and the cloud server while protecting privacy.

[0093] ​Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0094] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0095] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0097] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An emotion recognition method based on a multi-branch fusion attention mechanism, characterized in that: include: Collecting target body data and regional environment data, performing data preprocessing and key frame extraction on the target body data and regional environment data to obtain target body images and regional environment images; The target body image and the regional environment image are input into the pre-trained emotion recognition model to obtain the emotion recognition result. The process includes: The target ontology image is input into the target ontology branch network of the emotion recognition model to obtain the target ontology facial features and target ontology behavioral features, and the target facial features and target behavioral features are fused to obtain the target ontology emotional features f O ; Input the regional environment image into the environmental impact branch network of the emotion recognition model to obtain the environmental impact feature f E ; The environmental impact characteristics f E and the target’s emotional characteristics f O Perform feature fusion to obtain a feature vector f; and use the target emotion classifier of the emotion recognition model to classify the feature vector f and output an emotion recognition result.

2. The emotion recognition method according to claim 1, characterized in that: The method of performing data preprocessing and key frame extraction on target body data and regional environment data to obtain target body image and regional environment image includes: Calibrate the region of interest for each frame of the target body data and each frame of the regional environment data, and perform noise reduction and data enhancement processing on the calibrated target body data and target body data; locate and crop the region of interest in each frame of the target body data and each frame of the regional environment data and mark key points; The Lucas-Kanade optical flow method is used to calculate the average optical flow of the region of interest in each frame; the frame where the average optical flow is greater than the set threshold T is used to extract the key frame, and the target The target body image and the regional environment image are inserted into the candidate frame priority queue according to the optical flow intensity, so as to limit the order in which the target body image and the regional environment image are input into the pre-trained emotion recognition model.

3. The emotion recognition method according to claim 1, characterized in that: The method of inputting the target ontology image into the target ontology branch network of the emotion recognition model to obtain the target ontology facial features and the target ontology behavioral features includes: Input the target ontology image into the target ontology facial feature extraction unit of the target ontology branch network to obtain the target ontology facial features; input the target ontology image into the target ontology behavior feature extraction unit of the target ontology branch network to obtain the target ontology behavior features; The target body facial feature extraction unit and the target body behavior feature extraction unit include a CNN network module and an attention mechanism module; the CNN network module is sequentially provided with a convolution layer, a batch normalization layer and a GELU activation function layer; the attention mechanism module is set to an ECA attention mechanism; The target body image is sequentially input into the convolution layer, batch normalization layer and GELU activation function layer to obtain the feature map Y Oi , and at the same time input the target ontology image into the attention mechanism module to obtain the attention weight ω i , the feature map Y Oi and the attention weight ω i Perform dot multiplication to output the target entity's facial features or target entity's behavioral features.

4. The emotion recognition method according to claim 1, characterized in that: Input the regional environment image into the environmental impact branch network of the emotion recognition model to obtain the environmental impact feature f E The methods include: The regional environment image is input into the FPN feature extraction module of the environmental impact branch network to obtain the feature pyramid P = {P2, P3, P4, P5}; the feature pyramid P = {P2, P3, P4, P5} is output to the global average pooling layer and two fully connected layers in sequence, and the environmental impact feature f is output. E .

5. The emotion recognition method according to claim 4, characterized in that: The method of inputting the regional environment image into the FPN feature extraction module of the environmental impact branch network to obtain the feature pyramid P = {P2, P3, P4, P5} includes: Inputting the regional environment image into the FPN feature extraction module of the environmental impact branch network, wherein the FPN feature extraction module includes a VGG network unit K1, a VGG network unit K2, a VGG network unit K3, a VGG network unit K4 and a VGG network unit K5; The regional environment image is sequentially downsampled by the VGG network unit K1, the VGG network unit K2, the VGG network unit K3, the VGG network unit K4 and the VGG network unit K5, and a feature map C2, a feature map C3, a feature map C4 and a feature map C5 are outputted by the VGG network unit K2, the VGG network unit K3, the VGG network unit K4 and the VGG network unit K5 respectively; The feature map C5 is upsampled by the bicubic interpolation method to obtain the feature map P5; the feature map C4 is upsampled by the bicubic interpolation method and added to the corresponding position of the feature map P5 to obtain the feature map P4; the feature map C3 is upsampled by the bicubic interpolation method and added to the corresponding position of the feature map P4 to obtain the feature map P3; the feature map C2 is upsampled by the bicubic interpolation method and added to the corresponding position of the feature map P3 to obtain the feature map P2; The feature map P2, feature map P3, feature map P4 and feature map P5 constitute a feature pyramid P = {P2, P3, P4, P5}.

6. The emotion recognition method according to claim 1, characterized in that: The training process of the emotion recognition model includes: Collect training samples to construct a training data set, and use the training data set to train the emotion recognition model, and output the emotion recognition training results through the emotion recognition model; use the cross entropy loss function to calculate the loss value Loss(w) between the emotion recognition training results and the emotion labels of the training samples; based on the loss value Loss(w), use the Adam optimizer to adaptively adjust the emotion recognition model parameters; repeat the iterative emotion recognition The model training process continues until the set number of iterations is reached and the trained emotion recognition model is output.

7. The emotion recognition method according to claim 6, characterized in that: The cross entropy loss function is used to calculate the loss value Loss(w) between the emotion recognition training results and the emotion labels of the training samples. The expression formula is: In the formula, m represents the number of training samples in the training data set, d represents the number of emotion categories, Indicates whether the emotion label of the i-th training sample belongs to category j, h w (x (i) ) j It represents the predicted probability that the emotion recognition model maps the i-th training sample to emotion category j.

8. An emotion recognition system based on a multi-branch fusion attention mechanism, characterized in that: It includes a cloud server and an edge platform; the cloud server and the edge platform are connected in a two-way communication manner; and the edge platform is provided with a plurality of intelligent agents; The Actor network prediction sub-platform and the Critic network evaluation sub-platform are set in the intelligent body; the Critic network evaluation sub-platforms in each intelligent body are connected in bidirectional communication in sequence; the Actor network prediction sub-platform is used to input the target body image and the regional environment image into the pre-trained emotion recognition model to obtain the emotion recognition result; The Critic network evaluation sub-platform is used to calculate the loss value Loss(w) between the emotion recognition training results and the emotion labels of the training samples using the cross entropy loss function.

9. The emotion recognition system according to claim 8, characterized in that: The edge platform calculates the emotion recognition confidence of the emotion recognition model in each intelligent agent. When there is an emotion recognition model with a confidence lower than a set confidence threshold in the edge platform, the edge platform performs local training and outputs the trained edge platform parameters; the edge platform parameters include the optimal joint strategy H and the emotion recognition model parameters; The edge platform encodes the trained edge platform parameters and uploads them to the cloud server; The cloud server receives and decodes the edge platform parameters and sends the optimized edge platform parameters to other edge devices.

10. The emotion recognition system according to claim 9, characterized in that: The method for the edge platform to perform local training and output trained edge platform parameters includes: The training samples are collected to form a training data set, and the emotion recognition model in each intelligent agent is trained using the training data set; during the training process, a Markov quadruple is established based on the Markov decision method. i γ>; where S represents the data set of feature vectors f extracted by each emotion recognition model, and A represents the emotional state of the target entity; r i represents the reward function of the emotion recognition model in the i-th agent; γ represents the discount factor;​ According to the Markov quadruple i γ>Calculate the total training return of the emotion recognition model in all agents, with the goal of obtaining the maximum total training return, optimize the parameters and weights of the emotion recognition model in each agent, and repeat the optimization process of the edge platform on the weights of each agent until the set number of iterations is reached to output the optimal joint strategy H and the parameters of the trained emotion recognition model.​

Images