An emotion recognition method and system based on a multi-branch fusion attention mechanism

Abstract

The application discloses an emotion recognition method and system based on a multi-branch fusion attention mechanism, and comprises the following steps: inputting a target ontology image into a target ontology branch network of an emotion recognition model to obtain target ontology facial features and target ontology behavior features; fusing the target facial features and the target behavior features to obtain target ontology emotion features f o ; inputting a regional environment image into an environment influence branch network of the emotion recognition model to obtain environment influence features f E ; fusing the environment influence features f E and the target ontology emotion features f o to obtain a feature vector f; and classifying the feature vector f by using a target emotion classifier of the emotion recognition model to output an emotion recognition result; and the application fully utilizes emotion clues in a target regional environment to perform emotion recognition, and greatly improves the accuracy of emotion recognition.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image recognition, and particularly relates to an emotion recognition method and system based on a multi-branch fusion attention mechanism. BACKGROUND

[0002] Emotion recognition is one of the core tasks of intelligent perception technology. By accurately recognizing the emotional state of a target object, a computer system and an intelligent terminal can better understand its needs, thereby providing a more intelligent and personalized human-computer interaction experience.

[0003] There are mainly two kinds of common emotion perception methods: the first is contact detection, which mainly detects the physiological signals (such as electroencephalogram and muscle signals) of a target object, obtains emotion representation by analysis, and has the advantages of high detection reliability and real-time requirements, but has the disadvantages of contact with the target object, interference to the target object, and high cost of sensors and devices, which is not suitable for large-scale application.

[0004] The second is non-contact detection, which intelligently analyzes the facial, eye, and body features of a target object to accurately recognize the emotional state of the target object. However, existing emotion recognition networks mostly classify emotions by real-time analysis of facial features of a target object, and lack analysis of the influence of the target region environment and other individuals in the space-time environment on the emotion of the target object. SUMMARY

[0005] The present application provides an emotion recognition method and system based on a multi-branch fusion attention mechanism to solve the problem of insufficient use of emotion clues in the target region environment by existing emotion recognition methods and lack of analysis of the influence of the target region environment on the emotion of the target object.

[0006] To achieve the above-mentioned purposes, the technical solution adopted by the present application is:

[0007] The present application provides an emotion recognition method based on a multi-branch fusion attention mechanism, comprising:

[0008] Collecting target object data and region environment data, and performing data preprocessing and key frame extraction on the target object data and region environment data to obtain target object images and region environment images;

[0009] Inputting the target object images and region environment images into a pre-trained emotion recognition model to obtain an emotion recognition result, the process comprising:

[0010] inputting the target body image into a target body branch network of the emotion recognition model to obtain target body facial features and target body behavior features, f o ;

[0011] inputting the regional environment image into an environment influence branch network of the emotion recognition model to obtain environment influence features f E ; f E and the target body emotion features f o are fused to obtain a feature vector f; the target emotion classifier of the emotion recognition model is used to classify the feature vector f to output an emotion recognition result.

[0012] Preferably, the method for data preprocessing of the target body data and the regional environment data comprises:

[0013] The regions of interest in each frame of the target body data and each frame of the regional environment data are labeled, and the labeled target body data and the target body data are subjected to noise reduction and data enhancement processing.

[0014] Preferably, the method for key frame extraction of the preprocessed target body data and the regional environment data to obtain the target body image and the regional environment image comprises:

[0015] The regions of interest in each frame of the target body data and each frame of the regional environment data are positioned, cropped and key-point labeled;

[0016] The average optical flow of the region of interest in each frame is calculated by using the Lucas-Kanade optical flow method;

[0017] The frame in which the region of interest with an average optical flow greater than a set threshold T is extracted as a key frame to obtain the target body image and the regional environment image; the target body image and the regional environment image are inserted into a candidate frame priority queue according to the optical flow intensity, and used to limit the order of inputting the target body image and the regional environment image into the pre-trained emotion recognition model.

[0018] Preferably, the method for obtaining the target body facial features and the target body behavior features by inputting the target body image into the target body branch network of the emotion recognition model comprises:

[0019] The target body image is inputted into a target body facial feature extraction unit of the target body branch network to obtain the target body facial features; the target body image is inputted into a target body behavior feature extraction unit of the target body branch network to obtain the target body behavior features;

[0020] The target ontology face feature extraction unit and the target ontology behavior feature extraction unit comprise a CNN network module and an attention mechanism module; the CNN network module is sequentially provided with a convolution layer, a batch normalization layer and a GELU activation function layer; the attention mechanism module is an ECA attention mechanism;

[0021] The target ontology image is sequentially input into the convolution layer, the batch normalization layer and the GELU activation function layer to obtain a feature map Y Oi Meanwhile, the target ontology image is input into the attention mechanism module to obtain an attention weight ω i The feature map Y Oi and the attention weight ω i are point multiplied to output the target ontology face feature or the target ontology behavior feature.

[0022] Preferably, the regional environment image is input into the environment influence branch network of the emotion recognition model to obtain an environment influence feature f E The method comprises:

[0023] The regional environment image is input into the FPN feature extraction module of the environment influence branch network to obtain a feature pyramid P={P2, P3, P4, P5}; the feature pyramid P={P2, P3, P4, P5} is sequentially output to a global average pooling layer and two layers of fully connected layers to output the environment influence feature f E .

[0024] Preferably, the method for inputting the regional environment image into the FPN feature extraction module of the environment influence branch network to obtain the feature pyramid P={P2, P3, P4, P5} comprises:

[0025] The regional environment image is input into the FPN feature extraction module of the environment influence branch network, and the FPN feature extraction module comprises a VGG network unit K1, a VGG network unit K2, a VGG network unit K3, a VGG network unit K4 and a VGG network unit K5;

[0026] The regional environment image is sequentially subjected to down-sampling processing by the VGG network unit K1, the VGG network unit K2, the VGG network unit K3, the VGG network unit K4 and the VGG network unit K5, and the feature map C2, the feature map C3, the feature map C4 and the feature map C5 are respectively output by the VGG network unit K2, the VGG network unit K3, the VGG network unit K4 and the VGG network unit K5;

[0027] The feature map C5 is up-sampled by using a bicubic interpolation method to obtain a feature map P5; the feature map C4 is up-sampled by using the bicubic interpolation method and then added to the corresponding position of the feature map P5 to obtain a feature map P4; the feature map C3 is up-sampled by using the bicubic interpolation method and then added to the corresponding position of the feature map P4 to obtain a feature map P3; the feature map C2 is up-sampled by using the bicubic interpolation method and then added to the corresponding position of the feature map P3 to obtain a feature map P2.

[0028] The feature pyramid P = {P2, P3, P4, P5} is composed of the feature map P2, the feature map P3, the feature map P4 and the feature map P5.

[0029] Preferably, the training process of the emotion recognition model comprises:

[0030] The training samples are collected to construct a training data set, and the emotion recognition model is trained by using the training data set, and the emotion recognition training result is output by the emotion recognition model; the loss value Loss(w) between the emotion recognition training result and the emotion label of the training sample is calculated by using a cross-entropy loss function; the emotion recognition model parameters are adaptively adjusted based on the loss value Loss(w) and by using an Adam optimizer; the training process of the emotion recognition model is repeatedly iterated until the trained emotion recognition model is output after reaching a set iteration number.

[0031] Preferably, the loss value Loss(w) between the emotion recognition training result and the emotion label of the training sample is calculated by using a cross-entropy loss function, and the expression formula is:

[0032]

[0033] In the formula, m represents the number of training samples of the training data set, d represents the number of emotion categories, represents whether the emotion label of the i-th training sample belongs to the category j, j∈A, h w (x (i) ) j represents the prediction probability of the emotion recognition model mapping the i-th training sample to the emotion category j.

[0034] In a second aspect, the present application provides an emotion recognition system based on a multi-branch fusion attention mechanism, comprising a cloud server and an edge platform; the cloud server and the edge platform are in bidirectional communication connection; a plurality of intelligent agents are arranged in the edge platform;

[0035] The emotion recognition model is arranged in the Actor network prediction sub-platform; the Critic network evaluation sub-platform in each agent is connected in turn in a bidirectional communication mode; the Actor network prediction sub-platform is used for extracting target ontology images and regional environment images from the collected target ontology data and regional environment data, inputting the target ontology images and the regional environment images into a pre-trained emotion recognition model to obtain an emotion recognition result and outputting the emotion recognition result; and the Critic network evaluation sub-platform is used for calculating a loss value Loss(w) between an emotion recognition training result and a training sample emotion label by using a cross-entropy loss function.

[0036] Preferably, the edge platform calculates the emotion recognition confidence of the emotion recognition model in each agent, and when there is an emotion recognition model with a confidence lower than a set confidence threshold in the edge platform, the edge platform performs local training to output trained edge platform parameters; the edge platform parameters include an optimized joint strategy H and emotion recognition model parameters.

[0037] The edge platform encodes and uploads the trained edge platform parameters to a cloud server; the cloud server receives the edge platform parameters, decodes them, and distributes the optimized edge platform parameters to other edge devices.

[0038] Preferably, the method for the edge platform to perform local training to output trained edge platform parameters comprises:

[0039] The training samples are collected to construct a training data set, and the emotion recognition model in each agent is trained by using the training data set; in the training process, a Markov quadruple <S A r i γ> is established based on a Markov decision method; wherein S represents a data set of a feature vector f extracted by each emotion recognition model, A represents an emotion state of a target ontology, r i represents a reward function of the emotion recognition model in the i th agent, and γ represents a discount factor.

[0040] The training total return of the emotion recognition model in all agents is calculated according to the Markov quadruple <S A r i γ>, and the parameters and weights of the emotion recognition model in each agent are optimized to obtain a maximum training total return; the optimization process of the edge platform on the weights of each agent is repeated until a set iteration number is reached to output the optimized joint strategy H and the parameters of the trained emotion recognition model.

[0041] Preferably, the reward function r i of the emotion recognition model in the i th agent of the edge platform is expressed by the following formula:

[0042]

[0043] In the formula, C k = 0 indicates that the classification result of the emotion recognition model in the kth agent is wrong; C k = 1 indicates that the classification result of the emotion recognition model in the kth agent is wrong; N represents the number of all agents in the system; β i (t) represents the emotion classification reward of the ith agent at time slot t.

[0044] Preferably, the edge platform calculates the total reward of the training of the emotion recognition model in all agents according to a Markov quadruple <S, A, r i γ>, and the formula is as follows:

[0045]

[0046] In the formula, C represents the total reward of the training of the emotion recognition model in all agents; H represents the joint strategy of the edge platform; E{•} represents an expectation function.

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] The present application inputs the regional environment image into the environment influence branch network of the emotion recognition model to obtain the environment influence feature f E ; the environment influence feature f E and the target ontology emotion feature f o are fused to obtain a feature vector f; the target emotion classifier of the emotion recognition model is used to classify the feature vector f to output an emotion recognition result; the emotion clues in the target regional environment are fully utilized for emotion recognition, and the accuracy of emotion recognition is greatly improved.

[0049] The present application collects target ontology data and regional environment data, pre-processes the target ontology data and the regional environment data, and extracts key frames to obtain target ontology images and regional environment images; the target ontology images and the regional environment images are input into a pre-trained emotion recognition model to obtain an emotion recognition result, which can reduce the amount of data to be transmitted without losing key image change information, thereby reducing the load of network transmission. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is a flowchart of the emotion recognition method provided in Example 1;

[0051] Figure 2 is a relationship diagram of the target ontology branch network and the environment influence branch network provided in Example 1;

[0052] Figure 3 is a structural diagram of the target ontology face feature extraction unit or the target ontology behavior feature extraction unit provided in Embodiment 1;

[0053] Figure 4 is a structural diagram of the environmental impact branch network provided in Embodiment 1;

[0054] Figure 5 is a connection relationship diagram of the intelligent agents in the edge platform provided in Embodiment 2;

[0055] Figure 6 is a structural diagram of the emotion recognition system provided in Embodiment 2. DETAILED DESCRIPTION

[0056] The application will be further described below in conjunction with the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0057] Embodiment 1

[0058] As shown in Figure 1 , the present embodiment provides an emotion recognition method based on a multi-branch fusion attention mechanism, which comprises:

[0059] Target ontology data and regional environment data are collected, and the regions of interest in each frame of the target ontology data and the regional environment data are calibrated. The calibrated target ontology data and the target ontology data are subjected to noise reduction and data enhancement processing.

[0060] The method for extracting key frames from the preprocessed target ontology data and the regional environment data to obtain target ontology images and regional environment images comprises:

[0061] The regions of interest in each frame of the target ontology data and the regional environment data are positioned, cropped and key points are marked;

[0062] The average optical flow of the regions of interest in each frame is calculated using the Lucas-Kanade optical flow method;

[0063] The frame of the region of interest with an average optical flow greater than a set threshold T is subjected to key frame extraction to obtain target ontology images and regional environment images. The target ontology images and the regional environment images are inserted into a candidate frame priority queue according to the optical flow intensity, which is used to limit the order of inputting the target ontology images and the regional environment images into the pre-trained emotion recognition model. The key frame extraction method based on the regions of interest of the optical flow can reduce the amount of data transmitted without losing key image change information, thereby reducing the load of network transmission.

[0064] The target ontology image and the regional environment image are input into a pre-trained emotion recognition model to obtain an emotion recognition result, and the process includes:

[0065] The method for inputting the target ontology image into the target ontology branch network of the emotion recognition model to obtain the target ontology facial feature and the target ontology behavior feature includes:

[0066] The target ontology image is input into a target ontology facial feature extraction unit of the target ontology branch network to obtain the target ontology facial feature, and the target ontology image is input into a target ontology behavior feature extraction unit of the target ontology branch network to obtain the target ontology behavior feature.

[0067] The target ontology facial feature extraction unit and the target ontology behavior feature extraction unit include a CNN network module and an attention mechanism module; the CNN network module is sequentially provided with a convolution layer, a batch normalization layer and a GELU activation function layer; and the attention mechanism module is an ECA attention mechanism.

[0068] The activation function expression of the GELU activation function layer is:

[0069] GELU(x)=xΦ(x)

[0070]

[0071] In the formula, x is an input parameter, and Φ(x) is a probability function of a Gaussian normal distribution.

[0072] The target ontology image is sequentially input into the convolution layer, the batch normalization layer and the GELU activation function layer to obtain a feature map Y Oi Meanwhile, the target ontology image is input into the attention mechanism module to obtain an attention weight ω i The feature map Y Oi and the attention weight ω i are point multiplied to output the target ontology facial feature or the target ontology behavior feature; and the target facial feature and the target behavior feature are fused to obtain a target ontology emotion feature f o .

[0073] The regional environment image is input into an environment influence branch network of the emotion recognition model to obtain an environment influence feature f E , and the specific method includes:

[0074] The regional environment image is input into an FPN feature extraction module of the environment influence branch network, and the FPN feature extraction module includes a VGG network unit K1, a VGG network unit K2, a VGG network unit K3, a VGG network unit K4 and a VGG network unit K5.

[0075] The region environment image is sequentially down-sampled by the VGG network unit K1, the VGG network unit K2, the VGG network unit K3, the VGG network unit K4 and the VGG network unit K5, and the VGG network unit K2, the VGG network unit K3, the VGG network unit K4 and the VGG network unit K5 output feature maps C2, C3, C4 and C5 respectively;

[0076] The feature map C5 is up-sampled by using the bicubic interpolation method to obtain a feature map P5; the feature map C4 is up-sampled by using the bicubic interpolation method, and then added to the corresponding position of the feature map P5 to obtain a feature map P4; the feature map C3 is up-sampled by using the bicubic interpolation method, and then added to the corresponding position of the feature map P4 to obtain a feature map P3; the feature map C2 is up-sampled by using the bicubic interpolation method, and then added to the corresponding position of the feature map P3 to obtain a feature map P2;

[0077] The feature pyramid P = {P2, P3, P4, P5} is composed of the feature maps P2, P3, P4 and P5; the feature pyramid P = {P2, P3, P4, P5} is sequentially output to the global average pooling layer and the two fully connected layers, and the environmental influence feature f E .

[0078] The environmental influence feature f E and the target ontology emotion feature f o are fused to obtain a feature vector f, which can be expressed as: f = concatnate [f o , f E ]; the target emotion classifier of the emotion recognition model is used to classify the feature vector f, and an emotion recognition result is output.

[0079] The training process of the emotion recognition model includes:

[0080] Training samples are collected to construct a training data set, and the emotion recognition model is trained using the training data set, and an emotion recognition training result is output by the emotion recognition model; a cross-entropy loss function is used to calculate the loss value Loss(w) between the emotion recognition training result and the training sample emotion label; the expression formula is:

[0081]

[0082] In the formula, m represents the number of training samples in the training data set, d represents the number of emotion categories, represents whether the emotion label of the i-th training sample belongs to the category j, j ∈ A, h w (x (i) ) j represents the prediction probability of the emotion recognition model mapping the i-th training sample to the emotion category j.

[0083] The emotion recognition model parameters are adaptively adjusted based on the loss value Loss(w) and using an Adam optimizer; the training process of the emotion recognition model is repeated until a set number of iterations is reached, and a trained emotion recognition model is output.

[0084] The emotion recognition model with the multi-branch fusion attention mechanism in the embodiment fuses emotion features in the target ontology and the target regional environment, and takes into account the emotion mutual influence relationship between the target ontologies, greatly improving the accuracy of emotion recognition; in the embodiment, the cross-entropy loss function is used to calculate the loss value Loss(w) between the emotion recognition training result and the training sample emotion label, and the emotion recognition model parameters are adaptively adjusted based on the loss value Loss(w) and using an Adam optimizer, improving the robustness of the emotion recognition model.

[0085] Embodiment 2

[0086] The embodiment provides an emotion recognition system based on a multi-branch fusion attention mechanism, comprising: a cloud server and an edge platform; the cloud server and the edge platform are in bidirectional communication connection; a plurality of intelligent agents are arranged in the edge platform;

[0087] The Actor network prediction sub-platform and the Critic network evaluation sub-platform are arranged in the intelligent agent; the emotion recognition model in embodiment one is arranged in the Actor network prediction sub-platform; the Critic network evaluation sub-platform in each intelligent agent is in bidirectional communication connection in turn; the Actor network prediction sub-platform is used for extracting target ontology images and regional environment images from collected target ontology data and regional environment data, inputting the target ontology images and the regional environment images into a pre-trained emotion recognition model to obtain an emotion recognition result and outputting the emotion recognition result; the Critic network evaluation sub-platform is used for calculating the loss value Loss(w) between the emotion recognition training result and the training sample emotion label using a cross-entropy loss function.

[0088] The edge platform calculates the emotion recognition confidence of the emotion recognition model in each intelligent agent, and when there is an emotion recognition model with a confidence lower than a set confidence threshold in the edge platform, the edge platform performs local training to output trained edge platform parameters, comprising:

[0089] Training samples are collected to construct a training data set, and the emotion recognition model in each intelligent agent is trained using the training data set; in the training process, a Markov quadruple <S A r i > is established based on a Markov decision method; wherein S represents a data set of the feature vector f extracted by each emotion recognition model, A represents the emotion state of the target ontology, and r irepresents the reward function of the emotion recognition model in the ith agent; and represents a discount factor;

[0090] The emotion state A of the target ontology can be expressed as:

[0091] A={ 'angry', 'disgust', 'fear', 'happy','sad','surprise', 'neutral'}

[0092] The reward function r of the emotion recognition model in the ith agent of the edge platform i , and the expression formula is:

[0093]

[0094] In the formula, C k =0 represents that the classification result of the emotion recognition model in the kth agent is wrong; C k =1 represents that the classification result of the emotion recognition model in the kth agent is wrong; N represents the number of all agents in the system; and β i (t) represents the emotion classification reward of the ith agent at time slot t.

[0095] The edge platform calculates the total reward of the emotion recognition model of all agents according to the Markov quadruple <S, A, r i , and the expression formula is:

[0096]

[0097] In the formula, represents the total reward of the emotion recognition model of all agents; H represents the joint strategy of the edge platform; and E{·} represents an expectation function.

[0098] The parameters and weights of the emotion recognition model of each agent are optimized to obtain the maximum total reward, and the optimization process of the edge platform on the weights of each agent is repeated and iterated until the set iteration number is reached to output the edge platform parameters; the edge platform parameters include the optimized joint strategy H and the parameters of the trained emotion recognition model; the multi-agent is used for reinforcement learning emotion fusion training, and the robustness of the emotion recognition model is improved.

[0099] The edge platform encodes and uploads the trained edge platform parameters to the cloud server; this process does not contain any data containing user privacy. The transmission process uses methods such as quantization and sparsity to compress and encode the model weight matrix, reducing the transmission bandwidth and storage space; the cloud server receives the edge platform parameters and decodes them, and distributes the optimized edge platform parameters to other edge devices; based on the emotion perception edge-cloud collaborative framework, the emotion recognition model optimization of the edge platform and the cloud server can be realized under the premise of privacy protection.

[0100] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) having computer-usable program code embodied therein.

[0101] The application is described with reference to flowcharts and / or block diagrams that illustrate the methods, apparatus (systems), and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions that are executed by the processor of the computer or other programmable data processing apparatus generate means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowcharts and / or block diagrams.

[0102] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction means, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowcharts and / or block diagrams.

[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowcharts and / or block diagrams.

[0104] The above description is only the preferred embodiment of the present application, it should be pointed out that for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and variations can be made, which should be considered as the protection scope of the present application.

Claims

1. A multi-branch fusion attention mechanism-based emotion recognition method, characterized in that, The method comprises the following steps: Collecting target body data and regional environment data, and performing data preprocessing and key frame extraction on the target body data and the regional environment data to obtain target body images and regional environment images; Inputting the target body images and the regional environment images into a pre-trained emotion recognition model to obtain an emotion recognition result, the process comprising: Inputting the target body images into a target body branch network of the emotion recognition model to obtain target body facial features and target body behavior features, specifically comprising: Inputting the target body images into a target body facial feature extraction unit of the target body branch network to obtain target body facial features, and inputting the target body images into a target body behavior feature extraction unit of the target body branch network to obtain target body behavior features; The target body facial feature extraction unit and the target body behavior feature extraction unit comprise a CNN network module and an attention mechanism module; the CNN network module is sequentially provided with a convolution layer, a batch normalization layer and a GELU activation function layer; and the attention mechanism module is an ECA attention mechanism; The target ontology image is sequentially input into a convolution layer, a batch normalization layer and a GELU activation function layer to obtain a feature map Meanwhile, the target ontology image is input into an attention mechanism module to obtain an attention weight The feature map and the attention weight are point multiplied to output a target ontology face feature or a target ontology behavior feature; fusing the target facial feature and the target behavioral feature to obtain a target ontology emotional feature ; The regional environment image is input into an environment influence branch network of the emotion recognition model to obtain environment influence features , and specifically comprises: The regional environment image is input to a FPN feature extraction module of the environment influence branch network, and the FPN feature extraction module comprises VGG network units , a VGG network unit , a VGG network unit , a VGG network unit , and a VGG network unit ; The VGG network unit The VGG network unit The VGG network unit The VGG network unit The VGG network unit The region environment image is sequentially subjected to down-sampling processing, and the VGG network unit The VGG network unit The VGG network unit The VGG network unit Respectively output feature maps Feature maps Feature maps And feature maps ; adopting bicubic interpolation method to the feature map to obtain the feature map ; adopting bicubic interpolation method to the feature map after upsampling and the feature map corresponding positions are added to obtain the feature map ; adopting bicubic interpolation method to the feature map after upsampling and the feature map corresponding positions are added to obtain the feature map ; adopting bicubic interpolation method to the feature map after upsampling and the feature map corresponding positions are added to obtain the feature map ; from the feature map , the feature map , the feature map , and the feature map comprise a feature pyramid ; outputting the feature pyramid to a global average pooling layer and two fully connected layers in sequence to output an environmental influence feature outputting the feature pyramid to a global average pooling layer and two fully connected layers in sequence to output an environmental influence feature ; Environmental impact characteristics and target emotional characteristics Perform feature fusion to obtain feature vector ; Using the target emotion classifier of the emotion recognition model to classify the feature vector Perform classification and output emotion recognition results.

2. The emotion recognition method of claim 1, wherein, The method for obtaining the target body images and the regional environment images by performing data preprocessing and key frame extraction on the target body data and the regional environment data comprises: Performing region of interest calibration on each frame of picture in the target body data and each frame of picture in the regional environment data, and performing noise reduction and data enhancement processing on the calibrated target body data and the target body data; positioning and cropping the region of interest in each frame of picture of the target body data and each frame of picture of the regional environment data, and performing key point labeling; Calculating the average optical flow of the region of interest in each frame of picture by using a Lucas-Kanade optical flow method; performing key frame extraction on the frame of picture in which the region of interest with an average optical flow greater than a set threshold T is located to obtain the target body images and the regional environment images; and inserting the target body images and the regional environment images into a candidate frame priority queue according to the optical flow intensity, so as to limit the order of inputting the target body images and the regional environment images into the pre-trained emotion recognition model. 3.The emotion recognition method of claim 1, wherein, The training process of the emotion recognition model comprises: Collecting training samples to construct a training data set, and training the emotion recognition model by using the training data set, and outputting an emotion recognition training result by the emotion recognition model; calculating a loss value Loss(w) between the emotion recognition training result and a training sample emotion label by using a cross-entropy loss function; adaptively adjusting the emotion recognition model parameters based on the loss value Loss(w) and using an Adam optimizer; repeating the training process of the emotion recognition model until a set number of iterations is reached to output a trained emotion recognition model.

4. The emotion recognition method of claim 3, wherein, The loss value Loss(w) between the emotion recognition training result and the training sample emotion label is calculated by using a cross-entropy loss function, and the expression formula is: ； In the formula, denotes the number of training samples of the training data set, denotes the number of emotion categories, denotes whether the emotion label of the th training sample belongs to the category , denotes the prediction probability of the emotion recognition model mapping the th training sample to the emotion category .

5. An emotion recognition system based on multi-branch fusion attention mechanism, characterized in that, The system comprises a cloud server and an edge platform; the cloud server and the edge platform are in bidirectional communication connection; the edge platform is provided with a plurality of intelligent agents; and the intelligent agents are used to execute the emotion recognition method in any one of claims 3 to 4. The system comprises a cloud server and an edge platform; the cloud server and the edge platform are in bidirectional communication connection; the edge platform is provided with a plurality of intelligent agents; and the intelligent agents are used to execute the emotion recognition method in any one of claims 3 to 4. The intelligent agent is internally provided with an Actor network prediction sub-platform and a Critic network evaluation sub-platform; the Critic network evaluation sub-platforms in each intelligent agent are sequentially bidirectionally connected; the Actor network prediction sub-platform is used for inputting a target ontology image and a regional environment image into a pre-trained emotion recognition model to obtain an emotion recognition result; The Critic network evaluation sub-platform is used for calculating a loss value Loss(w) between an emotion recognition training result and a training sample emotion label by using a cross-entropy loss function.

6. The emotion recognition system of claim 5, wherein, The edge platform calculates an emotion recognition confidence of the emotion recognition model in each intelligent agent, and when there is an emotion recognition model with a confidence lower than a set confidence threshold in the edge platform, the edge platform performs local training and outputs trained edge platform parameters; the edge platform parameters include an optimized joint strategy H and emotion recognition model parameters; The edge platform encodes and uploads the trained edge platform parameters to a cloud server; The cloud server receives the edge platform parameters and decodes them, and distributes the optimized edge platform parameters to other edge devices.

7. The emotion recognition system of claim 6, wherein, The method for the edge platform to perform local training and output trained edge platform parameters includes: The training samples are collected to construct a training data set, and the emotion recognition model in each agent is trained by using the training data set; in the training process, a Markov quadruple is established based on a Markov decision method ; wherein S represents a data set of a feature vector extracted by each emotion recognition model ; A represents an emotion state of a target ontology; ; represents a reward function of the emotion recognition model in the i th agent; ; represents a discount factor; According to the Markov four-tuple The total training return of the emotion recognition model in all agents is calculated, the parameters and weights of the emotion recognition model in each agent are optimized with the goal of obtaining the maximum total training return, the optimization process of the edge platform on the weights of each agent is repeated iteratively until a set number of iterations is reached to output the optimized joint strategy H and the parameters of the trained emotion recognition model.

Images