A method, device, equipment and medium for emotion recognition based on multimodal information fusion of fuzzy brain and hybrid experts

Abstract

A method, device, equipment and medium for emotion recognition based on multimodal information fusion of fuzzy brain and hybrid experts, relating to the field of emotion recognition. The method comprises: obtaining initial data. Inputting data into a deep feature extraction fusion model to obtain fusion features. Inputting features into a hybrid expert recognition model to obtain emotion recognition results. The hybrid expert recognition model comprises an expert network, a gated network and a fusion network. The expert network comprises multiple expert models. The gated network comprises a gated model. Both the expert model and the gated model are interval type II fuzzy brain emotion learning networks. The interval type II fuzzy brain emotion learning network comprises an input layer, an amygdala network, an orbitofrontal cortex network and an output layer. The input layer transmits input signals to the amygdala network and the orbitofrontal cortex network, which jointly regulate the output. The amygdala network comprises a perception layer and an excitation layer. The perception layer converts the input layer signal into a fuzzy set. The excitation layer performs fuzzy reasoning on the perception layer signal to obtain an excitation signal.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal information fusion and recognition, and in particular to an emotion recognition method, device, equipment and medium based on multimodal information fusion of fuzzy brain and hybrid experts. Background Art

[0002] With the advancement of technology, the fusion processing of multiple modal information including human sensory signals, biological signals, sensor signals, etc. has become a research direction that has attracted much attention, and has demonstrated superior performance in many fields. For example: in the field of intelligent traffic management, combining multiple modalities such as images, sonar and radar to identify and predict vehicle behavior to improve safety and efficiency. In medical diagnosis, the fusion of image and voice features can assist doctors in understanding the patient's psychological state, such as emotions. Challenges and opportunities faced by multimodal fusion technology: How to efficiently process and analyze information while ensuring its consistency and reliability requires further exploration.

[0003] The main goal of multimodal feature fusion is to reduce the heterogeneity between modalities and maintain the integrity of the specific semantics of each modality. There are two main ways to fuse features: fusion at the end and fusion while extracting. To achieve effective cross-modal information fusion, it is crucial to make full use of the complementary spatial semantic information between multiple layers. However, existing research has not yet perfected the research strategy on how to utilize the spatial semantic correlations at various levels between different modalities.

[0004] Therefore, it has become an urgent problem to study how to effectively extract complementary features from various modalities in a multi-layer manner, and then improve the quality and expressiveness of the fused features to improve the effect of multimodal data recognition in emotion recognition. Summary of the invention

[0005] The present invention provides an emotion recognition method, device, equipment and medium based on multimodal information fusion of fuzzy brain and hybrid experts to improve at least one of the above technical problems.

[0006] In a first aspect, the present invention provides an emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts, which comprises steps S1 to S3.

[0007] S1. Obtaining initial data, wherein the initial data at least includes data of two different modalities: image data and audio data.

[0008] S2. Input the initial data into a pre-built deep feature extraction fusion model to extract features from data of different modalities and fuse them during the extraction process to obtain fused features. The deep feature extraction fusion model is used to cross-extract features and fuse features.

[0009] S3, input the fused features into a pre-built hybrid expert recognition model for classification to obtain the emotion recognition result of the initial data. The hybrid expert recognition model includes an expert network, a gating network and a fusion network. The expert network includes multiple expert models for extracting recognition features based on the fused features. The gating network includes a gating model for calculating the weights of each recognition feature based on the fused features. The fusion network is used to fuse each recognition feature into the emotion recognition result according to the weights.

[0010] Preferably, the emotion categories of the emotion recognition include anger, disgust, fear, happiness, sadness, surprise, and neutral.

[0011] Preferably, the expert model and the gated model are both interval type-2 fuzzy brain emotion learning networks. The interval type-2 fuzzy brain emotion learning network includes an input layer, an amygdala network, an orbitofrontal cortex network and an output layer. The input layer is used to receive external input signals and transmit the signals to the amygdala network and the orbitofrontal cortex network. The amygdala network includes a perception layer and an excitation layer. The perception layer is used to receive signals from the input layer and convert them into fuzzy sets. The excitation layer is used to perform fuzzy reasoning based on the input signals from the perception layer to obtain excitation signals. The orbitofrontal cortex network is used to jointly regulate the output with the amygdala network. The output layer is used to output the final output.

[0012] In an optional embodiment, the deep feature extraction fusion model includes a feature extraction network, a horizontal connection network and a feature splicing network.

[0013] The number of the feature extraction networks is the same as the number of modes in the initial data, and they are used to extract features from data of different modes respectively. The feature extraction network includes a plurality of first feature extraction modules arranged in cascade, and is used to perform multi-level feature extraction.

[0014] The number of the first feature extraction modules in the horizontal connection network and the feature extraction network is the same. The multiple horizontal connection networks are respectively connected to the first feature extraction modules of the same level of different feature extraction networks. The horizontal connection network includes a horizontal connection module and a second feature extraction module. The horizontal connection module is used to horizontally connect the features extracted by the first feature extraction module of the same level. The second feature extraction module is used to extract features from the features after horizontal connection to obtain horizontal connection features of each level.

[0015] The feature concatenation network is used to concatenate the horizontally connected features of each level into the fused feature.

[0016] In an optional embodiment, the number of the first feature extraction modules is four. The first feature extraction module includes a cascaded two-dimensional convolution layer, a LeakyRelu activation layer, and a maximum pooling layer. The convolution kernels of the two-dimensional convolution layers of the four cascaded first feature extraction modules are 16, 32, 64, and 128, respectively.

[0017] The second feature extraction module includes a two-dimensional convolutional layer, an activation function layer, a maximum pooling layer, a two-dimensional convolutional layer, an activation function, a maximum pooling layer, a flattening layer, and a fully connected layer connected in sequence.

[0018] The feature stitching network is configured to perform vertical stitching.

[0019] In an optional implementation, the model of the interval type-2 fuzzy brain emotion learning network is: In the formula, is the output of the model of interval type-2 fuzzy brain emotion learning network, is the output of the amygdala network, is the output of the orbitofrontal cortex network.

[0020] The model of the amygdala network is: In the formula, is the output of the amygdala network, and are the output values ​​of the left and right points of the amygdala after fuzzy reasoning, Calculate the number of blocks in the amygdala network, is the output of the left point of the perception layer, is the output of the right point of the perception layer, The amygdala The linear weight of the computational blocks, and are the upper and lower bounds of the amygdala linear weight, respectively.

[0021] The model of the orbitofrontal cortex network is: In the formula, is the output of the orbitofrontal cortex network, and are the output values ​​of the left and right points of the orbitofrontal cortex after fuzzy reasoning, Calculate the number of blocks in the orbitofrontal cortex network, is the number of input features, For the Input features Located in the orbitofrontal cortex The membership of the computational blocks, and are the upper and lower bounds of the orbitofrontal cortex membership, The orbitofrontal cortex The linear weight of the computational blocks, and are the upper and lower bounds of the linear weights of the orbitofrontal cortex, respectively.

[0022] In an optional implementation, the model of the perception layer is: In the formula, For the Input features Located in the amygdala The membership of the computational blocks, and are the upper and lower bounds of the amygdala membership, is an exponential function with a natural constant as its base. For the Input features, is the variance of the interval type II Gaussian membership function in the amygdala network, is the initial mean of the interval type II Gaussian membership function in the amygdala network, and They are the upper and lower bounds of the initial mean of the interval type-II Gaussian membership function in the amygdala network, respectively.

[0023] In an optional implementation, the model of the excitation layer is: In the formula, is the upper bound of fuzzy reasoning, is the lower bound of fuzzy reasoning, is the number of input features.

[0024] In an optional implementation, the loss function of the expert network is: In the formula, is the loss of the expert network, is the number of expert models, Output of the gating network to the The gate value of the expert model, is the true label value, For the The output of the expert model, is the penalty coefficient of expert network loss, For the The relevant penalty function of the expert model, It is the integrated combined output.

[0025] In an optional embodiment, the loss function of the gating network is: In the formula, is the loss of the gating network, is the estimate of the posterior probability, is the gate value, For the An expert model can generate an estimate of the posterior probability of the desired output, Output of the gating network to the The gate value of the expert model, For the The output of the expert model, For the The relevant penalty function of the expert model, For the The gating value of the expert model.

[0026] In an optional embodiment, the reward signal of the gating model is: In the formula, For the The reward signal corresponding to the expert model, Output of the gating network to the The gate value of the expert model, is an exponential function with a natural constant as its base. is the true label value, For the The output of the expert model, For transposition, is the number of expert models.

[0027] In an optional implementation, the weight update method of the expert model is:

[0028] .

[0029] .

[0030] In the formula, is the weight of the hidden layer of the amygdala network in the expert model, for The update amount, is the degradation rate of momentum, For the An expert model can generate an estimate of the posterior probability of the desired output, is the learning rate of the amygdala network in the expert model, For input features, is a function that takes the maximum value of two numbers, is the true label value, is the output of the amygdala network in the expert model, is the weight of the hidden layer of the orbitofrontal cortex network in the expert network, for The update amount, is the learning rate of the orbitofrontal cortex network in the expert network, For the The output of an expert model.

[0031] In an optional implementation, the weight update method of the gating model is:

[0032] .

[0033] .

[0034] In the formula, is the weight of the hidden layer of the amygdala network in the gating model, for The update amount, is the learning rate of the amygdala network in the gating model, As a reward signal, is the output of the amygdala network in the gating model, The gated network is the weight of the hidden layer of the orbitofrontal cortex network, for The update amount, is the learning rate of the orbitofrontal cortex network in the gating network, is the gate value.

[0035] In an optional implementation, the fuzzy parameter update model is:

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] ;

[0043] ;

[0044] ;

[0045] ;

[0046] In the formula, is the number of iterations, is the partial derivative, is the loss of the expert network, and are the upper and lower bounds of the initial mean of the interval type II Gaussian membership function in the amygdala network, and are the upper and lower bounds of the initial mean of the interval type II Gaussian membership function in the orbitofrontal cortex network, is the learning rate of the initial mean, is the variance of the interval type II Gaussian membership function in the amygdala network, is the variance of the interval type II Gaussian membership function in the orbitofrontal cortex network, is the learning rate of the variance, and are the upper and lower bounds of the linear weight of the amygdala, is the learning rate of the linear weights of the amygdala, and are the upper and lower bounds of the linear weight of the orbitofrontal cortex, is the learning rate for the linear weights of the orbitofrontal cortex.

[0047] In the second aspect, the present invention provides an emotion recognition device based on multimodal information fusion of fuzzy brain hybrid experts, which includes an initial data acquisition module, a feature fusion extraction module and a feature recognition module.

[0048] The initial data acquisition module is used to acquire initial data, wherein the initial data at least includes data of two different modes: image data and audio data.

[0049] The feature fusion extraction module is used to input the initial data into a pre-built deep feature extraction fusion model to extract features from data of different modalities and fuse them during the extraction process to obtain fusion features. The deep feature extraction fusion model is used to cross-extract features and fuse features.

[0050] A feature recognition module is used to input the fused features into a pre-built hybrid expert recognition model for classification to obtain the emotion recognition result of the initial data. The hybrid expert recognition model includes an expert network, a gating network and a fusion network. The expert network includes multiple expert models for extracting recognition features based on the fused features. The gating network includes a gating model for calculating the weights of each recognition feature based on the fused features. The fusion network is used to fuse each recognition feature into the emotion recognition result according to the weights.

[0051] The expert model and the gated model are both interval type-2 fuzzy brain emotion learning networks. The interval type-2 fuzzy brain emotion learning network includes an input layer, an amygdala network, an orbitofrontal cortex network and an output layer. The input layer is used to receive external input signals and transmit the signals to the amygdala network and the orbitofrontal cortex network. The amygdala network includes a perception layer and an excitation layer. The perception layer is used to receive signals from the input layer and convert them into fuzzy sets. The excitation layer is used to perform fuzzy reasoning based on the input signals from the perception layer and obtain excitation signals. The orbitofrontal cortex network is used to jointly regulate the output with the amygdala network. The output layer is used to output the final output.

[0052] In a third aspect, the present invention provides an emotion recognition device based on multimodal information fusion of fuzzy brain and mixed experts, which includes a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement an emotion recognition method based on multimodal information fusion of fuzzy brain and mixed experts as described in any paragraph of the first aspect.

[0053] In a fourth aspect, the present invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to perform an emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts as described in any paragraph of the first aspect.

[0054] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0055] The present invention provides an emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts. Through a deep feature extraction fusion model (network), the information of the network can flow effectively, giving full play to the advantages of multimodal and multi-layer features. It can reduce the complexity of the network structure, make full use of the connection between the modes, and give full play to the advantages of multimodal data. Through a hybrid expert classification system based on an interval type-II fuzzy brain emotion learning network, the complementary nonlinear correlation of features is fully utilized. It designs a suitable gating network for specific downstream tasks and dynamically allocates expert model resources, so that the hybrid expert recognition model can flexibly adapt to different input data distributions and task requirements, thereby improving the performance and efficiency of the model, so that it has better accuracy and adaptability when processing complex multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the specific implementation methods of the present invention. It should be understood that the following drawings only show certain specific implementation methods of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 It is a flowchart of an emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts.

[0058] Figure 2 It is a logical block diagram of an emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts.

[0059] Figure 3 It is a traditional multimodal feature extraction architecture.

[0060] Figure 4 It is a multi-layer multimodal feature extraction architecture of the present invention.

[0061] Figure 5 It is a network structure diagram of the multi-layer multimodal feature extraction architecture of the present invention.

[0062] Figure 6 It is the detailed internal structure of the feature extraction block of the multi-layer multimodal feature extraction architecture of the present invention.

[0063] Figure 7 It is the structural diagram of the fuzzy brain hybrid expert classification system. DETAILED DESCRIPTION

[0064] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0065] Example 1, please refer to Figures 1 to 7 The first embodiment of the present invention provides an emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts, which includes steps S1 to S3.

[0066] S1. Obtaining initial data, wherein the initial data at least includes data of two different modalities: image data and audio data.

[0067] S2. Input the initial data into a pre-built deep feature extraction fusion model to extract features from data of different modalities and fuse them during the extraction process to obtain fused features. The deep feature extraction fusion model is used to cross-extract features and fuse features.

[0068] S3, inputting the fusion feature into a pre-built hybrid expert recognition model for classification to obtain the emotion recognition result of the initial data. Preferably, the emotion categories of the emotion recognition include anger, disgust, fear, happiness, sadness, surprise, and neutral.

[0069] The hybrid expert recognition model includes an expert network, a gating network and a fusion network. The expert network includes multiple expert models for extracting recognition features according to the fusion features. The gating network includes a gating model for calculating the weights of each recognition feature according to the fusion features. The fusion network is used to fuse each recognition feature into the emotion recognition result according to the weights.

[0070] Preferably, the expert model and the gated model are both interval type-2 fuzzy brain emotion learning networks. The interval type-2 fuzzy brain emotion learning network includes an input layer, an amygdala network, an orbitofrontal cortex network and an output layer. The input layer is used to receive external input signals and transmit the signals to the amygdala network and the orbitofrontal cortex network. The amygdala network includes a perception layer and an excitation layer. The perception layer is used to receive signals from the input layer and convert them into fuzzy sets. The excitation layer is used to perform fuzzy reasoning based on the input signals from the perception layer to obtain excitation signals. The orbitofrontal cortex network is used to jointly regulate the output with the amygdala network. The output layer is used to output the final output.

[0071] The present invention provides an emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts. Through a deep feature extraction fusion model (network), the information of the network can flow effectively, giving full play to the advantages of multimodal and multi-layer features. It can reduce the complexity of the network structure, make full use of the connection between the modes, and give full play to the advantages of multimodal data. Through a hybrid expert classification system based on an interval type-II fuzzy brain emotion learning network, the complementary nonlinear correlation of features is fully utilized. It designs a suitable gating network for specific downstream tasks and dynamically allocates expert model resources, so that the hybrid expert recognition model can flexibly adapt to different input data distributions and task requirements, thereby improving the performance and efficiency of the model, so that it has better accuracy and adaptability when processing complex multimodal data.

[0072] Based on the above embodiments, in an optional embodiment of the present invention, as Figure 4As shown, the deep feature extraction fusion model includes a feature extraction network, a horizontal connection network and a feature splicing network.

[0073] The number of feature extraction networks is the same as the number of modalities in the initial data, and they are used to extract features from data of different modalities respectively. The feature extraction network includes a plurality of first feature extraction modules arranged in cascade, which are used to perform multi-level extraction of features. Preferably, the number of the first feature extraction modules is four. The first feature extraction module includes a two-dimensional convolutional layer, a LeakyRelu activation layer and a maximum pooling layer arranged in cascade. The convolution kernels of the two-dimensional convolutional layers of the four first feature extraction modules arranged in cascade are 16, 32, 64 and 128 respectively.

[0074] The number of the first feature extraction modules in the horizontal connection network and the feature extraction network is the same. A plurality of the horizontal connection networks are respectively connected to the first feature extraction modules at the same level of different feature extraction networks. The horizontal connection network includes a horizontal connection module and a second feature extraction module. The horizontal connection module is used to horizontally connect the features extracted by the first feature extraction module at the same level. The second feature extraction module is used to extract features from the features after horizontal connection to obtain horizontal connection features of each level. Preferably, the second feature extraction module includes a two-dimensional convolutional layer, an activation function layer, a maximum pooling layer, a two-dimensional convolutional layer, an activation function, a maximum pooling layer, a flattening layer, and a fully connected layer connected in sequence.

[0075] The feature splicing network is used to splice the horizontally connected features of each level into the fused feature. Preferably, the feature splicing network is configured to perform vertical splicing.

[0076] Specifically, the embodiment of the present invention proposes a deep feature extraction fusion model (network) designed for multimodal data (such as Figure 4 As shown in Figure 2, replacing the traditional feature extraction network (such as Figure 3 The model of first extraction and then fusion is used to explore the strategies and methods of combining complementary spatial information between multiple modalities with different attributes, so that the information of the network can flow effectively and give full play to the advantages of multimodal and multi-layer features. The model obtained by the study can not only reduce the complexity of the network structure, but also make full use of the connection between the various modalities, and give full play to the advantages of multimodal data.

[0077] like Figure 3As shown in the figure, in most current multimodal recognition studies, different models are used for feature extraction for each modality, and multiple top-level feature vectors are concatenated and fused at the end. Although this method is easy to implement, it does not establish the spatiotemporal correlation between the modalities during the feature extraction process, and it also fails to combine different shallow and deep semantic features. In many current multimodal application scenarios with high recognition complexity, this simple fusion solution has great limitations. When faced with the problem of high model complexity, it cannot fully utilize the advantages of multimodal data, and there is no close connection between the models, resulting in results that are insufficient to meet high-accuracy classification requirements.

[0078] To solve this problem, the present invention designs a deep neural network architecture (such as Figure 4 As shown in the figure, a hybrid network with multiple connections between layers is constructed, and each modality is integrated during the feature extraction process to cope with the high complexity of multimodal feature recognition and to help the recognition accuracy and adaptability of the recognition system.

[0079] Specifically, the embodiment of the present invention improves the traditional CNN feature extraction network and proposes a deep feature extraction fusion model (such as Figure 4 As shown in Figure 2). Compared with the traditional fusion method, the network architecture has made adjustments in the fusion method and fusion time. By building multiple CNN structure flows in one network, the function of simultaneously extracting features from multiple modal data is realized. At the same time, the model is scalable and can be applied to various modal data, and the model complexity will not increase significantly when the number of modalities increases.

[0080] like Figure 3 The existing model shown in FIG. 1 extracts features layer by layer through a convolutional neural network, and concatenates the feature vectors extracted by the two models at the end of the network. The two models have no intersection in the intermediate scale of feature extraction, while the deep feature extraction fusion model (such as Figure 4 (as shown) is to establish a connection between two branches during the extraction process, and fuse the outputs of the convolution blocks of corresponding scales. In the selection of the solution for multi-layer multimodal feature fusion, the embodiment of the present invention uses horizontal connections as mixed connections in different modal structure flows, and can obtain joint representations of multiple modalities at different scales. These fused features of different scales will be pushed into multiple feature extraction blocks again for further feature extraction. This method of fusing in multiple stages of feature extraction makes full use of the spatial information of different layers and establishes the correlation between different modalities.

[0081] According to the network architecture proposed above, the internal layer structure of the network is designed. The detailed structure of each layer of the proposed deep feature extraction fusion model and the second feature extraction module The internal structure is shown in the attached Figure 5 and Figure 6 As shown in the figure, each feature extraction network contains four convolution blocks, each of which consists of a two-dimensional convolution layer, a LeakyRelu activation layer, and a maximum pooling layer. However, as the number of network layers increases from shallow to deep, the number of convolution kernels in each convolution block changes, which are 16, 32, 64, and 128 respectively.

[0082] The network horizontally concatenates and fuses the outputs of convolution blocks of the same scale on different branches, and the fused feature vector is sent to the second feature extraction module again. Further extract features. It contains two convolutional layers, an activation function layer and a maximum pooling layer, and is flattened before being input into a fully connected layer. Output Said that eventually Vertical splicing is performed to obtain the final fused feature vector.

[0083] The embodiment of the present invention proposes to use the Gaussian interval type II fuzzy brain emotion learning model (network) as the expert model of the hybrid expert system to form a new hybrid expert system (such as Figure 7 As shown in the figure), the complementary nonlinear correlation of features is fully utilized, and by designing a suitable gating network and dynamically allocating expert model resources, the hybrid expert model can flexibly adapt to different input data distributions and task requirements, thereby improving the performance and efficiency of the model and enabling it to have better accuracy and adaptability when processing complex multimodal data.

[0084] On the basis of the above embodiment, in an optional embodiment of the present invention, the model of the interval type-2 fuzzy brain emotion learning network is: In the formula, is the output of the model of interval type-2 fuzzy brain emotion learning network, is the output of the amygdala network, is the output of the orbitofrontal cortex network.

[0085] In this embodiment, the interval type-2 fuzzy brain emotion learning network includes an input layer, an amygdala network, an orbitofrontal cortex network and an output layer.

[0086] The model of the amygdala network is: In the formula, is the output of the amygdala network, and are the output values ​​of the left and right points of the amygdala after fuzzy reasoning, Calculate the number of blocks in the amygdala network, is the output of the left point of the perception layer, is the output of the right point of the perception layer, The amygdala The linear weight of the computational blocks, and are the upper and lower bounds of the amygdala linear weight, respectively.

[0087] The model of the orbitofrontal cortex network is: In the formula, is the output of the orbitofrontal cortex network, and are the output values ​​of the left and right points of the orbitofrontal cortex after fuzzy reasoning, Calculate the number of blocks in the orbitofrontal cortex network, is the number of input features, For the Input features Located in the orbitofrontal cortex The membership of the computational blocks, and are the upper and lower bounds of the orbitofrontal cortex membership, The orbitofrontal cortex The linear weight of the computational blocks, and are the upper and lower bounds of the linear weights of the orbitofrontal cortex, respectively.

[0088] It is understood that a horizontal line above a symbol in a formula indicates an upper limit value of the symbol, and a horizontal line below a symbol in a formula indicates a lower limit value of the symbol.

[0089] The amygdala network includes a perception layer and an excitation layer.

[0090] Preferably, the model of the perception layer is: In the formula, For the Input features Located in the amygdala The membership of the computational blocks, and are the upper and lower bounds of the amygdala membership, is an exponential function with a natural constant as its base. For the Input features, is the variance of the interval type II Gaussian membership function in the amygdala network, is the initial mean of the interval type II Gaussian membership function in the amygdala network, , and They are the upper and lower bounds of the initial mean of the interval type-II Gaussian membership function in the amygdala network, respectively.

[0091] Preferably, the model of the excitation layer is: In the formula, is the upper bound of fuzzy reasoning, is the lower bound of fuzzy reasoning, is the number of input features.

[0092] The maximum weight matching point of the left brain calculated by the KM algorithm and the right brain maximum weight matching point , then according to the upper bound of fuzzy reasoning , the lower bound of fuzzy reasoning , the maximum weight matching point of the left brain and the right brain maximum weight matching point , get the output of the left point of the perception layer And the output of the right point of the perception layer . This algorithm can significantly improve the computational efficiency of the model.

[0093] The specific expression is as follows: In the formula, Amygdala The fuzzy inference output of the excitation layer of the computational block.

[0094] Based on the above embodiment, in an optional embodiment of the present invention, the loss function of the expert network is: In the formula, is the loss of the expert network, is the number of expert models, Output of the gating network to the The gate value of the expert model, is the true label value, For the The output of the expert model, is the penalty coefficient of expert network loss, For the The relevant penalty function of the expert model, It is the integrated combined output.

[0095] The embodiment of the present invention adds a relevant penalty term to the error function and introduces a control parameter so that the deviation-variance-covariance is close to the optimal balance. Minimize The error of each network is negatively correlated with the error of the rest of the whole. This parameter is used to adjust the intensity of the penalty. When it is 0, there is no relevant penalty term in the error function of each network, and each network is trained independently. The purpose of this penalty function is to encourage different individual networks to learn different parts of the training data in order to better learn the entire training data.

[0096] Based on the above embodiment, in an optional embodiment of the present invention, the loss function of the gating network is: In the formula, is the loss of the gating network, is the estimate of the posterior probability, is the gate value, For the An expert model can generate an estimate of the posterior probability of the desired output, Output of the gating network to the The gate value of the expert model, For the The output of the expert model, For the The relevant penalty function of the expert model, For the The gating value of the expert model.

[0097] In the study of brain emotion learning models, the definition of reward signal REW is one of the key points. This signal is flexible and needs to be determined according to the specific system type in order to achieve the optimal performance of the model. For example, when the network is applied to control systems, the reward signal is generally related to the control error, frequency deviation or other control signals, such as sliding mode control signals.

[0098] Similarly, for classification problems, it is also necessary to define a suitable reward signal. In order to make the interval type-2 fuzzy brain emotion learning network suitable for the learning strategy of the hybrid expert network, this patent develops an independent reward signal for the interval type-2 fuzzy brain emotion learning network model as a gated network. , whose mathematical meaning is the posterior probability that each expert can generate the desired output.

[0099] Based on the above embodiment, in an optional embodiment of the present invention, the reward signal of the gate model is: In the formula, For the The reward signal corresponding to the expert model, Output of the gating network to the The gate value of the expert model, is an exponential function with a natural constant as its base. is the true label value, For the The output of the expert model, For transposition, is the number of expert models.

[0100] The main feature that distinguishes the brain emotion learning model from common intelligent algorithms is the special weight update rule. In the hybrid expert architecture, the expert network and the gated network usually use different weight update methods.

[0101] Based on the above embodiment, in an optional embodiment of the present invention, similar to the negative correlation learning method, the introduced penalty term provides a control parameter that can approach the optimal balance in the bias-variance-covariance trade-off. In the hybrid expert recognition model proposed in this embodiment, each expert is an interval type-2 fuzzy brain emotion learning network, and its weight is updated according to the following rules.

[0102] .

[0103] .

[0104] In the formula, is the weight of the hidden layer of the amygdala network in the expert model, for The update amount, is the momentum degradation rate, For the An expert model can generate an estimate of the posterior probability of the desired output, is the learning rate of the amygdala network in the expert model, For input features, is a function that takes the maximum value of two numbers, is the true label value, is the output of the amygdala network in the expert model, is the weight of the hidden layer of the orbitofrontal cortex network in the expert network, for The update amount, is the learning rate of the orbitofrontal cortex network in the expert network, For the The output of an expert model.

[0105] On the basis of the above embodiment, in an optional embodiment of the present invention, based on the error function of the gating network, the weight of the gating network is determined according to the following rules using the BP error algorithm.

[0106] .

[0107] .

[0108] In the formula, is the weight of the hidden layer of the amygdala network in the gating model, for The update amount, is the learning rate of the amygdala network in the gating model, As a reward signal, is the output of the amygdala network in the gating model, The gated network is the weight of the hidden layer of the orbitofrontal cortex network, for The update amount, is the learning rate of the orbitofrontal cortex network in the gating network, is the gate value.

[0109] In the traditional brain emotion learning network model, the calculation of sensory input is located in the sensory cortex, and the calculation results are sent directly to the orbitofrontal cortex and amygdala space. There is no learning process inside the sensory cortex.

[0110] On the basis of the above embodiment, in an optional embodiment of the present invention, after adding fuzzy logic, it is necessary to update the fuzzy parameters, ie, the mean and variance of the Gaussian function, which means that the sensory cortex needs to have a learning rule.

[0111] The embodiment of the present invention uses the following update formula to adjust the parameters in the Gaussian function:

[0112] ;

[0113] ;

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] ;

[0119] ;

[0120] ;

[0121] ;

[0122] In the formula, is the number of iterations, is the partial derivative, is the loss of the expert network, and are the upper and lower bounds of the initial mean of the interval type II Gaussian membership function in the amygdala network, and are the upper and lower bounds of the initial mean of the interval type II Gaussian membership function in the orbitofrontal cortex network, is the learning rate of the initial mean, is the variance of the interval type II Gaussian membership function in the amygdala network, is the variance of the interval type II Gaussian membership function in the orbitofrontal cortex network, is the learning rate of the variance, and are the upper and lower bounds of the linear weight of the amygdala, is the learning rate of the linear weights of the amygdala, and are the upper and lower bounds of the linear weight of the orbitofrontal cortex, is the learning rate for the linear weights of the orbitofrontal cortex.

[0123] The hybrid expert recognition model in the emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts of the present invention is as follows: Figure 7 As shown in the figure. Its operation process can be summarized as follows: First, the fused multimodal features are received from the deep feature extraction fusion model (network). Then, the features are simultaneously input to the expert network and the gated network for training. During the training process, the expert classifiers compete with each other for different input features, and the gated network determines the proportion of each expert's decision in the final decision based on the judgment error of each expert classifier. As can be seen from the figure, in this system, each expert classifier processes the data in parallel, and increasing the number of expert classifiers does not affect the computational complexity of the model. Therefore, the number of expert classifiers can be appropriately adjusted according to actual needs.

[0124] In the feature fusion stage at the front end of the system, the embodiment of the present invention constructs an expandable advanced deep feature extraction framework to extract complementary spatial feature information from various modalities, and fully utilizes the advantages of multimodal data to alleviate the gradient vanishing problem that may occur when processing each modal data separately; in the decision-making stage at the end of the system, the reward signal of the brain emotion learning model is improved, and Gaussian interval type-2 fuzzy logic is added. It is used as an expert model in the hybrid expert network, and the nonlinear correlation between multimodal data is used to design an adaptive gating network, which greatly improves the accuracy and generalization of recognition, and provides new ideas and methods for the study of emotion recognition with multimodal information fusion.

[0125] Embodiment 2: The present invention provides an emotion recognition device based on multimodal information fusion of fuzzy brain hybrid experts, which includes an initial data acquisition module, a feature fusion extraction module and a feature recognition module.

[0126] The initial data acquisition module is used to acquire initial data, wherein the initial data at least includes data of two different modes: image data and audio data.

[0127] The feature fusion extraction module is used to input the initial data into a pre-built deep feature extraction fusion model to extract features from data of different modalities and fuse them during the extraction process to obtain fusion features. The deep feature extraction fusion model is used to cross-extract features and fuse features.

[0128] A feature recognition module is used to input the fused features into a pre-built hybrid expert recognition model for classification to obtain the emotion recognition result of the initial data. The hybrid expert recognition model includes an expert network, a gating network and a fusion network. The expert network includes multiple expert models for extracting recognition features based on the fused features. The gating network includes a gating model for calculating the weights of each recognition feature based on the fused features. The fusion network is used to fuse each recognition feature into the emotion recognition result according to the weights.

[0129] The expert model and the gated model are both interval type-2 fuzzy brain emotion learning networks. The interval type-2 fuzzy brain emotion learning network includes an input layer, an amygdala network, an orbitofrontal cortex network and an output layer. The input layer is used to receive external input signals and transmit the signals to the amygdala network and the orbitofrontal cortex network. The amygdala network includes a perception layer and an excitation layer. The perception layer is used to receive signals from the input layer and convert them into fuzzy sets. The excitation layer is used to perform fuzzy reasoning based on the input signals from the perception layer and obtain excitation signals. The orbitofrontal cortex network is used to jointly regulate the output with the amygdala network. The output layer is used to output the final output.

[0130] Embodiment 3, the present invention provides an emotion recognition device based on multimodal information fusion of fuzzy brain and hybrid experts, which includes a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement an emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts as described in any paragraph of Embodiment 1.

[0131] Embodiment 4: The present invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a method for emotion recognition based on multimodal information fusion of fuzzy brain and hybrid experts as described in any paragraph of Embodiment 1.

[0132] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0134] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code. It should be noted that in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0135] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0136] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0137] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0138] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0139] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method of emotion recognition based on multimodal information fusion of fuzzy brain and hybrid experts, It is characterized in that Include: Acquire initial data; wherein the initial data at least includes data in two different modes: image data and audio data; Inputting the initial data into a pre-built deep feature extraction and fusion model to extract features from data of different modalities respectively and fuse them during the extraction process to obtain fused features; wherein the deep feature extraction and fusion model is used to cross-extract features and fuse features; The fused features are input into a pre-built hybrid expert recognition model for classification to obtain the emotion recognition result of the initial data; wherein the hybrid expert recognition model includes an expert network, a gating network and a fusion network; the expert network includes a plurality of expert models for extracting recognition features according to the fused features; the gating network includes a gating model for calculating the weights of each recognition feature according to the fused features; the fusion network is used to fuse each recognition feature into the emotion recognition result according to the weights; The expert model and the gated model are both interval type II fuzzy brain emotion learning networks; the interval type II fuzzy brain emotion learning network includes an input layer, an amygdala network, an orbitofrontal cortex network and an output layer; the input layer is used to receive external input signals and transmit the signals to the amygdala network and the orbitofrontal cortex network; the amygdala network includes a perception layer and an excitation layer; the perception layer is used to receive signals from the input layer and convert them into fuzzy sets; the excitation layer is used to perform fuzzy reasoning according to the input signals from the perception layer to obtain excitation signals; the orbitofrontal cortex network is used to jointly adjust the output with the amygdala network; the output layer is used to output the final output; The model of the interval type-2 fuzzy brain emotion learning network is: In the formula, is the output of the model of interval type-2 fuzzy brain emotion learning network, is the output of the amygdala network, is the output of the orbitofrontal cortex network; The model of the amygdala network is: In the formula, is the output of the amygdala network, and are the output values ​​of the left and right points of the amygdala after fuzzy reasoning, Calculate the number of blocks in the amygdala network, is the output of the left point of the perception layer, is the output of the right point of the perception layer, Amygdala The linear weight of the computational blocks, and are the upper and lower bounds of the linear weight of the amygdala, respectively; The model of the orbitofrontal cortex network is: In the formula, is the output of the orbitofrontal cortex network, and are the output values ​​of the left and right points of the orbitofrontal cortex after fuzzy reasoning, Calculate the number of blocks in the orbitofrontal cortex network, is the number of input features, For the Input features Located in the orbitofrontal cortex The membership of the computational blocks, and are the upper and lower bounds of the orbitofrontal cortex membership, The orbitofrontal cortex The linear weight of the computational blocks, and are the upper and lower bounds of the linear weight of the orbitofrontal cortex, respectively; The model of the perception layer is: In the formula, For the Input features Located in the amygdala The membership of the computational blocks, and are the upper and lower bounds of the amygdala membership, is an exponential function with a natural constant as its base. For the Input features, is the variance of the interval type II Gaussian membership function in the amygdala network, is the initial mean of the interval type II Gaussian membership function in the amygdala network, and They are the upper and lower bounds of the initial mean of the interval type-II Gaussian membership function in the amygdala network, respectively; The model of the incentive layer is: In the formula, is the upper bound of fuzzy reasoning, is the lower bound of fuzzy reasoning, is the number of input features.

2. According to claim 1, a method for emotion recognition based on multimodal information fusion of fuzzy brain and hybrid experts, It is characterized in that The emotion categories of the emotion recognition include anger, disgust, fear, happiness, sadness, surprise, and neutral; The deep feature extraction fusion model includes a feature extraction network, a horizontal connection network and a feature splicing network; The number of the feature extraction networks is the same as the number of modes in the initial data, and they are respectively used to extract features from data of different modes; the feature extraction network comprises a plurality of first feature extraction modules arranged in cascade, and they are used to perform multi-level extraction of features; The number of the first feature extraction modules in the horizontal connection network and the feature extraction network is the same; the plurality of the horizontal connection networks are respectively connected to the first feature extraction modules at the same level of different feature extraction networks; The horizontal connection network includes a horizontal connection module and a second feature extraction module; The horizontal connection module is used to horizontally connect the features extracted by the first feature extraction module at the same level; The second feature extraction module is used to extract features after horizontal connection to obtain horizontal connection features of each level; The feature concatenation network is used to concatenate the horizontally connected features of each level into the fused feature.

3. The emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts according to claim 2, It is characterized in that The number of the first feature extraction modules is four; The first feature extraction module includes a cascaded two-dimensional convolution layer, a LeakyRelu activation layer, and a maximum pooling layer; the convolution kernels of the two-dimensional convolution layers of the four cascaded first feature extraction modules are 16, 32, 64, and 128 respectively; The second feature extraction module includes a two-dimensional convolutional layer, an activation function layer, a maximum pooling layer, a two-dimensional convolutional layer, an activation function, a maximum pooling layer, a flattening layer, and a fully connected layer connected in sequence; The feature stitching network is configured to perform vertical stitching.

4. The emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts according to any one of claims 1 to 3, It is characterized in that The loss function of the expert network is: In the formula, is the loss of the expert network, is the number of expert models, Output of the gating network to the The gate value of the expert model, is the true label value, For the The output of the expert model, is the penalty coefficient of expert network loss, For the The relevant penalty function of the expert model, It is the integrated combined output; The loss function of the gating network is: In the formula, is the loss of the gating network, is the estimate of the posterior probability, is the gate value, For the An expert model can generate an estimate of the posterior probability of the desired output, Output of the gating network to the The gate value of the expert model, For the The output of the expert model, For the The relevant penalty function of the expert model, For the The gating value of the expert model.

5. The emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts according to any one of claims 1 to 3, It is characterized in that The reward signal of the gated model is: In the formula, For the The reward signal corresponding to the expert model, Output of the gating network to the The gate value of the expert model, is an exponential function with a natural constant as its base. is the true label value, For the The output of the expert model, For transposition, is the number of expert models; The weight update method of the expert model is: ； ； In the formula, is the weight of the hidden layer of the amygdala network in the expert model, for The update amount, is the momentum degradation rate, For the An expert model can generate an estimate of the posterior probability of the desired output, is the learning rate of the amygdala network in the expert model, For input features, is a function that takes the maximum value of two numbers, is the true label value, is the output of the amygdala network in the expert model, is the weight of the hidden layer of the orbitofrontal cortex network in the expert network, for The update amount, is the learning rate of the orbitofrontal cortex network in the expert network, For the The output of the expert model; The weight update method of the gating model is: ； ； In the formula, is the weight of the hidden layer of the amygdala network in the gating model, for The update amount, is the learning rate of the amygdala network in the gating model, As a reward signal, is the output of the amygdala network in the gating model, The gated network is the weight of the hidden layer of the orbitofrontal cortex network, for The update amount, is the learning rate of the orbitofrontal cortex network in the gating network, is the gate value; The fuzzy parameter update model is: ; ; ; ; ; ; ; ; ; ; In the formula, is the number of iterations, is the partial derivative, is the loss of the expert network, and are the upper and lower bounds of the initial mean of the interval type II Gaussian membership function in the amygdala network, and are the upper and lower bounds of the initial mean of the interval type II Gaussian membership function in the orbitofrontal cortex network, is the learning rate of the initial mean, is the variance of the interval type II Gaussian membership function in the amygdala network, is the variance of the interval type II Gaussian membership function in the orbitofrontal cortex network, is the learning rate of the variance, and are the upper and lower bounds of the linear weight of the amygdala, is the learning rate of the linear weights of the amygdala, and are the upper and lower bounds of the linear weight of the orbitofrontal cortex, is the learning rate for the linear weights of the orbitofrontal cortex.

6. An emotion recognition device based on multimodal information fusion of fuzzy brain hybrid experts, It is characterized in that Include: An initial data acquisition module, used to acquire initial data; wherein the initial data at least includes data of two different modalities: image data and audio data; A feature fusion extraction module is used to input the initial data into a pre-built deep feature extraction fusion model to extract features from data of different modalities respectively and fuse them during the extraction process to obtain fusion features; wherein the deep feature extraction fusion model is used to cross-extract features and fuse features; A feature recognition module, used for inputting the fused features into a pre-built hybrid expert recognition model for classification to obtain the emotion recognition result of the initial data; wherein the hybrid expert recognition model includes an expert network, a gating network and a fusion network; the expert network includes a plurality of expert models for extracting recognition features according to the fused features; the gating network includes a gating model for calculating the weights of each recognition feature according to the fused features; the fusion network is used for fusing each recognition feature into the emotion recognition result according to the weights; The expert model and the gated model are both interval type II fuzzy brain emotion learning networks; the interval type II fuzzy brain emotion learning network includes an input layer, an amygdala network, an orbitofrontal cortex network and an output layer; the input layer is used to receive external input signals and transmit the signals to the amygdala network and the orbitofrontal cortex network; the amygdala network includes a perception layer and an excitation layer; the perception layer is used to receive signals from the input layer and convert them into fuzzy sets; the excitation layer is used to perform fuzzy reasoning according to the input signals from the perception layer to obtain excitation signals; the orbitofrontal cortex network is used to jointly adjust the output with the amygdala network; the output layer is used to output the final output; The model of the interval type-2 fuzzy brain emotion learning network is: In the formula, is the output of the model of interval type-2 fuzzy brain emotion learning network, is the output of the amygdala network, is the output of the orbitofrontal cortex network; The model of the amygdala network is: In the formula, is the output of the amygdala network, and are the output values ​​of the left and right points of the amygdala after fuzzy reasoning, Calculate the number of blocks in the amygdala network, is the output of the left point of the perception layer, is the output of the right point of the perception layer, Amygdala The linear weight of the computational blocks, and are the upper and lower bounds of the linear weight of the amygdala, respectively; The model of the orbitofrontal cortex network is: In the formula, is the output of the orbitofrontal cortex network, and are the output values ​​of the left and right points of the orbitofrontal cortex after fuzzy reasoning, Calculate the number of blocks in the orbitofrontal cortex network, is the number of input features, For the Input features Located in the orbitofrontal cortex The membership of the computational blocks, and are the upper and lower bounds of the orbitofrontal cortex membership, The orbitofrontal cortex The linear weight of the computational blocks, and are the upper and lower bounds of the linear weight of the orbitofrontal cortex, respectively; The model of the perception layer is: In the formula, For the Input features Located in the amygdala The membership of the computational blocks, and are the upper and lower bounds of the amygdala membership, is an exponential function with a natural constant as its base. For the Input features, is the variance of the interval type II Gaussian membership function in the amygdala network, is the initial mean of the interval type II Gaussian membership function in the amygdala network, and They are the upper and lower bounds of the initial mean of the interval type-II Gaussian membership function in the amygdala network, respectively; The model of the incentive layer is: In the formula, is the upper bound of fuzzy reasoning, is the lower bound of fuzzy reasoning, is the number of input features.

7. An emotion recognition device based on multimodal information fusion of fuzzy brain hybrid experts, It is characterized in that It includes a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement an emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts as described in any one of claims 1 to 5.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the emotion recognition method based on multimodal information fusion of fuzzy brain and hybrid experts as described in any one of claims 1 to 5.

Images