A multimodal emotion recognition method and device

Abstract

The present invention provides a multimodal emotion recognition method and device, including: preprocessing the original multimodal data sample, obtaining the primary features of each single modality and inputting them into the single modality feature extraction network respectively, extracting the high-level features of each single modality, and after one-dimensionalization, respectively inputting them into each single modality recognition network, training the single modality model composed of each single modality feature extraction network and the single modality recognition network; obtaining the weight of each single modality feature extraction network, and extracting the activation degree of all neurons in each single modality recognition network; judging the type of each neuron according to its activation degree, and obtaining the unique connection constraints between different neurons; based on the obtained weight of each single modality feature extraction network and the extracted connection constraints, establishing a complete multimodal emotion recognition model and completing the training, which is used to recognize multimodal emotions. The present invention realizes cross-modal interaction and can effectively recognize multimodal emotions.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal recognition, and in particular to a multimodal emotion recognition method and device. Background Art

[0002] Multimodal recognition can effectively utilize information from multiple different modalities. This information complements each other and can effectively alleviate the problems faced by single-modal recognition, such as noisy data and non-universality. Therefore, it has made extensive progress and achieved good performance in various multimodal recognition tasks (such as brain-like multimodal emotion recognition).

[0003] For multimodal recognition, how to effectively utilize information from multiple modalities is a key issue. To solve this problem, multimodal fusion is a key means. With the development of deep learning, more and more methods are used to fuse multimodal information, such as data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion is to combine input data from different modalities at the data level to merge into more information-rich data, then extract features from the merged data and establish a classification model to identify and classify the extracted features; feature-level fusion refers to creating a combined feature for multiple input features extracted from different data modalities, and then using a classifier to identify the combined feature; in decision-level fusion, a separate classification system classifies the data from each modality into relevant classes, and then combines the scores of each classification system to obtain the final recognition result. However, most of these deep learning methods do not take into account the interaction between different modalities, which is called cross-modal interaction. Cross-modal interaction enables the information of each modality to influence and complement each other, which is crucial for effective multimodal fusion. Therefore, the recognition accuracy of traditional deep learning methods in multimodal recognition tasks still needs to be improved due to the neglect of cross-modal interaction. Summary of the invention

[0004] The present invention provides a multimodal emotion recognition method and device to solve the problems existing in the prior art. The technical solution provided by the present invention is as follows:

[0005] In one aspect, a multimodal emotion recognition method is provided, the method comprising:

[0006] S1. Preprocessing the original multimodal data sample to obtain primary features of each single modality included in the original multimodal data sample;

[0007] S2, inputting the primary features of each single modality into a single modality feature extraction network to extract high-level features of each single modality;

[0008] S3, after converting the high-level features of each unimodality into one dimension, inputting them into each unimodality recognition network respectively, and training the unimodality model composed of each unimodality feature extraction network and the unimodality recognition network;

[0009] S4, obtaining the weight of each unimodal feature extraction network from the trained unimodal model, and extracting the activation degree of all neurons in each unimodal recognition network;

[0010] S5. Determine the type of each neuron according to its activation degree, and obtain unique connection constraints between different neurons according to the type of each neuron;

[0011] S6. Based on the obtained weights of each unimodal feature extraction network and the extracted connection constraints, a complete multimodal emotion recognition model is established and trained to obtain a trained multimodal emotion recognition model for recognizing multimodal emotions.

[0012] Optionally, the S1 specifically includes:

[0013] The original multimodal data sample is a video clip with sound. Several frames of pictures are extracted at equal intervals. The facial contour of a person is extracted from each frame of the picture as the primary feature of the visual modality. express;

[0014] Extract the Mel frequency spectrum coefficient features of the audio of the video clip as the primary features of the auditory modality, and use express.

[0015] Optionally, each unimodal feature extraction network in S2 is a pulse neural network using a classic LIF model as a neuron model, and each network is composed of a convolutional layer and a pooling layer, which is used to further extract high-level features of the visual modality or the auditory modality from the primary features of the visual modality and the auditory modality, respectively. The high-level features of these two modalities are respectively used and express.

[0016] Optionally, each unimodal recognition network in S3 is a pulse neural network using a classic LIF model as a neuron model, and each network consists of two fully connected layers and one output layer, and is used to recognize the emotion expressed in each unimodal data;

[0017] The neuron model of each unimodal recognition network adds an adaptive activation threshold, and the threshold is set by an ordinary differential equation;

[0018] In the first fully connected layer of the visual modality recognition network, the membrane potential of the i-th neuron is updated as follows:

[0019]

[0020]

[0021]

[0022] In the first fully connected layer in the auditory modality recognition network, the membrane potential of the i-th neuron is updated as follows:

[0023]

[0024]

[0025]

[0026] Where C is capacitance, g is conductance, V i (t) represents the membrane potential of the i-th neuron at time t, S i (t) represents the activation mark of this neuron, V1 represents the resting potential, V2 represents the reset potential, V th represents the activation threshold, Represents the weight of the jth neuron in the high-level features of the input visual modality or auditory modality to the i-th neuron in the first layer of the fully connected network, and the dynamic threshold a i (t) is accumulated during the period from resetting to membrane potential discharge. As the discharge frequency increases, the threshold increases, and as the discharge frequency decreases, the threshold decreases. α, β, and γ are all hyperparameters.

[0027] Optionally, extracting the activation levels of all neurons in each unimodal recognition network in S4 specifically includes:

[0028] In each unimodal recognition network, for each sample, the membrane potential of all spiking neurons in the lth layer at all times is recorded and stored, using Indicates that the activation degree of neurons is represented by the membrane potential matrix, where i represents the i-th sample and l represents the l-th layer.

[0029] Optionally, judging the type of each neuron according to its activation degree in S5 specifically includes:

[0030] By averaging the activation of neurons over time scales and samples, we obtain the spike pattern, denoted as The formula is:

[0031]

[0032]

[0033] Among them, Meant Indicates the average calculation on the time scale, It means to average over the samples;

[0034] according to Get multisensory neurons, using the set M l express:

[0035]

[0036] Among them, Top means finding the neuron with the strongest pulse level according to the pulse pattern of the neuron, ρ is a hyperparameter, indicating that the neuron with the highest activation degree ranking is selected as the neuron with the strongest discharge level, and the multisensory neuron in the lth layer is represented by M l It means that it is the intersection of the most activated visual neurons and auditory neurons;

[0037] A single sensory neuron is the difference between all neurons in layer l and multisensory neurons:

[0038]

[0039]

[0040] in, represent all neurons in layer l, They represent single sensory neurons in layer l respectively.

[0041] Optionally, the step S5 of obtaining unique connection constraints between different neurons according to the type of each neuron specifically includes:

[0042] The connection weight W1 between each single-modal high-level feature and the first-layer fully connected layer neurons of each single-modal recognition network, and the connection weight W2 between the first and second layers of fully connected neurons are used to establish connection constraint masks for these two weights according to the following rules: 1 , mask 2 , mask 1 , mask 2 It is a matrix filled with 1 or 0, where 1 and 0 respectively indicate whether there is a connection between neurons. The connection weight of i→j is determined according to the following rules:

[0043] a) If neuron i represents a single sensory neuron, and neuron j represents a single sensory neuron of the same modality, then the connection weight from i to j exists,

[0044] b) If neuron i represents a single sensory neuron and neuron j represents a multisensory neuron, then the connection weight from i to j exists.

[0045] c) If neuron i represents a multisensory neuron and neuron j represents a single sensory neuron, then the connection weight from i to j exists.

[0046] In other cases, the connection weight of i→j does not exist.

[0047] Optionally, the complete multimodal emotion recognition model in S6 includes each unimodal feature extraction network and the multimodal recognition network;

[0048] The multimodal recognition network has a similar architecture to each single-modal recognition network, including two fully connected layers and one output layer, except that:

[0049] a) Each fully connected layer of neurons in a unimodal recognition network has a fixed number of neurons, but the number of neurons in a fully connected layer in a multimodal recognition network is The change is the sum of the number of the two extracted single sensory neurons and multisensory neurons, and the calculation formula is:

[0050]

[0051] b) The layers in the multimodal recognition network are sparsely connected, and the extracted connection constraint mask is used 1 , mask 2 , the connection weight W1 between each single-modal high-level feature and the first layer of fully connected neurons, and the connection weight W2 between the first layer and the second layer of fully connected neurons, mask 1 , mask 2 Calculate the Hadamard product with W1 and W2 respectively, and make regular restrictions on the connections between layers. The calculation formula is:

[0052]

[0053]

[0054] Among them, * represents the Hadamard product, Represents connection constraint mask 1 The element in the i-th row and j-th column of this matrix, W 1，i，j represents the element in the i-th row and j-th column of the matrix of connection weight W1, Represents connection constraint mask 2 The element in the i-th row and j-th column of this matrix, W 2，i，j Represents the element in the i-th row and j-th column of the matrix of connection weight W2.

[0055] Optionally, in S6, a complete multimodal emotion recognition model is trained, specifically including:

[0056] The original multimodal data samples are subjected to unimodal data preprocessing and primary feature extraction. The unimodal primary features are respectively input into the unimodal feature extraction network to obtain unimodal high-level features. The unimodal high-level features are converted into one dimension and then spliced. The spliced ​​features are input into the multimodal recognition network. The DFA algorithm is used to train the entire model. During the training process, the weight of each unimodal feature extraction network is fixed, that is, the trained weight in S3, and only the weights in the multimodal recognition network are trained.

[0057] In another aspect, a multimodal emotion recognition device is provided, the device comprising:

[0058] A first acquisition module, used for preprocessing the original multimodal data sample to obtain primary features of each single modality included in the original multimodal data sample;

[0059] A first extraction module, used for inputting the primary features of each single modality into a single modality feature extraction network to extract high-level features of each single modality;

[0060] A first training module is used to convert the high-level features of each single modality into one dimension and input them into each single modality recognition network respectively, so as to train the single modality model composed of each single modality feature extraction network and the single modality recognition network;

[0061] The second acquisition module is used to obtain the weight of each unimodal feature extraction network from the trained unimodal model and extract the activation degree of all neurons in each unimodal recognition network;

[0062] The third acquisition module is used to determine the type of each neuron according to its activation degree, and obtain unique connection constraints between different neurons according to the type of each neuron;

[0063] The second training module is used to establish and train a complete multimodal emotion recognition model based on the weights of each unimodal feature extraction network and the extracted connection constraints, so as to obtain a trained multimodal emotion recognition model for recognizing multimodal emotions.

[0064] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the above-mentioned multimodal emotion recognition method.

[0065] On the other hand, a computer-readable storage medium is provided, wherein instructions are stored in the storage medium, and the instructions are loaded and executed by a processor to implement the above-mentioned multimodal emotion recognition method.

[0066] Compared with the prior art, the above technical solution has at least the following beneficial effects:

[0067] The present invention is inspired by the theory of neuronal diversity in the human brain in the process of perceiving multimodal information, realizes cross-modal interaction, and has a higher multimodal emotion recognition accuracy than existing brain-like methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 A flow chart of a multimodal emotion recognition method provided by an embodiment of the present invention;

[0070] Figure 2 A flow chart of a method for identifying a unimodal model using the auditory modality as an example provided by an embodiment of the present invention;

[0071] Figure 3 A schematic diagram of determining the type of each neuron in a single-modal recognition network according to the activation degree of each neuron provided by an embodiment of the present invention;

[0072] Figure 4 A schematic diagram of designing unique connection constraints between neurons according to the type of each neuron and applying them to a multimodal recognition network provided by an embodiment of the present invention;

[0073] Figure 5 A flow chart of a method for identifying multimodal emotions to be identified provided by an embodiment of the present invention;

[0074] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] like Figure 1 As shown, an embodiment of the present invention provides a multimodal emotion recognition method, the method comprising:

[0077] S1. Preprocessing the original multimodal data sample to obtain primary features of each single modality included in the original multimodal data sample;

[0078] S2, inputting the primary features of each single modality into a single modality feature extraction network to extract high-level features of each single modality;

[0079] S3, after converting the high-level features of each unimodality into one dimension, inputting them into each unimodality recognition network respectively, and training the unimodality model composed of each unimodality feature extraction network and the unimodality recognition network;

[0080] S4, obtaining the weight of each unimodal feature extraction network from the trained unimodal model, and extracting the activation degree of all neurons in each unimodal recognition network;

[0081] S5. Determine the type of each neuron according to its activation degree, and obtain unique connection constraints between different neurons according to the type of each neuron;

[0082] S6. Based on the obtained weights of each unimodal feature extraction network and the extracted connection constraints, a complete multimodal emotion recognition model is established and trained to obtain a trained multimodal emotion recognition model for recognizing multimodal emotions.

[0083] The human brain is able to interact with information from multiple modalities such as vision and hearing to achieve accurate and efficient perceptual recognition. Neuroscience research shows that this ability of the brain originates from a variety of neurons, including single sensory neurons and multisensory neurons. The characteristic of these neurons is that they respond differently to different sensory stimuli. Specifically, single sensory neurons only respond to a single sensory stimulus, while multisensory neurons respond to stimulation from multiple senses. Because these neurons have different response characteristics, the human brain can effectively capture cross-modal interactions and thus has superior multimodal recognition capabilities. The embodiment of the present invention draws on the neuronal diversity behind the advantages of the human brain in multimodal recognition, and proposes a multimodal emotion recognition method inspired by neuronal diversity. The following is combined with Figure 2-Figure 5 , a multimodal emotion recognition method provided by an embodiment of the present invention is described in detail, including:

[0084] S1. Preprocessing the original multimodal data sample to obtain primary features of each single modality included in the original multimodal data sample;

[0085] Optionally, the S1 specifically includes:

[0086] The original multimodal data sample is a video clip with sound. Several frames of pictures (for example, 15 frames) are extracted at equal intervals. The facial contour of a person is extracted from each frame of the picture (the picture can be compressed to a size of 28*28) as the primary feature of the visual modality. express;

[0087] The dimension of each frame in the embodiment of the present invention is 784, so the final dimension of the initial feature of the visual modality of each sample is

[0088] Extract the Mel-scale Frequency Cepstral Coefficients (MFCC) features of the audio of the video clip as the primary features of the auditory modality and use express.

[0089] The average number of frames of all speech in the embodiment of the present invention is 280 frames, and the feature dimension of each frame is 12. Therefore, the initial feature of the auditory modality of each sample is The final dimension is

[0090] The specific methods for extracting primary features of the visual modality and the auditory modality are both existing technologies and will not be described in detail here.

[0091] The embodiment of the present invention takes the visual modality and the auditory modality as examples to illustrate the method of the embodiment of the present invention, but the multimodal data of the embodiment of the present invention may also include data in other modalities such as text. The embodiment of the present invention does not limit the multimodal data, and all are within the protection scope of the embodiment of the present invention.

[0092] S2, inputting the primary features of each single modality into a single modality feature extraction network to extract high-level features of each single modality;

[0093] Optionally, each unimodal feature extraction network (visual modality extraction network and auditory modality extraction network) in S2 is a spiking neural network (SNN) using a classic LIF model as a neuron model, and each network is composed of a convolutional layer and a pooling layer, and is used to further extract high-level features of the visual modality or the auditory modality from the primary features of the visual modality and the auditory modality, respectively. The high-level features of these two modalities are respectively used and express.

[0094] S3, after the high-level features of each unimodal are converted into one dimension (specifically, the conversion can be performed using a Flatten operation), the high-level features are respectively input into each unimodal recognition network (visual modality recognition network and auditory modality recognition network), and the unimodal model composed of each unimodal feature extraction network and the unimodal recognition network is trained;

[0095] Optionally, each unimodal recognition network in S3 is a pulse neural network using a classic LIF model as a neuron model, and each network consists of two fully connected layers and one output layer, and is used to recognize the emotion expressed in each unimodal data;

[0096] In each unimodal recognition network of the embodiment of the present invention, each fully connected layer has a fixed number of neurons (for example, 200), and the number of neurons in the output layer is consistent with the number of emotion labels in the training data set.

[0097] The embodiment of the present invention takes into account the plasticity of neurons in a unimodal recognition network, and the neuron model of each unimodal recognition network adds an adaptive activation threshold, and the threshold is set by an ordinary differential equation;

[0098] In the first fully connected layer of the visual modality recognition network, the membrane potential of the i-th neuron is updated as follows:

[0099]

[0100]

[0101]

[0102] In the first fully connected layer in the auditory modality recognition network, the membrane potential of the i-th neuron is updated as follows:

[0103]

[0104]

[0105]

[0106] Where C is capacitance, g is conductance, V i (t) represents the membrane potential of the i-th neuron at time t, S i (t) represents the activation mark of this neuron, V1 represents the resting potential, V2 represents the reset potential, V th represents the activation threshold, Represents the weight of the jth neuron in the high-level features of the input visual modality or auditory modality to the i-th neuron in the first layer of the fully connected network, and the dynamic threshold a i(t) is accumulated during the period from reset to membrane potential discharge. As the discharge frequency increases, the threshold value increases accordingly. As the discharge frequency decreases, the threshold value decreases accordingly. α, β, and γ are all hyperparameters (without practical significance. In the embodiment of the present invention, α=0.9, β=0.1, and γ=1 are set);

[0107] For the unimodal model, direct feedback alignment (DFA) is used for supervised training based on the samples in the training set and their emotion labels (the emotion category to which the sample belongs that has been manually labeled) to obtain a trained unimodal model.

[0108] The recognition method of the single modal model (taking the auditory modality as an example) of the embodiment of the present invention is as follows: Figure 2 shown.

[0109] S4, obtaining the weight of each unimodal feature extraction network from the trained unimodal model, and extracting the activation degree of all neurons in each unimodal recognition network;

[0110] For the trained unimodal model, save the weights of each unimodal feature extraction network used to extract high-level features of the visual modality and the auditory modality, respectively using En (v) ,En (a) express.

[0111] Optionally, extracting the activation levels of all neurons in each unimodal recognition network in S4 specifically includes:

[0112] In each unimodal recognition network, for each sample, the membrane potential of all spiking neurons in the lth layer at all times is recorded and stored, using Indicates that the activation degree of neurons is represented by the membrane potential matrix, where i represents the i-th sample and l represents the l-th layer.

[0113] S5. Determine the type of each neuron according to its activation degree, and obtain unique connection constraints between different neurons according to the type of each neuron;

[0114] Inspired by the fact that single sensory neurons and multisensory neurons in the human brain have different responses to different stimuli, the embodiment of the present invention determines the type of each neuron in the single modality recognition network according to its activation degree, specifically:

[0115] The response characteristics of neuronal diversity in the human brain are as follows:

[0116] In the human brain, multisensory information can be effectively processed and recognized, which depends on different neurons that show different responses to sensory input. In the brain areas responsible for integrating multimodal information in the human brain, various types of neurons, including multisensory neurons and unisensory neurons, are prevalent. Multisensory neurons are defined as neurons that respond to stimuli from more than one sense, while unisensory neurons are those that respond to only one sensory stimulus. Unisensory neurons can be further divided into visual and auditory unisensory neurons. For multisensory neurons, when the external stimuli it receives have a common source, its response significantly exceeds the response to any single sensory stimulus, while the response of unisensory neurons to multisensory stimuli is not significantly different from that to unisensory stimuli.

[0117] Alternatively, if Figure 3 As shown, the type of each neuron is determined according to its activation degree in S5, which specifically includes:

[0118] By averaging the activation of neurons over time scales and samples, we obtain the spike pattern, denoted as The formula is:

[0119]

[0120]

[0121] Among them, Mean t Indicates the average calculation on the time scale, It means to average over the samples;

[0122] according to Get multisensory neurons, using the set M l express:

[0123]

[0124] Among them, Top means finding the neuron with the strongest pulse level according to the pulse pattern of the neuron, ρ is a hyperparameter (ranging from 0 to 1), indicating that the neurons with the highest activation level are selected as the neurons with the highest discharge level (for example, if ρ is 0.5, the top 50% of neurons are selected), and the multisensory neurons in the lth layer are represented by M l It means that it is the intersection of the most activated visual neurons and auditory neurons;

[0125] A single sensory neuron is the difference between all neurons in layer l and multisensory neurons:

[0126]

[0127]

[0128] in, represent all neurons in layer l, They represent single sensory neurons in layer l respectively.

[0129] Optionally, in S5, unique connection constraints between different neurons are obtained according to the type of each neuron (inspired by the response characteristics of single sensory neurons and multisensory neurons in the human brain), specifically including:

[0130] The connection weight W1 between each single-modal high-level feature and the first-layer fully connected layer neurons of each single-modal recognition network, and the connection weight W2 between the first and second layers of fully connected neurons are used to establish connection constraint masks for these two weights according to the following rules: 1 , mask 2 , mask 1 , mask 2 It is a matrix filled with 1 or 0, where 1 and 0 respectively indicate whether there is a connection between neurons. The connection weight of i→j is determined according to the following rules:

[0131] a) If neuron i represents a single sensory neuron (visual modality or auditory modality), and neuron j represents a single sensory neuron of the same modality, then the connection weight from i to j exists,

[0132] b) If neuron i represents a single sensory neuron (visual modality or auditory modality) and neuron j represents a multisensory neuron, then the connection weight from i to j exists,

[0133] c) If neuron i represents a multisensory neuron and neuron j represents a single sensory neuron (visual modality or auditory modality), then the connection weight from i to j exists,

[0134] In other cases, the connection weight of i→j does not exist.

[0135] Through these constraints, cross-modal interaction is achieved. For example, auditory features Connected to the multisensory neurons M1 in the first fully connected layer, and then connected to the visual neurons in the second fully connected layer One-way transmission from hearing to vision is achieved. The same is true for one-way transmission from vision to hearing, thus achieving cross-modal interaction.

[0136] S6. Based on the obtained weights of each unimodal feature extraction network and the extracted connection constraints, a complete multimodal emotion recognition model is established and trained to obtain a trained multimodal emotion recognition model for recognizing multimodal emotions.

[0137] Optionally, the complete multimodal emotion recognition model in S6 includes each unimodal feature extraction network and the multimodal recognition network;

[0138] The multimodal recognition network has a similar architecture to each unimodal recognition network, including two fully connected layers and one output layer, except that:

[0139] a) Each fully connected layer of neurons in a unimodal recognition network has a fixed number of neurons (e.g. 200), but the number of neurons in a fully connected layer in a multimodal recognition network is The change is the sum of the number of the two extracted single sensory neurons and multisensory neurons, and the calculation formula is:

[0140]

[0141] b) The layers in the multimodal recognition network are sparsely connected, and the extracted connection constraint mask is used 1 , mask 2 , the connection weight W1 between each single-modal high-level feature and the first layer of fully connected neurons, and the connection weight W2 between the first layer and the second layer of fully connected neurons, mask 1 , mask 2 Calculate the Hadamard product with W1 and W2 respectively, and make regular restrictions on the connections between layers. The calculation formula is:

[0142]

[0143]

[0144] Among them, * represents the Hadamard product, Represents connection constraint mask 1 The element in the i-th row and j-th column of this matrix, W 1，i，j represents the element in the i-th row and j-th column of the matrix of connection weight W1, Represents connection constraint mask 2 The element in the i-th row and j-th column of this matrix, W 2，i，j Represents the element in the i-th row and j-th column of the matrix of connection weight W2.

[0145] The embodiment of the present invention designs unique connection constraints between neurons according to the type of each neuron and applies them to the multimodal recognition network, such as Figure 4 shown.

[0146] Optionally, in S6, a complete multimodal emotion recognition model is trained, specifically including:

[0147] The original multimodal data samples are subjected to unimodal data preprocessing and primary feature extraction. The unimodal primary features are respectively input into the unimodal feature extraction network to obtain unimodal high-level features. The unimodal high-level features are converted into one dimension and then spliced. The spliced ​​features are input into the multimodal recognition network. The DFA algorithm is used to train the entire model. During the training process, the weight of each unimodal feature extraction network is fixed, that is, the trained weight in S3, and only the weights in the multimodal recognition network are trained.

[0148] like Figure 5 As shown, when the method of the embodiment of the present invention is used to identify the multimodal emotion to be identified, it includes:

[0149] Preprocessing the original multimodal data to be identified to obtain primary features of each single modality (visual modality and auditory modality) included in the original multimodal data sample to be identified;

[0150] Inputting the primary features of each single modality into a single modality feature extraction network to extract high-level features of each single modality;

[0151] The high-level features of each single modality are one-dimensionalized and concatenated, and input into a multimodal recognition network to identify the multimodal emotions expressed by the original multimodal data to be identified.

[0152] The embodiment of the present invention further provides a multimodal emotion recognition device, the device comprising:

[0153] A first acquisition module, used for preprocessing the original multimodal data sample to obtain primary features of each single modality included in the original multimodal data sample;

[0154] A first extraction module, used for inputting the primary features of each single modality into a single modality feature extraction network to extract high-level features of each single modality;

[0155] A first training module is used to convert the high-level features of each single modality into one dimension and input them into each single modality recognition network respectively, so as to train the single modality model composed of each single modality feature extraction network and the single modality recognition network;

[0156] The second acquisition module is used to obtain the weight of each unimodal feature extraction network from the trained unimodal model and extract the activation degree of all neurons in each unimodal recognition network;

[0157] The third acquisition module is used to determine the type of each neuron according to its activation degree, and obtain unique connection constraints between different neurons according to the type of each neuron;

[0158] The second training module is used to establish and train a complete multimodal emotion recognition model based on the weights of each unimodal feature extraction network and the extracted connection constraints, so as to obtain a trained multimodal emotion recognition model for recognizing multimodal emotions.

[0159] A multimodal emotion recognition device provided in an embodiment of the present invention has a functional structure corresponding to a multimodal emotion recognition method provided in an embodiment of the present invention, which will not be described in detail here.

[0160] Figure 6 It is a structural diagram of an electronic device 600 provided in an embodiment of the present invention. The electronic device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 601 and one or more memories 602, wherein the memory 602 stores instructions, and the instructions are loaded and executed by the processor 601 to implement the steps of the above-mentioned multimodal emotion recognition method.

[0161] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to perform the multimodal emotion recognition method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0162] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0163] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal emotion recognition method, characterized in that: The method comprises: S1. Preprocessing the original multimodal data sample to obtain primary features of each single modality included in the original multimodal data sample; S2, inputting the primary features of each single modality into a single modality feature extraction network to extract high-level features of each single modality; S3, after converting the high-level features of each unimodality into one dimension, inputting them into each unimodality recognition network respectively, and training the unimodality model composed of each unimodality feature extraction network and the unimodality recognition network; S4. Obtain the weight of each unimodal feature extraction network from the trained unimodal model, and extract the activation degree of all neurons in each unimodal recognition network; S5. Determine the type of each neuron according to its activation degree, and obtain unique connection constraints between different neurons according to the type of each neuron; S6. Based on the obtained weights of each unimodal feature extraction network and the extracted connection constraints, a complete multimodal emotion recognition model is established and trained to obtain a trained multimodal emotion recognition model for recognizing multimodal emotions; The extraction of activation levels of all neurons in each single-modal recognition network in S4 specifically includes: In each unimodal recognition network, for each sample, the membrane potential of all spiking neurons in the lth layer at all times is recorded and stored, using Indicates that the activation degree of neurons is represented by the membrane potential matrix, where i represents the i-th sample and l represents the l-th layer; The step S5 of determining the type of each neuron according to its activation degree specifically includes: By averaging the activation of neurons over time scales and samples, we obtain the spike pattern, denoted as The formula is: Among them, Mean t Indicates that the average is calculated on the time scale, It means to average over the samples; according to Get multisensory neurons, using the set M l express: Among them, Top means finding the neuron with the strongest pulse level according to the pulse pattern of the neuron, ρ is a hyperparameter, indicating that the neuron with the highest activation degree ranking is selected as the neuron with the strongest discharge level, and the multisensory neuron in the lth layer is represented by M l It means that it is the intersection of the most activated visual neurons and auditory neurons; A single sensory neuron is the difference between all neurons in layer l and multisensory neurons: in, represent all neurons in layer l, They represent single sensory neurons in layer l respectively; In S5, unique connection constraints between different neurons are obtained according to the type of each neuron, specifically including: The connection weight W1 between each single-modal high-level feature and the first-layer fully connected layer neurons of each single-modal recognition network, and the connection weight W2 between the first and second layers of fully connected neurons are used to establish connection constraint masks for these two weights according to the following rules: 1 , mask 2 , mask 1 , mask 2 It is a matrix filled with 1 or 0, where 1 and 0 respectively indicate whether there is a connection between neurons. The connection weight of i→j is determined according to the following rules: a) If neuron i represents a single sensory neuron, and neuron j represents a single sensory neuron of the same modality, then the connection weight from i to j exists, b) If neuron i represents a single sensory neuron, and neuron , represents a multisensory neuron, then the connection weight of i→j exists, c) If neuron i represents a multisensory neuron, and neuron , represents a single sensory neuron, then the connection weight i→j exists, In other cases, the connection weight of i→j does not exist.

2. The method according to claim 1, characterized in that The S1 specifically includes: The original multimodal data sample is a video clip with sound. Several frames of pictures are extracted at equal intervals. The facial contour of a person is extracted from each frame of the picture as the primary feature of the visual modality. express; Extract the Mel frequency spectrum coefficient features of the audio of the video clip as the primary features of the auditory modality, and use express.

3. The method according to claim 2, characterized in that Each single-modal feature extraction network in S2 is a pulse neural network that uses the classic LIF model as a neuron model. Each network consists of a convolutional layer and a pooling layer, which is used to further extract high-level features of the visual modality or the auditory modality from the primary features of the visual modality and the auditory modality. The high-level features of these two modalities are respectively used and express.

4. The method according to claim 3, characterized in that: Each unimodal recognition network in S3 is a pulse neural network that uses the classic LIF model as a neuron model. Each network consists of two fully connected layers and one output layer, and is used to recognize the emotions expressed in each unimodal data. The neuron model of each unimodal recognition network adds an adaptive activation threshold, and the threshold is set by an ordinary differential equation; In the first fully connected layer of the visual modality recognition network, the membrane potential of the i-th neuron is updated as follows: In the first fully connected layer in the auditory modality recognition network, the membrane potential of the i-th neuron is updated as follows: Where C is capacitance, g is conductance, V i (t) represents the membrane potential of the i-th neuron at time t, S i (t) represents the activation mark of this neuron, V1 represents the resting potential, V2 represents the reset potential, V th represents the activation threshold, Represents the weight of the ith neuron in the high-level features of the input visual modality or auditory modality to the ith neuron in the first layer of the fully connected network, dynamic threshold a i (t) is accumulated during the period from resetting to membrane potential discharge. As the discharge frequency increases, the threshold increases, and as the discharge frequency decreases, the threshold decreases. α, β, and γ are all hyperparameters.

5. The method according to claim 1, characterized in that The complete multimodal emotion recognition model in S6 includes each unimodal feature extraction network and a multimodal recognition network; The multimodal recognition network has a similar architecture to each single-modal recognition network, including two fully connected layers and one output layer, except that: a) Each fully connected layer of neurons in a unimodal recognition network has a fixed number of neurons, but the number of neurons in a fully connected layer in a multimodal recognition network is The change is the sum of the number of the two extracted single sensory neurons and multisensory neurons, and the calculation formula is: b) The layers in the multimodal recognition network are sparsely connected, and the extracted connection constraint mask is used 1 , mask 2 , the connection weight W1 between each single-modal high-level feature and the first layer of fully connected neurons, and the connection weight W2 between the first layer and the second layer of fully connected neurons, mask 1 , mask 2 Calculate the Hadamard product with W1 and W2 respectively, and make regular restrictions on the connections between layers. The calculation formula is: Among them, * represents the Hadamard product, Represents connection constraint mask 1 The element in the i-th row and j-th column of this matrix, W 1，i，j represents the element in the i-th row and j-th column of the matrix of connection weight W1, Represents connection constraint mask 2 The element in the i-th row and j-th column of this matrix, W 2，i，j Represents the element in the i-th row and j-th column of the matrix of connection weight W2.

6. The method according to claim 1, characterized in that In S6, a complete multimodal emotion recognition model is trained, including: The original multimodal data samples are subjected to unimodal data preprocessing and primary feature extraction. The unimodal primary features are respectively input into the unimodal feature extraction network to obtain unimodal high-level features. The unimodal high-level features are converted into one dimension and then spliced. The spliced ​​features are input into the multimodal recognition network. The DFA algorithm is used to train the entire model. During the training process, the weight of each unimodal feature extraction network is fixed, that is, the trained weight in S3, and only the weights in the multimodal recognition network are trained.

7. A multimodal emotion recognition device, characterized in that: The device comprises: A first acquisition module, used for preprocessing the original multimodal data sample to obtain primary features of each single modality included in the original multimodal data sample; A first extraction module, used for inputting the primary features of each single modality into a single modality feature extraction network to extract high-level features of each single modality; A first training module is used to convert the high-level features of each single modality into one dimension and input them into each single modality recognition network respectively, so as to train the single modality model composed of each single modality feature extraction network and the single modality recognition network; The second acquisition module is used to obtain the weight of each unimodal feature extraction network from the trained unimodal model and extract the activation degree of all neurons in each unimodal recognition network; The third acquisition module is used to determine the type of each neuron according to its activation degree, and obtain unique connection constraints between different neurons according to the type of each neuron; The second training module is used to establish and train a complete multimodal emotion recognition model based on the weights of each unimodal feature extraction network and the extracted connection constraints, so as to obtain a trained multimodal emotion recognition model for recognizing multimodal emotions; The second acquisition module is specifically used for: In each unimodal recognition network, for each sample, the membrane potential of all spiking neurons in the lth layer at all times is recorded and stored, using Indicates that the activation degree of neurons is represented by the membrane potential matrix, where i represents the i-th sample and l represents the l-th layer; The third acquisition module is specifically used for: By averaging the activation of neurons over time scales and samples, we obtain the spike pattern, denoted as The formula is: Among them, Mean t Indicates the average calculation on the time scale, It means to average over the samples; according to Get multisensory neurons, using the set M l express: Among them, Top means finding the neuron with the strongest pulse level according to the pulse pattern of the neuron, ρ is a hyperparameter, indicating that the neuron with the highest activation degree ranking is selected as the neuron with the strongest discharge level, and the multisensory neuron in the lth layer is represented by M l It means that it is the intersection of the most activated visual neurons and auditory neurons; A single sensory neuron is the difference between all neurons in layer l and multisensory neurons: in, represent all neurons in layer l, They represent single sensory neurons in layer l respectively; The third acquisition module is further used for: The connection weight W1 between each single-modal high-level feature and the first-layer fully connected layer neurons of each single-modal recognition network, and the connection weight W2 between the first and second layers of fully connected neurons are used to establish connection constraint masks for these two weights according to the following rules: 1 , mask 2 , mask 1 , mask 2 It is a matrix filled with 1 or 0, where 1 and 0 respectively indicate whether there is a connection between neurons. The connection weight of i→j is determined according to the following rules: a) If neuron i represents a single sensory neuron, and neuron j represents a single sensory neuron of the same modality, then the connection weight from i to j exists, b) If neuron i represents a single sensory neuron, and neuron , represents a multisensory neuron, then the connection weight of i→j exists, c) If neuron i represents a multisensory neuron and neuron j represents a single sensory neuron, then the connection weight from i to j exists. In other cases, the connection weight of i→j does not exist.

Images