A method for emotion discrimination based on dual-path dual-dimensional convolutional neural network

Abstract

The present invention provides an emotion discrimination method based on a dual-path dual-dimensional convolutional neural network, belonging to the technical field of emotion judgment; it solves the problem of low emotion recognition accuracy caused by lack of processing of physiological signals in existing emotion recognition methods; it comprises the following steps: collecting and filtering physiological signals of subjects, and taking emotion expression as labels; training a multi-scale one-dimensional convolutional neural network and a multi-scale two-dimensional convolutional neural network; implicitly crossing the features obtained by passing the physiological signals through the multi-scale one-dimensional convolutional neural network and the multi-scale two-dimensional convolutional neural network; processing the crossed features through an attention mechanism, assigning different weights to the crossed features of each dimension, and then connecting the classification layer for classification, so as to construct a complete dual-path dual-dimensional convolutional neural network model; conducting confidence fusion on the emotion prediction results of the multi-scale one-dimensional convolutional neural network, the multi-scale two-dimensional convolutional neural network and the dual-path dual-dimensional convolutional neural network; the present invention is applied to emotion discrimination.

Description

Technical Field

[0001] The present invention provides an emotion discrimination method based on a dual-path dual-dimensional convolutional neural network, belonging to the technical field of emotion computing based on physiological signals. Background Art

[0002] Emotions are a series of reactions to a person's psychological and physiological state, and they are closely related to a person's health. With the development of artificial intelligence technology, emotion recognition technology has also been greatly improved. Initially, researchers mostly used non-physiological signals such as facial expressions, voice intonation, and body features to extract recognition features, but since these features can be considered disguised, the recognition effect is not ideal. As research continues to deepen, people have found that emotions are closely related to people's physiological and psychological activities. In addition, thanks to the rapid development of wearable devices, researchers are currently more inclined to use physiological signals for emotion recognition research.

[0003] Emotion recognition has a wide range of applications, including detecting and displaying user emotions in real time in the metaverse and virtual reality technology, detecting users' daily emotions, detecting and improving emotions in medical treatment to enhance treatment effects, detecting emergencies and improving soldiers' emotions in the military field, and assisting in case analysis in the police field.

[0004] Chinese patent application 202110133224.X proposes a driver emotion recognition system based on a two-layer neural network model, which integrates electrocardiogram signals, facial electromyography signals and brain wave signals to obtain more comprehensive driver information. However, although the above technical solution integrates some physiological signal characteristics for emotion recognition, its application field is relatively limited, and there is no confidence calculation for the judged emotions, and the judgment result has a large error. Chinese patent application 202110371851.7 proposes an emotion recognition method and system based on a convolutional neural network, which analyzes and processes EEG signals to judge people's emotional state, but the accuracy of emotion recognition using only brain wave signals remains to be discussed. Summary of the invention

[0005] In order to solve the problem of low emotion recognition accuracy caused by lack of processing of physiological signals in existing emotion recognition methods, the present invention proposes an emotion discrimination method based on a dual-path dual-dimensional convolutional neural network.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: an emotion discrimination method based on a dual-path dual-dimensional convolutional neural network, comprising the following steps:

[0007] Step 1: Use peripheral physiological signal detection equipment to collect the subject's heart rate signal, blood oxygen signal, blood pressure signal, and body temperature signal, and filter them through wavelet transform to obtain the signals of the four channels to be judged. The emotional expression is obtained as a label by the subject filling out a questionnaire;

[0008] Step 2: Divide the collected physiological signals into a training set and a test set, and train a multi-scale one-dimensional convolutional neural network and a multi-scale two-dimensional convolutional neural network respectively;

[0009] Step 3: implicitly cross the features of the physiological signal obtained by wavelet transform filtering through a multi-scale one-dimensional convolutional neural network and a multi-scale two-dimensional convolutional neural network to obtain a new feature vector;

[0010] Step 4: The crossed features are processed by the attention mechanism, different weights are assigned to the crossed features in each dimension, and then connected to the classification layer for classification to build a complete two-way two-dimensional convolutional neural network model;

[0011] Step 5: Confidence fusion is performed on the emotion prediction results of the multi-scale one-dimensional convolutional neural network and the multi-scale two-dimensional convolutional neural network in step 2 and the emotion prediction results of the two-way two-dimensional convolutional neural network in step 4, and new emotion confidence results are obtained through different weights;

[0012] Step 6: Use the collected heart rate, blood pressure, blood oxygen and body temperature signals to train a two-way two-dimensional convolutional neural network model and a confidence score calculation method, and then use the trained model to predict unknown emotional states to achieve emotion discrimination and confidence score calculation.

[0013] The specific process of step 1 is as follows:

[0014] Step 1.1: The subjects were equipped with peripheral physiological signal detection equipment to monitor their heart rate, blood oxygen, blood pressure, and body temperature, and emotional stimulation materials were played to the subjects to make their emotions fluctuate significantly. After the emotional stimulation materials were played, the subjects were asked to fill in a questionnaire, which included valence scores and arousal scores.

[0015] Step 1.2: Filter the noise of the original four physiological signals by wavelet transform to obtain the peripheral physiological signal to be used. The heart rate is [x 11 ,x 12 ,…x 1n ], blood oxygen is [x 21 ,x 22 ,…x 2n ], blood pressure is [x 31 ,x 32 ,…x 3n ], body temperature is [x41 ,x 42 ,…x 4n ];

[0016] Then, the subjects’ scores were divided into four emotions according to the valence scores and arousal scores, namely high valence and high arousal (HVHA), high valence and low arousal (HVLA), low valence and high arousal (LVHA), and low valence and low arousal (LVLA);

[0017] Step 1.3: The physiological signals of the emotions stimulated by each emotion-stimulating material of the subject and the corresponding emotion label of the subject are combined into a sample in the form of:

[0018]

[0019]

[0020] In the above formula: x represents the physiological signal of the emotion stimulated by each emotion-stimulating material of the subject, and y represents the corresponding emotion label of the subject.

[0021] The steps of wavelet transformation in step 1.2 are as follows:

[0022] Assume f(n) is the discrete sampling data of the original signal f(t). For a one-dimensional discrete signal f(n), n = 0, 1…N-1, its wavelet transform is:

[0023]

[0024] In the above formula: Wf(j, k) is the wavelet coefficient, j is the scale and expansion of the wavelet function in the frequency domain, and k is the translation of the function in the time domain;

[0025] Then use the recursive implementation method with the dual-scaling equation:

[0026] Sf(j+1,k)=Sf(j,k)*h(j,k);

[0027] Wf(j+1,k)=Sf(j,k)*g(j,k);

[0028] In the above formula: g and h correspond to the wavelet functions and the low-pass and high-pass filters of the scaling function φ(t), sf(0, k) is the original signal f(n), Sf(j, k) is the scaling function, Wf(j, k) is the wavelet function, then the reconstruction formula of the wavelet transform is:

[0029] Sf(j-1,k)=Sf(j,k)*h(j,k)+Wf(j,k)*g(j,k).

[0030] The multi-scale one-dimensional convolutional neural network includes 1 ordinary convolution layer, 3 multi-scale convolution layers, and 2 multi-scale downsampling layers. The multi-scale convolution layer includes 4 branches, wherein branch 1 includes 1 convolution layer, and the scale of the convolution kernel is 1×1;

[0031] Branch 2 contains two convolutional layers, 1×1 and 3×1;

[0032] Branch 3 contains three convolutional layers, namely 1×1, 3×1 and 5×1;

[0033] Branch 4 contains the downsampling operation, and the final one directly connects the features of each branch together.

[0034] The two-dimensional convolutional neural network adopts the RSNet neural network, wherein the convolution layer of the two-dimensional convolutional neural network includes 3 convolutional layers, the convolution kernel size of the first convolution layer is 16×16 and the step size is 8, the convolution kernel size of the second convolution layer is 5×5 and the step size is 2, the convolution kernel size of the third convolution layer is 3×3 and the step size is 1, each convolution layer is followed by an activation function layer and a pooling layer, and a batch normalization layer is added at the same time, the pooling kernel size is 2×2, and the step size is 2.

[0035] In step 4, the weight of the feature vector after the feature crossover is allocated by the attention mechanism, and the weight calculation formula is as follows:

[0036]

[0037] In the above formula: c i represents the weight, q i represents the i-th element in the query element, α ij represents the normalized similarity, α ij =softmax(e ij ), e ij represents the similarity between elements i and j, and softmax(·) represents the normalized exponential function.

[0038] The confidence fusion calculation in step 5 adopts the improved DS combination rule, and the absolute assignment function after fusion is defined as:

[0039]

[0040] In the above formula: variable ω i is the weight coefficient of each network, with a value range of [0,1], and A i represents the i-th recognition frame, i represents the i-th signal source, A represents the recognition frame, m i represents the credibility distribution of the i-th signal source, and n represents the total number of signal sources.

[0041] The beneficial effects of the present invention compared with the prior art are as follows:

[0042] First, the present invention uses easily collected peripheral physiological signals and is more practical when used with wearable mobile devices;

[0043] Second, the present invention uses a multi-scale one-dimensional convolutional neural network and a two-dimensional convolutional neural network to extract one-dimensional features and two-dimensional features for splicing, thereby improving the effect of emotion discrimination;

[0044] Third, the present invention uses a multi-network emotion discrimination confidence score calculation method to eliminate errors and improve the credibility of the confidence score. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described below in conjunction with the accompanying drawings:

[0046] Figure 1 is a flow chart of the method of the present invention;

[0047] Figure 2 A schematic diagram of the structure of a multi-scale convolutional layer of the multi-scale one-dimensional convolutional neural network proposed in the present invention;

[0048] Figure 3 This is a working principle diagram of the attention mechanism of the present invention. DETAILED DESCRIPTION

[0049] like Figure 1 The figure shows a flow chart of the emotion discrimination method based on a dual-path dual-dimensional convolutional neural network proposed by the present invention, which respectively uses a multi-scale one-dimensional convolutional neural network and a multi-scale two-dimensional convolutional neural network to extract and analyze physiological signals, and recalculates based on the confidence results obtained by the two networks to eliminate errors, improve the discrimination effect of emotion recognition and increase the credibility of the confidence score. The specific steps are as follows:

[0050] Step 1: Use peripheral physiological signal detection equipment to collect the subject's heart rate signal, blood oxygen signal, blood pressure signal, and body temperature signal, and filter them through wavelet transform to obtain the signals of the four channels to be judged. The emotional expression is obtained as a label by the subject filling out a questionnaire;

[0051] Step 2: Divide the collected physiological signals into a training set and a test set, and train a multi-scale one-dimensional convolutional neural network and a multi-scale two-dimensional convolutional neural network respectively to achieve the effect of emotion discrimination and confidence calculation;

[0052] Step 3: implicitly cross the features of the original input physiological signal obtained by the multi-scale one-dimensional convolutional neural network and the multi-scale two-dimensional convolutional neural network to obtain a new feature vector;

[0053] Step 4: The crossed features are processed by the attention mechanism, different weights are assigned to the crossed features in each dimension, and then connected to the classification layer for classification to build a complete two-way two-dimensional convolutional neural network model;

[0054] Step 5: Confidence fusion is performed on the emotion prediction results of the multi-scale one-dimensional convolutional neural network and the multi-scale two-dimensional convolutional neural network in step 2 and the emotion prediction results of the two-way two-dimensional convolutional neural network in step 4, and new emotion confidence results are obtained through different weights;

[0055] Step 6: Use the collected heart rate, blood pressure, blood oxygen and body temperature signals to train a two-way two-dimensional convolutional neural network model and a confidence score calculation method, and then use the trained model to predict unknown emotional states to achieve emotion discrimination and confidence score calculation.

[0056] Wherein step 1 specifically comprises the following steps:

[0057] Step 1.1: Have the subjects wear peripheral physiological signal detection equipment to monitor their heart rate, blood oxygen, blood pressure, and body temperature, and play emotional arousal materials to make their emotions fluctuate significantly. After the emotional arousal materials are played, the subjects are asked to fill out a questionnaire, which mainly includes valence scores and arousal scores, and the scoring range is [1,9].

[0058] Step 1.2: Filter the noise of the original four physiological signals by wavelet transform to obtain the peripheral physiological signal to be used. The heart rate is [x 11 ,x 12 ,…x 1n ], blood oxygen is [x 21 ,x 22 ,…x 2n ], blood pressure is [x 31 ,x 32 ,…x 3n ], body temperature is [x 41 ,x 42 ,…x 4n Then, according to the valence score and arousal score, the subjects' scores were divided into four types of emotions, namely high valence and high arousal (HVHA), high valence and low arousal (HVLA), low valence and high arousal (LVHA), and low valence and low arousal (LVLA). Valence indicates the positivity of emotions, and arousal indicates the intensity of emotions. The confidence is calculated as the probability of obtaining the four types of emotions, and the confidence interval is [0,1].

[0059] Step 1.3: The physiological signals of the emotions stimulated by each emotion-stimulating material of the subject and the corresponding emotion label of the subject are combined into a sample in the form of:

[0060]

[0061]

[0062] In the above formula: x represents the physiological signal of the emotion stimulated by each emotion-stimulating material of the subject, and y represents the corresponding emotion label of the subject.

[0063] The entire data set consists of<X,Y> .

[0064] The original heart rate signal, blood oxygen signal, blood pressure signal, and body temperature signal in step 1.1 are processed by wavelet transform to remove noise, making the original physiological signal smoother and more accurate. According to the algorithm, if f(n) is the discrete sampling data of the original signal f(t), then for the one-dimensional discrete signal f(n), n = 0, 1...N-1, its wavelet transform is:

[0065]

[0066] In the above formula: Wf(j, k) is the wavelet coefficient, j is the scale and expansion of the wavelet function in the frequency domain, and k is the translation of the function in the time domain;

[0067] Then use the recursive implementation method with the help of the dual-scale equation:

[0068] Sf(j+1,k)=Sf(j,k)*h(j,k);

[0069] Wf(j+1,k)=Sf(j,k)*g(j,k);

[0070] In the above formula: g and h correspond to the wavelet functions and the low-pass and high-pass filters of the scaling function φ(t), sf(0, k) is the original signal f(n), Sf(j, k) is the scaling function, Wf(j, k) is the wavelet function, then the reconstruction formula of the wavelet transform is:

[0071] Sf(j-1,k)=Sf(j,k)*h(j,k)+Wf(j,k)*g(j,k).

[0072] Wherein step 2 specifically comprises the following steps:

[0073] Step 2.1: First, all data sets<X,Y> Divided into training set and test set, respectively<X1,Y1> and<X2,Y2> .

[0074] Step 2.2, then build a multi-scale one-dimensional convolutional neural network and a multi-scale two-dimensional convolutional neural network respectively, where the multi-scale one-dimensional convolutional neural network structure includes 1 ordinary convolution layer, 3 multi-scale convolution layers, and 2 multi-scale downsampling layers, where the multi-scale convolution layers and multi-scale downsampling layers appear alternately. The multi-scale two-dimensional convolutional neural network uses the RSNet neural network.

[0075] like Figure 2 The figure shows a schematic diagram of the structure of the multi-scale convolution layer of the multi-scale one-dimensional convolutional neural network proposed in the present invention. The multi-scale convolution layer of the multi-scale one-dimensional convolutional neural network contains 4 branches, among which branch 1 contains 1 convolution layer, the scale of the convolution kernel is 1×1, and its main function is to control the number of output channels. Branch 2 contains 2 convolution layers, 1×1 and 3×1 respectively. Branch 3 contains 3 convolution layers, 1×1, 3×1 and 5×1 respectively. Branch 4 contains a downsampling operation to remove redundant information and better retain the specific information of the features. The last operation directly connects the features of each branch together.

[0076] The convolution layer of the two-dimensional convolutional neural network contains 3 convolution layers. The convolution kernel size of the first convolution layer is 16×16 and the step size is 8. The convolution kernel size of the second convolution layer is 5×5 and the step size is 2. The convolution kernel size of the third convolution layer is 3×3 and the step size is 1. Each convolution layer is followed by an activation function layer and a pooling layer. At the same time, a batch normalization layer is added to prevent overfitting in the training process. The maximum pooling layer reduces the data dimension and retains the main feature information. The pooling kernel size is 2×2 and the step size is 2.

[0077] The loss function of the multi-scale one-dimensional convolutional neural network and the multi-scale two-dimensional convolutional neural network of the present invention uses the cross entropy loss function (Cross Entropy Loss Function) most commonly used in convolutional neural networks, and the specific expression is:

[0078]

[0079] In the above formula, p(x) is the target value distribution and q(x) is the estimated value distribution.

[0080] Step 2.3, add the training set to the multi-scale one-dimensional convolutional neural network and the multi-scale two-dimensional convolutional neural network for training. First, forward propagation calculates the loss function of the data in each iteration; then back propagation uses the gradient descent method and the chain rule to update the parameters; repeat forward propagation and back propagation until the loss function converges and no longer decreases, end the training process, and use the test set to verify the effectiveness of the model.

[0081] The method of implicitly crossing the features obtained by the multi-scale one-dimensional convolutional neural network and the multi-scale two-dimensional convolutional neural network of the original input physiological signal in step 3 is the DeepFM algorithm. First, the core of the DeepFM algorithm is to add a second-order factor decomposition term on the basis of the linear model to represent the intersection between features:

[0082]

[0083] In the above formula: y represents output, x i and x j Represent two different feature values, w0 represents the initial weight, w i Represents each feature x i The corresponding weight.

[0084] The core of DeepFM consists of two parts: FM and DNN. FM is used to characterize low-order cross features, and DNN is used to characterize high-order cross features. In DeepFM, FM and DNN share the Embedding layer, and finally the low-order and high-order parts are jointly trained to obtain the model output.

[0085] The attention mechanism method in step 4 is to focus on the weight distribution of the feature vector after feature crossover. The general expression of the attention mechanism can be written as:

[0086] O = softmax(QK T )V;

[0087] In a broad sense, it can be understood as a layer of perceptron consisting of the query matrix Q, the corresponding key item K, and the value item V to be weighted averaged. Figure 3 The following is a diagram showing the working principle of the attention mechanism. The calculation steps of the attention mechanism are as follows:

[0088] First, calculate the similarity between Q and K:

[0089] e ij =D(q i , p j );

[0090] In the above formula: D(·) represents the dot product function; q i 、p j Represent the query element and the i-th and j-th elements in the element address set respectively; e ij Represents the similarity between elements i and j.

[0091] Then normalize the similarity:

[0092] α ij =softmax(e ij );

[0093] Where: α ij represents the normalized similarity; softmax(·) represents the normalized exponential function.

[0094] Finally, perform weighted summation:

[0095]

[0096] In the formula, c i Represents weight.

[0097] The confidence fusion calculation method in step 5 uses the improved DS combination rule, and the absolute assignment function after fusion is defined as:

[0098]

[0099] In the above formula, the variable ω i is the weight coefficient of each network, with a value range of [0,1], and If the importance and reliability of a certain information source are extremely high, then the corresponding i is 1; on the contrary, if the importance and reliability of a network structure are extremely low, the corresponding ω i is 0.

[0100] Regarding the specific structure of the present invention, it should be noted that the connection relationship between the various component modules adopted in the present invention is definite and feasible. Except for special instructions in the embodiments, the specific connection relationship can bring about corresponding technical effects, and solve the technical problems raised by the present invention without relying on the execution of corresponding software programs. The components, modules, models of specific components, the connection methods between each other, and the conventional use methods and expected technical effects brought about by the above-mentioned technical features, except for specific instructions, all belong to the disclosed contents in patents, journal articles, technical manuals, technical dictionaries, and textbooks that can be obtained by technical personnel in this field before the application date, or belong to the existing technologies such as conventional technologies and common knowledge in this field, and there is no need to elaborate, so that the technical solution provided in this case is clear, complete, and feasible, and the corresponding physical products can be reproduced or obtained according to the technical means.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for emotion discrimination based on a dual-path dual-dimensional convolutional neural network, characterized in that: The steps include: Step 1: Use peripheral physiological signal detection equipment to collect the subject's heart rate signal, blood oxygen signal, blood pressure signal, and body temperature signal, and filter them through wavelet transform to obtain the signals of the four channels to be judged. The emotional expression is obtained as a label by the subject filling out a questionnaire; Step 2: Divide the collected physiological signals into a training set and a test set, and train a multi-scale one-dimensional convolutional neural network and a multi-scale two-dimensional convolutional neural network respectively; Step 3: implicitly cross the features of the physiological signal obtained by wavelet transform filtering through a multi-scale one-dimensional convolutional neural network and a multi-scale two-dimensional convolutional neural network to obtain a new feature vector; Step 4: The crossed features are processed by the attention mechanism, different weights are assigned to the crossed features in each dimension, and then connected to the classification layer for classification to build a complete two-way two-dimensional convolutional neural network model; Step 5: Confidence fusion is performed on the emotion prediction results of the multi-scale one-dimensional convolutional neural network and the multi-scale two-dimensional convolutional neural network in step 2 and the emotion prediction results of the two-way two-dimensional convolutional neural network in step 4, and new emotion confidence results are obtained through different weights; Step 6: Use the collected heart rate, blood pressure, blood oxygen and body temperature signals to train a two-way two-dimensional convolutional neural network model and a confidence score calculation method, and then use the trained model to predict unknown emotional states to achieve emotion discrimination and confidence score calculation.

2. The emotion discrimination method based on a dual-path dual-dimensional convolutional neural network according to claim 1, characterized in that: The specific process of step 1 is as follows: Step 1.1: The subjects were equipped with peripheral physiological signal detection equipment to monitor their heart rate, blood oxygen, blood pressure, and body temperature, and emotional stimulation materials were played to the subjects to make their emotions fluctuate significantly. After the emotional stimulation materials were played, the subjects were asked to fill in a questionnaire, which included valence scores and arousal scores. Step 1.2: Filter the noise of the original four physiological signals by wavelet transform to obtain the peripheral physiological signal to be used. The heart rate is [x 11 ,x 12 ,…x 1n ], blood oxygen is [x 21 ,x 22 ,…x 2n ], blood pressure is [x 31 ,x 32 ,…x 3n ], body temperature is [x 41 ,x 42 ,…x 4n ]; Then, the subjects’ scores were divided into four emotions according to the valence scores and arousal scores, namely high valence and high arousal (HVHA), high valence and low arousal (HVLA), low valence and high arousal (LVHA), and low valence and low arousal (LVLA); Step 1.3: The physiological signals of the emotions stimulated by each emotion-stimulating material of the subject and the corresponding emotion label of the subject are combined into a sample in the form of: In the above formula: x represents the physiological signal of the emotion stimulated by each emotion-stimulating material of the subject, and y represents the corresponding emotion label of the subject.

3. The emotion discrimination method based on a dual-path dual-dimensional convolutional neural network according to claim 2, characterized in that: The steps of wavelet transformation in step 1.2 are as follows: Assume f(n) is the discrete sampling data of the original signal f(t). For a one-dimensional discrete signal f(n), n = 0, 1…N-1, its wavelet transform is: In the above formula: Wf(j, k) is the wavelet coefficient, j is the scale and expansion of the wavelet function in the frequency domain, and k is the translation of the function in the time domain; Then use the recursive implementation method with the dual-scaling equation: Sf(j+1,k)=Sf(j,k)*h(j,k); Wf(j+1,k)=Sf(j,k)*g(j,k); In the above formula: g and h correspond to the wavelet functions and the low-pass and high-pass filters of the scaling function φ(t), sf(0, k) is the original signal f(n), Sf(j, k) is the scaling function, Wf(j, k) is the wavelet function, then the reconstruction formula of the wavelet transform is: Sf(j-1,k)=Sf(j,k)*h(j,k)+Wf(j,k)*g(j,k).

4. The emotion discrimination method based on a dual-path dual-dimensional convolutional neural network according to claim 1, characterized in that: The multi-scale one-dimensional convolutional neural network includes 1 ordinary convolution layer, 3 multi-scale convolution layers, and 2 multi-scale downsampling layers. The multi-scale convolution layer includes 4 branches, wherein branch 1 includes 1 convolution layer, and the scale of the convolution kernel is 1×1; Branch 2 contains two convolutional layers, 1×1 and 3×1; Branch 3 contains three convolutional layers, namely 1×1, 3×1 and 5×1; Branch 4 contains the downsampling operation, and the final one directly connects the features of each branch together.

5. The emotion discrimination method based on a dual-path dual-dimensional convolutional neural network according to claim 1, characterized in that: The two-dimensional convolutional neural network adopts the RSNet neural network, wherein the convolution layer of the two-dimensional convolutional neural network includes 3 convolutional layers, the convolution kernel size of the first convolution layer is 16×16 and the step size is 8, the convolution kernel size of the second convolution layer is 5×5 and the step size is 2, the convolution kernel size of the third convolution layer is 3×3 and the step size is 1, each convolution layer is followed by an activation function layer and a pooling layer, and a batch normalization layer is added at the same time, the pooling kernel size is 2×2, and the step size is 2.

6. The emotion discrimination method based on a dual-path dual-dimensional convolutional neural network according to claim 1, characterized in that: The weight calculation formula for the feature vector after feature crossover by the attention mechanism in step 4 is as follows: In the above formula: c i represents the weight, q i represents the i-th element in the query element, α ij represents the normalized similarity, α ij =softmax(e ij ), e ij represents the similarity between elements i and j, and softmax(·) represents the normalized exponential function.

7. The emotion discrimination method based on a dual-path dual-dimensional convolutional neural network according to claim 1, characterized in that: The confidence fusion calculation in step 5 adopts the improved DS combination rule, and the absolute assignment function after fusion is defined as: In the above formula: variable ω i is the weight coefficient of each network, with a value range of [0,1], and A i represents the i-th recognition frame, i represents the i-th signal source, A represents the recognition frame, m i represents the credibility distribution of the i-th signal source, and n represents the total number of signal sources.

Images