Multi-modal physiological signal emotion classification and recognition method and device

Abstract

The invention discloses a multi-modal physiological signal emotion classification and recognition method and device. The method comprises the following steps: acquiring a physiological signal set and a modal type; constructing a first model based on the modal type; training the first model by using the physiological signal set to obtain a first optimization model; constructing a third model based on the first optimization model and a preset second model; training the third model by using the physiological signal set and the tag sequence to obtain a third optimization model; processing the collected physiological signal to be analyzed by using the third optimization model to obtain an emotion probability vector; and determining a classification recognition result based on the emotion probability vector. The method can improve the emotion classification and recognition capability.

Description

Technical Field

[0001] The present invention belongs to the field of emotion classification and recognition, and specifically relates to a multi-modal physiological signal emotion classification and recognition method and device. Background Art

[0002] Emotions are not only various internal feelings and mental states of human beings, but also psychological and physiological activities generated by various external stimuli. In many fields, it is very important to correctly identify emotions. At the same time, a single physiological signal usually cannot contain all the information required for emotion classification. Therefore, more and more researchers are committed to studying emotion recognition and classification algorithms that integrate multimodal physiological signals.

[0003] Multimodal physiological signal fusion can utilize the complementary representations between different physiological signals to facilitate more accurate emotion classification and recognition, and has become a major trend in the field of emotion recognition and classification. However, existing methods use direct splicing to fuse features, and there are large differences between the patterns and features of different physiological signals. Therefore, there is a disadvantage of insufficient use of information in different modal physiological signals, which reduces the ability to classify and recognize emotions. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for multimodal physiological signal emotion classification and recognition, which is conducive to improving emotion classification and recognition capabilities.

[0005] In order to solve the above technical problems, an embodiment of the present invention discloses a multimodal physiological signal emotion classification and recognition method, the method comprising:

[0006] S1. Acquire a physiological signal set and a modality type; the physiological signal set includes a first signal sequence, a second signal sequence, a third signal sequence, a fourth signal sequence and a label sequence;

[0007] The first signal sequence and the third signal sequence both contain N first modal signals; the second signal sequence and the fourth signal sequence both contain N second modal signals; the first modal signal and the second modal signal are electroencephalogram signals, electrocardiogram signals or expression signals;

[0008] The tag sequence includes N tag values;

[0009] Said N is an integer greater than 1;

[0010] The value of the modality type is an integer from 1 to 6;

[0011] S2. constructing a first model based on the modality type;

[0012] S3. Using the physiological signal set, training the first model to obtain a first optimized model;

[0013] S4. Constructing a third model based on the first optimization model and the preset second model;

[0014] S5. Training the third model using the physiological signal set and the label sequence to obtain a third optimized model;

[0015] S6. Using the third optimization model, the collected physiological signal to be analyzed is processed to obtain an emotion probability vector; based on the emotion probability vector, a classification recognition result is determined;

[0016] The physiological signal to be analyzed includes the first modality signal and the second modality signal.

[0017] As an optional implementation manner, in the first aspect of the embodiment of the present invention, constructing the first model based on the modality type includes:

[0018] S21, presetting an EEG feature extraction module, an ECG feature extraction module, an expression feature extraction module and a comparative learning module;

[0019] S22: when the modality type is 1, determining that the first feature extraction module is the EEG feature extraction module, and determining that the second feature extraction module is the ECG feature extraction module;

[0020] When the modality type is 2, determining the first feature extraction module to be the electrocardiogram feature extraction module, and determining the second feature extraction module to be the electroencephalogram feature extraction module;

[0021] When the modality type is 3, determining that the first feature extraction module is the EEG feature extraction module, and determining that the second feature extraction module is the expression feature extraction module;

[0022] When the modality type is 4, determining that the first feature extraction module is the expression feature extraction module, and determining that the second feature extraction module is the EEG feature extraction module;

[0023] When the modality type is 5, determining that the first feature extraction module is the electrocardiogram feature extraction module, and determining that the second feature extraction module is the expression feature extraction module;

[0024] When the modality type is 6, determining that the first feature extraction module is the expression feature extraction module, and determining that the second feature extraction module is the electrocardiogram feature extraction module;

[0025] S23. Construct a first model based on the first feature extraction module, the second feature extraction module and the contrast learning module.

[0026] As an optional implementation manner, in the first aspect of the embodiment of the present invention, constructing a first model based on the first feature extraction module, the second feature extraction module and the contrastive learning module includes:

[0027] S231, configuring the input end of the first feature extraction module as the input end of the first model; connecting the output end of the first feature extraction module to the first input end of the contrastive learning module; the first feature extraction module is used to process the first signal sequence and the third signal sequence in sequence to obtain a first feature vector sequence and a third feature vector sequence respectively;

[0028] S232, connecting the output end of the second feature extraction module to the second input end of the contrastive learning module; the second feature extraction module is used to process the second signal sequence and the fourth signal sequence in sequence to obtain a second feature vector sequence and a fourth feature vector sequence respectively;

[0029] S233. Configure the output end of the contrastive learning module as the output end of the first model; the contrastive learning module is used to process the first feature vector sequence, the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence to obtain a contrast loss value.

[0030] As an optional implementation, in the first aspect of the embodiment of the present invention, the contrastive learning module includes a similarity calculation unit, a first loss calculation unit, a second loss calculation unit and a contrast loss calculation unit;

[0031] The similarity calculation unit is connected to the first feature extraction module, the second feature extraction module, the first loss calculation unit and the second loss calculation unit, and is used to process the first feature vector sequence, the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence using a similarity calculation model to obtain a first similarity matrix, a second similarity matrix and a third similarity matrix;

[0032] The first loss calculation unit is connected to the comparison loss calculation unit and is used to process the first similarity matrix and the second similarity matrix using a first loss calculation model to obtain a first loss value;

[0033] The second loss calculation unit is connected to the contrast loss calculation unit and is used to process the third similarity matrix using a second loss calculation model to obtain a second loss value;

[0034] The contrast loss calculation unit is used to calculate the sum of the first loss value and the second loss value to obtain the contrast loss value.

[0035] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the expression of the similarity calculation model is:

[0036]

[0037] Where, sim1 u,v ,sim2 u,v and sim3 u,v are the elements in the uth row and vth column in the first similarity matrix, the second similarity matrix and the third similarity matrix, respectively, where u and v are both integers, and 1≤u≤N, 1≤v≤N; is the uth first eigenvector in the first eigenvector sequence; and are the u-th and v-th second eigenvectors in the second eigenvector sequence respectively; and are respectively the uth and vth third eigenvectors in the third eigenvector sequence; is the vth fourth eigenvector in the fourth eigenvector sequence.

[0038] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the expression of the first loss calculation model is:

[0039]

[0040] Where, L c1 is the first loss value, exp is the exponential operation; sim1 a,b ,sim1 a,i and sim1 i,b are the elements in the ath row and bth column, the ath row and ith column, and the ith row and bth column in the first similarity matrix respectively; sim2 b,a ,sim2 b,i and sim2 i,a They are respectively the elements in the b-th row and a-th column, the b-th row and i-th column, and the i-th row and a-th column in the second similarity matrix; a, b and i are all integers, and 1≤a≤N, 1≤b≤N, 1≤i≤N.

[0041] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the expression of the second loss calculation model is:

[0042]

[0043] Where, L c2 is the second loss value, exp is the exponential operation; sim3 c,d ,sim3 c,j and sim3 j,d They are respectively the elements in the cth row and dth column, the cth row and jth column, and the jth row and dth column in the third similarity matrix; c, d and j are all integers, and 1≤c≤N, 1≤d≤N, 1≤j≤N.

[0044] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the third model is constructed based on the first optimization model and the preset second model:

[0045] S41, determining a first optimization extraction module as the first feature extraction module of the first optimization model; configuring an input end of the first optimization extraction module as a first input end of the third model;

[0046] S42, determining a second optimization extraction module as the second feature extraction module of the first optimization model; configuring an input end of the second optimization extraction module as a second input end of the third model;

[0047] S43, connecting the output end of the first optimization extraction module to the first input end of a preset summing module; connecting the output end of the second optimization extraction module to the second input end of the summing module; connecting the output end of the summing module to the input end of the second model;

[0048] S44. Configure the output end of the second model as the output end of the third model.

[0049] A second aspect of an embodiment of the present invention discloses a multimodal physiological signal emotion classification and recognition device, the device comprising:

[0050] A memory storing executable program code;

[0051] a processor coupled to the memory;

[0052] The processor calls the executable program code stored in the memory to execute part or all of the steps in the multimodal physiological signal emotion classification and recognition method disclosed in the first aspect of the embodiment of the present invention.

[0053] The third aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the multimodal physiological signal emotion classification and recognition method disclosed in the first aspect of an embodiment of the present invention.

[0054] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0055] The present invention utilizes cross-modal multiple contrastive learning to enable multiple correlations of physiological signals of different modalities to be considered in the emotion classification and recognition process, thereby improving the understanding and generalization capabilities of the model. By fusing feature vectors of different modalities, the emotion classification and recognition capabilities of multimodal physiological signals are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 It is a flow chart of a multimodal physiological signal emotion classification and recognition method disclosed in an embodiment of the present invention.

[0058] Figure 2 It is a structural diagram of a first model of a multimodal physiological signal emotion classification and recognition method disclosed in an embodiment of the present invention.

[0059] Figure 3 It is a structural diagram of a comparative learning module of a first model of a multimodal physiological signal emotion classification and recognition method disclosed in an embodiment of the present invention.

[0060] Figure 4 It is a structural diagram of the third model of a multimodal physiological signal emotion classification and recognition method disclosed in an embodiment of the present invention.

[0061] Figure 5 It is a structural diagram of a second model of a multimodal physiological signal emotion classification and recognition method disclosed in an embodiment of the present invention.

[0062] Figure 6 It is a structural schematic diagram of a multimodal physiological signal emotion classification and recognition device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0065] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0066] Embodiment 1

[0067] See also Figure 1 , Figure 1 : is a flow chart of a multimodal physiological signal emotion classification and recognition method disclosed in an embodiment of the present invention. Figure 1 The described multimodal physiological signal emotion classification and recognition method is applied to the field of emotion classification and recognition, such as emotion classification and recognition of patients or respondents, and the embodiments of the present invention are not limited thereto. Figure 1 As shown, the method includes:

[0068] S1. Acquire a physiological signal set and a modality type; the physiological signal set includes a first signal sequence, a second signal sequence, a third signal sequence, a fourth signal sequence and a label sequence;

[0069] The first signal sequence and the third signal sequence both include N first modal signals;

[0070] The second signal sequence and the fourth signal sequence both include N second modal signals;

[0071] It should be noted that, in the above-mentioned first signal sequence, second signal sequence, third signal sequence and fourth signal sequence, the first modal signal and the second modal signal with the same serial number are all collected in the same time period; in the first signal sequence and the second signal sequence, the first modal signal and the second modal signal with the same serial number are collected from the same subject; in the third signal sequence and the fourth signal sequence, the first modal signal and the second modal signal with the same serial number are collected from the same subject, and the subject is different from the subject corresponding to the first modal signal and the second modal signal with the same serial number in the first signal sequence and the second signal sequence.

[0072] The first modal signal and the second modal signal are electroencephalogram signals, electrocardiogram signals or expression signals;

[0073] Electroencephalogram (EEG) refers to the electrical signals generated by the activity of brain neurons. These signals are generated by the synchronous discharge of groups of neurons in the cerebral cortex and can be recorded non-invasively by placing electrodes on the scalp. With the help of EEG machines and portable EEG systems (such as ANT Neuro's eego TM Mylab system) and other EEG acquisition devices can collect the EEG signals of the subjects. The above EEG signals are obtained by filtering, denoising and normalizing the original EEG signals to improve the signal quality.

[0074] Electrocardiogram Signal (ECG or EKG for short) refers to the bioelectric signal generated by the successive excitement of the pacemaker, atrium, and ventricle, accompanied by the changes in the action potential of countless myocardial cells in each cardiac cycle. These bioelectric changes can be reflected on the surface of the body through the conductive tissue and body fluids around the heart, causing regular bioelectric changes in various parts of the body during each cardiac cycle, namely, cardiac potential. With the help of ECG collection devices such as electrocardiographs, portable ECG monitors, and multi-lead ECG recorders, the subject's ECG signal can be collected.

[0075] The expression signal is a set of pictures containing the subject's facial expressions, which are continuously collected by a camera set in front of the subject.

[0076] The tag sequence includes N tag values;

[0077] Optionally, the above label value is happy, sad, angry, anxious or afraid.

[0078] It should be noted that the dimension of the above emotion probability vector is equal to the number of all possible values ​​of the label value. Each component of the emotion probability vector represents the probability of the corresponding value of the label value.

[0079] It should be noted that the ii-th label value in the above-mentioned label sequence is used to simultaneously characterize the emotional state of the subject in the corresponding collection time period of the ii-th first modal signal in the first signal sequence, the ii-th second modal signal in the second signal sequence, the ii-th first modal signal in the third signal sequence and the ii-th second modal signal in the fourth signal sequence, where ii is an integer from 1 to N.

[0080] Said N is an integer greater than 1;

[0081] The value of the modality type is an integer from 1 to 6;

[0082] It should be noted that the above-mentioned modal types are used to represent the modal types of the first modal signal and the second modal signal; when the value of the modal type is 1 to 6, the types corresponding to (first modal signal, second modal signal) are (electroencephalogram signal, electrocardiogram signal), (electrocardiogram signal, electroencephalogram signal), (electroencephalogram signal, expression signal), (expression signal, electroencephalogram signal), (electrocardiogram signal, expression signal) and (expression signal, electrocardiogram signal).

[0083] S2. constructing a first model based on the modality type;

[0084] S3. Using the physiological signal set, training the first model to obtain a first optimized model;

[0085] S4. Constructing a third model based on the first optimization model and the preset second model;

[0086] S5. Training the third model using the physiological signal set and the label sequence to obtain a third optimized model;

[0087] S6. Using the third optimization model, the collected physiological signal to be analyzed is processed to obtain an emotion probability vector; based on the emotion probability vector, a classification recognition result is determined;

[0088] The physiological signal to be analyzed includes the first modality signal and the second modality signal.

[0089] It should be noted that the first type of signal and the second type of signal of the physiological signal to be analyzed are synchronously collected from the subject to be analyzed.

[0090] In an optional embodiment, the above-mentioned constructing the first model based on the modality type includes:

[0091] S21, presetting an EEG feature extraction module, an ECG feature extraction module, an expression feature extraction module and a comparative learning module;

[0092] It should be noted that the parameters of the above-mentioned EEG feature extraction module, ECG feature extraction module, expression feature extraction module and contrast learning module are all initialized to random values.

[0093] Optionally, the above-mentioned EEG feature extraction module is a DIMS-CNN model, which is not limited in the embodiment of the present invention.

[0094] The DIMS-CNN model is a deep learning model for EEG signal analysis. It uses a multi-scale convolutional neural network to extract the features of EEG signals. The advantage of the DIMS-CNN model is that it can process features of multiple scales at the same time, which makes it perform well in processing complex EEG signals. Through the multi-scale convolution layer, the model can capture the local details and global patterns of the signal, which is particularly important for the analysis of EEG signals. In addition, the deep structure of the model enables it to learn the complex and abstract features of the signal, thereby improving the accuracy of classification and prediction. The model includes the following structures connected in sequence: (1) Input layer: Receives EEG signals converted into image form (the EEG signals can be converted into time-frequency images through wavelet transform). (2) Multi-scale convolution layer: Contains multiple convolution layers, each of which uses convolution kernels of different sizes to capture features of different scales. For example, 3x3, 5x5 and 7x7 convolution kernels may be used simultaneously. (3) Activation layer: Such as ReLU layer, used to introduce nonlinearity and enhance the expressiveness of the model. (4) Pooling layer: such as max pooling, used to reduce the spatial dimension of features, extract important features, and reduce the amount of computation. (5) Fully connected layer: At the end of the network, the learned features are mapped to the final classification or regression task.

[0095] Optionally, the above-mentioned ECG feature extraction module is a VGG16net model, which is not limited in this embodiment of the present invention.

[0096] VGG16 is a classic deep learning model proposed by the Visual Geometry Group of the University of Oxford, mainly used for image recognition tasks. The model is characterized by the use of multiple small-sized (3x3) convolution kernels and the construction of a deeper network structure by stacking, which enables it to capture the hierarchical features of the image.

[0097] The network structure of VGG16 consists of the following modules connected in sequence: (1) Input layer: accepts 224x224x3 images as input; (2) Convolutional layer: contains multiple convolutional blocks, each of which consists of 2-3 convolutional layers, and each convolutional layer is usually followed by a ReLU activation function; (3) Pooling layer: after each convolutional block, a 2x2 pooling kernel is used to perform a maximum pooling operation to reduce the size of the feature map; (4) Fully connected layer: after the convolution and pooling operations, the data is flattened and classified through three fully connected layers.

[0098] In order to use the VGG16 model to extract ECG signal features, it is necessary to convert the ECG signal into an image and input it into the model to obtain the corresponding feature vector. The following methods can be used to convert the ECG signal into an image:

[0099] (a) Time domain waveform diagram: The time domain waveform of the ECG signal is directly displayed as an image, where the horizontal axis represents time and the vertical axis represents voltage.

[0100] (b) Spectrogram: The ECG signal is converted into the frequency domain through Fourier transform and then the spectrogram is plotted, which can help analyze the frequency components of the signal.

[0101] (c) Time-frequency representation: Using time-frequency analysis methods such as wavelet transform, the ECG signal is converted into a time-frequency diagram (such as a scalar diagram or a spectrogram), which can simultaneously display the time domain and frequency domain characteristics of the signal.

[0102] Optionally, the above-mentioned expression feature extraction module is a RepLKNet model, which is not limited in the embodiment of the present invention.

[0103] RepLKNet is a novel convolutional neural network (CNN) architecture that revisits the design of large convolution kernels in modern CNNs. Inspired by Visual Transformers (ViTs), the model proposes to use a small number of large convolution kernels instead of stacking multiple small convolution kernels. The model can automatically learn the feature representation of expression images to identify different emotional states. Due to its large receptive field and shape preference, RepLKNet is able to capture key visual patterns in expression images, which is particularly important for expression recognition.

[0104] S22: when the modality type is 1, determining that the first feature extraction module is the EEG feature extraction module, and determining that the second feature extraction module is the ECG feature extraction module;

[0105] When the modality type is 2, determining the first feature extraction module to be the electrocardiogram feature extraction module, and determining the second feature extraction module to be the electroencephalogram feature extraction module;

[0106] When the modality type is 3, determining that the first feature extraction module is the EEG feature extraction module, and determining that the second feature extraction module is the expression feature extraction module;

[0107] When the modality type is 4, determining that the first feature extraction module is the expression feature extraction module, and determining that the second feature extraction module is the EEG feature extraction module;

[0108] When the modality type is 5, determining that the first feature extraction module is the electrocardiogram feature extraction module, and determining that the second feature extraction module is the expression feature extraction module;

[0109] When the modality type is 6, determining that the first feature extraction module is the expression feature extraction module, and determining that the second feature extraction module is the electrocardiogram feature extraction module;

[0110] S23. Construct a first model based on the first feature extraction module, the second feature extraction module and the contrast learning module.

[0111] In another optional embodiment, Figure 2 As shown, the first model is constructed based on the first feature extraction module, the second feature extraction module and the contrast learning module, including:

[0112] S231, configuring the input end of the first feature extraction module as the input end of the first model; connecting the output end of the first feature extraction module to the first input end of the contrastive learning module; the first feature extraction module is used to process the first signal sequence and the third signal sequence in sequence to obtain a first feature vector sequence and a third feature vector sequence respectively;

[0113] S232, connecting the output end of the second feature extraction module to the second input end of the contrastive learning module; the second feature extraction module is used to process the second signal sequence and the fourth signal sequence in sequence to obtain a second feature vector sequence and a fourth feature vector sequence respectively;

[0114] S233. Configure the output end of the contrastive learning module as the output end of the first model; the contrastive learning module is used to process the first feature vector sequence, the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence to obtain a contrast loss value.

[0115] In another optional embodiment, Figure 3 As shown, the above-mentioned contrast learning module includes a similarity calculation unit, a first loss calculation unit, a second loss calculation unit and a contrast loss calculation unit;

[0116] The similarity calculation unit is connected to the first feature extraction module, the second feature extraction module, the first loss calculation unit and the second loss calculation unit, and is used to process the first feature vector sequence, the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence using a similarity calculation model to obtain a first similarity matrix, a second similarity matrix and a third similarity matrix;

[0117] The first loss calculation unit is connected to the comparison loss calculation unit and is used to process the first similarity matrix and the second similarity matrix using a first loss calculation model to obtain a first loss value;

[0118] The second loss calculation unit is connected to the contrast loss calculation unit and is used to process the third similarity matrix using a second loss calculation model to obtain a second loss value;

[0119] The contrast loss calculation unit is used to calculate the sum of the first loss value and the second loss value to obtain the contrast loss value.

[0120] In yet another optional embodiment, the similarity calculation model is expressed as:

[0121]

[0122] Where, sim1 u,v ,sim2 u,v and sim3 u,v are the elements in the uth row and vth column in the first similarity matrix, the second similarity matrix and the third similarity matrix, respectively, where u and v are both integers, and 1≤u≤N, 1≤v≤N; is the uth first eigenvector in the first eigenvector sequence; and are the u-th and v-th second eigenvectors in the second eigenvector sequence respectively; and are respectively the uth and vth third eigenvectors in the third eigenvector sequence; is the vth fourth eigenvector in the fourth eigenvector sequence.

[0123] It should be noted that the above-mentioned first similarity matrix describes the similarity between the first signal sequence and the fourth signal sequence; the above-mentioned second similarity matrix describes the similarity between the second signal sequence and the third signal sequence; and the above-mentioned third similarity matrix describes the similarity between the sum of the first signal sequence and the fourth signal sequence and the sum of the second signal sequence and the third signal sequence.

[0124] It can be seen that, with the help of the first similarity matrix, the second similarity matrix and the third similarity matrix, the similarity between signals of different modal types is described from different angles, which is helpful to improve the accuracy of emotion classification and recognition results.

[0125] In yet another optional embodiment, the expression of the first loss calculation model is:

[0126]

[0127] Where, L c1 is the first loss value, exp is the exponential operation; sim1 a,b ,sim1 a,i and sim1 i,b are the elements in the ath row and bth column, the ath row and ith column, and the ith row and bth column in the first similarity matrix respectively; sim2 b,a ,sim2 b,i and sim2 i,a They are respectively the elements in the b-th row and a-th column, the b-th row and i-th column, and the i-th row and a-th column in the second similarity matrix; a, b and i are all integers, and 1≤a≤N, 1≤b≤N, 1≤i≤N.

[0128] In yet another optional embodiment, the expression of the second loss calculation model is:

[0129]

[0130] Where, L c2 is the second loss value, exp is the exponential operation; sim3 c,d ,sim3 c,j and sim3 j,d They are respectively the elements in the cth row and dth column, the cth row and jth column, and the jth row and dth column in the third similarity matrix; c, d and j are all integers, and 1≤c≤N, 1≤d≤N, 1≤j≤N.

[0131] It should be noted that the above first model is trained by gradient descent method.

[0132] In yet another optional embodiment, the above-mentioned using the physiological signal set to train the first model to obtain the first optimization model includes:

[0133] S31, initializing the execution number N1 to 1;

[0134] S32, using the first model, processing the first signal sequence, the second signal sequence, the third signal sequence, and the fourth signal sequence to obtain a contrast loss value;

[0135] S33, using the contrast loss value as the loss function value, and using the back propagation algorithm to calculate the gradient value of each parameter in the first model;

[0136] S34, using the gradient value of each parameter, updating the corresponding parameter in the first model; adding 1 to the value of N1;

[0137] S35. Repeat S32 to S34 N1 times until the comparison loss value is less than the preset loss threshold, or N1 is equal to the preset maximum number of executions.

[0138] In yet another optional embodiment, Figure 4 As shown, the third model is constructed based on the first optimization model and the preset second model, including:

[0139] S41, determining a first optimization extraction module as the first feature extraction module of the first optimization model; configuring an input end of the first optimization extraction module as a first input end of the third model for inputting the first type of signal;

[0140] S42, determining a second optimization extraction module as the second feature extraction module of the first optimization model; configuring an input end of the second optimization extraction module as a second input end of the third model for inputting the second type of signal;

[0141] S43, connecting the output end of the first optimization extraction module to the first input end of a preset summing module; connecting the output end of the second optimization extraction module to the second input end of the summing module; connecting the output end of the summing module to the input end of the second model;

[0142] It should be noted that the above-mentioned summing module is used to calculate the sum of the feature vectors output by the first optimization extraction module and the second optimization module, which is not limited in the embodiment of the present invention.

[0143] S44. Configure the output end of the second model as the output end of the third model.

[0144] In yet another optional embodiment, Figure 5 As shown, the second model includes a first convolution module, an activation module, a pooling module, a second convolution module and a linear module:

[0145] The input end of the first convolution module is configured as the input end of the second model; the output end of the first convolution module is connected to the input end of the activation module; the output end of the activation module is connected to the input end of the pooling module; the output end of the pooling module is connected to the input end of the second convolution module; the output end of the second convolution module is connected to the input end of the linear module; the output end of the linear module is configured as the output end of the second model, and outputs the classification result.

[0146] It should be noted that the first convolution module and the second convolution module mentioned above both include one or more convolution layers.

[0147] Preferably, the above activation module is constructed based on the RELU activation function.

[0148] It should be noted that the above-mentioned linear module is constructed based on the fully connected layer, and is used to perform a linear transformation on the output result of the second convolution module to obtain an emotion probability vector, which is not limited in the embodiment of the present invention.

[0149] In yet another optional embodiment, the third model is trained using the physiological signal set and the label sequence to obtain a third optimized model, including:

[0150] S51, combining the first modal signal, the third modal signal and the label value with the same sequence number in the first signal sequence, the third signal sequence and the label sequence to obtain a first sample group; the first sample group includes N training samples;

[0151] It should be noted that the training samples in the first sample group are (first modal signal, third modal signal, label value).

[0152] S52, combining the second signal sequence, the fourth signal sequence and the second modal signal, the fourth modal signal and the label value with the same sequence number in the label sequence to obtain a second sample group; the second sample group includes N training samples;

[0153] It should be noted that the training samples in the first sample group are (second modal signal, fourth modal signal, label value).

[0154] S53, merging the first sample group and the second sample group to obtain a training sample set;

[0155] S54: Based on the training sample set, the third optimization model is trained using a gradient descent method to obtain a third optimization model.

[0156] Optionally, the third optimization model adopts a cross entropy loss function during training, and the expression is:

[0157]

[0158] Where MM is the number of categories, y k,c is an indicator variable. If the kth sample belongs to category c, then y k,c =1, otherwise y k,c =0; p k,c is the probability that the kth sample output by the third optimization model belongs to category c.

[0159] In yet another optional embodiment, the above-mentioned determining the classification recognition result based on the emotion probability vector includes:

[0160] Obtain the component with the largest value in the emotion probability vector to obtain the maximum component;

[0161] Determine the label value corresponding to the maximum component of the classification recognition result.

[0162] It can be seen that the multimodal physiological signal emotion classification and recognition method described in the embodiment of the present invention utilizes cross-modal multiple contrast learning, so that the multiple correlations of physiological signals of different modalities can be considered in the emotion classification and recognition process, thereby improving the understanding and generalization capabilities of the model, and by fusing the feature vectors of different modalities, the emotion classification and recognition capabilities of multimodal physiological signals are improved.

[0163] Embodiment 2

[0164] See also Figure 6 , Figure 6 : is a schematic diagram of a multi-modal physiological signal emotion classification and recognition device disclosed in an embodiment of the present invention. Figure 6 The described multimodal physiological signal emotion classification and recognition device can be applied to the field of emotion classification and recognition, such as emotion classification and recognition of patients or respondents, and the embodiments of the present invention are not limited thereto. Figure 6 As shown, the multimodal physiological signal emotion classification and recognition device may include the following parts:

[0165] A memory 201 storing executable program codes;

[0166] a processor 202 coupled to the memory 201;

[0167] The processor 202 calls the executable program code stored in the memory 201 to execute the steps in the multimodal physiological signal emotion classification and recognition method described in the first embodiment.

[0168] Embodiment 3

[0169] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the multimodal physiological signal emotion classification and recognition method described in the first embodiment.

[0170] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0171] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, a magnetic disk storage, a magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0172] Finally, it should be noted that the multimodal physiological signal emotion classification and recognition method and device disclosed in the embodiment of the present invention only discloses the preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal physiological signal emotion classification and recognition method, characterized in that: include: S1. Acquire a physiological signal set and a modality type; the physiological signal set includes a first signal sequence, a second signal sequence, a third signal sequence, a fourth signal sequence and a label sequence; The first signal sequence and the third signal sequence both contain N first modal signals; the second signal sequence and the fourth signal sequence both contain N second modal signals; the first modal signal and the second modal signal are electroencephalogram signals, electrocardiogram signals or expression signals; The tag sequence includes N tag values; Said N is an integer greater than 1; The value of the modality type is an integer from 1 to 6; S2. constructing a first model based on the modality type; S3. Using the physiological signal set, training the first model to obtain a first optimized model; S4. Constructing a third model based on the first optimization model and the preset second model; S5. Training the third model using the physiological signal set and the label sequence to obtain a third optimized model; S6. Using the third optimization model, the collected physiological signal to be analyzed is processed to obtain an emotion probability vector; based on the emotion probability vector, a classification recognition result is determined; The physiological signal to be analyzed includes the first modality signal and the second modality signal.

2. The multimodal physiological signal emotion classification and recognition method according to claim 1 is characterized in that: The constructing of the first model based on the modality type includes: S21, presetting an EEG feature extraction module, an ECG feature extraction module, an expression feature extraction module and a comparative learning module; S22: when the modality type is 1, determining that the first feature extraction module is the EEG feature extraction module, and determining that the second feature extraction module is the ECG feature extraction module; When the modality type is 2, determining the first feature extraction module to be the electrocardiogram feature extraction module, and determining the second feature extraction module to be the electroencephalogram feature extraction module; When the modality type is 3, determining that the first feature extraction module is the EEG feature extraction module, and determining that the second feature extraction module is the expression feature extraction module; When the modality type is 4, determining that the first feature extraction module is the expression feature extraction module, and determining that the second feature extraction module is the EEG feature extraction module; When the modality type is 5, determining that the first feature extraction module is the electrocardiogram feature extraction module, and determining that the second feature extraction module is the expression feature extraction module; When the modality type is 6, determining that the first feature extraction module is the expression feature extraction module, and determining that the second feature extraction module is the electrocardiogram feature extraction module; S23. Construct a first model based on the first feature extraction module, the second feature extraction module and the contrast learning module.

3. The multimodal physiological signal emotion classification and recognition method according to claim 2, characterized in that: The constructing a first model based on the first feature extraction module, the second feature extraction module and the contrastive learning module comprises: S231, configuring the input end of the first feature extraction module as the input end of the first model; connecting the output end of the first feature extraction module to the first input end of the contrastive learning module; the first feature extraction module is used to process the first signal sequence and the third signal sequence in sequence to obtain a first feature vector sequence and a third feature vector sequence respectively; S232, connecting the output end of the second feature extraction module to the second input end of the contrastive learning module; the second feature extraction module is used to process the second signal sequence and the fourth signal sequence in sequence to obtain a second feature vector sequence and a fourth feature vector sequence respectively; S233. Configure the output end of the contrastive learning module as the output end of the first model; the contrastive learning module is used to process the first feature vector sequence, the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence to obtain a contrast loss value.

4. The multimodal physiological signal emotion classification and recognition method according to claim 3 is characterized in that: The contrast learning module includes a similarity calculation unit, a first loss calculation unit, a second loss calculation unit and a contrast loss calculation unit; The similarity calculation unit is connected to the first feature extraction module, the second feature extraction module, the first loss calculation unit and the second loss calculation unit, and is used to process the first feature vector sequence, the second feature vector sequence, the third feature vector sequence and the fourth feature vector sequence using a similarity calculation model to obtain a first similarity matrix, a second similarity matrix and a third similarity matrix; The first loss calculation unit is connected to the comparison loss calculation unit and is used to process the first similarity matrix and the second similarity matrix using a first loss calculation model to obtain a first loss value; The second loss calculation unit is connected to the contrast loss calculation unit and is used to process the third similarity matrix using a second loss calculation model to obtain a second loss value; The contrast loss calculation unit is used to calculate the sum of the first loss value and the second loss value to obtain the contrast loss value.

5. The multimodal physiological signal emotion classification and recognition method according to claim 4, characterized in that: The expression of the similarity calculation model is: Where, sim1 u,v ,sim2 u,v and sim3 u,v are the elements in the uth row and vth column in the first similarity matrix, the second similarity matrix and the third similarity matrix, respectively, where u and v are both integers, and 1≤u≤N, 1≤v≤N; is the uth first eigenvector in the first eigenvector sequence; and are the u-th and v-th second eigenvectors in the second eigenvector sequence respectively; and are respectively the uth and vth third eigenvectors in the third eigenvector sequence; is the vth fourth eigenvector in the fourth eigenvector sequence.

6. The multimodal physiological signal emotion classification and recognition method according to claim 4, characterized in that: The expression of the first loss calculation model is: Where, L c1 is the first loss value, exp is the exponential operation; sim1 a,b ,sim1 a,i and sim1 i,b are the elements in the ath row and bth column, the ath row and ith column, and the ith row and bth column in the first similarity matrix respectively; sim2 b,a ,sim2 b,i and sim2 i,a are the elements in the b-th row and a-th column, the b-th row and i-th column, and the i-th row and a-th column in the second similarity matrix respectively; a, b and i are all integers, and 1≤a≤N, 1≤b≤N, 1≤i≤N.

7. The multimodal physiological signal emotion classification and recognition method according to claim 4, characterized in that: The expression of the second loss calculation model is: Where, L c2 is the second loss value, exp is the exponential operation; sim3 c,d ,sim3 c,j and sim3 j,d They are respectively the elements in the cth row and dth column, the cth row and jth column, and the jth row and dth column in the third similarity matrix; c, d and j are all integers, and 1≤c≤N, 1≤d≤N, 1≤j≤N.

8. The multimodal physiological signal emotion classification and recognition method according to claim 1, characterized in that: Based on the first optimization model and the preset second model, a third model is constructed: S41, determining a first optimization extraction module, which is the first feature extraction module of the first optimization model; Configuring the input end of the first optimization extraction module as the first input end of the third model; S42, determining a second optimization extraction module, which is the second feature extraction module of the first optimization model; Configuring the input end of the second optimization extraction module as the second input end of the third model; S43, connecting the output end of the first optimization extraction module to the first input end of a preset summing module; connecting the output end of the second optimization extraction module to the second input end of the summing module; connecting the output end of the summing module to the input end of the second model; S44. Configure the output end of the second model as the output end of the third model.

9. A multimodal physiological signal emotion classification and recognition device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the multimodal physiological signal emotion classification and recognition method as described in any one of claims 1-8.

10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, which, when called, are used to execute the multimodal physiological signal emotion classification and recognition method as described in any one of claims 1-8.

Images