Multimodal physiological signal semantic alignment method and system for emotion recognition

Abstract

The present application provides a method and system for semantic alignment of multimodal physiological signals for emotion recognition. The method obtains multimodal physiological signals and extracts feature representations of the multimodal physiological signals based on time segments, wherein the multimodal physiological signals are composed of multiple single-modal physiological signals. Then, semantic alignment is performed on the semantics of every two single-modal physiological signals in the same sample based on the feature representation to obtain a bidirectional semantic alignment representation between the single-modal physiological signals. The bidirectional semantic alignment representation is input into the emotion recognition model cross-validated by the test set to screen out the optimal emotion recognition model, and the target emotion is identified based on the optimal emotion recognition model. The method maximizes the semantic relevance of each single-modal physiological signal through semantic alignment, which is beneficial to improving the accuracy of emotion recognition. When screening the optimal model, the training set and the test set are crossed to improve the reliability of the optimal model.

Description

Technical Field

[0001] The present application relates to the technical field of emotion recognition, and in particular to a multimodal physiological signal semantic alignment method and system for emotion recognition. Background Art

[0002] Emotion is a state that integrates human feelings, thoughts, and behaviors, encompassing both psychological and physiological responses to stimuli from various sources. Emotions can manifest in both psychological and physiological ways. In daily life, humans can sense each other's emotions. Technological advancements have enabled human-computer interaction to be applied in multiple scenarios to enhance the user experience. However, the accuracy of machine-recognized human emotions is limited.

[0003] Machines can identify human emotions through single-modal signals such as gaze, voice, and gestures. However, this can lead to inaccurate results due to the controllability and camouflage of movements. Physiological signals, on the other hand, are less controllable and less camouflageable, and can provide more objective and authentic emotional information. Therefore, it is possible to combine physiological signals such as electroencephalograms (EEGs), electromyograms (EMGs), and electrocardiograms (ECGs) to improve emotion recognition accuracy.

[0004] When emotion recognition is performed through a combination of multiple single-modal signals, due to the differences in signal types and the sampling rates of the signal acquisition equipment, the signal durations of the various single-modal signals obtained by the signal acquisition equipment will be somewhat different. Different signals express different semantics in each time period, resulting in the semantic misalignment of the time periods of multiple single-modal signals during the emotion recognition process, making the emotion recognition results inaccurate. Summary of the Invention

[0005] The present application provides a multimodal physiological signal semantic alignment method and system for emotion recognition, which solves the problem that when recognizing emotions through multimodal physiological signals, emotions are not fully recognized based on the semantic correlation between multiple single-modal physiological signals, resulting in low emotion recognition accuracy.

[0006] In a first aspect, the present application provides a multimodal physiological signal semantic alignment method for emotion recognition, the method comprising:

[0007] Acquiring a multimodal physiological signal, and extracting a feature representation of the multimodal physiological signal based on a time segment; the multimodal physiological signal includes a plurality of single-modal physiological signals; the feature representation is used to represent the semantics of the single-modal physiological signal;

[0008] According to the feature representation, semantic alignment is performed on the semantics of each two single-modality physiological signals to obtain a semantic alignment representation; the semantic alignment representation includes a bidirectional semantic alignment representation;

[0009] Inputting the bidirectional semantic alignment representation into the emotion output network of the emotion recognition model to obtain emotion recognition result information;

[0010] Based on the leave-one-subject crossover method, the accuracy of the emotion recognition model is calculated, the optimal emotion recognition model is screened from the emotion recognition models, and the optimal emotion recognition model is output; among them, the emotion recognition model with the highest accuracy is confirmed as the optimal emotion recognition model.

[0011] In some feasible embodiments, the steps of acquiring multimodal physiological signals and extracting feature representations of the multimodal physiological signals based on time segments include:

[0012] dividing the multimodal physiological signal into a plurality of single-modal physiological signals according to the physiological signal category;

[0013] performing windowing processing on the single modality physiological signal to divide the single modality physiological signal into a plurality of single modality physiological signal segments;

[0014] Features of the single modality physiological signal segment are extracted, and the feature representation is calculated based on the features.

[0015] In some feasible embodiments, after dividing the multimodal physiological signal into a plurality of single-modal physiological signals according to the physiological signal category, the method further includes:

[0016] Dividing all single-modality physiological signals into multiple batches, and obtaining the maximum sequence length of the characteristic sequence of the single-modality physiological signal segments in each batch; the sequence length is positively correlated with the signal duration of the single-modality physiological signal;

[0017] Taking the maximum sequence length in each batch as a reference value, the sequence length of the single modality physiological signal segments in each batch is padded to the maximum sequence length so that the length of the feature sequence of the single modality physiological signal segments in each batch is the same.

[0018] In some feasible embodiments, the step of calculating the feature representation based on the feature includes:

[0019] The features of the single modality physiological signal are input into a long short-term memory network to obtain a feature representation of the single modality physiological signal; the features of the single modality physiological signal are represented by a sequence consisting of features of multiple single modality physiological signal segments; the long short-term memory network is connected to the normalization layer in a residual connection manner.

[0020] In some feasible embodiments, the step of performing semantic alignment on the semantics of the multiple single-modality physiological signals according to the feature representation includes:

[0021] Performing a linear transformation on the characteristic representation of the single modality physiological signal to obtain a plurality of row vector groups;

[0022] Based on the cross attention mechanism, an attention score matrix is ​​calculated according to the multiple row vector groups; the attention score matrix is ​​used to represent the unidirectional semantic association between each two single modality physiological signals;

[0023] Performing modal alignment on the attention score matrix to obtain an alignment matrix; the modal alignment is used to filter modal segments with low semantic relevance;

[0024] According to the alignment matrix, a bidirectional semantic alignment representation is calculated and obtained; the bidirectional semantic alignment representation is used to represent the degree of bidirectional semantic association between two single-modality physiological signals.

[0025] In some feasible embodiments, the step of performing modality alignment according to the attention score matrix to obtain an alignment matrix includes:

[0026] Setting an adaptive modality alignment threshold, where the adaptive modality alignment threshold is used to filter the values ​​in the attention score matrix for padding the sequence length; the adaptive modality alignment threshold is the inverse of the number of columns in the attention score matrix;

[0027] If the value of any element in the attention score matrix is ​​less than the adaptive modality alignment threshold, the element with a value less than the adaptive modality alignment threshold is replaced with zero.

[0028] In some feasible embodiments, the step of calculating and obtaining a bidirectional semantic alignment representation according to the alignment matrix includes:

[0029] Calculating a unidirectional semantic alignment representation based on the alignment matrix; the unidirectional semantic alignment representation is used to represent the degree of unidirectional semantic association between single-modality physiological signals;

[0030] Perform average pooling on multiple unidirectional semantic alignment representations to eliminate the temporal dimension differences between each unidirectional semantic representation and obtain unidirectional semantic alignment representations with the same shape;

[0031] Concatenate every two unidirectional semantically aligned representations to obtain multiple bidirectional semantically aligned representations.

[0032] In some feasible embodiments, the step of inputting the bidirectional semantic alignment representation into the emotion output network of the emotion recognition model to obtain emotion recognition result information includes:

[0033] splicing the bidirectional semantic alignment representations;

[0034] The spliced ​​bidirectional semantic alignment representations are input into the emotion output network respectively; the emotion recognition network model includes a fully connected layer and a softmax function layer; the fully connected layer and the softmax function layer are connected in sequence; the spliced ​​bidirectional semantic alignment representations are input by the fully connected layer; the softmax function layer is used to output emotion recognition result information.

[0035] In a second aspect, the present application also provides a multimodal physiological signal semantic alignment system for emotion recognition, the system comprising a preprocessing module and a training module;

[0036] The preprocessing module is used to obtain a multimodal physiological signal and extract a feature representation of the multimodal physiological signal based on a time segment; the multimodal physiological signal includes a plurality of single-modal physiological signals; the feature representation is used to represent the semantics of the single-modal physiological signal;

[0037] The preprocessing module is further configured to perform semantic alignment on the semantics of each two single modality physiological signals based on the feature representation to obtain a semantic alignment representation; the semantic alignment representation includes a bidirectional semantic alignment representation;

[0038] The training module is used to input the bidirectional semantic alignment representation into the emotion output network of the emotion recognition model to obtain emotion recognition result information;

[0039] The training module is also used to screen out the optimal emotion recognition model from the emotion recognition models by calculating the accuracy of the emotion recognition models based on the leave-one-subject crossover method, and output the optimal emotion recognition model; wherein, the emotion recognition model with the highest accuracy is confirmed as the optimal emotion recognition model.

[0040] It can be seen from the above technical solution that the present application provides a multimodal physiological signal semantic alignment method and system for emotion recognition. The method obtains multimodal physiological signals and extracts feature representations of the multimodal physiological signals based on time segments, wherein the multimodal physiological signals are composed of multiple single-modal physiological signals. Then, semantic alignment is performed on the semantics of every two single-modal physiological signals in the same sample based on the feature representation to obtain a bidirectional semantic alignment representation between the single-modal physiological signals. The bidirectional semantic alignment representation is input into the emotion recognition model cross-validated by the test set to screen out the optimal emotion recognition model, and the target emotion is identified according to the optimal emotion recognition model. The method maximizes the semantic relevance of each single-modal physiological signal through semantic alignment, which is beneficial to improving the accuracy of emotion recognition. When screening the optimal model, the training set and the test set are crossed to improve the reliability of the optimal model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 Flowchart of the multimodal physiological signal semantic alignment method for emotion recognition provided in an embodiment of the present application;

[0043] Figure 2 A flow chart of feature characterization training provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a bidirectional semantic alignment representation provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of emotion recognition based on bidirectional semantic alignment representation provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of a network for obtaining feature representations based on multimodal physiological signal feature training provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of linear transformation execution provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of the calculation alignment matrix provided in an embodiment of the present application;

[0049] Figure 8 A schematic diagram of a bidirectional semantic alignment representation calculated according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following embodiments are not intended to represent all possible implementations consistent with the present application. They are merely examples of systems and methods consistent with certain aspects of the present application, as detailed in the claims.

[0051] In fields such as deep learning and neural networks, emotion recognition models can be built to identify the target person's emotions based on the collected motion and physiological signals. Each collected signal can be used as a single modal signal, and multiple single modal signals can be combined to form a multimodal signal.

[0052] Recognizing a target person's emotions using a single modal signal, especially when the signal is action-based, can be challenging due to the camouflage inherent in the target person's movements. Furthermore, using a single modal physiological signal to identify emotions fails to incorporate the semantic information that can be extracted from the signal to express the emotion. Therefore, it's necessary to collect multimodal signals from the target person, i.e., to identify the target person's emotions based on multiple single modal signals.

[0053] Since different locations in the nervous system respond to the same stimulus in different orders and durations, the corresponding occurrence time of each single modality physiological signal is different. In addition, the sampling rate of the physiological signal equipment used to collect physiological signals may also be different, so the signal duration of each single modality physiological signal collected is different, and therefore the semantic features expressed in each time period in the single modality physiological signal are also different. When identifying emotions based on multimodal physiological signals, the degree of correlation between the semantic information of multiple single modality physiological signals is limited, resulting in a low degree of semantic alignment between multiple single modality physiological signals, which in turn reduces the accuracy of identifying emotions based on multimodal physiological signals.

[0054] Based on the above problems, Figure 1 As shown, some embodiments of the present application provide a multimodal physiological signal semantic alignment method for emotion recognition, the method comprising:

[0055] S100: Acquire multimodal physiological signals, and extract feature representations of the multimodal physiological signals based on time segments;

[0056] like Figure 2 As shown, the multimodal physiological signal includes multiple single-modal physiological signals, and the feature representation is used to characterize the semantics of the single-modal physiological signal. The multimodal physiological signal can be obtained by simultaneously collecting multiple single-modal physiological signals. For example, at the same time, the EEG signal of the target person is collected by an EEG signal collection device, and the ECG signal of the target person is collected by an ECG signal collection device. The two single-modal physiological signals obtained by collection can form a multimodal physiological signal for emotion recognition. In addition, the behavioral modality obtained by collection can be used to assist the multimodal physiological signal in emotion recognition.

[0057] The feature representation of multimodal physiological signals is a collection of feature representations of multiple single-modal physiological signals. By extracting the feature representations of multiple single-modal physiological signals and mining their semantics, the semantic alignment of multiple single-modal physiological signals in the same batch of samples can help improve the accuracy of emotion recognition of the target person.

[0058] Among them, multimodal physiological signals include multiple single-modal physiological signals. Due to the different signal sources and signal durations, it is not possible to directly extract features. It is necessary to extract features from multiple single-modal physiological signals separately. In some embodiments, the steps of acquiring multimodal physiological signals and extracting feature representations of the multimodal physiological signals based on time segments include:

[0059] dividing the multimodal physiological signal into a plurality of single-modal physiological signals according to the physiological signal category;

[0060] Different types of physiological signals require different collection methods. These types of physiological signals can be categorized using various methods, such as the device and interface information used to collect them. For example, when collecting EEG information from a target person, an EEG acquisition device is used. The connection interface between the EEG acquisition device and the terminal device can be determined, as can the source of the EEG data transmitted through the connection interface. Therefore, multimodal physiological signals can be divided into multiple single-modal physiological signals based on the connection interface information.

[0061] It is understood that multimodal physiological signals can also be divided into multiple single-modal physiological signals based on various features such as data characteristics and device information of the acquisition device. The basis for dividing multimodal physiological signals into multiple single-modal physiological signals is not specifically limited; any method that can effectively distinguish single-modal physiological signals and facilitate subsequent feature extraction, completion, and splicing of the data can serve as the basis for dividing the signals into multiple single-modal physiological signals.

[0062] Different types of single-modal physiological signals have corresponding signal durations of different types. Different signal durations also result in different features contained in the signal generation process, that is, different feature shapes. In order to make the feature shapes of multiple single-modal physiological signals the same and reduce the probability of training not being able to proceed normally due to different feature shapes, the sequence lengths of multiple single-modal physiological signals need to be supplemented to the same level. When extracting features from multiple single-modal physiological signals, all single-modal physiological signals can be divided into multiple batches, and feature extraction can be performed on the single-modal physiological signals. The division of multiple batches of single-modal physiological signals can reduce the training time of the model; at the same time, learning the features of a batch of multiple samples each time helps the model converge quickly, which is beneficial to improving the accuracy of emotion recognition.

[0063] The single modality physiological signal may be divided into a plurality of single modality physiological signal segments by performing windowing processing on the single modality physiological signal.

[0064] By extracting features from the collected physiological signal data and extracting multiple features, we can increase the training precision of the emotion recognition model and improve the accuracy of emotion recognition. Furthermore, the response intensity and latency of physiological signal data expressing the same semantic meaning may vary from one modality to another. Therefore, when extracting features through windowing, the time window parameters must be adjusted based on the modality of the physiological signal data.

[0065] In some embodiments, a non-overlapping sliding time window can be used to divide the signals of different modalities into multiple time segments, and features are extracted from each segment. The types of features extracted include but are not limited to expected value, standard deviation, skewness, power, differential entropy, and power spectral density.

[0066] During feature extraction, features within multiple time segments need to be extracted one by one. This can easily affect feature extraction efficiency when the amount of training data is large. When using a sliding time window for feature extraction, parameters such as the extraction execution interval and the minimum sliding unit can be set to adapt to the varying data volumes of different modal signals, thereby reducing feature extraction time and improving the training speed of the emotion recognition model.

[0067] After dividing the multiple single-modality physiological signals according to the time window, since the signal durations of the multiple single-modality physiological signals themselves may vary to a certain extent, the multiple single-modality physiological signals obtained in each batch may have certain differences in the time dimension. In order to obtain a more accurate semantic alignment effect between the multiple single-modality physiological signals, in some embodiments, after dividing the multi-modality physiological signals into multiple single-modality physiological signals according to physiological signal categories, the following is further included:

[0068] Obtaining sequence lengths of the multiple single-modality physiological signals;

[0069] Dividing all single-modality physiological signals into multiple batches, and obtaining the maximum sequence length of the characteristic sequence of the single-modality physiological signal segments in each batch; the sequence length is positively correlated with the signal duration of the single-modality physiological signal;

[0070] Taking the maximum sequence length in each batch as a reference value, the sequence length of the single modality physiological signal segments in each batch is padded to the maximum sequence length so that the length of the feature sequence of the single modality physiological signal segments in each batch is the same.

[0071] Sequence length is positively correlated with signal duration, and therefore can reflect the characteristics of each modality's physiological signal. Different sequence lengths indicate different characteristic shapes for each modality's physiological signal, hindering feature extraction and subsequent processing such as splicing and analysis. Therefore, it is necessary to pad the sequence lengths to the same level.

[0072] Extracting the maximum sequence length and using it as the padding criterion can ensure that the features of the corresponding single-modality physiological signal are not lost. At the same time, it can also make the sequence lengths of each single-modality physiological signal the same, which is beneficial for feature extraction, splicing, and analysis.

[0073] In some embodiments, when obtaining feature representations based on features and inputting the feature representations into model training to obtain the relationship between features and emotions, LSTM (Long Short Term Memory) can be selected as a training network. Therefore, based on the input characteristics of the LSTM network, the sequence lengths of the single modal physiological signals need to be padded to the same level. It should be understood that the partial data used to padded the sequence lengths have certain characteristics, so as to prevent the supplementary data from affecting the emotion recognition results in the subsequent emotion recognition process.

[0074] Extracting features of the single modality physiological signal and calculating the feature representation based on the features. In some embodiments, the step of calculating the feature representation based on the features includes:

[0075] The features of the single modality physiological signal are input into a long short-term memory network to obtain a feature representation of the single modality physiological signal; the features of the single modality physiological signal are represented by a feature composition of multiple single modality physiological signal segments; the long short-term memory network is sequentially connected to a residual layer and a normalization layer.

[0076] After a single modality physiological signal is divided into multiple single modality physiological signal segments, the features of the multiple single modality physiological signal segments can form a feature sequence of the single modality physiological signal. The feature sequence can be input into a long short-term memory network for learning and training to obtain a feature representation of the single modality physiological signal.

[0077] like Figure 5 As shown, by inputting the features of a single modality physiological signal into an LSTM network, it is possible to learn the temporal variation of the single modality physiological signal, specifically the relationship between features within different time segments and emotion types. To prevent overfitting and exploding / vanishing gradients during LSTM network learning, each layer in the LSTM network is connected to a normalization layer using a residual connection. The normalization layer prevents exploding / vanishing gradients, while the residual layer mitigates poor test accuracy caused by overfitting.

[0078] In some embodiments, the features extracted from the A-modality can be expressed as:

[0079]

[0080] in, is the sequence length of A modal sample, D A is the characteristic dimension of the A modal sample, is the feature set of modality A samples, i and j represent the jth sample of the i-th batch. After inputting the features of modality A into the LSTM network, the feature representation of modality A can be obtained:

[0081] H A i =Multi_layer LSTM A (X A i );

[0082] Among them, H A i is the feature representation of modality A, Multi_layer LSTM A (X A i ) indicates that the feature set of modal A is input into the multi-layer LSTM network for training and learning.

[0083] After obtaining the feature representation of modality A, the correlation between the features can be calculated based on the feature representation. The correlation can be expressed as the degree of alignment of the semantics corresponding to each single modal signal. Based on the degree of semantic alignment between the single modal signals, the corresponding emotion can be identified.

[0084] S200: performing semantic alignment on the semantics of every two single-modality physiological signals according to the feature representation to obtain a semantically aligned representation;

[0085] like Figure 3 As shown, the feature representation includes the changing relationship between time and features in multiple single-modal physiological signals. It can be understood that it also includes the semantic change law. The semantic alignment representation represents the unidirectional / bidirectional alignment information in a single-modal physiological signal. In some embodiments, the multimodal physiological signal includes modality A and modality B, and the semantic alignment representations of A and B are obtained as follows:

[0086]

[0087] Among them, O A i It is a one-way semantic representation from modality A to modality B, specifically a representation of a certain segment in modality A, which is obtained by weightedly considering the alignment information of all segments in modality B; B iIt is a unidirectional semantic representation from modality B to modality A. Specifically, it is a weighted representation of a segment in modality B, taking into account the information of all segments in modality B. bs is the number of samples in a single batch.

[0088] Based on the unidirectional semantic representation, a bidirectional semantic representation can be further derived. Bidirectional semantic representation more accurately describes the semantic relationship between physiological signals of a single modality. Therefore, using bidirectional semantic representation as the input value of emotion recognition can improve the accuracy of emotion recognition.

[0089] The feature representation includes multiple single modality physiological signal feature representations, each corresponding to a different semantics. Figure 7 As shown, in order to train and learn the semantic relevance of multiple single-modality physiological signals that change over time, it is necessary to perform semantic alignment on the semantics of the multiple single-modality physiological signals based on the feature representation, and the steps include:

[0090] Performing a linear transformation on the characteristic representation of the single modality physiological signal to obtain a plurality of row vector groups;

[0091] Based on the cross attention mechanism, an attention score matrix is ​​calculated according to the multiple row vector groups; the attention score matrix is ​​used to represent the unidirectional semantic association between single modality physiological signals;

[0092] Performing modal alignment on the attention score matrix to obtain an alignment matrix; the modal alignment is used to filter modal segments with low semantic relevance;

[0093] In some embodiments, the semantic representation is calculated based on a cross-attention mechanism. Figure 6 As shown, it is necessary to calculate the query row vector group Q required in the cross attention mechanism based on the feature representation A i , key row vector group K A i and the value row vector, V A i ,in, D K D is the length of the query row vector group and the key row vector group obtained after performing linear transformation on the feature representation. V is the length of the row vector group after linear transformation.

[0094] Q A i , K A i , V A i are all sets of row vector groups corresponding to feature representations, and the calculation formulas for the linear transformation results of a single row vector group are:

[0095]

[0096] in is a trainable linear transformation parameter matrix.

[0097] Based on the cross-attention mechanism, local attention information of feature representation can be obtained. When the value row vector group and the weight row vector group (query row vector group and key row vector group) exist at the same time, the local input features of the feature representation can be observed.

[0098] After obtaining the row vector group through linear transformation, the attention score matrix corresponding to the single modality physiological signal can be obtained. In some embodiments, taking modality A and modality B as examples, their attention score matrices are:

[0099]

[0100] The calculation formulas for each sample element in the attention score matrix are:

[0101]

[0102] Among them, row softmax means performing a softmax operation on the rows in the attention score matrix so that the sum of all elements in the row is 1. During the calculation process, the larger the value of the element in each row, the higher the semantic relevance of multiple single modal data expressions. It can be understood that s AB Each row calculates the semantic relevance between a certain time segment of modality A and all time segments of modality B, s BA Each row calculates the semantic relevance between a certain time segment of modality B and all time segments in modality A.

[0103] The attention score matrix may contain some data with low semantic relevance. This data will increase computation time and reduce the efficiency of emotion recognition. Modal alignment refers to filtering out the data with low semantic relevance. The steps include:

[0104] Setting an adaptive modality alignment threshold, wherein the adaptive modality alignment threshold is used to filter values ​​with low semantic relevance in the attention score matrix;

[0105] If the value of any element in the attention score matrix is ​​less than the adaptive modality alignment threshold, the element with a value less than the adaptive modality alignment threshold is replaced with zero.

[0106] like Figure 8 As shown, using the A mode and B mode in the above embodiment, after filtering the data with low semantic relevance, the unidirectional alignment matrix pointing from mode A to mode B is obtained respectively. One-way alignment matrix pointing from mode B to mode A Their respective calculation formulas are:

[0107]

[0108] in It is an L A i ×L B i However, after removing the padding, the meaningful portion is a block matrix. Data with low semantic relevance contributes less to emotion recognition. This type of data also exhibits certain numerical characteristics, which allows filtering out data with low semantic relevance by setting an adaptive modal alignment threshold.

[0109] For example, The number of columns is L B i , so the adaptive threshold can be set to the inverse of the number of columns of the block matrix Parts of the block matrix with values ​​below the threshold indicate that the corresponding semantic relevance is lower than the average. Therefore, these elements can be considered to have low semantic relevance, which can reduce the accuracy and efficiency of emotion recognition. Therefore, these elements can be replaced with zero elements to reduce interference with training and thus improve the accuracy of the model and emotion recognition.

[0110] The alignment matrix can be used to obtain a unidirectional semantic alignment representation, and a bidirectional semantic alignment representation can be calculated and obtained based on the unidirectional semantic alignment representation. The bidirectional semantic alignment representation is used to characterize the degree of bidirectional semantic association between two single modality physiological signals. In some embodiments, the step of calculating and obtaining the bidirectional semantic alignment representation based on the alignment matrix includes:

[0111] Calculating a one-way semantic alignment representation according to the alignment matrix;

[0112] Perform average pooling on multiple unidirectional semantic alignment representations to eliminate the temporal dimension differences between each unidirectional semantic representation and obtain unidirectional semantic alignment representations with the same shape;

[0113] Concatenate every two unidirectional semantically aligned representations to obtain multiple bidirectional semantically aligned representations.

[0114] The unidirectional semantic alignment representation is used to characterize the unidirectional semantic association degree of a single modality physiological signal. Continuing with the above embodiment, taking modality A and modality B as examples, based on their respective alignment matrices and the calculation formula for the unidirectional semantic alignment representation of each element in the alignment matrix:

[0115]

[0116] The one-way semantic alignment representation of modality A and modality B can be obtained through the one-way alignment representation calculation formula:

[0117]

[0118] The bidirectional alignment representation can be obtained by concatenating the unidirectional semantic alignment representation. Because the length of the sliding time window used to extract the features for calculating the unidirectional / bidirectional semantic alignment representation may be different, the shape of the unidirectional semantic alignment representation may be different. In order to obtain the bidirectional semantic alignment representation by concatenation, it is necessary to perform average pooling on the temporal dimension of the unidirectional semantic alignment representation to obtain the unidirectional semantic alignment representation with the same shape:

[0119]

[0120] Based on the unidirectional semantic alignment representations with the same shape, we can perform splicing and obtain the bidirectional alignment representations of modality A and modality B:

[0121]

[0122] Average pooling can prevent the problem of different shapes of unidirectional semantically aligned representations that prevent them from being combined to form bidirectional semantically aligned representations, thereby improving the stability of the training process. After calculating the bidirectional semantically aligned representations, they can be input into the emotion recognition network for emotion recognition.

[0123] S300: Inputting the bidirectional semantic alignment representation into an emotion output network of an emotion recognition model to obtain emotion recognition result information.

[0124] The emotion output network can perform emotion recognition based on the bidirectional semantic alignment representation, such as Figure 4 As shown, the steps of inputting the bidirectional semantic alignment representation into the emotion output network of the emotion recognition model to obtain emotion recognition result information include:

[0125] splicing the bidirectional semantic alignment representations;

[0126] The spliced ​​bidirectional semantic alignment representations are input into the emotion recognition output network respectively;

[0127] The number of bidirectional semantic alignment representations synthesized from multiple single semantic alignment representations is different. Therefore, when inputting into the emotion recognition network, multiple bidirectional semantic alignment representations need to be spliced ​​so that all valid training data can be input into the emotion recognition network for recognition, thereby improving the accuracy of emotion recognition.

[0128] The emotion output network includes a fully connected layer and a softmax function layer. The fully connected layer and the softmax function layer are sequentially connected. The concatenated bidirectional semantic alignment representation is input to the fully connected layer. The softmax function layer is used to output the emotion recognition result information.

[0129] It should be noted that the emotion recognition output network can be multiple networks of the same type. Multiple emotion recognition results can be obtained through multiple networks of the same type. The accuracy of multiple emotion recognition results can reflect the accuracy of the corresponding emotion recognition model, thereby screening the optimal emotion recognition model.

[0130] S400: Based on the leave-one-subject crossover method, by calculating the accuracy of the emotion recognition model, the optimal emotion recognition model is screened from the emotion recognition models, and the optimal emotion recognition model is output; among them, the emotion recognition model with the highest accuracy is confirmed as the optimal emotion recognition model.

[0131] In some embodiments, a leave-one-subject-out crossover method is used to screen models. That is, the training and test sets are rotated. Each emotion recognition model uses the same training and test set, but each emotion recognition model is validated using a different combination of training and test sets.

[0132] The training set of each emotion recognition model can be used as a test set for other emotion recognition models. Similarly, the test set of each emotion recognition model can be used as a training set for other emotion recognition models. For example, if there are n datasets that can be used as training and test sets, and the first dataset can be used as the test set for the first emotion recognition model, then the other n-1 datasets can be used as training sets for the first emotion recognition model. For another example, if the second dataset can be used as the test set for the second emotion recognition model, then the other n-1 datasets can be used as training sets for the second emotion recognition model. These n-1 training sets include the dataset that served as the test set in the first dataset.

[0133] Therefore, when selecting an emotion recognition model, the total number of datasets corresponds to the total number of emotion recognition models to be verified. That is, n datasets that can be used as training and test sets correspond to n emotion recognition models to be verified. By using a common dataset as both the test and training sets, and validating each emotion recognition model separately, we can effectively select the optimal emotion recognition model from multiple similar emotion recognition models.

[0134] Based on the above-mentioned multimodal physiological signal semantic alignment method for emotion recognition, some embodiments of the present application also provide a multimodal physiological signal semantic alignment system for emotion recognition, the system including a preprocessing module and a training module;

[0135] The preprocessing module is used to obtain a multimodal physiological signal and extract a feature representation of the multimodal physiological signal based on a time segment; the multimodal physiological signal includes a plurality of single-modal physiological signals; the feature representation is used to represent the semantics of the single-modal physiological signal;

[0136] The preprocessing module is further configured to perform semantic alignment on the semantics of the plurality of single modality physiological signals according to the feature representation to obtain a semantic alignment representation; the semantic alignment representation includes a bidirectional semantic alignment representation;

[0137] The training module is used to input the bidirectional semantic alignment representation into the emotion output network of the emotion recognition model to obtain emotion recognition result information;

[0138] The training module is also used to screen out the optimal emotion recognition model from the emotion recognition models by calculating the accuracy of the emotion recognition models based on the leave-one-subject crossover method, and output the optimal emotion recognition model; wherein, the emotion recognition model with the highest accuracy is confirmed as the optimal emotion recognition model.

[0139] It can be seen from the above technical solution that the present application provides a multimodal physiological signal semantic alignment method and system for emotion recognition. The method obtains multimodal physiological signals and extracts feature representations of the multimodal physiological signals based on time segments, wherein the multimodal physiological signals are composed of multiple single-modal physiological signals. Then, semantic alignment is performed on the semantics of every two single-modal physiological signals in the same sample based on the feature representation to obtain a bidirectional semantic alignment representation between the single-modal physiological signals. The bidirectional semantic alignment representation is input into the emotion recognition model cross-validated by the test set to screen out the optimal emotion recognition model, and the target emotion is identified according to the optimal emotion recognition model. The method maximizes the semantic relevance of each single-modal physiological signal through semantic alignment, which is beneficial to improving the accuracy of emotion recognition. When screening the optimal model, the training set and the test set are crossed to improve the reliability of the optimal model.

[0140] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without expending creative work shall fall within the scope of protection of this application.

Claims

1. A multimodal physiological signal semantic alignment method for emotion recognition, characterized by: include: Acquiring multimodal physiological signals, and extracting feature representations of the multimodal physiological signals based on time segments; The multimodal physiological signal includes a plurality of single-modal physiological signals; The feature representation is used to represent the semantics of the single modality physiological signal; According to the feature representation, semantic alignment is performed on the semantics of each two single-modality physiological signals to obtain a semantic alignment representation; the semantic alignment representation includes a bidirectional semantic alignment representation; Inputting the bidirectional semantic alignment representation into the emotion output network of the emotion recognition model to obtain emotion recognition result information; Based on the leave-one-subject crossover method, the accuracy of the emotion recognition model is calculated, and the optimal emotion recognition model is screened from the emotion recognition models, and the optimal emotion recognition model is output; among them, the emotion recognition model with the highest accuracy is confirmed as the optimal emotion recognition model; The step of performing semantic alignment on the semantics of every two single-modality physiological signals includes: Performing a linear transformation on the characteristic representation of the single modality physiological signal to obtain a plurality of row vector groups; Based on the cross attention mechanism, an attention score matrix is ​​calculated according to the multiple row vector groups; the attention score matrix is ​​used to represent the unidirectional semantic association between each two different single modality physiological signals; Performing modal alignment on the attention score matrix to obtain an alignment matrix; the modal alignment is used to filter modal segments with low semantic relevance; According to the alignment matrix, a bidirectional semantic alignment representation is calculated and obtained; the bidirectional semantic alignment representation is used to represent the degree of bidirectional semantic association between two single-modality physiological signals.

2. The method according to claim 1, characterized in that The steps of acquiring multimodal physiological signals and extracting feature representations of the multimodal physiological signals based on time segments include: dividing the multimodal physiological signal into a plurality of single-modal physiological signals according to the physiological signal category; performing windowing processing on the single modality physiological signal to divide the single modality physiological signal into a plurality of single modality physiological signal segments; Features of the single modality physiological signal segment are extracted, and the feature representation is calculated based on the features.

3. The method according to claim 2, characterized in that After dividing the multimodal physiological signal into a plurality of single-modal physiological signals according to the physiological signal category, the method further includes: Dividing all single-modality physiological signals into multiple batches, and obtaining the maximum sequence length of the characteristic sequence of the single-modality physiological signal segments in each batch; the sequence length is positively correlated with the signal duration of the single-modality physiological signal; Taking the maximum sequence length in each batch as a reference value, the sequence length of the single modality physiological signal segments in each batch is padded to the maximum sequence length so that the length of the feature sequence of the single modality physiological signal segments in each batch is the same.

4. The method according to claim 3, characterized in that The step of calculating the feature representation according to the feature comprises: The features of the single modality physiological signal are input into a long short-term memory network to obtain a feature representation of the single modality physiological signal; the features of the single modality physiological signal are represented by a sequence consisting of features of multiple single modality physiological signal segments; the long short-term memory network is connected to the normalization layer in a residual connection manner.

5. The method according to claim 1, characterized in that Performing modality alignment on the attention score matrix to obtain an alignment matrix includes: Setting an adaptive modality alignment threshold, wherein the adaptive modality alignment threshold is used to filter data with low semantic relevance in the attention score matrix; the adaptive modality alignment threshold is the inverse of the number of columns in the attention score matrix; If the value of any element in the attention score matrix is ​​less than the adaptive modality alignment threshold, the element with a value less than the adaptive modality alignment threshold is replaced with zero.

6. The method according to claim 1, characterized in that The steps of calculating and obtaining a bidirectional semantic alignment representation according to the alignment matrix include: Calculating a unidirectional semantic alignment representation based on the alignment matrix; the unidirectional semantic alignment representation is used to represent the degree of unidirectional semantic association between single-modality physiological signals; Perform average pooling on multiple unidirectional semantic alignment representations to eliminate the temporal dimension differences between each unidirectional semantic representation and obtain unidirectional semantic alignment representations with the same shape; Concatenate every two unidirectional semantically aligned representations to obtain multiple bidirectional semantically aligned representations.

7. The method according to claim 1, characterized in that The step of inputting the bidirectional semantic alignment representation into the emotion output network of the emotion recognition model to obtain emotion recognition result information includes: splicing the bidirectional semantic alignment representations; The spliced ​​bidirectional semantic alignment representations are respectively input into the emotion output network; the emotion output network includes a fully connected layer and a softmax function layer; the fully connected layer and the softmax function layer are connected in sequence; the spliced ​​bidirectional semantic alignment representations are input by the fully connected layer; the softmax function layer is used to output emotion recognition result information.

8. A multimodal physiological signal semantic alignment system for emotion recognition, characterized by: include: Preprocessing module, training module; The preprocessing module is used to obtain multimodal physiological signals and extract feature representations of the multimodal physiological signals based on time segments; The multimodal physiological signal includes a plurality of single-modal physiological signals; The feature representation is used to represent the semantics of the single modality physiological signal; The preprocessing module is further configured to perform semantic alignment on the semantics of each two single modality physiological signals based on the feature representation to obtain a semantic alignment representation; the semantic alignment representation includes a bidirectional semantic alignment representation; The training module is used to input the bidirectional semantic alignment representation into the emotion output network of the emotion recognition model to obtain emotion recognition result information; The training module is further configured to screen out an optimal emotion recognition model from the emotion recognition models by calculating the accuracy of the emotion recognition models based on a leave-one-subject crossover method, and output the optimal emotion recognition model; wherein the emotion recognition model with the highest accuracy is determined to be the optimal emotion recognition model; The step of performing semantic alignment on the semantics of every two single-modality physiological signals includes: Performing a linear transformation on the characteristic representation of the single modality physiological signal to obtain a plurality of row vector groups; Based on the cross attention mechanism, an attention score matrix is ​​calculated according to the multiple row vector groups; the attention score matrix is ​​used to represent the unidirectional semantic association between each two different single modality physiological signals; Performing modal alignment on the attention score matrix to obtain an alignment matrix; the modal alignment is used to filter modal segments with low semantic relevance; According to the alignment matrix, a bidirectional semantic alignment representation is calculated and obtained; the bidirectional semantic alignment representation is used to represent the degree of bidirectional semantic association between two single-modality physiological signals.

Images