A multi-modal human gait emotion recognition method

Abstract

The application discloses a multi-modal human gait emotion recognition method, and particularly relates to the field of emotion recognition. The gait emotion data of a subject is collected by stimulating the subject to produce emotions through a head-mounted VR device, and using an inertial sensor node worn on the leg of the subject and an electroencephalogram sensor node worn on the head of the subject. The gait emotion data is segmented and processed by using a sliding window method to obtain gait emotion samples. The inertial gait emotion samples are converted into frequency domain representations by using a fast Fourier transform method, and the electroencephalogram gait emotion samples are converted into time-frequency domain representations by using a wavelet continuous transform method to obtain gait emotion images. The convolutional neural network model based on the channel attention mechanism is obtained by using the training data to perform model training and parameter optimization. The inertial gait emotion feature matrix, the electroencephalogram gait emotion feature matrix and the fusion gait emotion feature matrix are respectively taken as the input of a full connection classifier, and the corresponding decision layer fusion mechanism is established to perform human gait emotion recognition. The application can effectively overcome the influence of the single mode on the gait emotion recognition effect, and effectively improve the human gait emotion recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotion recognition, and more particularly to a multi-modal human gait emotion recognition method. BACKGROUND

[0002] With the development of artificial intelligence, the progress of science and technology and economy, human-computer interaction becomes more and more frequent. In addition to simply operating a computer through a mouse and a keyboard, people hope that computers can understand and process human emotions. Most of the information people communicate and spread is emotional. As a key part of affective computing, the research of emotion recognition enables computers to receive and process signals representing human emotions, thereby inferring emotional states and realizing human-centered human-computer interaction. Gait is the posture and movement pattern of the human body during normal walking, which has obvious individual differences and uniqueness. Gait recognition is the premise of gait emotion recognition. There is evidence that human emotions are expressed to some extent through walking. Changes in emotions will lead to changes in gait movement characteristics. Gait is non-intrusive and acceptable, so emotion recognition based on gait is effective.

[0003] Emotion expression itself is a diversified process, and the study of a single emotion state is not enough. With the improvement of multi-source heterogeneous information fusion theory, the complementation and promotion of multi-modal information can make up for the defects of a single mode, and are more conducive to the realization of emotion recognition. Multi-modal emotion recognition has gradually become a popular research trend. The combination and optimization of multi-source information is called data fusion, which can be divided into input-level fusion, feature-level fusion and decision-level fusion. With the rapid development of deep neural networks, deep learning has shown superior performance in the field of emotion recognition. It can learn the most effective features directly from input data without manual extraction. SUMMARY

[0004] In order to overcome the defects that the single input mode of the prior art cannot be applied to all human gait emotion recognition and single modal has certain classification error, the embodiment of the present application provides a multi-modal human gait emotion recognition method.

[0005] To achieve the above purpose, the present application provides the following technical scheme: a multi-modal human gait emotion recognition method, the specific recognition steps are as follows:

[0006] Step S1: stimulating the measured person to produce e kinds of emotions through a head-mounted VR device, collecting inertia gait emotion data and electroencephalogram gait emotion data of v measured persons by using n inertia sensor nodes worn on the legs of the measured person and m electroencephalogram sensor nodes on the head of the measured person;

[0007] Step S2: Use the sliding window method to segment the inertial gait emotion data and EEG gait emotion data collected by each sensor node at fixed intervals to obtain all inertial gait emotion samples and EEG gait emotion samples.

[0008] Step S3: Convert the inertial gait emotion sample obtained in step S2 into a frequency domain representation using the Fast Fourier Transform method to obtain an inertial gait emotion image; convert the EEG gait emotion sample obtained in step S2 into a time-frequency domain representation using the wavelet continuous transform method to obtain an EEG gait emotion image.

[0009] Step S4: Divide the inertial gait emotion image and EEG inertial gait emotion image obtained in step S3 into training data and test data respectively. Use the training data to train the model and optimize the parameters to obtain a convolutional neural network model based on the channel attention mechanism.

[0010] Step S5: Using the convolutional neural network model based on channel attention mechanism obtained in step S4, extract the features of the inertial gait emotion image obtained in step S3 to obtain the inertial gait emotion feature matrix, and extract the features of the EEG gait emotion image obtained in step S3 to obtain the EEG gait emotion feature matrix.

[0011] Step S6: Perform feature fusion processing on the inertial gait emotion feature matrix and the EEG gait emotion feature matrix obtained in step S5 to obtain the fused gait emotion feature matrix;

[0012] Step S7: Use the inertial gait emotion feature matrix and EEG gait emotion feature matrix obtained in Step S5, and the fused gait emotion feature matrix obtained in Step S6 as inputs to the fully connected classifier to obtain the corresponding predicted labels, establish the corresponding decision layer fusion mechanism, and perform human gait emotion recognition.

[0013] In a preferred embodiment, step S2 specifically includes the following steps:

[0014] S2.1: Using the sliding window method, the inertial gait emotion data and EEG gait emotion data collected by the n inertial sensor nodes and m EEG sensor nodes worn by each subject are divided into multiple inertial gait emotion data segments and EEG gait emotion data segments of the same length. Each gait emotion data segment is used as a sample to obtain all inertial gait emotion samples and EEG gait emotion samples.

[0015] S2.2: The parameters in the sliding window method include the window size and the sliding step size. Let the window size be len and the sliding step size between two adjacent windows be μ, so as to obtain T windows. Where T = f(l / μlen)-1, l represents the total length of gait sentiment data, and f() represents the floor function, which takes the nearest integer less than the calculation result.

[0016] S2.3: For the inertial gait emotion data segment within the t-th (t=1,2,…,T) window collected by the k-th (k=1,2,…,n) inertial sensor node, the x-axis acceleration data is used to... y-axis acceleration data z-axis acceleration data x-axis angular velocity data y-axis angular velocity data z-axis angular velocity data x-axis magnetic field data y-axis magnetic field data z-axis magnetic field data Composition, can be represented as The EEG gait and emotion data segment collected by the j-th (j=1,2,…,m) EEG sensor node within the t-th (t=1,2,…,T) window is denoted as eeg. k，t .

[0017] In a preferred embodiment, step S3 specifically includes:

[0018] S3.1: The inertial gait emotion sample is converted from time domain representation to frequency domain representation using the Fast Fourier Transform method to obtain a frequency domain inertial gait emotion sample with a size of len×9. The inertial gait emotion sample and the frequency domain gait inertial emotion sample are then concatenated along the time dimension to obtain a time-frequency inertial gait emotion sample with a size of 2len×9. The sample data values ​​are mapped to the gray values ​​in the image to form a gray value matrix, thus obtaining the inertial gait emotion image.

[0019] S3.2: The time-domain representation of EEG gait emotion samples is converted to a time-frequency domain representation using wavelet continuous transform. Through N-level wavelet decomposition, M frequency bands are obtained, where M = 2. N To obtain time-frequency domain EEG gait emotion samples with a size of len×M, the sample data values ​​are mapped to gray values ​​in the image to form a gray value matrix, thereby obtaining EEG gait emotion images.

[0020] In a preferred embodiment, step S5 specifically includes:

[0021] S5.1: Convolutional neural networks mainly include convolutional layers, pooling layers, and fully connected layers. In order to examine the important correlation features between multiple channels of inertial sensor signals hidden in inertial gait emotion images, various convolutional kernels of different sizes are used. The stride of the convolutional kernels and the pooling parameters are adjusted to obtain correlation feature matrices between different numbers of channels. The matrices are then concatenated according to the channel dimension. Using the same convolutional kernel parameters and network structure, important correlation features between multiple frequency bands of wavelet signals of each EEG sensor in EEG gait emotion images are extracted.

[0022] S5.2: The feature matrix F obtained after passing through the convolutional neural network can be represented as F = {F1, F2, ..., F...} c}, where c represents the number of channels. Using the channel attention mechanism, we can obtain a c-dimensional vector Atn = {Atn1, Atn2, ..., Atn}. c}, where each value belongs to [0,1], to obtain the weight corresponding to each channel. The weights are multiplied by the feature matrix F×Atn to obtain the inertial gait emotion feature matrix and the EEG gait emotion feature matrix.

[0023] In a preferred embodiment, step S7 specifically includes: the dimensions of the inertial gait emotion feature matrix and the EEG gait emotion feature matrix are respectively fl. ine and fl eeg The two feature matrices are concatenated and normalized to form a new high-dimensional synthetic feature representation with size fl. ine +fl eeg To obtain a fused gait emotion feature matrix.

[0024] In a preferred embodiment, step S7 specifically includes:

[0025] S7.1: Using P-fold cross-validation, the F1 score of e emotions for o classification models is obtained as the contribution rate;

[0026] S7.2: Based on the contribution rate, establish the following evaluation matrix for the decision-making layer fusion model:

[0027]

[0028] Where R represents the evaluation matrix, r eo This represents the contribution rate of the e-th emotion to the o-th classification model;

[0029] S7.3: The contrast intensity S of the h-th (h=1,2,…,o) classification model h Conflict Index C h Expressed in terms of standard deviation and correlation coefficient, respectively:

[0030]

[0031] Among them, s h r represents the contrast intensity of the h-th classification model. ih Let e ​​represent the contribution rate of the e-th emotion to the h-th classification model. C represents the mean contribution rate of the e-th emotion in the h-th classification model. h c represents the conflict index of the h-th classification model. jh This represents the correlation coefficient between the j-th and h-th classification models;

[0032] The information content of the e-th sentiment in the h-th classification model can be obtained using the following formula:

[0033] Y h =S h ×C h

[0034] Among them, Y h This represents the amount of information about the e-th emotion in the h-th classification model;

[0035] The weight of the e-th sentiment in the h-th classification model is obtained using the following formula:

[0036]

[0037] Where, γ eh This represents the weight of the e-th emotion in the h-th classification model;

[0038] The output of the e-th emotion for the h-th classification model can be obtained using the following formula:

[0039]

[0040] Among them, Γ e This indicates that the test sample was assigned to the e-th emotion;

[0041] S7.4: Establish a corresponding decision-making level fusion mechanism to perform human gait emotion recognition.

[0042] This invention provides a multimodal human gait emotion recognition method. It utilizes a head-mounted VR device to stimulate the subject to generate emotions, and employs inertial sensor nodes and EEG sensor nodes to collect gait emotion data, converting it into gait emotion images to construct a human gait emotion dataset. At the emotion recognition level, a convolutional neural network model based on channel attention is proposed. This model effectively combines channel attention mechanisms and convolutional neural network models to automatically extract deep features from gait emotion images. An end-to-end network structure reduces engineering complexity, and lightweight design of the network structure and parameters further reduces model complexity and improves algorithm performance. At the fusion level, the classification decisions of the inertial gait emotion feature matrix, the EEG gait emotion feature matrix, and the fused gait emotion feature matrix are fused, with the output weights of each model obtained from the evaluation matrix. The algorithm proposed in this invention effectively overcomes the impact of using a single modality in gait emotion recognition, significantly improving the robustness of the system and the recognition accuracy of the human gait emotion recognition method. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0044] Figure 2 This is a schematic diagram of the convolutional neural network model structure of the present invention.

[0045] Specific implementation measures

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] This invention provides, for example Figure 1 The present invention provides a multimodal human gait emotion recognition method, the specific recognition steps of which are as follows: Step S1: Stimulate the subject to generate e kinds of emotions through a head-mounted VR device, and collect v kinds of inertial gait emotion data and EEG gait emotion data of the subject by using n inertial sensor nodes worn on the subject's legs and m EEG sensor nodes on the head.

[0048] Specifically, the test subject wears a head-mounted VR device to watch videos tailored to their individual emotions, entering a specific emotional state before beginning walking. Then, inertial gait emotional data and EEG gait emotional data are collected using n inertial sensor nodes fixed to the subject's legs and m EEG sensor nodes fixed to the head. Each inertial sensor node consists of an xyz triaxial accelerometer, an xyz triaxial gyroscope, and an xyz triaxial magnetometer, collecting xyz triaxial acceleration data, xyz triaxial angular velocity data, and xyz triaxial magnetic field data, respectively. For the k-th (k = 1, 2, ..., n) inertial sensor node, the collected xyz triaxial acceleration data can be expressed as... The xyz triaxial angular velocity data can be expressed as follows: The xyz triaxial magnetic field data can be represented as follows: For j (j = 1, 2, ..., m) EEG sensor nodes, the collected EEG data can be represented as eeg k The EEG gait emotion data collected by the EEG sensor node and the inertial gait emotion data collected by the inertial sensor node can constitute gait emotion data WE = {eeg, acc, anv, maf}.

[0049] Step S2: Use the sliding window method to segment the inertial gait emotion data and EEG gait emotion data collected by each sensor node at fixed intervals to obtain all inertial gait emotion samples and EEG gait emotion samples.

[0050] The sliding window method was used to segment the inertial gait emotion data and EEG gait emotion data collected by the n inertial sensor nodes and m EEG sensor nodes worn by each subject into multiple inertial gait emotion data segments and EEG gait emotion data segments of the same length. Each gait emotion data segment was used as a sample to obtain all inertial gait emotion samples and EEG gait emotion samples.

[0051] The parameters in the sliding window method include the window size and the sliding step size. Let the window size be len and the sliding step size between two adjacent windows be μ, so as to obtain t windows. Here, T = f(l / μlen)-1, l represents the total length of gait sentiment data, and f() represents the floor function, which takes the nearest integer less than the calculated result.

[0052] For the inertial gait emotion data segment within the t-th (t=1,2,…,T) window collected by the k-th (k=1,2,…,n) inertial sensor node, the x-axis acceleration data... y-axis acceleration data z-axis acceleration data x-axis angular velocity data y-axis angular velocity data z-axis angular velocity data x-axis magnetic field data y-axis magnetic field data z-axis magnetic field data Composition, can be represented as The EEG gait and emotion data segment collected by the j-th (j=1,2,…,m) EEG sensor node within the t-th (t=1,2,…,T) window is denoted as eeg. k，t ;

[0053] Step S3: Convert the inertial gait emotion samples obtained in Step S2 into a frequency domain representation using the Fast Fourier Transform (FFT) method to obtain an inertial gait emotion image; convert the EEG gait emotion samples obtained in Step S2 into a time-frequency domain representation using the wavelet continuous transform (WFT) method to obtain an EEG gait emotion image. Specific steps include:

[0054] The inertial gait emotion samples are converted from time-domain representation to frequency-domain representation using the Fast Fourier Transform (FFT) method to obtain frequency-domain inertial gait emotion samples with a size of len×9. The spectral information of the signal is used to reflect the properties of the signal. The inertial gait emotion samples and frequency-domain gait inertial emotion samples are concatenated along the time dimension, and logarithmic operations are used to increase the data difference to obtain time-frequency inertial gait emotion samples with a size of 2len×9. The data is normalized, and the numerical values ​​are mapped to [0,255]. The sample data values ​​are mapped to the gray values ​​in the image to form a gray value matrix to obtain the inertial gait emotion image.

[0055] The time-domain representation of EEG gait emotion samples is transformed into a time-frequency domain representation using wavelet continuous transform. The temporal and spatial frequency localization of the signal is analyzed, and M frequency bands are obtained through N-level wavelet decomposition, where M=2. N To obtain time-frequency domain EEG gait emotion samples with a size of len×M, the data is normalized and the numerical values ​​are mapped to [0,255]. The sample data values ​​are mapped to the gray values ​​in the image to form a gray value matrix, thereby obtaining the EEG gait emotion image.

[0056] Step S4: Divide the inertial gait emotion image and EEG inertial gait emotion image obtained in step S3 into training data and test data respectively. Use the training data to train the model and optimize the parameters to obtain a convolutional neural network model based on the channel attention mechanism.

[0057] Specifically, the inertial gait emotion images and EEG inertial gait emotion images obtained in step S3 are divided into training data and test data, respectively. For example, the B-fold cross-validation method is used to optimize the network parameters. Specifically, the training data is divided into B parts, of which B-1 parts are used for training the model and optimizing the parameters, and the remaining part is used to verify the effect of this experiment. Each part of the data is used as validation data. The training process is iterated B times in total. The parameters obtained with the highest accuracy in the B training iterations are the optimized parameters, so as to obtain a convolutional neural network model based on channel attention mechanism that can accurately identify each human gait emotion. The trained model is tested using test data, and precision, recall, and F1 score are used as model evaluation factors.

[0058] Step S5: Using the convolutional neural network model based on channel attention mechanism obtained in step S4, extract the features of the inertial gait emotion image obtained in step S3 to obtain the inertial gait emotion feature matrix, and extract the features of the EEG gait emotion image obtained in step S3 to obtain the EEG gait emotion feature matrix. The specific steps include:

[0059] Figure 2 The convolutional neural network model based on channel attention mechanism shown includes a convolutional neural network and an attention mechanism. The convolutional neural network mainly includes convolutional layers, pooling layers, and fully connected layers. To examine the important correlation features between multiple channels of inertial sensor signals hidden in inertial gait emotion images, multiple convolutional kernels of different sizes are used in parallel, namely 1×3, 5×5, 9×9, and 13×13. The stride of the convolutional kernels and the pooling parameters are adjusted to obtain correlation feature matrices between single channels, 5 channels, 9 channels, and 13 channels. These matrices are then concatenated according to the channel dimension to form an Inception structure. Two convolutional layers are then used to further optimize and reduce the dimensionality of the features. Using the same convolutional kernel parameters and network structure, important correlation features between single frequency bands, 5 frequency bands, 9 frequency bands, and 13 frequency bands of wavelet signals from each EEG sensor in the EEG gait emotion image are extracted.

[0060] S5.2: The feature matrix F obtained after passing through the convolutional neural network can be represented as F = {F1, F2, ..., F...} c}, where c represents the number of channels; using the channel attention mechanism, a c-dimensional vector Atn = {Atn1, Atn2, ..., Atn} can be obtained. c}, where each value belongs to [0,1], to obtain the adaptive weight corresponding to each channel. The weights are multiplied by the feature matrix F×Atn to obtain the inertial gait emotion feature matrix and the EEG gait emotion feature matrix.

[0061] Step S6: Perform feature fusion processing on the inertial gait emotion feature matrix and the EEG gait emotion feature matrix obtained in step S5 to obtain the fused gait emotion feature matrix;

[0062] Specifically, the dimensions of the inertial gait emotion feature matrix and the EEG gait emotion feature matrix after flattening are fl and fl respectively. ine and fl eeg The two feature vectors are concatenated and then normalized. The normalization is performed using the min-max normalization method. Among them, y i It is the normalized data, x i The i-th data point from the original data is used to form a new high-dimensional synthetic feature representation with size fl. ine +fl eeg To obtain gait emotion feature vectors.

[0063] Step S7: The inertial gait emotion feature matrix and EEG gait emotion feature matrix obtained in Step S5, and the fused gait emotion feature matrix obtained in Step S6 are used as inputs to a fully connected classifier to obtain corresponding predicted labels. A corresponding decision-level fusion mechanism is established to perform human gait emotion recognition. The specific steps include:

[0064] Through P-fold cross-validation, the F1 scores of e emotions for o classification models are obtained as the contribution rates.

[0065] Based on the contribution rate, the following evaluation matrix for the decision-making level fusion model is established:

[0066]

[0067] Where R represents the evaluation matrix, r co This represents the contribution rate of the e-th emotion to the o-th classification model;

[0068] The contrast intensity S of the h-th (h=1,2,…,o) classification model h Conflict Index C h Expressed in terms of standard deviation and correlation coefficient, respectively:

[0069]

[0070] Among them, S h r represents the contrast intensity of the h-th classification model. ih This represents the contribution rate of the i-th emotion to the h-th classification model. C represents the mean contribution rate of the e-th emotion in the h-th classification model. h c represents the conflict index of the h-th classification model. jh

[0071] This represents the correlation coefficient between the j-th and h-th classification models;

[0072] The information content of the e-th sentiment in the h-th classification model can be obtained using the following formula:

[0073] Y h =S h ×C h

[0074] Among them, Y h This represents the amount of information about the e-th emotion in the h-th classification model;

[0075] The weight of the e-th sentiment in the h-th classification model is obtained using the following formula:

[0076]

[0077] Where, γ eh This represents the weight of the e-th emotion in the h-th classification model;

[0078] The output of the e-th emotion for the h-th classification model can be obtained using the following formula:

[0079]

[0080] Among them, Γ e This indicates that the test sample was assigned to the e-th emotion;

[0081] Establish a corresponding decision-making level fusion mechanism to conduct human gait emotion recognition.

[0082] The specific implementation method is as follows:

[0083] For example, an EEG sensor and an inertial sensor encapsulating an accelerometer, gyroscope, and magnetometer were used to capture gait-emotional movements and achieve close-range gait-emotional data acquisition. A gait-emotional data acquisition platform was built using these sensors. Two inertial sensors and one EEG sensor were fixed to the subject's thigh and head, respectively, and connected to a host computer via Bluetooth. Sixteen healthy volunteers were recruited to build the gait-emotional dataset, with an equal male-to-female ratio. An external media stimulus method was used to generate emotions. Each volunteer watched a video on a VR device five minutes beforehand. While immersed in the video, they began walking according to their own habits, continuously watching the video during the walk to collect data on neutral, happy, and fearful gait emotions.

[0084] Walking is a cyclical and repetitive process. The entire time series of gait emotion data is divided into multiple sub-time series to increase the number of gait emotion samples and achieve more refined human gait emotion recognition. A sliding window method is used for data segmentation. The choice of window size affects the overall recognition performance of the system. The number of gait emotion data points collected per second is used as the window size. There is a 50% overlap between adjacent windows to ensure that the data in adjacent windows are correlated and that no information is lost, thus obtaining all inertial gait emotion samples and EEG gait emotion samples.

[0085] Inertial gait emotion samples are represented using both time-domain and frequency-domain methods. Time-domain representation maps the inertial gait emotion samples to pixels in a grayscale image, normalizes the data, and maps the numerical values ​​to the range [0, 255]. Each data value corresponds to a grayscale pixel in the image. The inertial sensor's sampling frequency is 50 Hz, resulting in a 9×50 time-domain gait inertial emotion image. Frequency-domain representation applies a Fast Fourier Transform along the time dimension to the time-domain inertial gait emotion image, using spectral information to reflect the data properties. Logarithmic operations are performed to enhance data differentiation, resulting in a frequency-domain gait inertial emotion image. Finally, the time-domain and frequency-domain inertial gait emotion images are concatenated along the time dimension to obtain the final inertial gait emotion image.

[0086] Electroencephalogram (EEG) signals are typically unstable and non-stationary signals, and representation using only the frequency domain or time domain is insufficient. Time-frequency domain representation combines the characteristics and advantages of both, providing a better reflection of EEG gait and emotional data. Wavelet transform is used to decompose and represent the EEG signals. Through N-level wavelet decomposition, M frequency bands are obtained, where M = 2n N The sampling frequency of the EEG sensor is 512 Hz. We can use continuous wavelet transform to convert the 1×512 signal into the required 75×512 matrix to obtain EEG gait emotion images.

[0087] Convolutional neural network (CNN) models based on attention mechanisms include feature extraction modules and attention modules. Pure convolutional layers are used as the backbone for extracting deep features, reducing the complexity of the network structure. After obtaining the multidimensional feature matrix, channel attention is used to focus on salient parts and find the most critical components from complex data.

[0088] Network models for extracting inertial gait emotion images and EEG gait emotion images, such as Figure 2As shown, to detect key features of hidden correlation patterns among the 9 channels and 75 frequency bands of the sensor in the input gait emotion image, for the initial structure, 128 convolutional kernels with a stride of 1 and sizes of 1×3, 5×5, 9×9, and 13×13 are used to detect features in different channels of the inertial gait emotion image and different frequency bands of the EEG gait emotion image. Two convolutional layers are then used to optimize the features. Each convolutional layer performs 2D convolution, followed by a batch normalization layer, a ReLU activation function, and a max-pooling downsampling layer. After the second max-pooling downsampling layer, the feature vector is connected to a fully connected layer through a flattening layer, flattening the feature matrix into a one-dimensional feature vector. A softmax layer is then used to obtain the probability distribution for each human gait emotion category.

[0089] The feature matrix F obtained after passing through the convolutional neural network can be represented as F = {F1, F2, ..., F}. c}, where c represents the number of channels; using the channel attention mechanism, a c-dimensional vector Atn = {Atn1, Atn2, ..., Atn} can be obtained. c}, where each value belongs to [0,1], to obtain the adaptive weight corresponding to each channel. The weights are multiplied by the feature matrix F×Atn to obtain the inertial gait emotion feature matrix and the EEG gait emotion feature matrix.

[0090] The dimensions of the inertial gait emotion feature matrix and the EEG gait emotion feature matrix are 64×3×24 and 64×8×11, respectively. The two feature matrices are concatenated and stitched together in a flattened layer to obtain a new feature representation, called the fused gait emotion feature matrix. The two concatenated features must have the same numerical scale to balance the fused features. Temporal min-max normalization is applied to the fused gait emotion feature matrix. Different features have different dimensions, according to the formula... A linear transformation is performed on the data to map it to the range [0,1] to eliminate the effect of dimensionality, where y i It is the normalized data, x i It is the i-th data in the original data.

[0091] The obtained inertial gait emotion images and EEG inertial gait emotion images were divided into training data and test data, respectively. An 8-fold cross-validation method was used to optimize the network parameters. Specifically, the training data was divided into eight equal parts, with seven parts used for model training and parameter optimization, and the remaining part used to validate the experimental results. Each part served as validation data, and the training process was iterated eight times. The parameters obtained with the highest accuracy in the eight training iterations were used as the optimized parameters to obtain a convolutional neural network model based on channel attention that can accurately identify each type of human gait emotion. The decision layer fusion mechanism proposed in this invention was used to classify and identify the test data, and precision, recall, and F1 score were used as model evaluation factors.

[0092] Then, experiments were conducted. The method provided in this embodiment of the invention was compared using three different human gait emotion recognition results under different evaluation indicators, as shown in Table 1:

[0093] Table 1 shows the recognition results of three types of human gait emotions using different evaluation metrics.

[0094]

[0095] As can be seen from Table 1, the method provided by the embodiments of the present invention is effective.

[0096] This invention discloses a multimodal method for human gait emotion recognition, specifically relating to the field of emotion recognition. It involves stimulating the subject with emotions using a head-mounted VR device, and collecting gait emotion data using inertial sensor nodes worn on the subject's legs and EEG sensor nodes on the head. The gait emotion data is segmented using a sliding window method to obtain gait emotion samples. The inertial gait emotion samples are converted to a frequency domain representation using Fast Fourier Transform, and the EEG gait emotion samples are converted to a time-frequency domain representation using Wavelet Continuous Transform to obtain gait emotion images. Training data is used for model training and parameter optimization to obtain a convolutional neural network model based on a channel attention mechanism. The inertial gait emotion feature matrix, the EEG gait emotion feature matrix, and the fused gait emotion feature matrix are used as inputs to a fully connected classifier, establishing a corresponding decision-level fusion mechanism for human gait emotion recognition. This invention effectively overcomes the impact of using a single modality on gait emotion recognition performance and significantly improves the accuracy of human gait emotion recognition.

[0097] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal human gait emotion recognition method, characterized in that: The specific identification steps are as follows: Step S1: Stimulate the subject to generate e kinds of emotions through a head-mounted VR device, and collect inertial gait emotion data and EEG gait emotion data of v subjects using n inertial sensor nodes worn on the subject's legs and m EEG sensor nodes on the head. Step S2: Use the sliding window method to segment the inertial gait emotion data and EEG gait emotion data collected by each sensor node at fixed intervals to obtain all inertial gait emotion samples and EEG gait emotion samples. Step S3: Convert the inertial gait emotion sample obtained in step S2 into a frequency domain representation using the Fast Fourier Transform method to obtain an inertial gait emotion image; convert the EEG gait emotion sample obtained in step S2 into a time-frequency domain representation using the wavelet continuous transform method to obtain an EEG gait emotion image. Step S4: Divide the inertial gait emotion image and EEG inertial gait emotion image obtained in step S3 into training data and test data respectively. Use the training data to train the model and optimize the parameters to obtain a convolutional neural network model based on the channel attention mechanism. Step S5: Using the convolutional neural network model based on channel attention mechanism obtained in step S4, extract the features of the inertial gait emotion image obtained in step S3 to obtain the inertial gait emotion feature matrix, and extract the features of the EEG gait emotion image obtained in step S3 to obtain the EEG gait emotion feature matrix. Step S6: Perform feature fusion processing on the inertial gait emotion feature matrix and the EEG gait emotion feature matrix obtained in step S5 to obtain the fused gait emotion feature matrix; Step S7: Use the inertial gait emotion feature matrix and EEG gait emotion feature matrix obtained in Step S5, and the fused gait emotion feature matrix obtained in Step S6 as inputs to the fully connected classifier to obtain the corresponding predicted labels, establish the corresponding decision layer fusion mechanism, and perform human gait emotion recognition.

2. The multimodal human gait emotion recognition method according to claim 1, characterized in that: The specific steps of step S2 include: S2.1: Using the sliding window method, the inertial gait emotion data and EEG gait emotion data collected by the n inertial sensor nodes and m EEG sensor nodes worn by each subject are divided into multiple inertial gait emotion data segments and EEG gait emotion data segments of the same length. Each gait emotion data segment is used as a sample to obtain all inertial gait emotion samples and EEG gait emotion samples. S2.2: The parameters in the sliding window method include the window size and the sliding step size. Let the window size be len and the sliding step size between two adjacent windows be μ, so as to obtain T windows. Where T = f(l / μlen)-1, l represents the total length of gait sentiment data, and f() represents the floor function, which takes the nearest integer less than the calculation result. S2.3: For the inertial gait emotion data segment within the t-th (t=1,2,…,T) window collected by the k-th (k=1,2,…,n) inertial sensor node, the x-axis acceleration data is used to... y-axis acceleration data z-axis acceleration data x-axis angular velocity data y-axis angular velocity data z-axis angular velocity data x-axis magnetic field data y-axis magnetic field data z-axis magnetic field data Composition, can be represented as The EEG gait and emotion data segment collected by the j-th (j=1,2,…,m) EEG sensor node within the t-th (t=1,2,…,T) window is denoted as eeg. k，t .

3. The multimodal human gait emotion recognition method according to claim 1, characterized in that: The specific steps of step S3 include: S3.1: The inertial gait emotion sample is converted from time domain representation to frequency domain representation using the Fast Fourier Transform method to obtain a frequency domain inertial gait emotion sample with a size of len×9. The inertial gait emotion sample and the frequency domain gait inertial emotion sample are then concatenated along the time dimension to obtain a time-frequency inertial gait emotion sample with a size of 2len×9. The sample data values ​​are mapped to the gray values ​​in the image to form a gray value matrix, thus obtaining the inertial gait emotion image. S3.2: The time-domain representation of EEG gait emotion samples is converted to a time-frequency domain representation using wavelet continuous transform. Through N-level wavelet decomposition, M frequency bands are obtained, where M = 2. N To obtain time-frequency domain EEG gait emotion samples with a size of len×M, the sample data values ​​are mapped to gray values ​​in the image to form a gray value matrix, thereby obtaining EEG gait emotion images.

4. The multimodal human gait emotion recognition method according to claim 1, characterized in that: The specific steps of step S5 include: S5.1: Convolutional neural networks mainly include convolutional layers, pooling layers, and fully connected layers. In order to examine the important correlation features between multiple channels of inertial sensor signals hidden in inertial gait emotion images, various convolutional kernels of different sizes are used. The stride of the convolutional kernels and the pooling parameters are adjusted to obtain correlation feature matrices between different numbers of channels. The matrices are then concatenated according to the channel dimension. Using the same convolutional kernel parameters and network structure, important correlation features between multiple frequency bands of wavelet signals of each EEG sensor in EEG gait emotion images are extracted. S5.2: The feature matrix F obtained after passing through the convolutional neural network can be represented as F = {F1, F2, ..., F...} c }, where c represents the number of channels. Using the channel attention mechanism, we can obtain a c-dimensional vector Atn = {Atn1, Atn2, ..., Atn}. c }, where each value belongs to [0,1], to obtain the weight corresponding to each channel. The weights are multiplied by the feature matrix F×Atn to obtain the inertial gait emotion feature matrix and the EEG gait emotion feature matrix.

5. The multimodal human gait emotion recognition method according to claim 1, characterized in that: The specific steps of step S6 include: the dimensions of the inertial gait emotion feature matrix and the EEG gait emotion feature matrix are respectively fl. ine and fl eeg The two feature matrices are concatenated and normalized to form a new high-dimensional synthetic feature representation with size fl. ine +fl eeg To obtain a fused gait emotion feature matrix.

6. The multimodal human gait emotion recognition method according to claim 1, characterized in that: The specific steps of step S7 include: S7.1: Using P-fold cross-validation, the F1 score of e emotions for o classification models is obtained as the contribution rate; S7.2: Based on the contribution rate, establish the following evaluation matrix for the decision-making layer fusion model: Where R represents the evaluation matrix, r eo This represents the contribution rate of the e-th emotion to the o-th classification model; S7.3: The contrast intensity S of the h-th (h=1,2,…,o) classification model h Conflict Index C h Expressed in terms of standard deviation and correlation coefficient, respectively: Among them, S h r represents the contrast intensity of the h-th classification model. ih Let e ​​represent the contribution rate of the e-th emotion to the h-th classification model. C represents the mean contribution rate of the e-th emotion in the h-th classification model. h c represents the conflict index of the h-th classification model. jh This represents the correlation coefficient between the j-th and h-th classification models; The information content of the e-th sentiment in the h-th classification model can be obtained using the following formula: AND h =S h ×C h Among them, Y h This represents the amount of information about the e-th emotion in the h-th classification model; The weight of the e-th sentiment in the h-th classification model is obtained using the following formula: Where, γ eh This represents the weight of the e-th emotion in the h-th classification model; The output of the e-th emotion for the h-th classification model can be obtained using the following formula: Among them, Γ e This indicates that the test sample was assigned to the e-th emotion; S7.4: Establish a corresponding decision-making level fusion mechanism to perform human gait emotion recognition.

Images