Multi-modal emotion judgment HMI man-machine interaction method

Abstract

The invention provides a multi-modal emotion judgment HMI man-machine interaction method, and belongs to the technical field of multi-modal man-machine interaction, and the method comprises the steps: collecting facial features, recognizing facial key feature points, and combining an expression classification algorithm to achieve the judgment of the facial features; signals are processed by collecting electroencephalogram signals, and emotion judgment is carried out by combining signal changes of different frequency bands. Time axis synchronization of the face and electroencephalogram signals is achieved by recording log information in real time. Through data processing, features are respectively extracted from facial expressions and EEG (electroencephalogram) signals, and then the features are combined into a new feature vector for comprehensive judgment. According to the method, the recognition and prediction precision is improved, and the emotion judgment failure caused by face camouflage of the tested person can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal human-computer interaction, and in particular to a multimodal emotion judgment HMI human-computer interaction method. Background Art

[0002] In human-computer interaction, accurately identifying the user's emotional state is crucial to providing a personalized and efficient interactive experience. Traditional human-computer interaction methods mainly rely on a single input method, such as keyboard, mouse or touch screen, which to a certain extent limits the natural interaction between users and devices, and often makes it difficult to accurately capture the user's emotions, limiting the quality and effect of the interaction. At present, although there are some emotion recognition methods, most of them are based on a single modality, such as judging emotions only by facial expressions or voice analysis, and their accuracy and reliability are limited. There are some problems with existing multimodal human-computer interaction methods. For example, low data processing efficiency, inaccurate feature extraction, poor modal fusion effect, etc., these problems limit the widespread application of multimodal human-computer interaction technology. At the same time, the existing emotion recognition system also has shortcomings in real-time and interactivity with users, and it is difficult to meet the growing demand for intelligent interaction. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a multi-modal emotion judgment HMI human-computer interaction method.

[0004] The technical solution of the present invention is:

[0005] A multimodal emotion judgment HMI human-computer interaction method collects facial features, identifies key facial feature points and combines expression classification algorithms to realize facial feature judgment; collects electroencephalogram signals to realize signal processing and combines signal changes in different frequency bands to perform emotion judgment. The time axis synchronization of facial and electroencephalogram signals is achieved by real-time recording of log information. Through data processing, features are extracted from facial expressions and EEG signals respectively, and then these features are combined into a new feature vector for comprehensive judgment.

[0006] Furthermore,

[0007] By collecting EEG signals, the signal can be processed and emotion judgment can be made based on the changes in signals in different frequency bands.

[0008] The time axis synchronization of facial and EEG signals is achieved by recording log information in real time;

[0009] Through data processing, features are extracted from facial expressions and EEG signals respectively, and then these features are combined into a new feature vector for comprehensive judgment.

[0010] Going further,

[0011] The left and right branches are synchronized through the acquisition device, and the process is as follows:

[0012] (1) In the left branch, the EEG signals are collected from several channels in real time at a sampling rate of 128 Hz, and the raw data are saved. The raw data are further processed to obtain the characteristic vector A of the frequency band power of the EEG signal;

[0013] (2) In the right branch, by playing the video for a period of time, the camera collects the user's facial image in real time during the video playback process, uses face recognition to mark the positions of key points in the face, and generates a feature vector B;

[0014] (3) The feature vectors generated on the left and right sides are fused, and the fused vectors are input into the LSTM classifier, and finally the recognition results are output and transmitted to the HMI system for real-time display.

[0015] The facial features and EEG signal log synchronization are used, and the acquisition process is as follows:

[0016] (1) The subjects wore professional EEG acquisition equipment and watched the video played by the multimedia device in front of the camera, during which the subjects' emotional changes were observed in real time;

[0017] (2) When a change in the subject's facial features is observed, a log message with a timestamp will be recorded;

[0018] (3) Synchronous, the EEG signals are stored in a hard disk in a time-stamped form;

[0019] (4) The host computer will collect and record the EEG signal and facial features with log timestamps at the same time.

[0020] The process of facial feature collection and labeling is as follows:

[0021] The human face is marked with ten reference points (n1, n2, ..., n10), and the facial feature vector is composed of the timestamp ts in seconds and the relative distance R from the center point C. d The structure is shown in the following formula ①, and the facial feature vector FAFV is shown in formula ②

[0022]

[0023]

[0024] The processing of EEG signals should first extract the characteristic vector of signal frequency band power. Frequency band power refers to the signal power within the set frequency band range.

[0025] The process of extracting the frequency band power feature vector is:

[0026] (1) Filter the collected EEG signals and use fast Fourier transform to transform the signals from the time domain to the frequency domain to obtain the amplitude and phase of the signals at a specific frequency;

[0027] (2) The signal is divided into different independent frequency bands, and different bands represent different emotional characteristics;

[0028] (3) Calculate the power spectral density after FFT transformation. PSD gives the power distribution information of the signal at different frequencies. The power spectral density PSD(k) is calculated based on the discrete time series x(n) as shown in formula ③, where N is the signal length, including the entire wave period, n represents the discrete time series, and x(n) represents the sample value of the discrete time signal at time n, which is the original time domain signal. It represents a unit root on the complex plane and is an important unit for Fourier transform from time domain to frequency domain calculation. F represents frequency

[0029]

[0030] (4) Calculate the sum of the PSD values ​​at each frequency point within the specified frequency range of 4-50 Hz;

[0031] (5) Calculate the relative power spectrum density by dividing the PSD value corresponding to each frequency point by the sum of the PSD values ​​of all frequency points in the range of 4Hz-50Hz. The result is the relative band power rbp;

[0032] (6) Generate a timestamped band power feature vector (BPFV) as shown in formula ④, and calculate the relative band power rbp as shown in formula ⑤:

[0033] BPFV i =(ts,rbp1,rbp2,...,rbp n ) ④

[0034]

[0035] The fusion process of feature vectors:

[0036] (1) During the training process, the two feature vectors FAFV and BPFV are synchronized with the log to generate a fused feature vector FUFV, as shown in formula ⑥;

[0037] (2) Merging and aligning data streams: The two pairs of data, facial tags and EEG signals, are merged into one vector based on the ts timestamp. Data from different sources are synchronized in time so that they can be processed in the same time dimension and merged into a comprehensive multimodal vector MME. i , as shown in formula ⑦, it shows the merging and alignment of two streams;

[0038] (3) Multimodal Vector Metrics (MME) i It contains key information for judging emotions. After applying it to the long short-term memory (LTSM) classifier, the emotions at the moment can be identified; Multimodal vector MME i As shown in formula ⑥, the identified transient emotion value E i As shown in formula ⑧;

[0039]

[0040] MME i =(FAFV i ,BPFV i ) ⑦

[0041] E i =LSTM(MME i )⑧.

[0042] Data classification process:

[0043] The neural network RNN ​​is trained using long-term memory LSTM classification, and the emotions of the subjects are predicted based on the input fusion vector. By stacking multiple LSTM layers, the depth and expressiveness of the model can be increased. Each LSTM layer can learn feature representations at different levels, thereby improving the prediction accuracy of the model.

[0044] The specific process is:

[0045] (1) The fused vector is input into the embedding layer, and then into the 128-neuron LSTM layer A1, the 20% dropout layer B1, and then into step (2);

[0046] (2) Then input the second LSTM layer A2, with 64 neurons and 20% dropout layer B2, and go to step (3);

[0047] (3) Then input the third LSTM layer A3, with 32 neurons and 20% dropout layer B3, and go to step (4)

[0048] (4) The output of the third dropout layer is passed through a fully connected layer and an activation function to obtain the probability distribution of each sample belonging to different emotion categories, thereby realizing the prediction of emotions. The calculation process is as follows:

[0049] Suppose the matrix input into the fully connected layer is F, with a dimension of (batch_size, 32*seq_len), where batch_size represents the number of samples processed by the model at one time during training, seq_len represents the length of each data discretization, 32 represents the number of features of each sample, and the weight matrix w of the fully connected layer isfc and the bias vector b fc , assuming that n emotions are to be predicted, then w fc The dimension of b is (32*seq_len,n), fc The dimension is n, and the calculation formula of the full-connected layer is:

[0050] O=F×W fc +b fc

[0051] The dimension of O is (batch_size,n)

[0052] The softmax activation function is used to convert the output of the fully connected layer into a probability distribution. For each sample i in category j, the calculation formula is:

[0053]

[0054] For each sample, a probability vector of length n is obtained, which indicates the probability that the sample belongs to different emotion categories; the category with the largest probability value is the emotion category of the sample predicted by the model.

[0055] The beneficial effects of the present invention are

[0056] By combining facial recognition with EEG signal collection and adopting multimodal feature fusion, the accuracy of recognition and prediction is improved, which can solve the problem of emotional judgment failure caused by facial disguise of the subject. In addition, the system can be applied to other intelligent application fields, such as online education, where students can detect their preferences for learning a certain subject based on their emotional changes, so as to adjust their learning methods in a targeted manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0058] Figure 2 It is a schematic diagram of the process of synchronously collecting facial features and EEG signals;

[0059] Figure 3 It is a schematic diagram of face landmarks. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0061] The present invention proposes a multimodal emotion judgment HMI human-computer interaction method, which can realize the judgment of multiple emotions instead of relying solely on the facial expressions of the subject, and display the detection results in real time on the human-computer interaction interface HMI. The system collects facial expressions and electroencephalogram signals, synchronizes the information of the two, and comprehensively judges the emotions of the subject through multimodal feature vector fusion and LSTM classifier.

[0062] The present invention proposes a method for combining facial feature recognition and EEG signal data sources to perform emotion recognition and display it in HMI (Human-Machine Interaction). The camera collects facial images and professional equipment collects EEG signals. The two methods generate their own feature vectors respectively, and then combine the two feature vectors to create a fusion vector, and finally make a comprehensive judgment. The system multimodal real-time acquisition and recognition system flow chart is shown in the attached figure. Figure 1 shown.

[0063] Users need to wear professional data collection equipment, including Figure 1 The left and right branches are carried out simultaneously, and the detailed description of the process is as follows:

[0064] 1. In the left branch, professional equipment collects EEG signals from several channels in real time at a sampling rate of 128 Hz, saves the original data, and further processes the original data to obtain the characteristic vector A of the frequency band power of the EEG signal.

[0065] 2. In the right branch, by playing the video for a period of time, the camera collects the user's facial image in real time during the video playback process, uses face recognition to mark the positions of key points in the face, and generates a feature vector B.

[0066] 3. The feature vectors generated on the left and right sides are fused, and the fused vectors are input into the LSTM classifier, and finally the recognition results are output and transmitted to the HMI system for real-time display.

[0067] In the data collection process, the present invention proposes a method of synchronizing facial features and EEG signal log. The collection process is as shown in the attached figure. Figure 2 The detailed description is as follows:

[0068] 1. The subjects wear professional EEG acquisition equipment and watch the video played by multimedia equipment in front of a high-definition camera. During this time, observers observe the subjects' emotional changes in real time.

[0069] 2. When the observer observes that the subject has a change in facial features, a log message with a timestamp will be recorded

[0070] 3. Synchronously, the EEG signals will also be stored in the hard disk in a time-stamped form.

[0071] 4. The host computer will collect and record the EEG signal and facial features with log timestamps at the same time, thus realizing the synchronous collection of facial features and EEG signal data.

[0072] The process of facial feature collection and labeling is as follows:

[0073] As attached Figure 3 As shown, ten reference points (n1, n2, ..., n10) are marked on the face of a person, and the facial feature vector is composed of a timestamp ts in seconds and a relative distance R from the center point C. d The structure is shown in the following formula ①, and the facial feature vector FAFV is shown in formula ②

[0074]

[0075]

[0076] Among them, C x represents the x-coordinate value of the center point C, Cy represents the y-coordinate value of the center point C, and R di Indicates the relative distance between the i-th point and the center point C, n i,x Indicates the x-coordinate value of the i-th reference point, n i,y Indicates the y-coordinate value of the i-th reference point.

[0077] The process of processing the EEG signal is to first extract the characteristic vector of the signal band power. The band power refers to the signal power within a specific frequency band. The process of extracting the band power characteristic vector is:

[0078] 1. Filter the collected EEG signal and use fast Fourier transform to transform the signal from time domain to frequency domain to obtain the amplitude and phase of the signal at a specific frequency.

[0079] 2. Divide the signal into different independent frequency bands, for example, extract the 8-13Hz signal from the EEG signal. 8-12Hz is the alpha wave in the EEG signal, 13-30Hz is the beta wave, and above 30Hz is the gamma wave. Different bands represent different emotional characteristics.

[0080] 3. (3) Calculate the power spectral density after FFT transformation. PSD gives the power distribution information of the signal at different frequencies. The power spectral density PSD(k) is calculated based on the discrete time series x(n) as shown in formula ③, where N is the signal length, including the entire wave period, n represents the discrete time series, x(n) represents the sample value of the discrete time signal at time n, which is the original time domain signal. It represents a unit root on the complex plane and is an important unit for Fourier transform from time domain to frequency domain calculation. F represents frequency

[0081]

[0082] 4. Calculate the sum of the PSD values ​​at each frequency point within the specified frequency range of 4-50Hz.

[0083] 5. Calculate the relative power spectral density by dividing the PSD value corresponding to each frequency point by the sum of the PSD values ​​of all frequency points in the range of 4Hz-50Hz. The result is the relative band power (rbp).

[0084] 6. Generate a timestamped band power feature vector (BPFV) as shown in formula ④, and calculate the relative band power (rbp) as shown in formula ⑤.

[0085] BPFV i =(ts,rbp1,rbp2,...,rbp n ) ④

[0086]

[0087] The fusion process of feature vectors:

[0088] 1. During the training process, the two feature vectors FAFV and BPFV are synchronized with the log to generate a fused feature vector FUFV, as shown in formula ⑥

[0089] 2. Merge and align data streams. The two pairs of data, facial tags and EEG signals, are merged into one vector based on the ts timestamp. The data from different sources are synchronized in time so that they can be processed in the same time dimension and merged into a comprehensive multimodal vector MME. i , as shown in formula ⑦, it demonstrates the merging and alignment of two streams.

[0090] 3. Multi-modal vector MME i It contains key information for judging emotions. After applying it to the long short-term memory (LTSM) classifier, the emotion at the moment can be identified. Multimodal vector MME i As shown in formula ⑥, the identified transient emotion value E i As shown in formula ⑧

[0091]

[0092] MME i =(FAFV i ,BPFV i ) ⑦

[0093] E i =LSTM(MME i )⑧.

[0094] Data classification process:

[0095] Use the long-term memory LSTM classification to train the neural network RNN ​​and predict the subject's emotions based on the input fusion vector. By stacking multiple LSTM layers, the depth and expressiveness of the model can be increased. Each LSTM layer can learn feature representations at different levels, thereby improving the prediction accuracy of the model. The specific process is:

[0096] 1. The fused vector is input into the embedding layer, and then into the LSTM layer A1 with 128 neurons and the 20% dropout layer B1, and then proceed to step 22. Then the second LSTM layer A2 is input, with 64 neurons and the 20% dropout layer B2, and then proceed to step 33. Then the third LSTM layer A3 is input, with 32 neurons and the 20% dropout layer B3, and then proceed to step 44. The output of the third dropout layer is passed through a fully connected layer and an activation function to obtain the probability distribution of each sample belonging to different emotion categories, thereby realizing the prediction of emotions. The specific operation process is as follows:

[0097] Assume that the matrix input into the fully connected layer is F, with a dimension of (batch_size, 32*seq_len), where batch_size represents the number of samples processed by the model at one time during training, seq_len represents the length of each data discretization, 32 represents the number of features of each sample, and the weight matrix w of the fully connected layer is fc and the bias vector b fc , assuming that n emotions are to be predicted, then w fc The dimension of b is (32*seq_len,n), fc The dimension is n, and the calculation formula of the full-connected layer is:

[0098] O=F×W fc +b fc

[0099] The dimension of O is (batch_size,n)

[0100] The softmax activation function is used to convert the output of the fully connected layer into a probability distribution. For each sample i in category j, the calculation formula is:

[0101]

[0102] For each sample, a probability vector of length n is obtained, which indicates the probability that the sample belongs to different emotion categories. The category with the largest probability value is the emotion category predicted by the model for the sample. For example, there are 5 emotion categories (n=5). For a sample, the probability vector obtained by the above calculation is [0.1, 0.3, 0.2, 0.15, 0.25]. Then the model predicts that the sample is most likely to belong to the emotion category corresponding to the probability of 0.3.

[0103] Further explanation:

[0104] 1. Embedding layer function: The embedding layer can convert high-dimensional input vectors into low-dimensional continuous vector space, so as to better represent the characteristics of the input data. The embedding layer can learn the potential representation of the input data and improve the performance of the model.

[0105] 2. Multi-layer LSTM layers: The first LSTM layer receives the output of the embedding layer and learns the preliminary feature representation of the input data. It can capture long-term dependencies in the input sequence and provide richer information for subsequent LSTM layers. The second and third LSTM layers further process the output of the previous layer and learn more advanced feature representations. As the number of layers increases, the model can learn more complex patterns and relationships.

[0106] 3. Dropout layer: Adding a Dropout layer after each LSTM layer can randomly discard a certain proportion of neurons. This helps prevent the model from being too dependent on certain specific neurons or features and improves the generalization ability of the model.

[0107] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A multimodal emotion judgment HMI human-computer interaction method is characterized by: By collecting facial features, identifying key facial feature points and combining expression classification algorithms, facial features can be judged; By collecting EEG signals, the signal can be processed and emotion judgment can be made based on the changes in signals in different frequency bands. The time axis synchronization of facial and EEG signals is achieved by recording log information in real time; Through data processing, features are extracted from facial expressions and EEG signals respectively, and then these features are combined into a new feature vector for comprehensive judgment.

2. The method according to claim 1, characterized in that The left and right branches are synchronized through the acquisition device, and the process is as follows: (1) In the left branch, the EEG signals are collected from several channels in real time at a sampling rate of 128 Hz, and the raw data are saved. The raw data are further processed to obtain the characteristic vector A of the frequency band power of the EEG signal; (2) In the right branch, by playing the video for a period of time, the camera collects the user's facial image in real time during the video playback process, uses face recognition to mark the positions of key points in the face, and generates a feature vector B; (3) The feature vectors generated on the left and right sides are fused, and the fused vectors are input into the LSTM classifier, and finally the recognition results are output and transmitted to the HMI system for real-time display.

3. The method according to claim 2, characterized in that The facial features and EEG signal log synchronization are used, and the acquisition process is as follows: (1) The subjects wore professional EEG acquisition equipment and watched the video played by the multimedia device in front of the camera, during which the subjects' emotional changes were observed in real time; (2) When a change in the subject's facial features is observed, a log message with a timestamp will be recorded; (3) Synchronous, the EEG signals are stored in a hard disk in a time-stamped form; (4) The host computer will collect and record the EEG signal and facial features with log timestamps at the same time.

4. The method according to claim 3, characterized in that The process of facial feature collection and labeling is as follows: The human face is marked with ten reference points (n1, n2, ..., n10), and the facial feature vector is composed of the timestamp ts in seconds and the relative distance R from the center point C. d The structure is shown in the following formula ①, and the facial feature vector FAFV is shown in formula ② Among them, C x represents the x-coordinate value of the center point C, Cy represents the y-coordinate value of the center point C, and R di Indicates the relative distance between the i-th point and the center point C, n i,x Indicates the x-coordinate value of the i-th reference point, n i,y Indicates the y coordinate value of the i-th reference point The processing of EEG signals should first extract the characteristic vector of signal frequency band power. Frequency band power refers to the signal power within the set frequency band range.

5. The method according to claim 4, characterized in that The process of extracting the frequency band power feature vector is: (1) Filter the collected EEG signals and use fast Fourier transform to transform the signals from the time domain to the frequency domain to obtain the amplitude and phase of the signals at a specific frequency; (2) The signal is divided into different independent frequency bands, and different bands represent different emotional characteristics; (3) Calculate the power spectral density after FFT transformation. PSD gives the power distribution information of the signal at different frequencies. The power spectral density PSD(k) is calculated based on the discrete time series x(n) as shown in formula ③, where N is the signal length, including the entire wave period, n represents the discrete time series, and x(n) represents the sample value of the discrete time signal at time n, which is the original time domain signal. It represents a unit root on the complex plane and is an important unit for Fourier transform from time domain to frequency domain calculation. F represents frequency (4) Calculate the sum of the PSD values ​​at each frequency point within the specified frequency range of 4-50 Hz; (5) Calculate the relative power spectrum density by dividing the PSD value corresponding to each frequency point by the sum of the PSD values ​​of all frequency points in the range of 4Hz-50Hz. The result is the relative band power rbp; (6) Generate a timestamped band power feature vector (BPFV) as shown in formula ④, and calculate the relative band power rbp as shown in formula ⑤: BPFV i =(ts,rbp1,rbp2,...,rbp n ) ④ 6. The method according to claim 2, characterized in that The fusion process of feature vectors: (1) During the training process, the two feature vectors FAFV and BPFV are synchronized with the log to generate a fused feature vector FUFV, as shown in formula ⑥; (2) Merging and aligning data streams: The two pairs of data, facial tags and EEG signals, are merged into one vector based on the ts timestamp. Data from different sources are synchronized in time so that they can be processed in the same time dimension and merged into a comprehensive multimodal vector MME. i , as shown in formula ⑦, it shows the merging and alignment of two streams; (3) Multimodal Vector Metrics (MME) i It contains key information for judging emotions. After applying it to the long short-term memory (LTSM) classifier, the emotions at the moment can be identified; Multimodal vector MME i As shown in formula ⑥, the identified transient emotion value E i As shown in formula ⑧; FUFV i =(ts,R d1 ,R d2 ,...R d10 ,rbp1,rbp2,...rbp n ) ⑥ MME i =(FAFV i ,BPFV i ) ⑦ E i =LSTM(MME i )⑧。 7. The method according to claim 2, characterized in that The long-term memory LSTM classification neural network RNN ​​is used to train the emotions of the subjects based on the input fusion vector; the depth and expression ability of the model are increased by stacking several LSTM layers, and each LSTM layer learns feature representations at different levels, thereby improving the prediction accuracy of the model.

8. The method according to claim 7, characterized in that The specific steps are: (1) The fused vector is input into the embedding layer, and then into the 128-neuron LSTM layer A1, the 20% dropout layer B1, and then into step (2); (2) Then input the second LSTM layer A2, with 64 neurons and 20% dropout layer B2, and go to step (3); (3) Then input the third LSTM layer A3, with 32 neurons and 20% dropout layer B3, and go to step (4) (4) The output of the third dropout layer is passed through a fully connected layer and an activation function to obtain the probability distribution of each sample belonging to different emotion categories, thereby realizing the prediction of emotions.

9. The method according to claim 8, characterized in that The operation process is as follows: Suppose the matrix input into the fully connected layer is F, with a dimension of (batch_size, 32*seq_len), where batch_size represents the number of samples processed by the model at one time during training, seq_len represents the length of each data discretization, 32 represents the number of features of each sample, and the weight matrix w of the fully connected layer is fc and the bias vector b fc , assuming that n emotions are to be predicted, then w fc The dimension of b is (32*seq_len,n), fc The dimension is n, and the calculation formula of the full-connected layer is: O=F×W fc +b fc The dimension of O is (batch_size,n) The softmax activation function is used to convert the output of the fully connected layer into a probability distribution. For each sample i in category j, the calculation formula is: For each sample, a probability vector of length n is obtained, which indicates the probability that the sample belongs to different emotion categories; the category with the largest probability value is the emotion category of the sample predicted by the model.

Images