Physiological emotion recognition method and system

Abstract

The invention discloses a physiological emotion recognition method and system, and relates to the technical field of emotion recognition, and the method comprises the steps: carrying out the preprocessing of collected original physiological signal data, so as to obtain feature vector data; inputting a weighting result of the feature vector data subjected to weighting processing into a preset bidirectional long-short term memory network to obtain first-level hidden state data; transferring the first-level hidden state data to a preset graph attention network to obtain second-level hidden state data; performing compensation processing according to a preset compensation mechanism and the first-level hidden state data to generate compensation feature data, and performing integration processing on the compensation feature data and the second-level hidden state data to generate final state data; inputting the final state data into a classification layer to obtain sentiment classification; according to the method, the time decay factor and a compensation mechanism are combined, the emotion excitation information loss caused by time decay is corrected, and the time dependency relationship between information in an emotion recognition task is optimized, so that the emotion recognition performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to a physiological emotion recognition method and system. Background Art

[0002] Through physiological reactions, people can truly reflect their inner emotional state, such as subtle changes in facial expressions, the ups and downs of language, body language (gestures and postures), and even subtle fluctuations in physiological signals, thereby helping themselves and others to better understand each other's emotions. In social interactions, physiological emotional signals also enhance the emotional connection between people. Therefore, physiological emotions have an important impact on interpersonal communication, stress regulation, and anxiety relief.

[0003] With the development of science and technology, how to make emotion recognition equipment more accurately and efficiently recognize the emotions generated by users has become the key to the development of current emotion recognition technology. At present, most emotion recognition technologies use signal features in static or short time windows, but ignore the fact that emotional states will continue to evolve over time. For example, a Chinese patent with announcement number CN107945790B proposes an emotion recognition method and an emotion recognition system, which obtains the corresponding emotion type through voice signal recognition. Although the above method can realize emotion recognition, research and practical application of the above method and existing technologies have found that the existing technologies have at least the following defects:

[0004] (1) Human emotions fluctuate, and the response of physiological signals to emotional fluctuations has a certain lag and emotional attenuation. That is, after the feedback of emotion recognition, the physiological reaction does not reach a climax immediately, but goes through a process of alternating between strong and weak. This dynamic characteristic in time is prone to recognition errors during emotion recognition, thereby reducing the accuracy of emotion recognition.

[0005] (2) Physiological emotions based on temporal changes are not taken into account. The overall changes of emotions cannot be accurately identified through instantaneous physiological emotion recognition, which further leads to a decrease in the accuracy of emotion recognition. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a physiological emotion recognition method and system.

[0007] A physiological emotion recognition method, the method comprising:

[0008] Preprocessing the collected original physiological signal data to obtain feature vector data;

[0009] The weighted result of the weighted processing of the feature vector data is input into a preset bidirectional long short-term memory network to obtain the first-level hidden state data;

[0010] Input the first-level hidden state data into a preset graph attention network to obtain second-level hidden state data;

[0011] Perform compensation processing based on a preset compensation mechanism and the first-level hidden state data to generate compensation feature data, and integrate the compensation feature data with the second-level hidden state data to generate final state data;

[0012] Input the final state data into a classification layer to obtain sentiment classification.

[0013] Furthermore, preprocess the collected original physiological signal data, including:

[0014] Perform windowing on the original physiological signal data, segment it according to the window size Dx and the sliding step S to obtain N sliding windows, where S < Dx, and each sliding window represents where i represents the i-th sliding window, represents the data at the t-th time step of the i-th sliding window;

[0015] For each sliding window X (i) Perform frequency band division, using the delta band from 1 Hz to 4 Hz, the theta band from 4 Hz to 8 Hz, the alpha band from 8 Hz to 12 Hz, the beta band from 12 Hz to 30 Hz, and the gamma band from 30 Hz to 75 Hz as division nodes. On the basis of dividing the frequency bands, extract differential entropy features and then perform normalization processing to obtain a feature matrix and output it as feature vector data.

[0016] Furthermore, the acquisition logic of the weighted result is as follows:

[0017] P1: Set a time decay factor λ, where λ is a constant between 0 and 1. Weight the N sliding windows based on the time decay factor λ to obtain a time decay weight w. The logical formula for the time decay weight w is: w (i) = λ (N-i) , where w (i) is the time decay weight of the i-th sliding window;

[0018] P2: Perform comprehensive processing on the time decay weight w and the feature vector data DE (i) to generate a weighted feature matrix and output it as the weighted result, where the weighted result has the logical formula: where, is the weighted result of the i-th sliding window;

[0019] P3: Weighted results of N sliding windows Packaged as X w ∈R N×C A two-dimensional tensor of , where C is the number of channels.

[0020] Furthermore, the logic for obtaining the first-level hidden state data is:

[0021] Q1: Process the weighted result X from 1 to N according to the partitioning order of the sliding window through the forward LSTM w , for the i-th sliding window, the forward LSTM calculates the forward hidden data, and the logical formula of the forward hidden data is: in, is the forward hidden data of the forward LSTM in the i-th sliding window, LSTM f is the forward LSTM unit, It is the forward hidden data of the forward LSTM in the i-1th sliding window;

[0022] Q2: Use backward LSTM to process the weighted result X in reverse order from N to 1 according to the partitioning order of the sliding window w , for the i-th sliding window, the backward LSTM calculates its backward hidden data, and the logical formula of the backward hidden data is: in, is the backward hidden data of the backward LSTM in the i-th sliding window, LSTM b is the backward LSTM unit, is the backward hidden data of the backward LSTM in the i+1th sliding window;

[0023] Q3: For the i-th sliding window, the forward hidden data and the backward hidden data are concatenated to obtain the bidirectional hidden connection data of Bi-LSTM. The logical formula of the bidirectional hidden connection data is: Among them, h i is the bidirectional hidden connection data of the i-th sliding window, To concatenate the forward hidden data and the backward hidden data;

[0024] Q4: In the time dimension, perform the processing steps Q1 to Q3 on the N sliding windows to generate an output matrix H containing the bidirectional hidden connection data of all sliding windows, H = [h1,h2,…,h N ]∈R N×2M , where 2M is the hidden layer dimension of Bi-LSTM, H is the output sequence of Bi-LSTM, which contains the output of N sliding windows. The output of each sliding window is a 2M-dimensional vector, and the output matrix H is output as the first-level hidden state data.

[0025] Furthermore, the acquisition logic of the secondary hidden state data is:

[0026] J1: According to the bidirectional hidden connection data in the first-level hidden state data, each channel in the C channels of each bidirectional hidden connection data is set as a single node, and the node feature data of each node is obtained. The matrix of the node feature data is: K i =[h i,1 ,h i,2 ,…h i,C ]; where h i,C is the node feature data of the Cth channel node of sliding window i;

[0027] J2: Organize all channel nodes into a graph G = (V, E), and use the node feature data of the C channels in each sliding window to form the nodes of graph G. The node feature data h i,C and the node feature data h of other channels in the i-th sliding window i,C' Connect based on physical distance;

[0028] J3: Calculate the feature data h of each node through the graph attention network i,C Its neighbor node feature data h i,c' The attention weight between , the logical formula of the attention weight is: Where a is the attention weight, α (i,C),(i,C') is the node feature data h i,C For the adjacent node feature data h i,C' , W is the trainable weight matrix for linear transformation node features, || is the concatenation operation of the vector, LeakyReLU is an activation function with a negative slope, N (i,C) is the node feature data h i,C Adjacent node feature data set;

[0029] J4: Based on attention weight α (i,C),(i,C') , for node feature data h i,C The node features of are weighted summed to obtain the new node feature data of the channel node in the i-th sliding window. The logical formula of the new node feature data is:

[0030]

[0031] Where σ represents the activation function, W is the same as the weight matrix in step J3, and h 'i,C is the channel node h i,C New node feature data after being processed by the graph attention network;

[0032] J5: Calculate the mean of the feature vectors of all channel nodes of the i-th sliding window to obtain the global feature data of a sliding window. The logical formula of the global feature data is: Wherein, β is an integer greater than 0 and less than C; the same steps are performed for each sliding window to generate corresponding global feature data, and a global feature data matrix arranged in the order of the sliding windows is obtained. The matrix formula of the global feature data matrix is: H'=[h'1,h'2…,h' N ], where H' is the output matrix after graph attention network processing, N is the total number of sliding windows, and the global feature data matrix H' is output as the secondary hidden state data.

[0033] Furthermore, the acquisition logic of the compensation feature data is:

[0034] According to the time attenuation factor λ, the lost data is calculated and the logical formula for compensating the characteristic data is: i =(1-λ)W r h i +b r , where r i For lost data, W r is the compensation weight matrix, b r is the bias term, h i is the bidirectional hidden connection data of the i-th sliding window, and λ is the time decay factor.

[0035] Furthermore, the compensation feature data is integrated with the secondary hidden state data to generate final state data, including:

[0036] According to the global feature data h' in the compensation feature data and the secondary hidden state data i Adjust to generate adjustment data c i , the c i The logical formula is: i =λh′ i +(1-λ)r i ; Based on the adjustment data c i and global feature data h′ i Perform serial fusion to calculate and obtain composite global feature data. The calculation formula of the composite global feature data is: i =λh′ i +(1-λ)r i +h′ i ; Perform the above processing on each channel node to generate a composite global feature data matrix H”=[h″1,h″2,…,h″ N ] and output the composite global feature data matrix as the final state data.

[0037] Furthermore, the final state data is input into the classification layer to obtain sentiment classification, including:

[0038] The final state data is input into the fully connected layer to obtain the score of each emotion category. The logical formula of the fully connected layer is: l =W c h″ i +b c ; Among them, y l is the original output of the classification layer, W c is the weight matrix of the classification layer, b c is the bias term of the classification layer, which converts the output y of the fully connected layer l The softmax is converted into a probability distribution to calculate the predicted probability of each category. The calculation formula of the predicted probability is: Among them, y p,i is the predicted probability of the i-th class, y l,i is the score of the i-th class.

[0039] A physiological emotion recognition system, used to perform any physiological emotion recognition method, the system comprising:

[0040] Preprocessing module: preprocess the collected original physiological signal data to obtain feature vector data;

[0041] Primary analysis module: input the weighted result of the feature vector data after weighted processing into the preset bidirectional long short-term memory network to obtain the primary hidden state data;

[0042] Secondary analysis module: used to pass the primary hidden state data to the preset graph attention network to obtain the secondary hidden state data;

[0043] Data compensation module: used to perform compensation processing according to the preset compensation mechanism and the first-level hidden state data, generate compensation feature data, integrate the compensation feature data with the second-level hidden state data, and generate final state data;

[0044] Sentiment classification module: used to input the final state data into the classification layer to obtain sentiment classification

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention realizes adaptive learning of key time windows by weighting the sliding window with a time decay factor. The application of the time decay factor enables the system to dynamically adjust the emotion weights of different time points by adjusting the value of the time decay factor, effectively distinguishing and emphasizing those key time periods that have a decisive influence on the overall emotional state, thereby significantly improving the accuracy and reliability of emotion recognition. In addition, the present invention ensures that the model can not only utilize important information at the current moment, but also retain and compensate for historical information weakened or lost by time by designing a compensation mechanism, thereby avoiding excessive weakening or loss of important emotional information due to time decay, and ensuring the comprehensiveness and continuity of the emotional analysis process. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0048] Figure 1 A flowchart of a physiological emotion recognition method provided in Example 1 of the present invention;

[0049] Figure 2 A module diagram of a physiological emotion recognition system provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Example 1

[0052] See also Figure 1 As shown, this embodiment discloses a physiological emotion recognition method, the method comprising:

[0053] S110: preprocessing the collected original physiological signal data to obtain feature vector data;

[0054] Specifically, the collected raw physiological signal data is preprocessed, including:

[0055] The original physiological signal data is windowed and segmented according to the window size Dx and the sliding step S to obtain N sliding windows, where S < Dx, and each sliding window represents where i represents the i-th sliding window, represents the data at the t-th time step of the i-th sliding window;

[0056] For each sliding window X (i) Frequency band division is performed. Using the delta band from 1 Hz to 4 Hz, the theta band from 4 Hz to 8 Hz, the alpha band from 8 Hz to 12 Hz, the beta band from 12 Hz to 30 Hz, and the gamma band from 30 Hz to 75 Hz as division nodes, differential entropy feature extraction is performed on the basis of the divided frequency bands, and then normalization processing is carried out to obtain the feature matrix and output it as feature vector data;

[0057] It should be noted that: the normalization processing includes but is not limited to Z-score standardization, L2 normalization, L1 normalization, and logarithmic normalization.

[0058] S120: Input the weighted result of the feature vector data after weighted processing into a preset bidirectional long short-term memory network to obtain first-level hidden state data;

[0059] Specifically, the acquisition logic of the weighted result is as follows:

[0060] P1: Set the time decay factor λ, where λ is a constant between 0 and 1. Weight the N sliding windows based on the time decay factor λ to obtain the time decay weight w. The logical formula for the time decay weight w is: w (i) = λ (N-i) where w (i) is the time decay weight of the i-th sliding window;

[0061] It should be noted that: the closer the time decay factor λ is to 1, the smaller the weight difference between the sliding windows; the closer the time decay factor λ is to 0, the weight of the sliding window will decay significantly;

[0062] P2: Perform comprehensive processing on the time decay weight w and the feature vector data DE (i) to generate the weighted feature matrix and output it as the weighted result, where the weighted result has the logical formula: where is the weighted result of the i-th sliding window;

[0063] It is important to understand that this step scales the features of the entire sliding window, reflecting the relative importance of the window in the time series.

[0064] P3: Weighted results of N sliding windows Packaged as X w ∈R N×C A two-dimensional tensor of , where C is the number of channels;

[0065] Specifically, the logic for obtaining the first-level hidden state data is:

[0066] Q1: Process the weighted result X from 1 to N according to the partitioning order of the sliding window through the forward LSTM w , for the i-th sliding window, the forward LSTM calculates the forward hidden data, and the logical formula of the forward hidden data is: in, is the forward hidden data of the forward LSTM in the i-th sliding window, LSTM f is the forward LSTM unit, It is the forward hidden data of the forward LSTM in the i-1th sliding window;

[0067] Q2: Use backward LSTM to process the weighted result X in reverse order from N to 1 according to the partitioning order of the sliding window w , for the i-th sliding window, the backward LSTM calculates its backward hidden data, and the logical formula of the backward hidden data is: in, is the backward hidden data of the backward LSTM in the i-th sliding window, LSTM b is the backward LSTM unit, is the backward hidden data of the backward LSTM in the i+1th sliding window;

[0068] Q3: For the i-th sliding window, the forward hidden data and the backward hidden data are concatenated to obtain the bidirectional hidden connection data of Bi-LSTM. The logical formula of the bidirectional hidden connection data is: Among them, h i is the bidirectional hidden connection data of the i-th sliding window, To concatenate the forward hidden data and the backward hidden data;

[0069] Q4: In the time dimension, perform the processing steps Q1 to Q3 on the N sliding windows to generate an output matrix H containing the bidirectional hidden connection data of all sliding windows, H = [h1,h2,…,h N ]∈R N×2M, where 2M is the hidden layer dimension of Bi-LSTM, H is the output sequence of Bi-LSTM, which contains the output of N sliding windows. The output of each sliding window is a 2M-dimensional vector, and the output matrix H is output as the first-level hidden state data.

[0070] S130: Sending the first-level hidden state data to a preset graph attention network to obtain second-level hidden state data;

[0071] Specifically, the acquisition logic of the secondary hidden state data is:

[0072] J1: According to the bidirectional hidden connection data in the first-level hidden state data, each channel in the C channels of each bidirectional hidden connection data is set as a single node, and the node feature data of each node is obtained. The matrix of the node feature data is: K i =[h i,1 ,h i,2 ,…h i,C ]; where h i,C is the node feature data of the Cth channel node of sliding window i;

[0073] J2: Organize all channel nodes into a graph G = (V, E), and use the node feature data of the C channels in each sliding window to form the nodes of graph G. The node feature data h i,C and the node feature data h of other channels in the i-th sliding window i,C′ Connect based on physical distance;

[0074] J3: Calculate the feature data h of each node through the graph attention network i,C Its neighbor node feature data h i,C' The attention weight between , the logical formula of the attention weight is: Where a is the attention weight, α (i,C),(i,C') is the node feature data h i,C For the adjacent node feature data h i,C' , W is the trainable weight matrix for linear transformation node features, || is the concatenation operation of the vector, LeakyReLU is an activation function with a negative slope, N (i,C) is the node feature data h i,C Adjacent node feature data set;

[0075] It should be noted that: the attention weight reflects the strength of the relationship between the feature data of a node and its adjacent node feature data; the graph attention network includes but is not limited to self-attention mechanism, graph convolution network, deep learning framework, graph embedding technology, regularization technology;

[0076] J4: Based on attention weight α (i,C),(i,C') , for node feature data h i,C The node features of are weighted summed to obtain the new node feature data of the channel node in the i-th sliding window. The logical formula of the new node feature data is:

[0077]

[0078] Where σ represents the activation function, W is the same as the weight matrix in step J3, and h' i,C is the channel node h i,C New node feature data after being processed by the graph attention network;

[0079] J5: Calculate the mean of the feature vectors of all channel nodes of the i-th sliding window to obtain the global feature data of a sliding window. The logical formula of the global feature data is: Wherein, β is an integer greater than 0 and less than C; the same steps are performed for each sliding window to generate corresponding global feature data, and a global feature data matrix arranged in the order of the sliding windows is obtained. The matrix formula of the global feature data matrix is: H'=[h'1,h'2…,h' N ], where H' is the output matrix after graph attention network processing, N is the total number of sliding windows, and the global feature data matrix H' is output as the secondary hidden state data.

[0080] S140: performing compensation processing according to a preset compensation mechanism and the first-level hidden state data to generate compensation feature data, and integrating the compensation feature data with the second-level hidden state data to generate final state data;

[0081] Specifically, the acquisition logic of the compensation feature data is:

[0082] According to the time attenuation factor λ, the lost data is calculated and the logical formula for compensating the characteristic data is: i =(1-λ)W r h i +b r , where r i For lost data, W r is the compensation weight matrix, b r is the bias term, h i is the bidirectional hidden connection data of the i-th sliding window, and λ is the time decay factor.

[0083] It should be noted that the compensation weight matrix W r Acquired through experimental records conducted by operators;

[0084] Specifically, the compensation feature data and the secondary hidden state data are integrated and processed to generate final state data, including:

[0085] According to the global feature data h' in the compensation feature data and the secondary hidden state data i Adjust to generate adjustment data c i , the c i The logical formula is: i =λh' i +(1-λ)r i ; Based on the adjustment data c i and global feature data h' i Perform serial fusion to calculate and obtain composite global feature data. The calculation formula of the composite global feature data is: i '=λh' i +(1-λ)r i +h' i ; Perform the above processing on each channel node to generate a composite global feature data matrix H”=[h″1'h″2,…,h″ N ], and output the composite global feature data matrix as the final state data;

[0086] S150: Input the final state data into the classification layer to obtain sentiment classification;

[0087] Specifically, the final state data is input into the classification layer to obtain sentiment classification, including:

[0088] The final state data is input into the fully connected layer to obtain the score of each emotion category. The logical formula of the fully connected layer is: l =W c h″ i +b c ; Among them, y l is the original output of the classification layer, W c is the weight matrix of the classification layer, b c is the bias term of the classification layer, which converts the output y of the fully connected layer l The softmax is converted into a probability distribution to calculate the predicted probability of each category. The calculation formula of the predicted probability is: Among them, y p,i is the predicted probability of the i-th class, y l,i is the score of the i-th class.

[0089] Example 2

[0090] See also Figure 2As shown, based on a unified inventive concept, this embodiment discloses a physiological emotion recognition system, which includes a preprocessing module, a primary analysis module, a secondary analysis module, a data compensation module, and an emotion classification module:

[0091] Preprocessing module S210: Preprocess the collected original physiological signal data to obtain feature vector data;

[0092] Specifically, preprocessing the collected original physiological signal data includes:

[0093] Perform windowing on the original physiological signal data, divide it according to the window size Dx and the sliding step S to obtain N sliding windows, where S < Dx, and each sliding window represents where i represents the i-th sliding window, represents the data of the t-th time step of the i-th sliding window;

[0094] For each sliding window X (i) Perform frequency band division, use the delta band from 1 Hz to 4 Hz, the theta band from 4 Hz to 8 Hz, the alpha band from 8 Hz to 12 Hz, the beta band from 12 Hz to 30 Hz, and the gamma band from 30 Hz to 75 Hz as division nodes, extract differential entropy features on the basis of dividing the frequency bands, and then perform normalization processing to obtain a feature matrix and output it as feature vector data.

[0095] Primary analysis module S220: Input the weighted result of the feature vector data after weighted processing into a preset bidirectional long short-term memory network to obtain primary hidden state data;

[0096] Specifically, the acquisition logic of the weighted result is as follows:

[0097] P1: Set the time decay factor λ, where λ is a constant between 0 and 1, and weight the N sliding windows based on the time decay factor λ to obtain the time decay weight w. The logical formula for the time decay weight w is: w (i) = λ (N-i) , where w (i) is the time decay weight of the i-th sliding window;

[0098] P2: Perform comprehensive processing on the time decay weight w and the feature vector data DE (i) to generate a weighted feature matrix and output it as the weighted result, where the weighted result has the logical formula: where, is the weighted result of the i-th sliding window;

[0099] P3: Weighted results of N sliding windows Packaged as X w ∈R N×C A two-dimensional tensor of , where c is the number of channels;

[0100] Specifically, the logic for obtaining the first-level hidden state data is:

[0101] Q1: Process the weighted result X from 1 to N according to the partitioning order of the sliding window through the forward LSTM w , for the i-th sliding window, the forward LSTM calculates the forward hidden data, and the logical formula of the forward hidden data is: in, is the forward hidden data of the forward LSTM in the i-th sliding window, LSTM f is the forward LSTM unit, It is the forward hidden data of the forward LSTM in the i-1th sliding window;

[0102] Q2: Use backward LSTM to process the weighted result X in reverse order from N to 1 according to the partitioning order of the sliding window w , for the i-th sliding window, the backward LSTM calculates its backward hidden data, and the logical formula of the backward hidden data is: in, is the backward hidden data of the backward LSTM in the i-th sliding window, LSTM b is the backward LSTM unit, is the backward hidden data of the backward LSTM in the i+1th sliding window;

[0103] Q3: For the i-th sliding window, the forward hidden data and the backward hidden data are concatenated to obtain the bidirectional hidden connection data of Bi-LSTM. The logical formula of the bidirectional hidden connection data is: Among them, h i is the bidirectional hidden connection data of the i-th sliding window, To concatenate the forward hidden data and the backward hidden data;

[0104] Q4: In the time dimension, perform the processing steps Q1 to Q3 on the N sliding windows to generate an output matrix H containing the bidirectional hidden connection data of all sliding windows, H = [h1,h2,…,h N ]∈R N×2M, where 2M is the hidden layer dimension of Bi-LSTM, H is the output sequence of Bi-LSTM, which contains the output of N sliding windows. The output of each sliding window is a 2M-dimensional vector, and the output matrix H is output as the first-level hidden state data.

[0105] Secondary analysis module S230: used to pass the primary hidden state data to a preset graph attention network to obtain secondary hidden state data;

[0106] Specifically, the acquisition logic of the secondary hidden state data is:

[0107] J1: According to the bidirectional hidden connection data in the first-level hidden state data, each channel in the C channels of each bidirectional hidden connection data is set as a single node, and the node feature data of each node is obtained. The matrix of the node feature data is: K i =[h i,1 ,h i,2 ,…h i,C ]; where h i,C is the node feature data of the Cth channel node of sliding window i;

[0108] J2: Organize all channel nodes into a graph G = (V, E), and use the node feature data of the C channels in each sliding window to form the nodes of graph G. The node feature data h i,C and the node feature data h of other channels in the i-th sliding window i,C' Connect based on physical distance;

[0109] J3: Calculate the feature data h of each node through the graph attention network i,c Its neighbor node feature data h i,c' The attention weight between , the logical formula of the attention weight is: Where a is the attention weight, α (i,c),(i,c') is the node feature data h i,C For the adjacent node feature data h i,C' , W is the trainable weight matrix for linear transformation node features, || is the concatenation operation of the vector, LeakyReLU is an activation function with a negative slope, N (i,C) is the node feature data h i,C Adjacent node feature data set,

[0110] J4: Based on attention weight α (i,C),(i,C′) , for node feature data h i,C The node features of are weighted summed to obtain the new node feature data of the channel node in the i-th sliding window. The logical formula of the new node feature data is:

[0111]

[0112] Where σ represents the activation function, W is the same as the weight matrix in step J3, and h′ i,C is the channel node h i,C New node feature data after being processed by the graph attention network;

[0113] J5: Calculate the mean of the feature vectors of all channel nodes of the i-th sliding window to obtain the global feature data of a sliding window. The logical formula of the global feature data is: Wherein, β is an integer greater than 0 and less than C; the same steps are performed for each sliding window to generate corresponding global feature data, and a global feature data matrix arranged in the order of the sliding windows is obtained. The matrix formula of the global feature data matrix is: H′=[h′1,h′2…,h′ N ], where H′ is the output matrix after being processed by the graph attention network, N is the total number of sliding windows, and the global feature data matrix H′ is output as the secondary hidden state data;

[0114] Data compensation module S240: used to perform compensation processing according to a preset compensation mechanism and the first-level hidden state data, generate compensation feature data, integrate the compensation feature data with the second-level hidden state data, and generate final state data;

[0115] Specifically, the acquisition logic of the compensation feature data is:

[0116] According to the time attenuation factor λ, the lost data is calculated and the logical formula for compensating the characteristic data is: i =(1-λ)W r h i +b r , where r i For lost data, W r is the compensation weight matrix, b r is the bias term, h i is the bidirectional hidden connection data of the i-th sliding window, and λ is the time decay factor.

[0117] Specifically, the compensation feature data and the secondary hidden state data are integrated and processed to generate final state data, including:

[0118] According to the global feature data h′ in the compensated feature data and the secondary hidden state data i Adjust to generate adjustment data c i , the c i The logical formula is: i =λh′i +(1-λ)r i ; Based on the adjustment data c i and global feature data h′ i Perform serial fusion to calculate and obtain composite global feature data. The calculation formula of the composite global feature data is: i =λh′ i +(1-λ)r i +h′ i ; Perform the above processing on each channel node to generate a composite global feature data matrix H”=[h″1,h″2,…,h” N ], and output the composite global feature data matrix as the final state data;

[0119] Sentiment classification module S250: used to input the final state data into the classification layer to obtain sentiment classification;

[0120] Specifically, the final state data is input into the classification layer to obtain sentiment classification, including:

[0121] The final state data is input into the fully connected layer to obtain the score of each emotion category. The logical formula of the fully connected layer is: l =W c h″ i +b c ; Among them, y l is the original output of the classification layer, W c is the weight matrix of the classification layer, b c is the bias term of the classification layer, which converts the output y of the fully connected layer l The softmax is converted into a probability distribution to calculate the predicted probability of each category. The calculation formula of the predicted probability is: Among them, y p,i is the predicted probability of the i-th class, y l,i is the score of the i-th class.

[0122] Real-time Example 3

[0123] In order to further illustrate the effectiveness of the present invention, the present invention is described from the implementation process of a single signal (EEG) and a multi-modal physiological signal (EEG, EMG, ECG and EOG), and the two methods and other steps are unified invention concepts;

[0124] Only the cross-fusion of two modalities is realized in the acquisition of multimodal signal feature attention mechanism, and the different operations when applying graph attention network processing are emphasized;

[0125] Among them, the processing method of a single signal (EEG) mainly focuses on the relationship between time steps and the weighting of features. The specific steps are the same as above, while multimodal signals (EEG, EMG, ECG and EOG) need to fully consider the features of different modalities and the semantic associations between modalities. To this end, a graph cross-attention network is used to calculate the feature expressions of different modal data through self-attention mechanism and cross-attention mechanism, thereby improving the model's capture and robustness of emotional features. The specific steps are as follows:

[0126] Step S310: constructing a graph structure;

[0127] The different modal data of each sliding window form corresponding nodes. There are N sliding windows, and EEG, EMG, ECG and EOG each have C E , C M , C C and C O channels, then there are a total of C E +C M +C C +C O nodes;

[0128] Among them, the feature of each node is expressed as Where m represents the mode, i represents the sliding window index, and j represents the channel index within the mode;

[0129] It should be noted that in this graph structure, the type of edges constructed are: edges are established between different channel nodes of the same mode and between corresponding channel nodes of different modes based on signal correlation.

[0130] Step S320: Formulate and calculate the self-attention of EEG, EMG, ECG and EOG multi-modal, the calculation formula is:

[0131]

[0132] Among them, Q, K and V come from the same mode; Represents the feature dimension.

[0133] Step S330: Use the cross attention mechanism to realize the interaction of EGG and EOG multi-modal data. The logic formula is:

[0134]

[0135] Among them, A EEG→EMG Represents the modal fusion features of EEG and EMG, A EEG→ECG represents the modality fusion features of EEG and ECG, A EEG→EOG represents the modality fusion features of EEG and EOG, A EMG→ECGrepresents the modal fusion features of EMG and ECG, A EMG→EOG represents the modal fusion features of EMG and EOG, A ECG→ECG Represents the modality fusion features of ECG and EOG.

[0136] Step S340: For the i-th sliding window, the fusion features obtained in step S320 are fused in a splicing manner to obtain the final multimodal feature representation, which is shown as follows:

[0137]

[0138] in, Represents feature concatenation.

[0139] Step S350: After performing the above operation on all sliding windows, a new feature representation matrix H′=[h′1,h′2…,h′ N ], where H′ is the output matrix after GAT processing and N represents the total number of sliding windows.

[0140] In summary, the present invention realizes adaptive learning of key time windows by weighting the sliding window with the time decay factor. The application of the time decay factor enables the system to dynamically adjust the emotion weights at different time points by adjusting the value of the time decay factor, effectively distinguishing and emphasizing those key time periods that have a decisive influence on the overall emotional state, thereby significantly improving the accuracy and reliability of emotion recognition. In addition, the present invention ensures that the model can not only utilize important information at the current moment, but also retain and compensate for historical information weakened or lost by time by designing a compensation mechanism, thereby avoiding excessive weakening or loss of important emotional information due to time decay, and ensuring the comprehensiveness and continuity of the emotional analysis process.

[0141] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through a wired network or a wireless network. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD) or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0142] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a physiological emotion recognition method and system. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0147] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A physiological emotion recognition method, characterized in that: The method comprises: Preprocessing the collected original physiological signal data to obtain feature vector data; The weighted result of the weighted processing of the feature vector data is input into a preset bidirectional long short-term memory network to obtain the first-level hidden state data; The first-level hidden state data is fed into the preset graph attention network to obtain the second-level hidden state data; Perform compensation processing according to a preset compensation mechanism and the first-level hidden state data to generate compensation feature data, and integrate the compensation feature data with the second-level hidden state data to generate final state data; The final state data is fed into the classification layer to obtain the sentiment classification.

2. A physiological emotion recognition method according to claim 1, characterized in that: Preprocess the collected raw physiological signal data, including: The original physiological signal data is windowed and divided according to the window size Dx and the sliding step size S to obtain N sliding windows, where S < Dx, and each sliding window represents Among them, i represents the i-th sliding window, Represents the data of the tth time step of the i-th sliding window; For each sliding window X (i) Frequency band division is performed, with the delta band of 1Hz to 4Hz, theta band of 4Hz to 8Hz, the alpha band of 8Hz to 12Hz, the beta band of 12Hz to 30Hz, and the gamma band of 30Hz to 75Hz as division nodes. Differential entropy feature extraction is performed on the basis of the divided frequency bands, and then normalization is performed to obtain the feature matrix And output it as feature vector data.

3. A physiological emotion recognition method according to claim 2, characterized in that: The logic for obtaining weighted results is: P1: Set the time decay factor λ, where λ is a constant between 0 and 1. Weight the N sliding windows based on the time decay factor λ to obtain the time decay weight w. The logical formula of the time decay weight w is: w (i) =λ (N-i) , where w (i) is the time decay weight of the i-th sliding window; P2: Combine the time decay weight w with the feature vector data DE (i) Perform comprehensive processing to generate a weighted feature matrix And output as a weighted result, where the weighted result The logical formula is: in, is the weighted result of the i-th sliding window; P3: Weighted results of N sliding windows Packaged as X w ∈R N×C A two-dimensional tensor of , where C is the number of channels.

4. A physiological emotion recognition method according to claim 3, characterized in that: The logic for obtaining the first-level hidden state data is: Q1: Process the weighted result X from 1 to N according to the partitioning order of the sliding window through the forward LSTM w , for the i-th sliding window, the forward LSTM calculates the forward hidden data, and the logical formula of the forward hidden data is: in, is the forward hidden data of the forward LSTM in the i-th sliding window, LSTM f is the forward LSTM unit, It is the forward hidden data of the forward LSTM in the i-1th sliding window; Q2: Use backward LSTM to process the weighted result X in reverse order from N to 1 according to the partitioning order of the sliding window w , for the i-th sliding window, the backward LSTM calculates its backward hidden data, and the logical formula of the backward hidden data is: in, is the backward hidden data of the backward LSTM in the i-th sliding window, LSTM b is the backward LSTM unit, is the backward hidden data of the backward LSTM in the i+1th sliding window; Q3: For the i-th sliding window, the forward hidden data and the backward hidden data are concatenated to obtain the bidirectional hidden connection data of Bi-LSTM. The logical formula of the bidirectional hidden connection data is: Among them, h i is the bidirectional hidden connection data of the i-th sliding window, To concatenate the forward hidden data and the backward hidden data; Q4: In the time dimension, perform the processing steps Q1 to Q3 on the N sliding windows to generate an output matrix H containing the bidirectional hidden connection data of all sliding windows, H = [h1, h2, ..., h N ]∈R N×2M , where 2M is the hidden layer dimension of Bi-LSTM, H is the output sequence of Bi-LSTM, which contains the output of N sliding windows. The output of each sliding window is a 2M-dimensional vector, and the output matrix H is output as the first-level hidden state data.

5. A physiological emotion recognition method according to claim 4, characterized in that: The acquisition logic of the secondary hidden state data is: J1: According to the bidirectional hidden connection data in the first-level hidden state data, each channel in the C channels of each bidirectional hidden connection data is set as a single node, and the node feature data of each node is obtained. The matrix of the node feature data is: K i =[h i，1 ,h i，2 ,…h i，C ]; where h i,C is the node feature data of the Cth channel node of sliding window i; J2: Organize all channel nodes into a graph G = (V, E), and use the node feature data of the C channels in each sliding window to form the nodes of graph G. The node feature data h i，C and the node feature data h of other channels in the i-th sliding window i，C′ Connect based on physical distance; J3: Calculate the feature data h of each node through the graph attention network i，C Its neighbor node feature data h i，C′ The attention weight between , the logical formula of the attention weight is: Where a is the attention weight, α (i，C)，(i，C′) is the node feature data h i，C For the adjacent node feature data h i，C′ , W is the trainable weight matrix for linear transformation node features, || is the concatenation operation of the vector, LeakyReLU is an activation function with a negative slope, N (i，C) is the node feature data h i，C Adjacent node feature data set; J4: Based on attention weight α (i，C)，(i，C′) , for node feature data h i，C The node features of are weighted summed to obtain the new node feature data of the channel node in the i-th sliding window. The logical formula of the new node feature data is: Where σ represents the activation function, W is the same as the weight matrix in step J3, and h′ i，C is the channel node h i，C New node feature data after being processed by the graph attention network; J5: Calculate the mean of the feature vectors of all channel nodes of the i-th sliding window to obtain the global feature data of a sliding window. The logical formula of the global feature data is: Wherein, β is an integer greater than 0 and less than C; the same steps are performed for each sliding window to generate corresponding global feature data, and a global feature data matrix arranged in the order of the sliding windows is obtained, and the matrix formula of the global feature data matrix is: H′=[h′1,h′2…,h′ N ], where H′ is the output matrix after graph attention network processing, N is the total number of sliding windows, and the global feature data matrix H′ is output as the secondary hidden state data.

6. A physiological emotion recognition method according to claim 5, characterized in that: The logic for obtaining compensation feature data is: According to the time attenuation factor λ, the lost data is calculated and the logical formula for compensating the characteristic data is: i =(1-λ)W r h i +b r , where r i For lost data, W r is the compensation weight matrix, b r is the bias term, h i is the bidirectional hidden connection data of the i-th sliding window, and λ is the time decay factor.

7. A physiological emotion recognition method according to claim 6, characterized in that: The compensation feature data is integrated with the secondary hidden state data to generate the final state data, including: According to the global feature data h′ in the compensated feature data and the secondary hidden state data i Adjust to generate adjustment data c i , the c i The logical formula is: i =λh′ i +(1-λ)r i ; Based on the adjustment data c i and global feature data h′ i Perform serial fusion to calculate and obtain composite global feature data. The calculation formula of the composite global feature data is: h″ i =λh′ i +(1-λ)r i +h′ i ; Perform the above processing on each channel node to generate a composite global feature data matrix H″=[h″1,h″2,…,h″ N ] and output the composite global feature data matrix as the final state data.

8. A physiological emotion recognition method according to claim 7, characterized in that: The final state data is fed into the classification layer to obtain sentiment classification, including: The final state data is input into the fully connected layer to obtain the score of each emotion category. The logical formula of the fully connected layer is: l =W c h″ i +b c ; Among them, y l is the original output of the classification layer, W c is the weight matrix of the classification layer, b c is the bias term of the classification layer, which converts the output y of the fully connected layer l The softmax is converted into a probability distribution to calculate the predicted probability of each category. The calculation formula of the predicted probability is: Among them, y p，i is the predicted probability of the i-th class, y l，i is the score of the i-th class.

9. A physiological emotion recognition system for executing any one of the physiological emotion recognition methods of claims 1-8, characterized in that: The system comprises: Preprocessing module: preprocess the collected original physiological signal data to obtain feature vector data; Primary analysis module: input the weighted result of the feature vector data after weighted processing into the preset bidirectional long short-term memory network to obtain the primary hidden state data; Secondary analysis module: used to pass the primary hidden state data to the preset graph attention network to obtain the secondary hidden state data; Data compensation module: used to perform compensation processing according to the preset compensation mechanism and the first-level hidden state data, generate compensation feature data, integrate the compensation feature data with the second-level hidden state data, and generate final state data; Sentiment classification module: used to input the final state data into the classification layer to obtain sentiment classification.

Images