A method and system for multi-vital sign emotion recognition based on temporal features

Abstract

This invention discloses a method and system for multi-vital sign emotion recognition based on temporal features. The method includes: performing temporal fusion and feature extraction on multiple vital signs using a deep convolutional neural network to obtain fused deep features; optimizing the fused deep features using an attention neural network model based on temporal feature capture to obtain attention fusion features; and performing dimensionality reduction feature extraction and mapping on the attention fusion features using a deep neural network model to obtain the emotion recognition result. This invention uses readily available multi-vital sign signals as the basis for emotion recognition and implements emotion recognition based on artificial intelligence algorithms.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to a method and system for multi-vital sign emotion recognition based on temporal features. Background Technology

[0002] Currently, both domestically and internationally, image-based visual information is still the primary factor for judging emotions, relying on facial expressions to determine feelings. However, this method is significantly affected by lighting angles and camera shooting angles, resulting in poor robustness. Furthermore, human facial expressions cannot accurately reflect true emotional states, leading to substantial discrepancies with reality. In addition, some researchers use electroencephalogram (EEG) signals to determine emotional states; however, EEG signal acquisition is extremely weak, easily affected by environmental interference, and places certain requirements on the subjects.

[0003] Furthermore, current research on emotion recognition systems based on multiple vital sign detection often references physiological signals such as respiration, electromyography (EMG), electroencephalography (EEG), electrooculography (EOG), and electrocardiography (ECG), employing simple data processing mapping and machine learning methods such as support vector machines to determine the subject's emotion. While physiological signals such as respiration, EMG, EOG, and ECG are also very important, the collection points for these signals cover almost every part of the subject's body, making the collection process cumbersome and difficult, thus significantly reducing its applicability and convenience. Summary of the Invention

[0004] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a method and system for emotion recognition based on time-domain features. It uses easily obtainable multi-vital sign signals, such as pulse, blood pressure, blood oxygen, and body temperature, as the basis for emotion recognition and realizes emotion recognition based on artificial intelligence algorithms.

[0005] To address the aforementioned technical problems, this invention discloses a multi-vital sign emotion recognition method based on time-domain features, comprising:

[0006] Based on deep convolutional neural networks, temporal fusion and feature extraction of multiple vital signs are performed to obtain fused deep features;

[0007] An attention-based neural network model based on temporal feature capture is used to optimize the fusion of deep features to obtain attention fusion features.

[0008] Based on a deep neural network model, the attention fusion features are reduced in dimensionality and then extracted and mapped to obtain the emotion recognition results.

[0009] In the aforementioned multi-vital sign emotion recognition method based on temporal features, a deep convolutional neural network is used to perform temporal fusion and feature extraction on multiple vital signs to obtain fused deep features, including:

[0010] Acquire pulse signal image features A within time period n n Blood pressure signal image features B n Blood oxygen signal image features C n and body temperature signal image features D n ;

[0011] The pulse signal image features A acquired within the time period n n Blood pressure signal image features B n Blood oxygen signal image features C n and body temperature signal image features D n The pulse segment feature sets are divided into equal groups according to time sequence. Blood pressure fragment feature set Blood oxygen fragment feature set and body temperature fragment feature set

[0012] pulse segment characteristics Blood pressure fragment characteristics Blood oxygen fragment characteristics and body temperature fragment characteristics The features of two adjacent segments are superimposed and then reassembled according to the time sequence to form a pulse reconstruction feature a. n Blood pressure remodeling characteristics b n Blood oxygen remodeling characteristics c n and body temperature recombination characteristics d n ;

[0013] Reconstructing pulse characteristics a n Blood pressure remodeling characteristics b n Blood oxygen remodeling characteristics c n and body temperature recombination characteristics d n Combined and spliced ​​to form the initial fusion feature J n ; Pulse signal image features A n Blood pressure signal image features B n Blood oxygen signal image features C n and body temperature signal image features D n Combined and spliced ​​to form the initial fusion original feature I n ;

[0014] Initial fusion original feature I n With initial fusion feature J n The inputs are fed into convolutional neural network 1 and convolutional neural network 2 respectively to obtain the initial fused depth features (DI). n and initial fusion deep features DJ n ;

[0015] The initial fusion depth original features DI nand initial fusion deep features DJ n The data is then further concatenated and fed into convolutional layer 1 and convolutional layer 2, respectively.

[0016] The output features of convolutional layer 1 are further input into the Sigmoid function for computation. The result is then multiplied element-wise with the output features of convolutional layer 2 to obtain the fused depth feature H. n .

[0017] In the aforementioned multi-vital sign emotion recognition method based on temporal features, pulse reconstruction feature a n Blood pressure remodeling characteristics b n Blood oxygen remodeling characteristics c n and body temperature recombination characteristics d n The calculation process is as follows:

[0018]

[0019]

[0020]

[0021]

[0022] In the aforementioned multi-vital sign emotion recognition method based on temporal features, the deep feature H is fused. n The calculation process is as follows:

[0023]

[0024] Where Sigmoid(·) represents the Sigmoid function operation, cov1 represents the operation of convolutional layer 1, and cov2 represents the operation of convolutional layer 2.

[0025] In the aforementioned multi-vital sign emotion recognition method based on temporal features, the attention neural network model based on temporal feature capture optimizes the fusion of deep features to obtain attention fusion features, including:

[0026] Fuse depth features H n The features are divided into fusion depth segment feature sets according to time order.

[0027] Will The input is fed into the convolutional layer r1 of the attention neural network model for temporal feature capture, and the output is then fed into the Softmax function of the same model for computation. Finally, the result is compared with... Perform element-wise multiplication to obtain a set of attention feature fragments.

[0028] Set of attention feature fragments The feature fragments within are concatenated and merged into a fused and recombined feature R. n , will integrate recombination features R n The input is fed into the convolutional layer r2 of the attention neural network model for capturing temporal features. The calculation result is then fed into the softmax function of the attention neural network model for capturing temporal features for computation. The result is then fused and reconstructed with the feature R. n Add them together to obtain attention fusion features.

[0029] In the aforementioned multi-vital sign emotion recognition method based on temporal features,

[0030]

[0031]

[0032] Where softmax(·) represents the operation of the Softmax function, covr1 represents the operation of convolutional layer r1, and covr2 represents the operation of convolutional layer r2.

[0033] In the aforementioned multi-vital sign emotion recognition method based on temporal features, a deep neural network model is used to perform dimensionality reduction feature extraction and mapping on attention fusion features to obtain emotion recognition results, including:

[0034] Attention fusion features The input is fed into the convolutional layer Y1 of the deep neural network model, then activated by the ReLU function, and then fed into the convolutional layer Y2 of the deep neural network model. Finally, it is fed into the Softmax function of the deep neural network model for computation. If a certain element of the feature normalized by the Softmax function of the deep neural network model is the maximum value, it indicates that the emotion j corresponding to the position j of that element is the emotion expressed by multiple vital signs at the current moment.

[0035] In the aforementioned multi-vital sign emotion recognition method based on temporal features,

[0036]

[0037] Where softmax(·) represents the Softmax function operation, covY1 represents the Y1 operation of the convolutional layer, covY2 represents the Y2 operation of the convolutional layer, and ReLU(·) represents the ReLU function operation.

[0038] In the above-mentioned multi-vital sign emotion recognition method based on time-domain features, the multi-vital signs include: pulse signal image features, blood pressure signal image features, blood oxygen signal image features, and body temperature signal image features.

[0039] Accordingly, this invention also discloses a multi-vital sign emotion recognition system based on temporal domain features, comprising:

[0040] The fusion deep feature extraction module is used to perform temporal fusion and feature extraction on multiple vital signs based on deep convolutional neural networks to obtain fused deep features;

[0041] The attention fusion feature extraction module is used to optimize the fusion deep features of the attention neural network model based on temporal feature capture to obtain attention fusion features;

[0042] The emotion recognition module is used to extract and map attention fusion features based on a deep neural network model to obtain emotion recognition results.

[0043] The present invention has the following advantages:

[0044] (1) The invention can realize the judgment and recognition of the user's emotional state in real time through the most easily obtainable vital signs signals.

[0045] (2) The invention makes signal acquisition in the process of human emotion recognition very simple and easy to operate, does not affect the emotions of the user, and ensures the accuracy of emotion recognition.

[0046] (3) This invention can not only improve the psychological quality of trainees and suppress negative emotions in daily training in a portable and simple way, but also help remind users of their real-time emotional state in actual application scenarios, and pay attention to adjustment and improvement. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the steps of a multi-vital sign emotion recognition method based on temporal features in an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of a deep feature extraction process according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of an attention fusion feature extraction process in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of emotion mapping matching based on fusion features in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments disclosed in the present invention will be described in further detail below with reference to the accompanying drawings.

[0052] One of the core ideas of this invention is to disclose a multi-vital sign emotion recognition method based on time-domain features, which mainly includes the following three aspects:

[0053] (1) By designing a feature extraction method for time-domain fusion of multiple vital signs, the physiological signals of multiple vital signs in the same period are arranged and combined in the form of feature vectors and input into a deep convolutional neural network for feature extraction. Through continuous iterative training, key information features are captured, which solves the problem that it is difficult to effectively extract important features of multiple vital signs in the same period.

[0054] (2) By designing an attention neural network model for capturing temporal features, the temporal feature vectors of the same time period after fusion are optimized, and the most critical feature changes in the fused feature vectors are captured, thus solving the problem of how multiple physiological signals reflect the most essential emotional state.

[0055] (3) By designing a feature-integrated emotion mapping matching method, the feature vector after attention capture optimization is extracted by a deep neural network model to reduce the dimensionality of the feature vector, and the reduced dimensionality feature vector is mapped to multiple emotional states through a classification function, thus solving the problem of emotion recognition based on multiple common and easily accessible physiological signals.

[0056] It is evident that the multi-vital sign emotion recognition method based on time-domain features described in this invention solves at least the following technical problems:

[0057] (1) The problem that it is difficult to effectively extract important features from multiple vital signs at the same time.

[0058] (2) The most fundamental reflection of emotional state by various physiological signals.

[0059] (3) The problem of emotion recognition based on a variety of common and easily accessible physiological signals.

[0060] like Figure 1 In this embodiment, the multi-vital sign emotion recognition method based on time-domain features may specifically include:

[0061] Step 1: Based on a deep convolutional neural network, perform temporal fusion and feature extraction on multiple vital signs to obtain fused deep features.

[0062] In this embodiment, multiple vital signs include, but are not limited to: pulse signal image features, blood pressure signal image features, blood oxygen signal image features, and body temperature signal image features.

[0063] like Figure 2 The specific methods for obtaining fused deep features are as follows:

[0064] Acquire pulse signal image features A within time period n n Blood pressure signal image features B n Blood oxygen signal image features C n and body temperature signal image features D n .

[0065] The pulse signal image features A acquired within the time period n n Blood pressure signal image features B n Blood oxygen signal image features C n and body temperature signal image features D n The pulse segment feature sets are divided into equal groups according to time sequence. Blood pressure fragment feature set Blood oxygen fragment feature set and body temperature fragment feature set

[0066] pulse segment characteristics Blood pressure fragment characteristics Blood oxygen fragment characteristics and body temperature fragment characteristics The features of two adjacent segments are superimposed and then reassembled according to the time sequence to form a pulse reconstruction feature a. n Blood pressure remodeling characteristics b n Blood oxygen remodeling characteristics c n and body temperature recombination characteristics d n .

[0067] Reconstructing pulse characteristics a n Blood pressure remodeling characteristics b n Blood oxygen remodeling characteristics c n and body temperature recombination characteristics d n Combined and spliced ​​to form the initial fusion feature J n ; Pulse signal image features A n Blood pressure signal image features B n Blood oxygen signal image features C n and body temperature signal image features D n Combined and spliced ​​to form the initial fusion original feature I n .

[0068] Initial fusion original feature I n With initial fusion feature J n The inputs are fed into convolutional neural network 1 and convolutional neural network 2 respectively to obtain the initial fused depth features (DI). n and initial fusion deep features DJ n .

[0069] The initial fusion depth original features DI n and initial fusion deep features DJn The data are then further concatenated and fed into convolutional layer 1 and convolutional layer 2, respectively.

[0070] The output features of convolutional layer 1 are further input into the Sigmoid function for computation. The result is then multiplied element-wise with the output features of convolutional layer 2 to obtain the fused depth feature H. n .

[0071] Preferred pulse reconstruction feature a n Blood pressure remodeling characteristics b n Blood oxygen remodeling characteristics c n and body temperature recombination characteristics d n The calculation process is as follows:

[0072]

[0073]

[0074]

[0075]

[0076] Preferably, the fusion depth feature H n The calculation process is as follows:

[0077]

[0078] Where Sigmoid(·) represents the Sigmoid function operation, cov1 represents the operation of convolutional layer 1, and cov2 represents the operation of convolutional layer 2.

[0079] Step 2: Based on the attention neural network model that captures temporal features, optimize the fusion deep features to obtain attention fusion features.

[0080] In this embodiment, as Figure 3 The specific methods for obtaining attention fusion features are as follows:

[0081] Fuse depth features H n The features are divided into fusion depth segment feature sets according to time order.

[0082] Will The input is fed into the convolutional layer r1 of the attention neural network model for temporal feature capture, and the output is then fed into the Softmax function of the same model for computation. Finally, the result is compared with... Perform element-wise multiplication to obtain a set of attention feature fragments.

[0083] Set of attention feature fragments The feature fragments within are concatenated and merged into a fused and recombined feature R. n , will integrate recombination features R n The input is fed into the convolutional layer r2 of the attention neural network model for capturing temporal features. The calculation result is then fed into the softmax function of the attention neural network model for capturing temporal features for computation. The result is then fused and reconstructed with the feature R. n Add them together to obtain attention fusion features.

[0084]

[0085]

[0086] Where softmax(·) represents the operation of the Softmax function, covr1 represents the operation of convolutional layer r1, and covr2 represents the operation of convolutional layer r2.

[0087] Step 3: Based on the deep neural network model, perform dimensionality reduction feature extraction and mapping on the attention fusion features to obtain the emotion recognition results.

[0088] In this embodiment, as Figure 4 The specific implementation method of emotion recognition is as follows:

[0089] Attention fusion features The input is fed into the convolutional layer Y1 of the deep neural network model, then activated by the ReLU function, and then fed into the convolutional layer Y2 of the deep neural network model. Finally, it is fed into the softmax function of the deep neural network model for computation. If a certain element of the features normalized by the softmax function of the deep neural network model is the maximum value, it indicates that the emotion j corresponding to the position j of that element represents the emotion expressed by the multiple vital signs at the current moment.

[0090]

[0091] Where softmax(·) represents the Softmax function operation, covY1 represents the Y1 operation of the convolutional layer, covY2 represents the Y2 operation of the convolutional layer, and ReLU(·) represents the ReLU function operation.

[0092] Based on the above embodiments, the present invention also discloses a multi-vital sign emotion recognition system based on temporal features, comprising: a fusion deep feature extraction module, used to perform temporal fusion and feature extraction on multiple vital signs based on a deep convolutional neural network to obtain fusion deep features; an attention fusion feature extraction module, used to optimize the fusion deep features based on an attention neural network model that captures temporal features to obtain attention fusion features; and an emotion recognition module, used to perform dimensionality reduction feature extraction and mapping on the attention fusion features based on a deep neural network model to obtain emotion recognition results.

[0093] As the system implementation is in contrast to the method implementation, it is described in a simpler way. For relevant details, please refer to the description in the method implementation section.

[0094] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

[0095] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. A multi-vital sign emotion recognition method based on time domain features, characterized in that, Comprise: Based on deep convolutional neural network, time domain fusion and feature extraction are performed on multiple vital signs to obtain fused deep features; including: obtaining pulse signal image features A n , blood pressure signal image features B n , blood oxygen signal image features C n and body temperature signal image features D n in a time period n; the obtained pulse signal image features A n , blood pressure signal image features B n , blood oxygen signal image features C n and body temperature signal image features D n in the time period n are respectively divided into pulse segment feature sets , blood pressure segment feature sets , blood oxygen segment feature sets and body temperature segment feature sets in time order; adjacent two segment features of pulse segment features , blood pressure segment features , blood oxygen segment features and body temperature segment features are superimposed, and are reconnected and spliced into pulse reorganization features a n , blood pressure reorganization features b n , blood oxygen reorganization features c n and body temperature reorganization features d n in time order; the pulse reorganization features a n , blood pressure reorganization features b n , blood oxygen reorganization features c n and body temperature reorganization features d n are combined and spliced into initial fusion features J n ; the pulse signal image features A n , blood pressure signal image features B n , blood oxygen signal image features C n and body temperature signal image features D n are combined and spliced into initial fusion original features I n ; the initial fusion original features I n and the initial fusion features J n are respectively input into convolutional neural network 1 and convolutional neural network 2 to obtain initial fusion deep original features DI n and initial fusion deep features DJ n ; the initial fusion deep original features DI n and the initial fusion deep features DJ n The further concatenated is respectively input into the convolution layer 1 and the convolution layer 2; the obtained output feature of the convolution layer 1 is further input into the Sigmoid function for operation, and then the operation result is multiplied with the obtained output feature of the convolution layer 2 according to elements, to obtain the fused deep feature H n ; An attention neural network model based on time domain feature capture is used to optimize the fusion depth feature to obtain attention fusion features. Based on a deep neural network model, the attention fusion features are dimensionally reduced and mapped to obtain emotion recognition results.

2. The method of claim 1, wherein, Pulse recombination feature a n Blood pressure recombination feature b n Blood oxygen recombination feature c n And body temperature recombination feature d n The calculation process is as follows: 3.The time-domain feature based multi-vital sign emotion recognition method of claim 1, wherein, Fusion of deep features H n The calculation process is as follows: Wherein, Sigmoid(·) represents the Sigmoid function operation, cov1 represents the convolution layer 1 operation, and cov2 represents the convolution layer 2 operation.

4. The method of claim 1, wherein, An attention neural network model based on time domain feature capture is used to optimize the fusion depth feature to obtain attention fusion features, comprising: The fusion depth feature H n is divided into a fusion depth segment feature set H Will The input is fed into the convolutional layer r1 of the attention neural network model for temporal feature capture, and the output is then fed into the Softmax function of the same model for computation. Finally, the result is compared with... Perform element-wise multiplication to obtain a set of attention feature fragments. The attention feature segment set The feature segment string in the attention feature segment set is combined and fused to obtain a fusion recombination feature R n The fusion recombination feature R n is input into a convolutional layer r2 of the attention neural network model for time domain feature capture, and the calculation result is input into a Softmax function of the attention neural network model for time domain feature capture for operation, and the operation result is added to the fusion recombination feature R n to obtain an attention fusion feature 5. The multi-vital sign emotion recognition method based on time domain features according to claim 4, characterized in that, Wherein, softmax(·) represents the Softmax function operation, covr1 represents the convolution layer r1 operation, and covr2 represents the convolution layer r2 operation.

6. The method of claim 4, wherein the time domain feature-based multi- vital sign emotion recognition method is characterized by, Based on a deep neural network model, the attention fusion features are dimensionally reduced and mapped to obtain emotion recognition results, comprising: Fusing attention features The input is input into the convolutional layer Y1 of the deep neural network model, and then activated by the ReLU function, and then input into the convolutional layer Y2 of the deep neural network model, and then input into the Softmax function of the deep neural network model for operation; the feature normalized by the Softmax function of the deep neural network model has a certain element as the maximum value, which indicates that the emotion j corresponding to the position j where the element is located is the emotion expressed by the current multi-life sign.

7. The multi-vital sign emotion recognition method based on time domain features according to claim 6, characterized in that, Wherein, softmax(·) represents the Softmax function operation, covY1 represents the convolution layer Y1 operation, covY2 represents the convolution layer Y2 operation, and ReLU(·) represents the ReLU function operation. 8.The time-domain feature based multi-vital sign emotion recognition method of claim 1, wherein, Multi-vital signs, including: pulse signal image features, blood pressure signal image features, blood oxygen signal image features, and body temperature signal image features.

9. A time-domain feature based multi-vital sign emotion recognition system for implementing the method as claimed in claim 1, characterized in that, Comprise: A fusion depth feature extraction module is used to perform time domain fusion and feature extraction on multi-vital signs based on a deep convolutional neural network to obtain fusion depth features; An attention fusion feature extraction module is used to optimize the fusion depth feature based on an attention neural network model based on time domain feature capture to obtain attention fusion features; An emotion recognition module is used to dimensionally reduce and map the attention fusion features based on a deep neural network model to obtain emotion recognition results.

Images