Portable electrocardio skin conductance bimodal emotion discrimination system

Abstract

The invention discloses a portable electrocardio skin conductivity bimodal emotion discrimination system, and belongs to the technical field of biomedical signal detection. Comprising a core control main board, a data analysis and evaluation module and a control module, wherein the core control main board is used for collecting electrocardio and skin conductance signals; the data analysis and evaluation module is used for processing the collected electrocardio and skin conductance signals and judging emotions; the data analysis and evaluation module is matched with the core control main board and controls the core control main board to collect the electrocardio and skin conductance signals; by the adoption of the system and a corresponding method, the system is simple in structure, small in size and integrated in design, meets the portable application requirement, supports real-time signal processing and diagnosis, is high in reliability and high in reliability, and can be applied to the field of wireless communication. The method is suitable for rapid and accurate emotion-related mental disease detection in a dynamic clinical environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of biomedical signal detection, in particular to a portable electrocardio-skin conductance dual-mode emotion discrimination system. BACKGROUND

[0002] With the continuous development of medical diagnosis technology, various biological signals (such as facial expression images, body movements, and electroencephalograms) have been relatively mature in accurately identifying and classifying human emotional states in automatic emotion recognition technology. However, due to the difficulty in capturing and obtaining these signals, it is necessary to complete the acquisition in a specific scene, and there are great obstacles in the data preparation stage.

[0003] And the traditional electrocardio signal (ECG) and skin conductance signal (GSR) acquisition equipment is large in size, expensive in cost and needs to be operated by professional personnel with medical knowledge, which is particularly prominent in the acquisition of electrocardiogram. The traditional signal processing method (such as wavelet transform and time domain analysis) has limitations in feature extraction and classification, and the traditional method of machine learning and deep learning also needs to consume a large amount of calculation cost (large number of parameters, high storage occupation, and slow inference speed) in physiological signal analysis. SUMMARY

[0004] The purpose of the present application is to provide a portable electrocardio-skin conductance dual-mode emotion discrimination system, which is simple in structure, small in size, integrated in design, meets the demand of portable application, supports real-time signal processing and diagnosis, and is suitable for rapid and accurate emotion-related mental illness detection in dynamic clinical environment.

[0005] To achieve the above purpose, the present application provides a portable electrocardio-skin conductance dual-mode emotion discrimination system, which comprises a core control mainboard for collecting electrocardio and skin conductance signals, a data analysis and evaluation module for processing the collected electrocardio and skin conductance signals and discriminating emotions, and a Bluetooth wireless receiver paired with the core control mainboard, which controls the core control mainboard to collect electrocardio and skin conductance signals and transmits the data collected by the core control mainboard to the data analysis and evaluation module.

[0006] The use method is as follows:

[0007] S1, collect electrocardio signals and skin conductance signals through the core control mainboard, and transmit them to the data analysis and evaluation module through the Bluetooth wireless receiver;

[0008] S2, after the data analysis and evaluation module receives the collected signals, pre-processes them;

[0009] S3, use the SAC-GAN method for data enhancement and realize the pre-training of the deep learning model;

[0010] S4, using the trained deep learning model to extract the features of the preprocessed electrocardio signals and skin conductance signals and to fuse them, obtaining the prediction result of emotion level discrimination and displaying it through the software program.

[0011] Preferably, the core control mainboard comprises a signal acquisition module for acquiring electrocardio signals and skin conductance signals, a main control chip for processing multi-channel physiological signals, a Bluetooth wireless connection module for maintaining communication and data transmission with the computer, a power supply, a charging module and a switch control module.

[0012] Preferably, the signal acquisition module comprises multi-channel electrodes for acquiring electrocardio signals and skin conductance signals, the electrocardio signal acquisition part of the multi-channel electrodes adopts 3-lead electrodes and is equipped with independent reference electrodes and bias electrodes, and the skin conductance signal acquisition part adopts 2-lead electrodes and uses the same reference electrodes and bias electrodes as the electrocardio signal acquisition part.

[0013] Preferably, the data analysis and evaluation module comprises a preprocessing module for preprocessing electrocardio signals and skin conductance signals, a feature extraction and fusion module for extracting and fusing electrocardio signals and skin conductance signals, and an upper computer comprising a deep learning model for discriminating the emotional state corresponding to the dual-modal physiological signals and a software program for displaying the emotional state.

[0014] Preferably, the deep learning model enriches the feature representation through a data enhancement technique, which is a SAC-GAN model method that fuses self-attention and conditional generative adversarial networks, and the deep learning model comprises a shallow feature extraction module, a global attention module, a residual connection feature extraction module and a channel attention module.

[0015] Preferably, the process of S1 is as follows:

[0016] S11, set the sampling frequency of the signal acquisition module, and then synchronously acquire electrocardio signals and skin conductance signals;

[0017] S12, use the main control chip to process the multi-channel physiological signals obtained during the signal acquisition process;

[0018] S13, transmit the acquired electrocardio signals and skin conductance signals to the data analysis and evaluation module through the Bluetooth wireless connection module, and transmit the physiological signals to the software program in the upper computer.

[0019] Preferably, the process of preprocessing in S2 is as follows:

[0020] S21, preprocess the electrocardio signals, remove high-frequency noise and power frequency interference through band-pass filtering and notch filtering, correct the baseline drift of the electrocardio signals, then perform standard normalization processing on the electrocardio signals, detect abnormal values in the synchronous 3-lead signals, and use an interpolation method to compensate for missing and abnormal signals;

[0021] S22, pre-process the skin conductance signal, remove high-frequency noise interference by band-pass filtering, then smooth the skin conductance signal in time domain, then normalize the signal, detect abnormal values in 2-lead signals, and use interpolation method to compensate for missing and abnormal signals.

[0022] Preferably, the process of training the deep learning model using the SAC-GAN method in S3 is as follows:

[0023] S31, use two public data sets and self-collected data sets to enrich feature representation through data enhancement technology to generate a number of effective data;

[0024] S32, use effective data to repeatedly train the deep learning neural network model double attention feature extraction residual chain module in the deep learning model;

[0025] S33, enhance the discriminant ability and generalization performance of different signal data through S33 enhancement module, and then complete the pre-training task of the model.

[0026] Preferably, the process of S4 is as follows:

[0027] S41, cut the pre-processed electrocardiogram signal and skin conductance signal into 5s time windows without overlap according to time sequence, and splice the time window sequences of the two signals in the channel dimension to obtain a merged time window sequence X concat , the merging process is as follows:

[0028] X concat =[X ECG ,X GSR ];

[0029] Where X ECG represents the time window sequence of the electrocardiogram signal, and X GSR represents the time window sequence of the skin conductance signal.

[0030] S42, input X concat into the shallow feature extraction module in the deep learning neural network model double attention feature extraction residual chain module to preliminarily extract relevant features for emotion discrimination, and raise the dimension of X concat data to 64 in the channel level, the process is as follows:

[0031] F s =Dropout(BN(Conv1D(Signal mix )),p=0.3;

[0032] Where F swherein, represents the shallow feature vector, Dropout(·) represents randomly discarding network neuron nodes with a probability of p=0.3, BN(·) represents a batch normalization operation, Conv1D(·) represents a 1D convolution operation;

[0033] S43, after preliminarily extracting the relevant features of emotion discrimination, a global attention module is used to deeply mine and learn the basis of the relevant features in S42, and the process is as follows:

[0034] S431, the global attention module is used to calculate the correlation coefficient between the input features;

[0035] S432, the obtained shallow feature vector is uniformly cut into smaller time window in 3 time sequence dimensions, and is regarded as a fine window, and the process is as follows:

[0036] F s ={f k1 ,f k2 ,f k3};

[0037] f s ∈R C*L ,f k1 ∈R C*L / 3 ,f k2 ∈R C*L / 3 ,f k3 ∈R C*L / 3 ;

[0038] S433, the three fine windows are input into the global attention module to perform segmented global attention operation, and the process is as follows:

[0039]

[0040] wherein, y i represents the output of the global attention operation on a single fine window, i represents the time sequence index of the output feature, j represents the time sequence index of the input feature which has an influence on the output, x represents the input feature map, f(x) represents a function for calculating the relationship between the features with indexes i and j, g(x) adjusts the feature representation of the input feature vector, C(x) represents normalization, and θ(x) and φ(x) represent convolution operation;

[0041] S434, the output of the global attention module is connected with the input through a residual connection to obtain the weighted output F TGA of the global attention module, and the process is as follows:

[0042]

[0043] wherein, represents cutting the input, input of the global attention mechanism residual connection, W w represents a 1D convolution operation, W θ , φ and W g all represent 1D convolution operations with the same operation as W w , Softmax(·) represents an exponential normalization function, and AvgPool(·) and MaxPool(·) represent global average pooling and global maximum pooling, respectively;

[0044] S44, all 2D convolution layers in the original ResNet architecture are replaced with 1D convolution operations using a residual connection feature extraction module, and multi-level feature extraction and abstraction are performed on the original 1D physiological signals to obtain residual connection output features f res ;

[0045] S45, a channel attention module is used to assign importance scores to the electrocardiogram signal and the skin conductance signal for emotion discrimination, and a prediction result of the emotion level discrimination is obtained, as follows:

[0046] S451, global maximum pooling and global average pooling operations are performed on the residual connection output features f res , respectively, to obtain significant feature response maximum value and global feature distribution average value f max and f avg ;

[0047] S452, f max and f avg are input into a shared multi-layer convolution to capture cross-channel nonlinear interactions, generating corresponding feature representations F max and F avg ;

[0048] S453, F max and F avg are added and then nonlinearly mapped through a Sigmoid activation function to generate a channel attention weight matrix M c ;

[0049] S454, the residual connection output features f res are scaled according to the attention weight matrix, and the original features are adaptively re-scaled to obtain an original feature vector;

[0050] S456, the original feature vector is transmitted to three linear layers, each followed by a ReLU activation function, and the maximum value is selected as the prediction result of the model for the input electrocardiogram signal and skin conductance signal emotion level discrimination using a Softmax function normalization;

[0051] S46, transmit the prediction result obtained by the deep learning model to the software program, and display the prediction result and the physiological signal through the software program.

[0052] Therefore, the portable electrocardio-skin conductance dual-modal emotion discrimination system has the following advantages:

[0053] 1. The ECG and GSR signals can be synchronously collected, high-quality bioelectric signals are provided, the signal clarity and accuracy are ensured through low-noise amplification technology, and the system is suitable for accurate evaluation of the emotional level.

[0054] 2. The system is miniaturized and integrated, meets the portable application requirements, supports real-time signal processing and diagnosis, and is suitable for rapid and accurate emotion-related mental illness detection in a dynamic clinical environment.

[0055] 3. Through the end-to-end feature-level fusion and the residual connection network algorithm with a double attention mechanism, the system can reduce the data processing steps, efficiently fuse the signal features from the two modalities, and improve the recognition accuracy of physiological state changes.

[0056] The technical solutions of the present application will be further described in detail through the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 FIG. 1 is a system principle schematic diagram of the portable electrocardio-skin conductance dual-modal emotion discrimination system of the present application;

[0058] Figure 2 FIG. 2 is a flowchart of the pre-processing and obtaining of the prediction result of the emotional level discrimination of the portable electrocardio-skin conductance dual-modal emotion discrimination system of the present application;

[0059] Figure 3 FIG. 3 is a signal acquisition module circuit diagram of the portable electrocardio-skin conductance dual-modal emotion discrimination system of the present application;

[0060] Figure 4 FIG. 4 is a circuit schematic diagram of the master control chip of the portable electrocardio-skin conductance dual-modal emotion discrimination system of the present application;

[0061] Figure 5 FIG. 5 is a circuit schematic diagram of the Bluetooth wireless connection module of the portable electrocardio-skin conductance dual-modal emotion discrimination system of the present application;

[0062] Figure 6 FIG. 6 is a power supply circuit schematic diagram of the portable electrocardio-skin conductance dual-modal emotion discrimination system of the present application. DETAILED DESCRIPTION

[0063] EMBODIMENT

[0064] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings. Obviously, the described embodiments are only some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0065] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0066] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0067] In the description of the present application, it should be noted that the terms "upper", "lower", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the present application is usually placed, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0068] In the description of the present application, it should also be noted that, unless otherwise explicitly specified and limited, the terms "arrange", "mount", "connect" should be understood broadly, for example, can be fixedly connected, can be detachably connected, or integrally connected; can be mechanically connected, can be electrically connected; can be directly connected, can be indirectly connected through an intermediate medium, can be the communication inside two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0069] The following will describe some embodiments of the present application in detail with reference to the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

[0070] As Figures 1-6As shown, the portable electrocardio-skin conductance dual-mode emotion discrimination system of the present application comprises a core control mainboard for collecting electrocardio and skin conductance signals, a data analysis and evaluation module for processing the collected electrocardio and skin conductance signals and discriminating emotions, and a Bluetooth wireless receiver paired with the core control mainboard to control the core mainboard to collect electrocardio and skin conductance signals and transmit the data collected by the core mainboard to the data analysis and evaluation module.

[0071] The use method is as follows:

[0072] S1, collect electrocardio signals and skin conductance signals through the core control mainboard, and transmit them to the data analysis and evaluation module through the Bluetooth wireless receiver;

[0073] S2, after the data analysis and evaluation module receives the collected signals, it pre-processes them;

[0074] S3, use the SAC-GAN method for data enhancement and realize the pre-training of the deep learning model;

[0075] S4, use the trained deep learning model to extract the features of the pre-processed electrocardio signals and skin conductance signals and fuse them to obtain the prediction result of emotion level discrimination and display it through the software program.

[0076] The core control mainboard comprises a signal collection module for collecting electrocardio and skin conductance signals, a main control chip for processing multi-channel physiological signals, a Bluetooth wireless connection module for maintaining communication and data transmission with the computer, a power supply, a charging module and a switch control module.

[0077] The signal collection module comprises multi-channel electrodes for collecting electrocardio signals and skin conductance signals, and dedicated reference electrodes and bias electrodes are provided for both types of physiological signals to ensure signal stability and anti-interference performance. The electrocardio signal collection part of the multi-channel electrode adopts 3-lead electrodes, and is equipped with independent reference electrodes and bias electrodes. The skin conductance signal collection part adopts 2-lead electrodes to indirectly collect signals, and uses the same reference electrodes and bias electrodes as the electrocardio signal collection part. In the process of collecting skin conductance signals, the signals of the 2-lead electrodes need to be subtracted to form 1-lead signals, which are the real skin conductance signals to be used subsequently. The placement positions of all electrodes are designed according to the human anatomical structure and in reference to international standards, aiming to ensure the accuracy of the signals and the integrity of the physiological characteristics.

[0078] (1) Electrode placement for electrocardio (ECG)

[0079] Surface patch electrodes are adopted, and electrode placement follows IEC60601-2-25 (Medical electrical equipment - Electrocardiograph devices - Part 2-25: Particular requirements for safety) and ISO 80601-2-25 (International Standardization Organization Electrocardiograph Devices Standard), from which two reference electrodes and three signal electrodes are selected for emotion discrimination. The specific positions include:

[0080] RA right arm electrode (right wrist or right subclavian) and RL right leg electrode (right lower leg or right lower abdomen).

[0081] The RA electrode signal exhibits low amplitude electrical activity due to its distance from the heart, mainly providing a stable reference potential; the RL electrode directs irrelevant current (such as electromyographic interference) into the ground, improving signal quality and excluding environmental electromagnetic interference.

[0082] V3 (5th intercostal, left midclavicular line right margin), V4 (5th intercostal, left midclavicular line), V5 (left anterior axillary line, same level as V4).

[0083] V3 is used to obtain the activity level of the anterior interventricular septum; V4 is used to obtain the activity level of the left ventricular anterior apex, with the most prominent R wave shape; V5 is used to obtain the activity level of the left ventricular lateral wall.

[0084] (2) Skin conductance (GSR) electrode placement

[0085] Surface patch electrodes are used, and the electrode positions refer to the guidelines of the International Union of Physiology (IUPS) and the Society for Psychophysiological Research (SPR), which include:

[0086] Distal phalangeal ventral of left index finger and middle finger.

[0087] The distal phalangeal ventral of the middle finger is considered as the reference electrode position, and the distal phalangeal ventral of the index finger is considered as the signal electrode. The signal level of the index finger minus the distal phalangeal ventral of the middle finger is the GSR signal.

[0088] Through the above electrode placement, high-quality acquisition of human ECG and GSR dual-mode signals can be achieved; combined with independent reference electrodes and bias electrode design, the interference of human electromyographic signals and the widely existing 50Hz power frequency interference are minimized, preliminary improving the accuracy and stability of the data.

[0089] The signal acquisition module uses high-precision analog front-end chip ADS1299. This chip has multi-channel signal synchronous acquisition function, supporting amplification and filtering of weak signals such as ECG and GSR.

[0090] Input protection design: each signal input pin of ADS1299 chip is connected in series with a filter capacitor and a protection resistor to prevent electrostatic discharge high amplitude pulse from damaging the chip and effectively suppress high frequency interference signals.

[0091] Power supply design: The signal acquisition module uses a switching regulator TPS60403 and a linear regulator LT3032 to generate different voltages required for the ADS1299 chip to work, ensuring low-power stable operation.

[0092] Data transmission interface: The 34th pin (DIN) and the 43rd pin (DOUT) of the ADS1299 chip are connected to the 17th pin (PA7) and the 16th pin (PA6) of the STM32F103C8T6 chip for sending and receiving instructions and data.

[0093] The signal acquisition module has an instruction-controlled acquisition function, which can continuously record ECG and GSR signals for a long time, and is suitable for emotional evaluation and analysis.

[0094] The core control mainboard uses an STM32F103C8T6 chip to control the entire hardware system, and uses a minimum control system to maintain the characteristics of small size and low power consumption. The Bluetooth wireless connection module uses an MX-02 chip to receive the start / stop instructions sent by the computer using low-power Bluetooth (BLE) technology. After the analog signal (physiological data) is converted into a digital signal, the Bluetooth wireless connection module transmits the processed digital signal to the computer. The 30th pin (PA9) and the 31st pin (PA10) of the STM32F103C8T6 chip are connected to the 8th pin (P03-RX) and the 9th pin (P03-TX) of the wireless Bluetooth module MX-02 through a serial communication interface (UART) to realize real-time data transmission.

[0095] The power supply includes a lithium battery charging circuit and a voltage stabilizing circuit. The lithium battery charging circuit uses a TP4057 chip as a lithium battery charging management chip. The TP4057 chip has high-efficiency charging management capability, which can automatically control the battery charging process, ensure the safety of battery charging and prolong the service life of the battery. The specific connection method is as follows:

[0096] The 4th pin of the TP4057 is connected to a 5V power supply to provide external power for the charging circuit; the 1st and 5th pins are connected in series with an LED lamp and a 1k resistor to indicate the charging status; the 3rd pin is connected to the positive electrode of a 3.7V lithium battery for charging the lithium battery; the 2nd pin is grounded; a 2k resistor is connected in series between the 6th pin and the ground to limit the maximum charging current to 500mA; and the ground connection of the charging circuit is completed. Through the management of the TP4057 chip, the charging process of the lithium battery is accurately controlled to ensure that the battery is charged within a safe voltage range, avoid overcharging or overdischarging, prolong the battery life and improve the system stability.

[0097] The voltage stabilizing circuit uses a TPS73633 low dropout linear regulator chip and supplies a 3.3V voltage to the VDD pin of the ADS1299 chip, the STM32F103C8T6 chip and the MX-02 Bluetooth chip to provide a stable voltage for the system. The specific connection mode is as follows:

[0098] The first pin of the TPS73633 chip is connected to the ground in series with a 1uF and a 100nF capacitor to reduce noise and improve power supply rejection ratio; the fourth pin is connected in series with a 1uF bypass capacitor to reduce output noise; the fifth pin is connected in series with a 1uF and a 100nF capacitor to the output 3.3V voltage, which is used to improve the load transient characteristics and reduce fluctuation noise, and provides power supply for the signal acquisition part, the main control chip and the Bluetooth transmission part.

[0099] The data analysis and evaluation module includes a preprocessing module for preprocessing electrocardio and skin conductance signals, a feature extraction and fusion module for extracting and fusing electrocardio and skin conductance signals, and an upper computer, which includes a deep learning model for distinguishing the corresponding emotional state of the dual-mode physiological signal and a software program for displaying the emotional state.

[0100] The deep learning model enriches the feature representation through data enhancement technology, which is a SAC-GAN model method that combines self-attention generative adversarial network and conditional generative adversarial network. The deep learning model includes a shallow feature extraction module, a global attention module, a residual connection feature extraction module and a channel attention module.

[0101] The data enhancement technology refers to a SAC-GAN model method that combines self-attention generative adversarial network (SAGAN) and conditional generative adversarial network (CGAN). The SAC-GAN model consists of two parts: discriminator and generator. The discriminator is used for the SAC-GAN model to automatically mine and learn the feature level contained in the dual-mode signal data and the corresponding emotional level label, and the potential association between them. The generator is used to generate dual-mode signal data with consistent association with the specified label, restore the feature representation of the dual-mode signal data, and enrich the data set used for pre-training of DARFEC.

[0102] Generator: The generator starts with a label embedding layer, followed by six transposed convolution layers, and inserts a self-attention module after the first two transposed convolution layers. The self-attention mechanism captures long-distance dependencies within the generated features, enabling the generator to perceive the association between different feature regions and generate samples with better internal consistency and category-specific features.

[0103] Discriminator: The discriminator starts with the label embedding layer and is composed of 6 separable convolutional layers. After the 3rd and 4th convolutional layers, self-attention modules are introduced to enhance the discriminator's ability to perceive the global feature structure of the input sample. By focusing on the interdependence between features at different positions in the input signal, the discriminator can better identify the differences between generated and real samples, improving its ability to learn the discriminant boundary between different classes and thus enhancing overall signal recognition performance. A channel attention module is introduced at the end of the discriminator, consistent with the foregoing, to strengthen the expression of signal features that are significantly distinctive of a particular class.

[0104] The aforementioned label embedding refers to the fusion of the bimodal signal Signal mix and the corresponding emotional level Label into a unified representation. First, a learnable lookup table LT is created to map discrete emotional level labels to fixed-size, continuous dense label embedding vectors M Embd , preserving semantic information. Subsequently, a fixed-size numerical value is selected as the embedding dimension (128 for the generator and 32 for the discriminator) to balance the amount of information in the label information and the embedding vector z (z is random noise in the generator and z is the bimodal signal in the discriminator). The label embedding vector M Embd is then concatenated with z to restore the original embedding dimension and avoid dimension expansion. Embedding label information into signal data in this way can enhance the generator's and discriminator's ability to generate and distinguish signals of different emotional levels.

[0105] The calculation formula of the aforementioned self-attention module is:

[0106] O SA (Q, K, V, γ) = γ·Attention(Q, K, V);

[0107] where Q, K, and V represent the Query, Key, and Value matrices, respectively, and γ is a scaling coefficient that is dynamically adjusted based on feature importance. Through this mechanism, the model can automatically learn and adjust γ during training, optimizing feature selection and reducing the impact of redundant features. Input signal features are first transformed into Q(x), K(x), and V(x) three feature spaces to calculate attention, where C represents the number of channels. Q(x) = W q ·x, K(x) = W k ·x, and V(x) = W v ·x. The three feature spaces Q(x), K(x), and V(x) represent three one-dimensional convolutional blocks with a kernel size of 1, and the output vectors of the three are halved and compressed in the channel layer. When calculating the feature value of the jth channel, the model's attention to the features of channel i is integrated into the [0, 1] space using a softmax function, represented as:

[0108]

[0109] The output of the self-attention layer is:

[0110] F SA =α j,i ·V(x i ),V(x i )=W v ·x i ;

[0111] where the weight matrix is used to calculate the attention score and adjust the feature representation. The output of the attention layer is weighted by a learnable scaling factor γ and added to the input feature vector through a residual connection to obtain the final output. This process is represented as:

[0112] O SA =γ·F SA +x;

[0113] where γ is the scaling factor and is initialized to 0. γ allows the network to control the influence of the self-attention mechanism on the output feature map.

[0114] The SAC-GAN model is used to learn the feature differences between different categories of signals and generate signals, which alleviates the class label imbalance problem of the two public datasets and the self-collected dataset used for pre-training DARFEC, so as to obtain more superior DARFEC performance.

[0115] The use method of the portable electrocardio-skin conductance dual-mode emotion judgment system comprises the following specific steps:

[0116] S1, collecting electrocardio signals and skin conductance signals through a core control mainboard, and transmitting the signals to a data analysis and evaluation module through a Bluetooth wireless receiver;

[0117] S11, setting the sampling frequency of the signal acquisition module to 250Hz, and then synchronously collecting electrocardio signals and skin conductance signals;

[0118] S12, using a main control chip to process the multi-channel physiological signals obtained in the signal acquisition process;

[0119] S13, transmitting the collected electrocardio signals and skin conductance signals to the data analysis and evaluation module through a Bluetooth wireless connection module, and transmitting the physiological signals to a software program in the upper computer.

[0120] S2, after the data analysis and evaluation module receives the collected signals, pre-processing the signals;

[0121] S21, preprocessing the electrocardiosignal, adopting 3-order Butterworth band-pass filtering and notch filtering of 0.5-45Hz to remove high-frequency noise and power frequency interference, and correcting the baseline drift of the electrocardiosignal, removing the slowly varying part in the signal, then standardizing and normalizing the electrocardiosignal to make the amplitudes consistent, synchronously detecting abnormal values of 3-lead signals, and using an interpolation method to compensate for missing and abnormal signals;

[0122] S22, preprocessing the skin conductance signal, adopting 3-order Butterworth band-pass filtering of 0.5-15Hz to remove unnecessary frequency components, then smoothing the skin conductance signal in the time domain to improve the stability of the signal, then standardizing and normalizing the signal to make the amplitudes consistent, synchronously detecting abnormal values of 2-lead signals, and using an interpolation method to compensate for missing and abnormal signals.

[0123] S3, using the SAC-GAN method to perform data augmentation and realize pre-training of the deep learning model;

[0124] S31, using two public data sets and self-collected data sets to enrich feature representation through data augmentation technology to generate a number of effective data;

[0125] S32, using the effective data to repeatedly train the deep learning neural network model double attention feature extraction residual chain (DARFEC) module in the deep learning model;

[0126] S33, enhancing the discriminant ability and generalization performance of different signal data through the S33 enhancement module, and then completing the pre-training task of the model.

[0127] S4, using the trained deep learning model to extract the features of the preprocessed electrocardiosignal and skin conductance signal and fuse them to obtain the prediction result of emotion level discrimination and display it through the software program.

[0128] S41, cutting the preprocessed electrocardiosignal and skin conductance signal into 5s time windows without overlap according to the time sequence, and splicing the time window sequences of the two signals in the channel dimension to obtain a merged time window sequence X concat , the merging process is as follows:

[0129] X concat =[X ECG ,X GSR ];

[0130] Wherein X ECG represents the time window sequence of the electrocardiosignal, and X GSR represents the time window sequence of the skin conductance signal.

[0131] S42, X concatThe shallow feature extraction module in the double-attention feature extraction residual chain module of the input deep learning neural network model preliminarily extracts relevant features for emotion discrimination, and X concat The data is dimensionally promoted to 64 at the channel level, retaining deeper feature representations and capturing more feature patterns that are helpful for auxiliary emotion discrimination, as follows:

[0132] F s = Dropout(BN(Conv1D(Signal mix ), p = 0.3);

[0133] where F s represents a shallow feature vector, Dropout(·) represents randomly discarding network neuron nodes with a probability of p = 0.3, BN(·) represents a batch normalization operation, and Conv1D(·) represents a 1D convolution operation; the shallow feature extraction module consists of two convolution layers, both with the same convolution kernel size of 3, output channel numbers of 8 and 64, respectively, and a probability of randomly discarding network neurons of 0.3.

[0134] S43, after preliminarily extracting relevant features for emotion discrimination, a global attention module is used to deeply mine and learn the basis of the relevant features in S42, as follows:

[0135] S431, the global attention module is used to calculate the correlation coefficients between the input features, and the time and space complexity of this step is O(n 2 ), in order to reduce the consumption of time and space resources by the model and improve the training efficiency.

[0136] S432, the obtained shallow feature vector is uniformly cut into smaller time window in 3 time sequence dimensions, which is regarded as a fine window, as follows:

[0137] F s = {f k1 ,f k2 ,f k3};

[0138] f s ∈ R C*L ,f k1 ∈ R C*L / 3 ,f k2 ∈ R C*L / 3 ,f k3 ∈ R C*L / 3 ;

[0139] S433, the three fine windows are input into the global attention module to perform segmented global attention operation, as follows:

[0140]

[0141] where y i represents the output of performing global attention operation on single tiny window, i represents the time index of output feature, j represents the time index of input feature which will have influence on output, x represents the input feature map, f(x) represents the function of calculating the relationship between features with index i and j, g(x) adjusts the feature representation of input feature vector, C(x) represents normalization, θ(x) and φ(x) represent convolution operation;

[0142] S434, the output of the global attention module is connected with the input through the residual connection, and the weighted output F TGA of the global attention module is obtained

[0143]

[0144] where represents the cut input, represents the input of the global attention mechanism residual connection, W w represents 1D convolution operation, W θ , W φ and W g all represent 1D convolution operation with the same operation as W w but non-shared parameters, Softmax(·) represents exponential normalization function, AvgPool(·) and MaxPool(·) represent global average pooling and global maximum pooling respectively;

[0145] S44, all 2D convolution layers in the original ResNet architecture are replaced by 1D convolution operation using residual connection feature extraction module, and multi-level feature extraction and abstraction are performed on the original 1D physiological signal to obtain the output feature f res of the residual connection;

[0146] In order to continue to excavate the potential features of the input vector, solve the gradient vanishing or explosion problem in the training of deep network, and improve the robustness of the optimal depth required for different tasks, we use the residual connection mechanism. Deep learning neural network has an optimal depth "threshold" for a specific task, and the model performance will decrease significantly beyond this threshold. Using the residual connection mechanism can achieve effective depth self-adaptive expansion or degradation through identity mapping in the training process.

[0147] The residual connection mechanism used reduces the network size, and the entire feature extraction part is composed of only four convolution modules and two down-sampling residual connection modules. Each residual module integrates convolution, batch normalization, ReLU activation function and residual connection. In order to enable the residual connection to quickly capture wide-area features at the beginning, efficient down-sampling is performed. Subsequently, the receptive field is gradually reduced, and more attention is paid to detailed information, avoiding the loss of high-dimensional feature information and enhancing the model's ability to aggregate contextual feature information on ECG and GSR signals; at the same time, the model complexity and computational overhead are greatly reduced, so that the model can automatically adjust the network depth by adjusting the residual connection parameters as needed. The residual connection mechanism used adjusts the size of the convolution kernel of each residual block, and the size of the convolution kernel of the four convolution blocks is adjusted to 7, 5, 3 and 3, respectively, while the size of the convolution kernel of the down-sampling module of the bypass connection is uniformly 1. The optimized residual connection structure pays more attention to balancing the representation ability, training stability and computational efficiency of the model, and provides more discriminative feature representation for subsequent processing tasks.

[0148] S45, using a channel attention module to assign an importance score to the electrocardiogram signal and the skin conductance signal for emotion discrimination, and obtaining a prediction result of the emotion level, the process being as follows:

[0149] S451, the output feature f res of the residual connection is respectively subjected to global maximum pooling and global average pooling operations to obtain the feature representing the maximum significant feature response f max and the average global feature distribution f avg ;

[0150] S452, inputting f max and f avg to a shared multi-layer convolution to capture cross-channel nonlinear interactions, to generate corresponding feature representations F max and F avg ;

[0151] S453, adding F max and F avg and then performing nonlinear mapping through a Sigmoid activation function to generate a channel attention weight matrix M c ;

[0152] S454, scaling the output feature f res of the residual connection according to the attention weight matrix and the weight corresponding to each channel to perform adaptive feature re-labeling on the original feature, to obtain an original feature vector;

[0153] S456, the original feature vector is transmitted to three linear layers, each linear layer is followed by a ReLU activation function, normalized using a Softmax function and selecting the maximum value as the prediction result of the model for the emotional level of the input electrocardiogram and skin conductance signal;

[0154] S46, the prediction result obtained by the deep learning model is transmitted to the software program, and the prediction result and physiological signal are displayed through the software program.

[0155] The software program in the upper computer provides an intuitive graphical interface to display the data fluctuations of various signals and the discrimination results. The specific functions are as follows:

[0156] Real-time data monitoring: after the Bluetooth receiver is successfully paired with the electrocardiogram and skin conductance portable acquisition device, the Bluetooth receiver receives data in real time and transmits it to the upper computer software program of the computer, and the software program displays the electrocardiogram and skin conductance signal data, so that the user can observe the changes of the signal in real time.

[0157] Signal visualization: the collected signal data is displayed in waveform through the graphical interface, and the data record is saved locally, which is convenient for users to understand and analyze the signal characteristics; at the same time, the signal quality is observed through the graphical interface to adjust the electrode position in time to optimize the data acquisition process.

[0158] Evaluation result display: based on the results analyzed by the aforementioned pre-trained deep learning neural network model, the real-time collected signal is subjected to data preprocessing and feature fusion steps, and then every 5s signal window is automatically cropped as the data basis for once emotional discrimination, and the emotional level discrimination result is displayed in real time, including the comprehensive evaluation of physiological signals, to help workers related to emotional mental illness to make diagnosis and decision or scientific research.

[0159] Therefore, the portable electrocardiogram and skin conductance dual-mode emotional discrimination system with the above structure has the following advantages:

[0160] 1. It can synchronously acquire ECG and GSR signals, provide high-quality bioelectric signals, and ensure the clarity and accuracy of the signals through low-noise amplification technology, which is suitable for accurate evaluation of emotional level;

[0161] 2. The system is miniaturized and integrated, meeting the needs of portable application, supporting real-time signal processing and diagnosis, and being suitable for rapid and accurate detection of emotional mental illness in dynamic clinical environment;

[0162] 3. Through the end-to-end feature-level fusion of the residual connection network algorithm with the dual attention mechanism, the system can reduce the data processing steps, efficiently fuse the signal features from two modalities, and improve the recognition accuracy of physiological state changes.

[0163] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit it, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can still be modified or replaced equivalently, and these modifications or equivalent replacements should not make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A portable ECG-skin conductance dual-modal emotion discrimination system, characterized in that: It includes a core control motherboard for collecting electrocardiogram and skin conductance signals, a data analysis and evaluation module for processing the collected electrocardiogram and skin conductance signals and determining emotions, and a Bluetooth wireless receiver that is paired with the core control motherboard to control the core motherboard to collect electrocardiogram and skin conductance signals and transmit the data collected by the core motherboard to the data analysis and evaluation module. Its usage method is as follows: S1. Collects ECG and skin conductance signals through the core control motherboard and transmits them to the data analysis and evaluation module via Bluetooth wireless receiver; S2. After receiving the collected signals, the data analysis and evaluation module preprocesses them. S3. Use the SAC-GAN method for data augmentation and implement a pre-trained deep learning model; S4. Use the trained deep learning model to extract and fuse the features of the preprocessed electrocardiogram and skin conduction signals to obtain the prediction results of emotion level discrimination and display them through software program.

2. The portable ECG-skin conductance dual-modal emotion discrimination system according to claim 1, characterized in that: The core control motherboard includes a signal acquisition module for collecting electrocardiogram and skin conductance signals, a main control chip for processing multi-channel physiological signals, a Bluetooth wireless connection module for communicating and transmitting data with the computer, a power supply module, a charging module, and a switch control module.

3. The portable ECG and skin conductance dual-modal emotion discrimination system according to claim 2, characterized in that: The signal acquisition module includes a multi-channel electrode for acquiring electrocardiogram (ECG) signals and skin conductance signals. The ECG signal acquisition part of the multi-channel electrode uses a 3-lead electrode and is equipped with an independent reference electrode and bias electrode. The skin conductance signal acquisition part uses a 2-lead electrode and uses the same reference electrode and bias electrode as the ECG signal acquisition part.

4. A portable ECG-skin conductance dual-modal emotion discrimination system according to claim 3, characterized in that: The data analysis and evaluation module includes a preprocessing module for preprocessing ECG and skin conductance signals, a feature extraction and fusion module for extracting and fusing ECG and skin conductance signals, and a host computer. The host computer includes a deep learning model for identifying the emotional state corresponding to the bimodal physiological signals and a software program for displaying the emotional state.

5. A portable ECG and skin conductance dual-modal emotion discrimination system according to claim 4, characterized in that: Deep learning models enrich feature representations through data augmentation techniques. Data augmentation techniques are a method of SAC-GAN model that integrates self-attention generative adversarial networks and conditional generative adversarial networks. The deep learning model includes a shallow feature extraction module, a global attention module, a residual connection feature extraction module, and a channel attention module.

6. A portable ECG-skin conductance dual-modal emotion discrimination system according to claim 5, characterized in that, The process of S1 is as follows: S11. Set the sampling frequency of the signal acquisition module, and then synchronously acquire ECG signals and skin conductance signals; S12. Use the main control chip to process the multi-channel physiological signals obtained during the signal acquisition process; S13. The collected electrocardiogram signals and skin conductance signals are transmitted to the data analysis and evaluation module via the Bluetooth wireless connection module, and the physiological signals are transmitted to the software program in the host computer.

7. A portable ECG-skin conductance dual-modal emotion discrimination system according to claim 6, characterized in that, The preprocessing process in S2 is as follows: S21. Preprocess the ECG signal by using bandpass filtering and notch filtering to remove high-frequency noise and power frequency interference, and perform baseline drift correction on the ECG signal. Then, perform standard normalization on the ECG signal, perform abnormal value detection on the synchronous 3-lead signal, and use interpolation method to compensate for lost and abnormal signals. S22. The skin conductance signal is preprocessed by using bandpass filtering to remove high-frequency noise interference. Then, the skin conductance signal is smoothed in the time domain and then normalized. Abnormal values ​​are detected by synchronizing the two-lead signals, and interpolation methods are used to compensate for lost and abnormal signals.

8. A portable ECG-skin conductance dual-modal emotion discrimination system according to claim 7, characterized in that, The process of training a deep learning model using the SAC-GAN method in S3 is as follows: S31. Using two public datasets and a self-collected dataset, data augmentation techniques are employed to enrich the feature representations and generate several effective datasets. S32. Repeatedly train the deep learning neural network model in the deep learning model using effective data to extract the residual chain module of dual attention feature extraction. S33 enhances the model's ability to distinguish different signal data and its generalization performance, thereby completing the model's pre-training task.

9. A portable ECG and skin conductance dual-modal emotion discrimination system according to claim 8, characterized in that, The process of S4 is as follows: S41. Divide the preprocessed ECG and skin conductance signals into non-overlapping 5-second time windows according to the time series, and concatenate the time window sequences of the two signals along the channel dimension to obtain a merged time window sequence X. concat The merging process is as follows: X concat =[X ECG ,X GSR ]; Where X ECG X represents the time window sequence of an electrocardiogram (ECG) signal. GSR This represents a time window sequence of skin conductance signals; S42, X concat The shallow feature extraction module in the dual-attention feature extraction residual chain module of the input deep learning neural network model initially extracts relevant features for emotion judgment, and then X... concat The data dimension at the channel level was increased to 64, as follows: F s =Dropout(BN(Conv1D(Signal mix )),p=0.3; Where F s This represents a shallow feature vector; Dropout(·) means randomly dropping network neuron nodes with a probability of p = 0.3; BN(·) means batch normalization operation; and Conv1D(·) means 1D convolution operation. S43. After initially extracting relevant features for emotion discrimination, the global attention module is used to deeply mine and learn the foundation of relevant features in S42. The process is as follows: S431. Use the global attention module to calculate the correlation coefficients between input features; S432. The obtained shallow feature vectors are uniformly cropped into smaller time windows in three time series dimensions, which are regarded as fine windows. The process is as follows: F s ={f k1 ,f k2 ,f k3 }; f s ∈R C*L ,f k1 ∈R C*L / 3 ,f k2 ∈R C*L / 3 ,f k3 ∈R C*L / 3 ; S433. Input the three small windows into the global attention module to perform segmented global attention operations, as follows: Where y i The output of the global attention operation is performed on a single small window, i represents the temporal index of the output feature, j represents the temporal index of the input feature that will affect the output, x represents the input feature map, f(x) represents the function that calculates the relationship between the features with indices i and j, g(x) adjusts the feature representation of the input feature vector, C(x) represents normalization, and θ(x) and φ(x) represent convolution operations. S434. Connect the output and input of the global attention module through a residual to obtain the weighted output F of the global attention module. TGA The process is as follows; in This indicates cut input. W represents the input to the residual connection of the global attention mechanism. w W represents a 1D convolution operation. θ W φ and W g Both indicate that they are related to W. w The same 1D convolution operation but with different parameters: Softmax(·) represents the exponential normalization function, and AvgPool(·) and MaxPool(·) represent global average pooling and global max pooling, respectively. S44. Using the residual connection feature extraction module, all 2D convolutional layers in the original ResNet architecture are replaced with 1D convolutional operations. Multi-level feature extraction and abstraction are performed on the original 1D physiological signal to obtain the output feature f of the residual connection. res ; S45. Using the channel attention module, assign importance scores to the ECG and skin conductance signals for emotion discrimination, and obtain the prediction results for emotion level discrimination. The process is as follows: S451, Output characteristics f of residual connection res Performing global max pooling and global average pooling operations respectively yields the maximum value of the response representing the significant feature and the average value of the global feature distribution f. max and f avg ; S452, f max and f avg Each input line is fed into a shared multi-layer convolution to capture non-linear interactions across channels, producing the corresponding feature representation F. max and F avg ; S453, F max and F avg After addition, a non-linear mapping is performed through a sigmoid activation function to generate the channel attention weight matrix M. c ; S454. Based on the attention weight matrix, the output features f of the residual connection are... res The weights corresponding to each channel are scaled, and the original features are adaptively recalibrated to obtain the original feature vector. S456. The original feature vector is passed to three linear layers, each followed by a ReLU activation function. The ReLU activation function is used to normalize the ReLU activation function and the maximum value is selected as the prediction result of the model for judging the emotional level of the input ECG signal and skin conductance signal. S46. Transmit the prediction results obtained from the deep learning model to the software program, and display the prediction results and physiological signals through the software program.

Images