Self-supervised emotion recognition method based on heart-brain joint codebook and related equipment

Abstract

The embodiment of the invention provides a self-supervised emotion recognition method based on a heart and brain combined codebook and related equipment, and belongs to the technical field of physiological signal processing and artificial intelligence. The method comprises the following steps: respectively defining heart beats of electrocardiosignals and electroencephalogram signal segments with equal lengths as words, and constructing sentences; a shared heart and brain joint codebook is created and trained, and electrocardio and electroencephalogram words are mapped to a unified discrete semantic space through vector quantization so as to learn cross-subject general characterization; then, discretizing a signal sentence by using the codebook, and combining space and position embedding and inputting a Transform encoder to carry out mask pre-training so as to learn context semantics of the signal; and finally, finely tuning the pre-training model for an emotion recognition task. According to the method, deep semantic fusion of heart and brain signals is realized through signal structuring and codebook sharing, dependence on labeled data is effectively overcome, and emotion recognition accuracy and cross-subject generalization ability are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of physiological signal processing and artificial intelligence, in particular to a self-supervised emotion recognition method based on a heart-brain joint codebook and related equipment. BACKGROUND

[0002] Emotion recognition has important application value in the fields of human-computer interaction, mental health monitoring and neuroscience. Compared with subjective questionnaires, physiological signals such as electrocardiogram (ECG) and electroencephalogram (EEG) can provide more objective and continuous emotion state representation. Heart rate variability (HRV) and other features in electrocardiogram signals are regulated by the autonomic nervous system and are important indicators reflecting emotional arousal; while electroencephalogram signals can capture the neural oscillation patterns of the brain under different emotional states (such as alpha waves and beta waves), and have the ability to represent emotional valence (positivity and negativity).

[0003] Currently, emotion recognition methods based on deep learning mainly face two challenges: 1) dependence on labeled data: mainstream supervised methods require a large amount of high-quality data labeled by experts, which is costly and easily affected by subjective factors. 2) Insufficient modality fusion and generalization ability: existing methods usually treat ECG and EEG as ordinary time series for processing, ignoring their inherent rhythm and morphology structure. Most multi-modal fusion methods stop at feature concatenation or decision-level fusion, failing to achieve deep semantic fusion. In addition, the model is sensitive to physiological differences between different individuals and has limited generalization ability.

[0004] Self-supervised learning (SSL) learns general representations from unlabeled data, providing a solution to the above problems. However, existing SSL methods (such as contrastive learning and reconstruction learning) are mostly directly applied to general sequences, failing to effectively utilize the hierarchical structure of ECG and EEG signals (such as heartbeats and brain rhythm), and lacking a mechanism to naturally fuse multi-modal information. SUMMARY

[0005] The main purpose of the embodiments of the present application is to propose a self-supervised emotion recognition method based on a heart-brain joint codebook and related equipment, which can learn discriminative, cross-modal emotion representations without relying on a large amount of labeled data, and significantly improve the generalization ability of the model in the cross-subject scenario.

[0006] To achieve the above purpose, one aspect of the embodiments of the present application proposes a self-supervised emotion recognition method based on a heart-brain joint codebook, which comprises: obtaining a multi-lead electrocardiogram signal and a multi-lead electroencephalogram signal of a user; detecting heartbeats of the electrocardiogram signal, regarding each detected heartbeat as an electrocardiogram word, and constructing an electrocardiogram sentence from a plurality of serialized electrocardiogram words; The electroencephalogram signal is segmented into multiple electroencephalogram segments of fixed length, each electroencephalogram segment is taken as an electroencephalogram word, and a plurality of electroencephalogram words are serialized to form an electroencephalogram sentence; A shared heart-brain joint codebook is constructed, and the codebook includes a plurality of learnable set vectors; Words in the electrocardiogram sentence and the electroencephalogram sentence are mapped into the heart-brain joint codebook, and a corresponding nearest neighbor set vector is found to obtain a discretized representation; Using the discretized representation, a word reconstruction task is performed by a decoder, and the heart-brain joint codebook is trained based on a reconstruction loss to learn a general representation across subjects; The trained heart-brain joint codebook is used to discretize the input electrocardiogram sentence and electroencephalogram sentence, and input into a Transformer-based sentence encoder, pre-trained through a masked language model task, so that the model learns the context semantic information of the signal; The pre-trained sentence encoder is used for a downstream sentiment recognition task, and a sentiment dimension is classified through fine-tuning.

[0007] In some embodiments, the maximum sequence length of the electrocardiogram sentence and the maximum sequence length of the electroencephalogram sentence are uniformly set to a fixed value L; when the actual length of the sentence is less than L, zero padding is used; when the actual length of the sentence is greater than L, truncation processing is performed.

[0008] In some embodiments, words in the electrocardiogram sentence and the electroencephalogram sentence are mapped into the heart-brain joint codebook through an encoder and a vector quantizer; The vector quantizer uses cosine similarity to find the nearest neighbor set vector for the encoding representation of each input word in the heart-brain joint codebook.

[0009] In some embodiments, the training steps of the heart-brain joint codebook specifically include: The electrocardiogram sentence or the electroencephalogram sentence is input into an encoder to obtain a continuous vector representation of each word; In the shared codebook, the closest codebook vector is found for each continuous vector, and it is replaced by the corresponding discrete set word embedding, wherein the similarity measure uses cosine similarity or L2 distance; The discrete set word embedding is input into a decoder to reconstruct the original word signal; The encoder, the decoder and the codebook are collectively trained using a reconstruction loss and a vector quantization loss, and the codebook update is stabilized using an exponential moving average strategy.

[0010] In some embodiments, the loss function L for training the heart-brain joint codebook is: L = reconstruction loss + codebook loss + commitment loss wherein the reconstruction loss is the mean square error between the model output and the original input; the codebook loss and commitment loss are used to constrain the distance between the encoder output and the codebook vector, and a stop gradient operation is adopted.

[0011] In some embodiments, before inputting the discretized sentence into the Transformer-based sentence encoder, the following steps are further included: adding a spatial embedding and a position embedding to each word embedding in the sentence; wherein the spatial embedding is used to distinguish different leads of the signal, and its dimension corresponds to the number of leads; the position embedding is used to represent the sequential position of the word in the sentence.

[0012] In some embodiments, the sentence input to the sentence encoder is represented as:

[0013] wherein, is the sentence discretized by the heart-brain joint codebook; is a spatial embedding used to distinguish different signal leads; is a position embedding used to represent the sequential position of the word in the sentence.

[0014] In some embodiments, the sentence encoder adopts layer normalization applied to the input of the multi-head self-attention mechanism In some embodiments, the fine-tuning process adopts a strategy of freezing the backbone parameters of the sentence encoder and optimizing only the top classification head.

[0015] To achieve the above object, another aspect of the embodiments of the present application proposes an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0016] To achieve the above object, another aspect of the embodiments of the present application proposes a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0017] To achieve the above object, another aspect of the embodiments of the present application proposes a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the above method.

[0018] Compared with the prior art, the present application has the following remarkable beneficial effects: 1) Innovative signal modeling: Analogizing heartbeats and EEG slices as "words", constructing "sentences", successfully migrating advanced ideas of natural language processing to physiological signal processing, realizing multi-level, structured representation learning of signals from micro-morphology to macro-rhythm.

[0019] 2) Deep modality fusion: Through the shared "heart-brain joint codebook", early fusion is realized at the discrete semantic level of ECG and EEG signals. The codebook naturally learns the joint distribution of the two modalities, enabling the model to implicitly utilize the complementary information between modalities, overcoming the limitations of traditional late fusion methods.

[0020] 3) Strong generalization ability: Through vector quantization, individual-specific signals are mapped to cross-subject general "set words", effectively eliminating the interference caused by individual physiological differences, significantly improving the model's performance on unseen subject data.

[0021] 4) High data efficiency: The self-supervised learning framework makes full use of a large amount of unlabeled physiological data, reducing the dependence on expensive expert labeling and reducing application costs.

[0022] 5) Strong practicality: The entire method flow is complete, its effectiveness is verified on public datasets, and efficient fine-tuning strategies are provided, facilitating deployment and application in practical scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a flowchart of a self-supervised emotion recognition method based on a heart-brain joint codebook in an embodiment of the present application; Figure 2 is a division diagram of ECG words and EEG words in an embodiment of the present application.

[0024] Figure 3 is a training flowchart of a heart-brain joint codebook in an embodiment of the present application.

[0025] Figure 4 is a structure flowchart of a sentence encoder in an embodiment of the present application.

[0026] Figure 5 is a hardware structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application and are not intended to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the present application. They are only examples of apparatuses and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification herein is for describing the embodiments of the present application only and is not intended to limit the present application.

[0029] Before the embodiments of the present application are described in detail, first, some nouns and terms involved in the embodiments of the present application are described, and the nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0030] 1) Electrocardiogram (ECG) is a technique for recording the electrical activity of the heart during each cardiac cycle on the body surface using an electrocardiograph.

[0031] 2) Electroencephalogram (EEG) is a detection technique for recording the spontaneous bioelectric potential patterns of the brain from the scalp using electronic instruments, mainly applied to the diagnosis of epilepsy, mental illness, etc.

[0032] Emotion recognition, as an interdisciplinary subject combining psychology, computer science and cognitive science, has broad application prospects in public safety, mental health and education. Compared with subjective reports, physiological signals can provide more objective emotional representations. Among them, the heart rate variability (HRV) reflected by electrocardiogram (ECG) signals is directly regulated by the autonomic nervous system and is an important indicator for depicting emotional state changes. Similarly, electroencephalogram (EEG) signals can capture specific neural oscillation patterns (such as alpha waves, beta waves, etc.) under different emotional states by recording the electrical activity of the cerebral cortex. Therefore, ECG and EEG have become the most critical data sources in physiological signal-based emotion recognition research due to their rich and complementary emotional information.

[0033] However, it is not easy to build efficient sentiment recognition models with these key data sources. Currently, the mainstream supervised deep learning methods face a major limitation when dealing with ECG and EEG signals: they rely on large-scale, expert-reviewed, annotated high-quality data. Moreover, even with enough data, these methods are often designed for specific tasks, which limits the model's generalization ability. To overcome these challenges, self-supervised learning (SSL) has proven effective by training on large amounts of unlabeled electrocardiogram and electroencephalogram recordings to learn general electrocardiogram and electroencephalogram representations, and then fine-tuning for specific downstream tasks.

[0034] Current SSL methods can be mainly divided into two categories: contrast-based methods and reconstruction-based methods. The core principle of contrast-based methods is to create positive and negative sample pairs, aiming to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. Reconstruction-based methods focus on training models to reconstruct the original input from partial or transformed data, thereby learning effective data representations. However, both of the above methods treat electrocardiogram and electroencephalogram as ordinary sequences, which has significant drawbacks, ignoring the multi-level information in electrocardiogram and electroencephalogram.

[0035] To solve the above problems, the present application proposes a self-supervised learning framework for electrocardio language and electroencephalogram language processing--Heart-Brain Emo. Specifically, the present application proposes a self-supervised emotion recognition method based on a heart-brain joint codebook, an electronic device, a storage medium and a program product. Taking electrocardio signals as an example, a significant feature of electrocardio signals is the clear visibility of heart rate patterns, in which patterns the heartbeat of an individual is easily recognized. The core concept of our framework is to regard a single heartbeat as a word and regard multi-lead electrocardio records as a sentence, thereby realizing self-supervised learning from the micro level of heartbeat morphology to the macro level of signal sequence to capture multi-level general representation. The method includes four key parts: (1) Tokenizer, which is used to construct an original electrocardio signal into an "electrocardio sentence", wherein each detected heartbeat is regarded as a basic component "word"; (2) Transformer Encoder, which uses self-attention to enhance the potential semantic extraction of the electrocardio sentence; (3) a vector quantization (VQ) method is used to construct an electrocardio codebook, and the model learns the discrete representation of the heartbeat at the morphology level by quantizing the input heartbeat (QRS token) into the nearest neighbor vector in the codebook and performing the reconstruction task; and (4) Masked electrocardio sentence pre-training, which can effectively drive the model to learn the rhythm representation of the electrocardio signal by randomly masking part of the tokens in the electrocardio sentence and letting the model perform the reconstruction task. Through these methods, our method can simultaneously learn the morphology level and rhythm level representation of the electrocardio signal, and can extract the potential semantic representation of the electrocardio sentence. Given that electroencephalogram signals lack typical waveform features such as electrocardio QRS waves, in order to ensure consistency with electrocardio tokens in the time scale, the present research adopts a slice processing of equal length to the QRS token for the electroencephalogram signal. The fixed length electroencephalogram signal is regarded as a "word", thereby constructing an original electroencephalogram signal into an "electroencephalogram sentence".

[0036] The embodiment of the application provides a self-supervised emotion recognition method based on a heart-brain joint codebook, and relates to the technical field of physiological signal processing and artificial intelligence. The self-supervised emotion recognition method based on the heart-brain joint codebook can be applied to a terminal, can also be applied to a server, and can be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a stand-alone physical server, can also be configured as a server cluster or a distributed system formed by multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network; the software can be an application for implementing the self-supervised emotion recognition method based on the heart-brain joint codebook, and the like, but is not limited to the above forms.

[0037] The application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0038] As shown in Figure 1 The embodiment provides a self-supervised emotion recognition method based on a heart-brain joint codebook, including the following steps: S1: signal preprocessing and sentence formation.

[0039] Specifically, the original multi-lead electrocardiogram signal is subjected to heart beat detection, each heart beat is regarded as an “electrocardiogram word”, and a sequence thereof forms an “electrocardiogram sentence”; the original multi-lead electroencephalogram signal is segmented into fixed-length segments equal to the heart beat duration, each segment is regarded as an “electroencephalogram word”, and a sequence thereof forms an “electroencephalogram sentence”. The sentence length is unified to a fixed value by zero padding or truncation.

[0040] S2: heart-brain joint codebook training.

[0041] Specifically, a shared codebook containing multiple learnable set vectors is constructed. The words in the electrocardiogram and electroencephalogram sentences are mapped to continuous vectors by an encoder, and then the nearest neighbor discrete set vector is found in the shared codebook by a vector quantizer. The codebook is trained to learn the cross-subject general discrete representation by a composite loss function containing reconstruction loss, codebook loss and commitment loss, and an exponential moving average strategy is adopted.

[0042] In some embodiments, the vector quantization process uses cosine similarity as the metric for nearest neighbor search.

[0043] S3: Sentence encoder pre-training.

[0044] Specifically, the input signal sentence is discretized using the trained codebook, and spatial embedding and position embedding are added. Then, the processed sentence is input into the Transformer-based sentence encoder, pre-trained by the mask language model task of randomly masking part of the words and making the model reconstruct the mask, so that the model learns the context semantic information of the signal.

[0045] In some embodiments, the dimension of the spatial embedding corresponds to the number of leads of the signal, which is used to distinguish the spatial information of different leads.

[0046] S4: Downstream task fine-tuning.

[0047] Specifically, the pre-trained sentence encoder is applied to downstream tasks such as emotion recognition. The strategy of freezing the backbone network and optimizing only the top classification head is adopted for fine-tuning to achieve efficient transfer and prevent overfitting.

[0048] In the following, the scheme of the embodiments of the present application is described in detail in conjunction with the drawings and specific application examples.

[0049] In this embodiment, first, the multi-lead electrocardiogram data is defined as Xecg∈R C1xT , Xeeg∈R C2xT , where C1 represents the number of electrocardiogram leads, C2 is the number of electroencephalogram electrodes (channels), and T represents the total time stamp. The configuration of the electrocardiogram leads follows the standard 12-lead electrocardiogram setting, and the electroencephalogram uses 14 leads.

[0050] (1) Generate electrocardiogram sentences and electroencephalogram sentences For electrocardiogram signals, a key concept of the method of the present embodiment is to regard heartbeats as words, so that it is crucial to segment the original electrocardiogram signal into "electrocardiogram sentences", as shown in (a) of Figure 2 For electroencephalogram signals, the electroencephalogram signal is cut into segments equal in length to a QRS token, thereby obtaining the corresponding "electroencephalogram sentences", as shown inFigure 2 (b) is shown.

[0051] After segmentation, the 12-lead single ECG words and 14-lead single EEG words are connected in order respectively to form the whole ECG sentence xecg∈R lxt and the whole EEG sentence xeeG∈R lxt , where denotes the sequence length, and t denotes the time window size. Considering the variability of different sample heart rates, the sequence length obtained is inconsistent. Similar to natural language processing, we set as the maximum length of the ECG sentence. If the ECG sentence length is less than , it is filled to by zero padding; if the ECG sentence length exceeds , the interval length is truncated to . In this embodiment, we set to 256 and t to 96. In addition, the EEG signal is also processed in the same way as the ECG signal.

[0052] (2) Vector quantization and reconstruction training The single ECG words and EEG words segmented according to the QRS wave and fixed length lack generalization characteristics, because each individual word is independent. This application aims to let Heart-Brain Emo learn cross-subject general representation in the subsequent pre-training phase. To achieve this goal, this embodiment introduces an ECG and EEG vocabulary - a codebook containing a set of ECG words and EEG words, and these set ECG words and EG words have cross-subject generalization ability. Wherein, the same type of words between different subjects should be consistent at the semantic level. Similar individual words from different subjects are mapped to the same discrete and compact collective ECG word, thereby overcoming the physiological differences between subjects.

[0053] 2.1) Vector quantization. First, define a heart-brain vocabulary V = {v i |i = 1,..., k} ∈ R kxd , where k is the number of set heart-brain words in the vocabulary, and d is the dimension of each set heart-brain word. For ECG signals, given an ECG signal Xecg∈R C1xT , a heartbeat is regarded as a word, so that the original ECG signal becomes an ECG sentence. After inputting the obtained ECG sentence into the Encoder, a set of ECG word representations P = {p i |i = 1,..., } is obtained. Then, use the quantizer to convert them into set ECG unit embeddings. The heart-brain vocabulary looks up each interval representation p iThe nearest neighbors of the query are found using cosine similarity to find the closest set of word embeddings. This procedure can be formulated as:

[0054] where v i represents the codebook vector, represents regularization. Correspondingly, the same procedure is applied to the EEG.

[0055] 2.2) Reconstruction training. After the output of the encoder is discretized, the discrete set of word embeddings is normalized to obtain , which is fed into the decoder to reconstruct the corresponding words. To make the updating of the codebook more stable, the exponential moving average (EMA) strategy is adopted in this embodiment. The mean square error (MSE) loss is used to guide the quantization and reconstruction process. Finally, the loss function used to train the codebook is defined as:

[0056] where represents the reconstructed word, and sg represents the stop gradient operation, which is defined as the identity function in the forward pass and has zero gradient. The specific process is shown in Figure 3 .

[0057] It should be particularly noted that the codebook of this embodiment can represent two modalities, one is electrocardiogram and the other is electroencephalogram. When training the codebook, the electrocardiogram and the electroencephalogram are input alternately, so that the codebook can better integrate the semantics of the two modalities.

[0058] (3) Sentence Encoder In order to more effectively capture the spatiotemporal features and potential semantic relationships in the electrocardiogram and electroencephalogram sentences, this embodiment designs a backbone network named Sentence Encoder. The backbone network is used for masking the electrocardiogram and electroencephalogram sentence pre-training and downstream task fine-tuning. The specific process is shown in Figure 4 .

[0059] Since the electrocardiogram and electroencephalogram codebook has been trained, during pre-training, inspired by the natural language field, the input signal sentence is first discretized, and the discrete vector in the codebook is used to replace the word in the input signal sentence. This step is similar to the embedding process in natural language. Then the discretized output is set with a mask, and the pre-training process is similar to the pre-training process of Bert, and the masked part is restored. In order to make the Sentence Encoder better capture the spatial information of the signal sentence, this embodiment proposes a spatial embedding set where n denotes the number of leads. In ECG signals, n takes the value of 12, while for EEG signals, n takes the value of 14. Both D and are learnable during training. Next, a cls token is added at the beginning of the sequence to enhance the representation. In addition, a list of position embeddings is introduced to enhance the sequential relationship between individual signal words. Thus, a signal sentence can be described by the following equation:

[0060] where, denotes the input signal sentence.

[0061] Finally, the signal sentence will be input into the Transformer Encoder. To ensure stability during training, this embodiment adopts layer normalization, which is applied to the input of the attention mechanism:

[0062]

[0063] where, , and denote the projection matrices of Q, K and V, denotes the size of each head in multi-head attention, and LN denotes layer normalization.

[0064] (4) Experiments This subsection demonstrates the results of the method proposed in this embodiment on the public datasets AMIGOS and DREAMER.

[0065] 4.1) Datasets The AMIGOS dataset contains electrocardiogram, electroencephalogram signals and emotion ratings of 40 participants. These data were collected when the participants watched emotionally evocative videos to induce emotional states. The videos varied in length, from 250 seconds to 14 minutes. After each experimental trial, the participants rated their emotional experience on a 1 to 9 scale, evaluating three key dimensions: valence (V), arousal (A) and dominance (D).

[0066] The DREAMER dataset contains electrocardiogram and electroencephalogram signals of 23 participants while watching 18 emotionally evocative video clips, each lasting 60 seconds. Before each emotionally evocative video was played, a neutral video clip was shown to help the participants return to a neutral emotional state. After each video was played, a self-assessment model was used to rate arousal and valence on a 1 to 5 scale.

[0067] 4.2) Experimental Setup The codebook is trained on two NVIDIA RTX 4090 GPUs in this embodiment experiment, mean squared error (MSE) is used as the loss function, and AdamW optimizer is used for parameter update. The weight decay coefficient is set to 0.01 in the optimizer to clearly distinguish the regularization term from the adaptive learning rate mechanism. The initial learning rate is set to and the learning rate is dynamically adjusted using the cosine annealing strategy.

[0068] In the pre-training and downstream task fine-tuning phase, the experiment is completed on an RTX 4090 GPU. The cross-entropy loss function is used, and the AdamW optimizer is also used. The learning rate in the pre-training phase is set to , and the cosine annealing scheduling strategy is used; the learning rate in the downstream task fine-tuning phase is set to During fine-tuning, this embodiment uses the strategy of "freezing the backbone and optimizing only the classification head", that is, only updating the parameters of the final classification layer, to enhance the generalization ability of the model and avoid overfitting.

[0069] 4.3) Experimental results Table 1 Experimental results on DREAMER and AMIGOS datasets, where V is Valence, A is Arousal, and D is Dominance.

[0070] Table 1

[0071] The experimental results are shown in Table 1, and the method proposed in this embodiment achieves good emotion recognition performance on the DREAMER and AMIGOS datasets. Taking the recognition of ECG signals for arousal as an example, it achieves an accuracy of 63.89% and 75.33% on the DREAMER and AMIGOS datasets, respectively, indicating that the method proposed in this embodiment has certain adaptability to changes in different data collection environments and experimental paradigms.

[0072] Analysis of electrocardiogram signals reveals their specific association with emotional dimensions. In the DREAMER dataset, ECG signals have the best recognition effect on dominance (D) (Accuracy: 72.22%, F1: 41.6%), which indicates that heart rate variability and other features regulated by the autonomic nervous system may be closely related to the perceived sense of control (dominance). In addition, ECG achieved an accuracy of 75.33% and an F1 score of 67.27% in recognizing arousal on the AMIGOS dataset, further confirming the potential advantage of ECG in distinguishing emotional intensity (high / low arousal).

[0073] In contrast, EEG signals exhibit different characteristics. In the AMIGOS dataset, the recognition accuracy of EEG on valence (V) and dominance (D) reaches 89.74% and 84.62%, respectively, which is particularly outstanding. This implies that the high-level neural activity of the cerebral cortex may have stronger representation ability for distinguishing the positive and negative nature of emotions (valence) and control-related feelings (dominance).

[0074] Most importantly, ECG and EEG signals exhibit clear and complementary performance patterns on different emotional dimensions and datasets. For example, in the AMIGOS dataset, ECG is better at recognizing arousal (A) (Acc: 75.33%), while EEG is better at recognizing valence (V) (Acc: 89.74%) and dominance (D) (Acc: 84.62%). This modality-dimension specificity strongly suggests that a single physiological signal is difficult to fully capture complex emotional states.

[0075] The method proposed in this embodiment simultaneously inputs ECG and EEG signals to train a shared codebook. This codebook naturally captures the complementary features between the two modalities by learning their joint distribution. Therefore, even without explicit fusion, the model can benefit from multi-modal information using the discrete representation in the codebook. This design enables the model to implicitly select the most relevant modality features when processing different emotional dimensions, thereby providing a richer and more robust feature basis for subsequent downstream tasks such as emotion classification.

[0076] (5) Summary In summary, this embodiment proposes a self-supervised learning method for Heart-Brain Emo, which discretizes ECG and EEG signals through a codebook. However, during codebook training, ECG and EEG signals are simultaneously input to train a shared codebook. By learning the joint distribution of the two modalities, the complementary features of ECG and EEG signals in emotion recognition are captured. During pre-training, the model is better able to understand the physiological and pathological states represented by the signals through mask settings, thereby achieving more accurate, robust, and intelligent heart-brain ECG signal analysis.

[0077] The embodiments of the present application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0078] It can be understood that the contents in the above method embodiments are applicable to the present device embodiments, and the present device embodiments specifically implement the same functions as the above method embodiments and achieve the same beneficial effects as the above method embodiments.

[0079] Referring to Figure 5 , Figure 5 The hardware structure of an electronic device is illustrated, and the electronic device includes: The processor 501 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application. The memory 502 can be implemented by a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), and the like. The memory 502 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 502 and are called and executed by the processor 501 to implement the above-mentioned method of the embodiments of the present application. The input / output interface 503 is configured to realize information input and output. The communication interface 504 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like). The bus 505 is configured to transmit information between various components (for example, the processor 501, the memory 502, the input / output interface 503, and the communication interface 504) of the device. The processor 501, the memory 502, the input / output interface 503, and the communication interface 504 are connected to each other through the bus 505 to realize the communication connection between the device.

[0080] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned method.

[0081] It can be understood that the content in the above-mentioned method embodiments is applicable to the present storage medium embodiments. The present storage medium embodiments specifically implement the same functions as the above-mentioned method embodiments, and achieve the same beneficial effects as the above-mentioned method embodiments.

[0082] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0083] The embodiment of the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements the method described above.

[0084] It can be understood that the contents in the above method embodiments are all applicable to the program product embodiments, the program product embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments. The executable computer program code or "code" for executing various embodiments can be written in a high-level programming language such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0085] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0086] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0087] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0088] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0089] The terms "first", "second", "third", "fourth" and the like in the description of this application and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover a changeable order, sequence or arrangement, if any. Further, the terms "comprising", "having", "including", and "containing" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product or apparatus that comprises, has, includes or contains a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, system, product or apparatus.

[0090] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including single or multiple combinations of items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0091] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0092] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0093] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0094] If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0095] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A self-supervised emotion recognition method based on a mind-brain joint codebook, characterized in that, The method includes the following steps: Acquire the user's multi-lead electrocardiogram (ECG) and multi-lead electroencephalogram (EEG) signals; The electrocardiogram (ECG) signal is subjected to heartbeat detection, and each detected heartbeat is treated as an ECG word. A series of sequential ECG words are then used to form an ECG sentence. The EEG signal is divided into multiple EEG segments, each EEG segment is treated as an EEG word, and multiple sequential EEG words are used to form an EEG sentence. Construct a shared mind-brain joint codebook, which contains multiple learnable set vectors; The words in the ECG and EEG sentences are mapped to the combined ECG and EEG codebook, and the corresponding nearest neighbor set vectors are found to obtain a discretized representation. Using the discretized representation, a word reconstruction task is performed through a decoder, and the mind-brain joint codebook is trained based on the reconstruction loss to learn a general representation across subjects. The input ECG and EEG sentences are discretized using a trained joint codebook and then fed into a Transformer-based sentence encoder. The encoder is pre-trained using a masked language model task to enable the model to learn the contextual semantic information of the signals. The pre-trained sentence encoder is used for downstream sentiment recognition tasks, and the classification of sentiment dimensions is achieved through fine-tuning.

2. The method according to claim 1, characterized in that, The maximum sequence length of the ECG sentence and the maximum sequence length of the EEG sentence are uniformly set to a fixed value L; when the actual sentence length is less than L, zero padding is used; when the actual sentence length is greater than L, truncation is performed.

3. The method according to claim 1, characterized in that, The words in the ECG and EEG sentences are mapped to the combined ECG and EEG codebook using an encoder and a vector quantizer. The vector quantizer uses cosine similarity to find the set vector of nearest neighbors for the encoded representation of each input word in the brain-heart joint codebook.

4. The method according to claim 1, characterized in that, The training steps for the aforementioned mind-brain joint codebook specifically include: The ECG or EEG sentence is input into an encoder to obtain a continuous vector representation of each word; In the shared codebook, for each continuous vector, find the closest codebook vector and replace it with the corresponding discrete set of word embeddings; The discrete set of words is embedded into an input decoder to reconstruct the original word signal; The encoder, decoder, and codebook are trained using reconstruction loss and vector quantization loss, and the codebook updates are stabilized using an exponential moving average strategy.

5. The method according to claim 1, characterized in that, The loss function L for training the heart-brain joint codebook is: L = Reconstruction Loss + Codebook Loss + Commitment Loss The reconstruction loss is the mean square error between the model output and the original input; the codebook loss and commitment loss are used to constrain the distance between the encoder output and the codebook vector, and a stopping gradient operation is employed.

6. The method according to claim 1, characterized in that, Before inputting the discretized sentence into the Transformer-based sentence encoder, the following steps are also included: Add spatial embeddings and positional embeddings to each word embedding in the sentence; The spatial embedding is used to distinguish different leads of the signal, and its dimension corresponds to the number of leads; the positional embedding is used to represent the sequential position of words in a sentence.

7. The method according to claim 1 or 6, characterized in that, The sentence representation input to the sentence encoder is as follows: in, It is a sentence after being discretized by the aforementioned heart-brain joint codebook; It is spatial embedding, used to distinguish different signal leads; It is a positional embedding used to represent the sequential position of words in a sentence.

8. The method according to claim 1, characterized in that, The sentence encoder employs layer normalization applied to the input of a multi-head self-attention mechanism; The fine-tuning process employs a strategy of freezing the backbone parameters of the sentence encoder and optimizing only the top classification head.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images