State emotion classification method, system, and model training method

Abstract

The application provides a state emotion classification method, system and model training method, which can be applied to the technical field of state emotion classification. The method comprises the following steps: encoding a physiological feature graph to obtain physiological encoding data, wherein the physiological feature graph is constructed according to physiological feature data extracted from physiological signal data of each of a plurality of physiological channels, and a physiological feature node in the physiological feature graph represents physiological feature data of a physiological channel; encoding an electroencephalogram feature graph to obtain electroencephalogram encoding data, wherein the electroencephalogram feature graph is constructed according to electroencephalogram feature data extracted from electroencephalogram signal data of each of a plurality of electroencephalogram channels, and an electroencephalogram feature node of the electroencephalogram feature graph represents electroencephalogram feature data of an electroencephalogram channel; fusing the physiological encoding data and the electroencephalogram encoding data to obtain multi-modal fusion encoding data; and obtaining a state classification result and an emotion classification result of a target object according to the multi-modal fusion encoding data.

Description

Technical Field

[0001] This invention relates to the field of state emotion classification technology, and more specifically, to a state emotion classification method, system, and model training method. Background Technology

[0002] With the rapid development of intelligent cockpits, wearable devices, and human-vehicle interaction systems, real-time monitoring of drivers' cognitive states has become a key element in improving safety and the human-vehicle interaction experience. This is especially true for the long-haul transportation industry, where drivers' cognitive states (such as focus and cognitive load) directly impact decision-making accuracy and reaction time.

[0003] However, the methods for assessing the cognitive state of drivers in related technologies suffer from problems such as strong lag, difficulty in quantification, and large individual differences. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a state emotion classification method, system, and model training method.

[0005] One aspect of this invention provides a state and emotion classification method, comprising: encoding a physiological feature map to obtain physiological encoded data, wherein the physiological feature map is constructed based on physiological feature data extracted from physiological signal data of a target object in multiple physiological channels, the physiological feature nodes in the physiological feature map represent the physiological feature data of the physiological channels, and the physiological feature edge weights of the physiological feature map represent the degree of correlation between the physiological feature data corresponding to two connected physiological feature nodes; encoding an electroencephalogram (EEG) feature map to obtain EEG encoded data, wherein the EEG feature map is constructed based on EEG feature data extracted from EEG signal data of the target object in multiple EEG channels, the EEG feature nodes in the EEG feature map represent the EEG feature data of the EEG channels, and the EEG feature edge weights of the EEG feature map represent the degree of correlation between the EEG feature data corresponding to two connected EEG feature nodes; fusing the physiological encoded data and the EEG encoded data to obtain multimodal fusion encoded data; and obtaining a state classification result and an emotion classification result for the target object based on the multimodal fusion encoded data.

[0006] Another aspect of this invention provides a method for training a state-emotion classification model, comprising: encoding a sample physiological feature map to obtain sample physiological encoded data, wherein the sample physiological feature map is constructed based on sample physiological feature data extracted from sample physiological signal data of a sample target object in multiple sample physiological channels, sample physiological feature nodes in the sample physiological feature map represent the sample physiological feature data of the sample physiological channel, and the sample physiological feature edge weights of the sample physiological feature map represent the degree of correlation between the sample physiological feature data corresponding to two connected sample physiological feature nodes; and encoding a sample electroencephalogram (EEG) feature map to obtain sample EEG encoded data, wherein the sample EEG feature map is constructed based on sample EEG feature data extracted from sample EEG signal data of the sample target object in multiple sample EEG channels, and the sample EEG feature map... The sample EEG feature nodes represent the sample EEG feature data of the sample EEG channel, and the sample EEG feature edge weights of the sample EEG feature map represent the degree of correlation between the sample EEG feature data corresponding to the two connected sample EEG feature nodes. The sample physiological coding data and the sample EEG coding data are fused to obtain sample multimodal fusion coding data. Based on the sample multimodal fusion coding data, the sample state classification result and sample emotion classification result of the sample target object are obtained. Based on the target loss function, a deep learning model is trained according to the sample state classification result, the sample emotion classification result, the sample multimodal fusion coding data, state emotion category features, the sample physiological coding data, the sample EEG coding data, and sample labels. The state emotion category features are obtained from the sample multimodal fusion coding data corresponding to the same sample label.

[0007] Another aspect of the present invention provides a state emotion classification system, including: a physiological signal data acquisition module configured to acquire physiological signal data of a target object; an electroencephalogram (EEG) signal data acquisition module configured to acquire EEG signal data of the target object; an electronic device including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described method.

[0008] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the method described above.

[0009] Another aspect of the present invention provides a computer program product including computer-executable instructions that, when executed, implement the method described above.

[0010] According to embodiments of the present invention, physiological feature maps and electroencephalogram (EEG) feature maps are constructed based on physiological feature data and EEG feature data, respectively, reflecting the characteristics of physiological feature data and EEG feature data in their respective channels and the degree of correlation between different channels. By encoding the physiological feature map to obtain physiological coded data and encoding the EEG feature map to obtain EEG coded data, features from the physiological feature map and EEG feature map can be further compressed and extracted. By fusing the physiological coded data and EEG coded data, data features can be enhanced from different modalities to obtain multimodal fused coded data, thereby improving the accuracy of state classification results and emotion classification results obtained based on multimodal fused coded data. Attached Figure Description

[0011] The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0012] Figure 1 A block diagram of a state emotion classification system applicable according to an embodiment of the present invention is shown.

[0013] Figure 2 A flowchart of a state emotion classification method according to an embodiment of the present invention is shown.

[0014] Figure 3 A block diagram of a state emotion classification device according to an embodiment of the present invention is shown.

[0015] Figure 4 A flowchart of a state emotion classification model training method according to an embodiment of the present invention is shown.

[0016] Figure 5 A block diagram of an electronic device suitable for implementing the methods described above, according to an embodiment of the present invention, is shown. Detailed Implementation

[0017] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0018] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0019] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0020] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0021] In the embodiments of this invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to maintain the security of user personal information and network security.

[0022] In the embodiments of the present invention, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0023] Cognitive activities are usually accompanied by significant neurophysiological changes. Electroencephalogram (EEG) signals can reflect the information processing in the cerebral cortex, while peripheral circulatory indicators such as blood pressure and heart rate can reflect the balance between the sympathetic and parasympathetic nervous systems. However, cognitive recognition methods in related technologies still face bottlenecks: 1. Single-signal methods have obvious limitations. Methods using EEG signals for recognition often rely on high-density leads or laboratory-grade systems, which are costly, complex to wear, and unsuitable for long-term mobile scenarios; while methods based solely on peripheral indicators such as heart rate and blood pressure are easy to obtain, they lack sensitivity to short-term cognitive fluctuations. 2. Simple feature fusion methods. The fusion of multi-source physiological signals is usually only at the feature-level splicing or linear weighting stage, making it difficult to characterize the nonlinear dependencies between modalities; the time series of multi-source physiological signals have asynchronous and drift problems, and direct cascading often leads to feature redundancy. 3. Insufficient individualization and adaptation. The basic physiological characteristics of different subjects vary significantly, such as resting heart rate, blood pressure level, and EEG power spectrum morphology, making it difficult for a unified model to generalize. In addition, cognitive state characteristics also dynamically shift with changes in task duration and environmental factors, and static models cannot cope with real-time drift.

[0024] In view of this, the present invention provides a state-emotion classification method. By encoding the physiological feature maps of cross-modal physiological modalities and the EEG feature maps of EEG modalities, respective physiological and EEG encoded data are obtained. The graph structure provides more features for multimodal data fusion. Unlike vector concatenation, the graph structure can characterize the topological relationships between channels of EEG signal data and physiological signal data such as blood pressure and heart rate in the form of nodes and edges, explicitly representing the interdependence of cross-modal data as a feature map. This allows the multimodal fusion encoded data to not only focus on changes in individual modal data but also capture the synchronization or coupling patterns between the brain, heart, and blood vessels.

[0025] Figure 1 A block diagram of a state emotion classification system applicable according to an embodiment of the present invention is shown.

[0026] like Figure 1 As shown, the state emotion classification system 100 may include a physiological signal data acquisition module 110, an electroencephalogram (EEG) signal data acquisition module 120, and an electronic device 130.

[0027] The physiological signal data acquisition module 110 is configured to acquire physiological signal data of the target object.

[0028] The EEG signal data acquisition module 120 is configured to acquire EEG signal data of the target object.

[0029] Electronic device 130 includes one or more processors and a memory. The memory is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0030] The physiological signal data acquisition module 110 can be integrated into a wearable physiological signal acquisition device. The physiological signal data can include various data such as heart rate data and blood pressure data. For example, the physiological signal data acquisition module can collect the driver's heart rate data and blood pressure data as physiological signal data.

[0031] The physiological signal data acquisition module may include a power supply module, a physiological measurement submodule, and a transmission submodule. The power supply module provides a stable voltage of 4.5V to 5.5V to the physiological signal data acquisition module. The physiological measurement submodule can acquire the surface pulse wave signal of the target object through an internal broadband high-sensitivity optical sensor, and directly process it into physiological signal data related to heart rate and blood pressure through built-in signal conditioning and algorithms. Heart rate-related data may include the instantaneous mean of the RR interval (RRI) sequence, the standard deviation of RRI, the power of low-frequency (PLF) of RRI, and the power of high-frequency (PHF) of RRI. Blood pressure-related data may include heart rate (HR), systolic pressure (SP), diastolic pressure (DP), and microvascular perfusion index (MPI). Finally, the physiological signal data is output as data packets to the electronic device 130 in real time through the transmission submodule via the Universal Asynchronous Receiver / Transmitter (UART) serial interface.

[0032] The EEG signal data acquisition module 120 can be integrated into a portable EEG cap. The EEG signal data acquisition module 120 may include an EEG power supply module, an EEG signal acquisition submodule, a data storage submodule, and a wireless transmission submodule.

[0033] The EEG power supply module provides operating voltage to the EEG signal data acquisition module 120. The EEG signal acquisition submodule acquires and processes the EEG signals of the target object to obtain EEG signal data. The data storage submodule stores the EEG signal data. The wireless transmission submodule transmits the EEG signal data to the electronic device 130.

[0034] The EEG signal acquisition submodule may include EEG acquisition electrodes and connectors, a high-precision bioelectrical signal acquisition unit, and a microcontroller unit (MCU). The EEG acquisition electrodes and connectors can be integrated onto a portable 8-channel dry electrode cap. The electrode distribution follows the standard lead electrode distribution: Fp1, Fp2, F3, F4, Cz, P3, P4, and Oz. The sampling rate can be 250Hz, used to acquire the target subject's EEG signals. The EEG acquisition electrodes and connectors can be connected to the high-precision bioelectrical signal acquisition unit via a flexible flat cable. The high-precision bioelectrical signal acquisition unit is used for bioelectrical signal acquisition and transmission. It may integrate a high common-mode rejection ratio analog input module, a low-noise programmable gain amplifier, and a high-resolution synchronous sampling analog-to-digital converter for EEG signal processing and amplification, as well as analog-to-digital signal conversion. The microcontroller unit controls the high-precision bioelectrical signal acquisition unit.

[0035] The electronic device 130 may include a first obtaining submodule, a second obtaining submodule, a third obtaining submodule and a fourth obtaining submodule.

[0036] The first submodule is used to encode the physiological feature map to obtain physiological encoded data.

[0037] Physiological feature maps are constructed based on physiological feature data extracted from physiological signal data of target objects in multiple physiological channels. Physiological feature nodes in the physiological feature map represent the physiological feature data of physiological channels, and the physiological feature edge weights of the physiological feature map represent the degree of correlation between the physiological feature data corresponding to the two connected physiological feature nodes.

[0038] The second submodule is used to encode the EEG feature map to obtain EEG encoded data.

[0039] EEG feature maps are constructed based on EEG feature data extracted from the EEG signal data of the target object in multiple EEG channels. The EEG feature nodes of the EEG feature map represent the EEG feature data of the EEG channel, and the EEG feature edge weights of the EEG feature map represent the degree of correlation between the EEG feature data corresponding to the two connected EEG feature nodes.

[0040] The third submodule is used to fuse physiological coding data and EEG coding data to obtain multimodal fused coding data.

[0041] The fourth submodule is used to obtain the state classification result and emotion classification result of the target object based on the multimodal fusion encoded data.

[0042] The specific implementations of the first, second, third, and fourth obtaining submodules can be found in the descriptions of other embodiments of the present invention.

[0043] Electronic device 130 can issue warnings based on the target object's state classification and emotion classification results. When the target object is a driver, the electronic device obtains power from the vehicle being driven. The overall power consumption of the electronic device can be ≤15W, meeting the requirements of a typical vehicle 12V power supply.

[0044] Figure 2 A flowchart of a state emotion classification method according to an embodiment of the present invention is shown.

[0045] like Figure 2 As shown, the method includes operations S210 to S240.

[0046] In operation S210, the physiological feature map is encoded to obtain physiological encoded data.

[0047] Physiological feature maps are constructed based on physiological feature data extracted from physiological signal data of target objects in multiple physiological channels. Physiological feature nodes in the physiological feature map represent the physiological feature data of physiological channels, and the physiological feature edge weights of the physiological feature map represent the degree of correlation between the physiological feature data corresponding to the two connected physiological feature nodes.

[0048] In operation S220, the EEG feature map is encoded to obtain EEG encoded data.

[0049] EEG feature maps are constructed based on EEG feature data extracted from the EEG signal data of the target object in multiple EEG channels. The EEG feature nodes of the EEG feature map represent the EEG feature data of the EEG channel, and the EEG feature edge weights of the EEG feature map represent the degree of correlation between the EEG feature data corresponding to the two connected EEG feature nodes.

[0050] In operation S230, physiological coding data and EEG coding data are fused to obtain multimodal fused coding data.

[0051] In operation S240, based on multimodal fusion encoding data, the state classification result and emotion classification result of the target object are obtained.

[0052] Encoding physiological feature maps allows for the compression of features within the maps, yielding physiologically encoded data. Physiological feature data can be extracted from physiological signal data from multiple physiological channels, processed, and used to obtain vectors representing physiological feature nodes. The correlation between the physiological feature data corresponding to any two physiological feature nodes is then used to determine the corresponding physiological feature edge weights. A physiological feature map is constructed based on the vectors of the physiological feature nodes and the physiological feature edge weights.

[0053] Encoding the EEG feature map allows for the compression of features within the map, yielding encoded EEG data. EEG feature data can be extracted from EEG signal data from multiple channels, processed to obtain vectors representing EEG feature nodes. The correlation between the EEG feature data corresponding to any two EEG feature nodes is then used to determine the corresponding EEG feature edge weights. An EEG feature map is constructed based on the vectors of the EEG feature nodes and their edge weights.

[0054] The fusion of physiological coding data and EEG coding data can be performed based on the data corresponding to each channel. Data with weights that are highly correlated with state and emotion classification in each channel can be fused to obtain multimodal fused coding data, which can contain more data that can highlight the correlation with state and emotion classification.

[0055] Multimodal fusion encoded data can be input into a pre-trained network to obtain the state classification results and emotion classification results of the target object.

[0056] According to embodiments of the present invention, physiological feature maps and electroencephalogram (EEG) feature maps are constructed based on physiological feature data and EEG feature data, respectively, reflecting the characteristics of physiological feature data and EEG feature data in their respective channels and the degree of correlation between different channels. By encoding the physiological feature map to obtain physiological coded data and encoding the EEG feature map to obtain EEG coded data, features from the physiological feature map and EEG feature map can be further compressed and extracted. By fusing the physiological coded data and EEG coded data, data features can be enhanced from different modalities to obtain multimodal fused coded data, thereby improving the accuracy of state classification results and emotion classification results obtained based on multimodal fused coded data.

[0057] Electroencephalogram (EEG) signal data and physiological signal data can be represented as follows.

[0058] (1);

[0059] in, This represents raw electroencephalogram (EEG) signal data and physiological signal data. Represents electroencephalogram (EEG) signal data. The number of brainwave channels for EEG signal data, for example, could be 8. The number of EEG sampling points representing the EEG channel can be 4000 (obtained with a sampling duration of 4s and downsampling from 1kHz to 250Hz). This represents data related to heart rate in physiological signal data. This represents data related to blood pressure in physiological signal data. The number of physiological sampling points in the physiological channel can be 400 (obtained by sampling for 4 seconds and resampling at 100Hz).

[0060] Physiological feature nodes in a physiological feature map can be multidimensional data extracted from physiological signal data using a non-overlapping window of a predetermined length. For example, extracting 8-dimensional features from physiological signal data with a sampling duration of 4 seconds using a non-overlapping window of a predetermined length of 0.5 seconds yields 8 physiological feature nodes. These 8-dimensional features include: the instantaneous mean of the RRI sequence. RRI standard deviation , RRI low frequency power , RRI high frequency power HR, SP, DP, MPI. The original vectors of physiological feature nodes can be obtained by concatenating features from each dimension. Then, the original vectors of physiological feature nodes are normalized and the first-order difference vector is calculated. The final data representation of the physiological feature nodes is as follows.

[0061] (2);

[0062] in, This represents the data of the i-th physiological feature node in the physiological feature map. This represents the original vector of the i-th physiological feature node in the physiological feature map. Let represent the first-order difference vector of the i-th physiological feature node in the physiological feature map.

[0063] The edge weights of physiological features can be calculated using a cosine similarity Gaussian kernel, as shown below.

[0064] (3);

[0065] (4);

[0066] in, This represents the weight of the physiological feature edge between the i-th and j-th physiological feature nodes in the physiological feature graph. This represents the cosine similarity between the i-th and j-th physiological feature nodes in the physiological feature map. The vector representing the i-th physiological feature node in the physiological feature map. Let represent the vector of the j-th physiological feature node in the physiological feature map. This represents the Gaussian kernel parameter, which can be 0.15.

[0067] According to an embodiment of the present invention, the state emotion classification method may further include: determining a first number of target physiological feature edge weights based on the sorting of at least one physiological feature edge weight; and obtaining a physiological feature map based on the target physiological feature edge weights and the physiological feature nodes corresponding to the target physiological feature edge weights.

[0068] The physiological feature edge weights of the physiological feature nodes can be sorted to determine a first number of target physiological feature edge weights. Based on the target physiological feature edge weights and the corresponding physiological feature nodes, a physiological feature graph is obtained. For example, if there are 8 physiological feature nodes and 7 neighbor nodes for each node (meaning each node has 7 physiological feature edge weights), these 7 weights can be sorted from largest to smallest, with a first number of 6. The top 6 weights from the sorted results are selected as target physiological feature edge weights, and the selection result is represented using a sparse adjacency matrix. , The number of physiological feature nodes is represented by the number of nodes. Then, based on the target physiological feature edge weights and the physiological feature nodes corresponding to the target physiological feature edge weights, a physiological feature graph based on sparse adjacency is obtained.

[0069] According to an embodiment of the present invention, the state emotion classification method may further include: determining the EEG feature edge weights between any two EEG channels based on the frequency domain EEG feature data and the graph domain EEG feature data included in the EEG feature data, wherein the frequency domain EEG feature data characterizes the energy distribution of the EEG feature data in multiple predetermined physiological frequency bands, and the graph domain EEG feature data characterizes the connectivity of the EEG sampling times corresponding to each of the multiple EEG sampling values ​​of the EEG signal data, and the connectivity of the EEG sampling times is the number of other EEG sampling times that can be connected based on the visibility relationship within the EEG channels; determining a second number of target EEG feature edge weights based on the ranking of at least one EEG feature edge weight; and obtaining an EEG feature map based on the target EEG feature edge weights and the EEG feature nodes related to the target EEG feature edge weights.

[0070] The physiological frequency bands of electroencephalogram (EEG) signals typically include Delta ( ) wave (0.5-4Hz), Theta ( ) wave (4-8Hz), Alpha ( ) wave (8-13Hz), Beta ( ) wave (13-30Hz) and Gamma ( ) waves (30-49Hz). Physiological frequency bands related to state and emotion include Wave, Wave, wave and The frequency band where the wave is located, therefore, can Wave, Wave, wave and The frequency bands corresponding to each wave are designated as predetermined physiological frequency bands. Spectral analysis can be performed on EEG signal data to obtain frequency domain EEG characteristic data.

[0071] Since EEG signal data is typically acquired continuously, there is a certain correlation between multiple EEG sample values ​​corresponding to a specific EEG channel, which can be considered as a visibility relationship within the EEG channel. Therefore, map-domain EEG feature data can be obtained based on this visibility relationship within the EEG channel. A preset number of blocking EEG sample times can be set between the current EEG sampling time and other EEG sampling times, for example, one. This can prevent excessive connectivity of EEG sampling times from leading to inaccurate map-domain EEG feature data results.

[0072] The EEG feature nodes in the EEG feature map can be EEG feature data corresponding to the EEG channels extracted from the EEG signal data. For example, the frequency domain EEG feature data and the map domain EEG feature data of the EEG signal data are calculated within a time window with a sampling duration of 4 seconds, and the result is as follows after concatenation.

[0073] (5);

[0074] in, This represents the data of the m-th EEG feature node in the EEG feature map. Respectively represent and Wave, Wave, wave and The power corresponding to each wave, This represents the graph domain EEG feature data of the m-th EEG feature node in the EEG feature map.

[0075] The edge weights of EEG features can be calculated using a cosine similarity Gaussian kernel, as shown below.

[0076] (6);

[0077] (7);

[0078] in, This represents the weight of the EEG feature edge between the m-th and n-th EEG feature nodes in the EEG feature map. The cosine similarity between the m-th and n-th EEG feature nodes in the EEG feature map is represented by . This indicates mutual information between sliding windows. The vector representing the m-th EEG feature node in the EEG feature map. The vector representing the m-th EEG feature node in the EEG feature map.

[0079] The weights of EEG feature edges of EEG feature nodes can be sorted to determine a second number of target EEG feature edge weights. Based on these target EEG feature edge weights and the corresponding EEG feature nodes, an EEG feature map is obtained. For example, if there are 8 EEG feature nodes and 7 neighbor nodes for each node (meaning each node has 7 EEG feature edge weights), these 7 weights can be sorted from largest to smallest. Let the second number be 6, and select the top 6 from the sorted results as target EEG feature edge weights. A sparse adjacency matrix can be used to represent the selection results. Then, based on the target EEG feature edge weights and the EEG feature nodes corresponding to the target EEG feature edge weights, an EEG feature map based on sparse adjacency is obtained.

[0080] According to embodiments of the present invention, filtering the order of physiological feature edge weights based on their ranking can reduce the representation content in the physiological feature map and decrease the computational load on the physiological feature map. Similarly, filtering the order of electroencephalogram (EEG) feature edge weights based on their ranking can also reduce the representation content in the EEG feature map and decrease the computational load on the EEG feature map.

[0081] The structures of physiological feature maps and EEG feature maps can be identical; therefore, the encoding process for physiological feature maps can also be the same as that for EEG feature maps. EEG and physiological feature maps can be encoded separately using a Graph-Transformer Dual-Branch Encoder (GT-Encoder). Each branch stacks M=4 GT-Encoders, with a hidden dimension of d=64 and a head count of H=6. Taking the EEG feature map as an example, the encoded EEG data is obtained through processing by a multi-head graph attention mechanism layer and an output layer.

[0082] The multi-head graph attention mechanism layer can be implemented using the Transformer self-attention mechanism, and the calculation formula is as follows.

[0083] (8);

[0084] (9);

[0085] in, This represents the output of the multi-head graph attention mechanism layer. , and This represents the learnable projection matrix corresponding to the query vector, key vector, and value vector, respectively. This represents the h attention heads in a multi-head attention mechanism. This represents the p-th attention head out of h attention heads, where h = 1, 2, ..., p. This represents the dimension of the key vector in the self-attention mechanism. This represents the projection weights corresponding to the projection matrix of the query vector in the self-attention mechanism.

[0086] Features of layer 0 (input layer) in the multi-head graph attention mechanism layer It can be the feature matrix of an EEG feature map. .

[0087] The output layer can include processing by feedforward networks, residual networks, and layer normalization networks, as shown below.

[0088] (10);

[0089] in, This represents the output of the (q+1)th layer. This represents the output of the q-th layer. This represents the intermediate state of layer q after processing by the multi-head graph attention mechanism layer and before processing by the feedforward network. This indicates feedforward network processing. This indicates residual network processing. Presentation layer normalized network processing, , These represent the intermediate weight matrix and the output weight matrix of the feedforward network, respectively. , These represent the intermediate and output parameters of the feedforward network, respectively.

[0090] The encoding process for physiological feature maps is similar and will not be elaborated upon here.

[0091] The Transformer self-attention mechanism can automatically allocate attention weights for different modalities and time slices in the temporal dimension, highlighting the features most relevant to the current cognitive task, and achieving multi-scale dependency capture through multi-head attention. Combining the local connectivity features of graph neural networks with the global dependency modeling capabilities of Transformers, the graph-transformer bi-branch encoder can achieve end-to-end learning of complex multimodal physiological signals while maintaining model interpretability.

[0092] According to embodiments of the present invention, fusing physiological coding data and EEG coding data to obtain multimodal fused coding data may include: performing cross-attention fusion on physiological coding data and EEG coding data to obtain intermediate EEG-physiological fused data; and fusing the intermediate EEG-physiological fused data and EEG coding data according to the feature dimensions of the intermediate EEG-physiological fused data and the feature dimensions of the EEG coding data to obtain multimodal fused coding data.

[0093] Cross-attention fusion can be performed on physiological coding data and EEG coding data to correlate the data between the physiological coding data and EEG coding data, and obtain intermediate EEG-physiological fusion data.

[0094] Intermediate EEG fusion data can be seen as data with more features that incorporate physiological coding data. Therefore, intermediate EEG fusion data and EEG coding data can be further fused in their respective feature dimensions to obtain multimodal fusion coding data.

[0095] According to an embodiment of the present invention, by performing cross-attention fusion on physiological coding data and EEG coding data, intermediate EEG-physiological fusion data after associating physiological modalities and EEG modalities can be obtained. Further fusing the intermediate EEG-physiological fusion data with the EEG coding data of the EEG modal can further enhance the feature association between the data of the EEG modal and the physiological modal.

[0096] According to embodiments of the present invention, performing cross-attention fusion on physiological coding data and EEG coding data to obtain intermediate EEG-physiological fusion data may include: performing cross-attention fusion on a query matrix determined based on EEG coding data and a key matrix determined based on physiological coding data to obtain a cross-attention weight matrix; using the cross-attention weight matrix to perform a weighted summation on the value matrix determined based on physiological coding data to obtain context information of the query matrix; and obtaining intermediate EEG-physiological fusion data based on the context information of the query matrix and the EEG coding data.

[0097] The query matrix determined by EEG coding data and the key matrix determined by physiological coding data are fused together at their respective nodes to obtain the cross-attention weight matrix.

[0098] By using a weighted summation of the cross-attention weight matrix and the value matrix, the contextual information of the query matrix can be obtained, which can then be fused with the EEG coding data to obtain intermediate EEG physiological fusion data. The cross-attention process can be represented as follows.

[0099] (11);

[0100] in, Represents the query matrix. Represents EEG coding data, The projection weights of the query matrix representing the EEG coding data. Represents the key matrix. Represents physiological coding data, The projected weights of the key matrix representing the physiological encoded data. Represents a value matrix, The projected weights of the value matrix representing the physiological encoded data. This indicates the number of nodes in the EEG coding data. The number of nodes in the physiological coding data is represented by d, and d represents the dimension of the query matrix. This represents the cross-attention weight matrix. This represents intermediate fusion data from electroencephalography.

[0101] According to embodiments of the present invention, cross-attention fusion can selectively focus on feature information in physiological coding data from different physiological modalities of EEG, thereby enabling more effective data enhancement.

[0102] According to an embodiment of the present invention, fusing the intermediate fusion data of electroencephalography (EEG) and the EEG-coded data based on the feature dimensions of the intermediate fusion data of EEG and the feature dimensions of the EEG-coded data to obtain multimodal fusion coded data may include: obtaining feature dimension-based gating data based on the feature dimensions of the intermediate fusion data of EEG and the feature dimensions of the EEG-coded data, wherein the gating data characterizes the components of the intermediate fusion data of EEG and the EEG-coded data used for fusion in the corresponding feature dimensions; and using the gating data to perform weighted fusion of the intermediate fusion data of EEG and the EEG-coded data to obtain multimodal fusion coded data.

[0103] Based on the feature dimensions of the intermediate EEG fusion data and the feature dimensions of the EEG coding data, the relative importance of each feature dimension for state-emotion classification can be determined. These feature dimensions can be those present in the output multimodal fusion coding data. Furthermore, based on the relative importance of each feature dimension of the intermediate EEG fusion data and the EEG coding data for state-emotion classification, the components used for fusion in the corresponding feature dimensions of the intermediate EEG fusion data and the EEG coding data can be determined, thus obtaining feature-dimensional gating data.

[0104] Based on the gating data, the intermediate fusion data of EEG physiology and the EEG coding data can be further weighted and fused to obtain multimodal fusion coding data.

[0105] According to embodiments of the present invention, by fusing intermediate electroencephalographic fusion data and electroencephalographic coding data through gating data, the component sizes of different data in the multimodal fusion coding data can be further controlled, thereby enabling the fusion of more effective data in the multimodal fusion coding data.

[0106] According to an embodiment of the present invention, obtaining gated data based on feature dimensions according to the feature dimensions of intermediate fusion data of electroencephalography (EEG) and the feature dimensions of EEG coding data may include: splicing the feature dimensions of intermediate fusion data of EEG and the feature dimensions of EEG coding data to obtain spliced ​​EEG data; and performing a linear transformation on the spliced ​​EEG data to obtain gated data.

[0107] The process of acquiring gating data is as follows.

[0108] (12);

[0109] in, Indicates gating data, This can represent a splicing operation based on the feature dimensions of intermediate fusion data of electroencephalography (EEG) and the feature dimensions of EEG coding data. This represents the weights of the respective feature dimensions of the intermediate fusion data of EEG and the EEG coding data. Represents a linear transformation. This represents the bias term for the gating data.

[0110] The process of fusing gated data with intermediate EEG physiological data and EEG coding data can be represented as follows.

[0111] (13);

[0112] in, This represents multimodal fusion encoded data. This indicates the EEG coding data involved in the fusion. This represents intermediate fusion data from the electroencephalogram (EEG) involved in the fusion process.

[0113] According to an embodiment of the present invention, obtaining the state classification result and emotion classification result of the target object based on the multimodal fusion coding data may include: obtaining first multimodal fusion processing data based on a first processing order and second multimodal fusion processing data based on a second processing order based on the multimodal fusion coding data; concatenating the first multimodal fusion processing data and the second multimodal fusion processing data to obtain multimodal fusion connection data; and obtaining the state classification result and emotion classification result of the target object based on the multimodal fusion connection data.

[0114] The first and second processing orders can be reversed in time; for example, they could be the forward and reverse processing orders of a bidirectional long short-term memory network. Multimodal fused encoded data can be stacked according to a sliding window of a certain length and represented as a sequence. The length of the sliding window can be 0.5s, T can be 8, and the total length is 4s.

[0115] The number of layers in the first and second order processing can be the same, or it can be three layers. Taking a three-layer bidirectional long short-term memory network as an example, the processing can be represented as follows.

[0116] (14);

[0117] in, , and This represents the hidden states of the three layers of a bidirectional long short-term memory network. This represents the multimodal fused connection data obtained after the final layer of processing and connection.

[0118] Multimodal fusion connection data can be normalized to obtain the joint probability of fatigue and emotion. The state classification result can include at least one of a conscious state, a mild fatigue state, and a severe fatigue state, while the emotion classification result can include at least one of a positive emotion and a sub-positive emotion. State classification results and emotion classification results can be mapped using state codes, for example: 0 → conscious state, 1 → mild fatigue state, 2 → severe fatigue state, 3 → sub-positive emotion, 4 → positive emotion. This is represented as follows.

[0119] (15)

[0120] (16);

[0121] (17);

[0122] in, Indicates the probability of fatigue. The weights representing the fatigue probability output. The parameter representing the fatigue probability output. Represents the probability of emotion. The weights representing the probability output of emotions. The parameter representing the probability output of emotion. This represents the joint probability of fatigue and mood.

[0123] The original 250Hz probability sequence can be downsampled to 10Hz and a 5-point moving average can be performed as follows.

[0124] (18);

[0125] in, Let represent the average fatigue-emotion joint probability in the t-th time period, and k represent the size of the sliding window.

[0126] Taking a sequence length of 4 seconds as an example, the mean fatigue-emotion joint probability is greater than the threshold θ for 3 consecutive seconds. user In certain circumstances, a corresponding level of warning can be triggered.

[0127] According to embodiments of the present invention, by processing the multimodal fusion encoded data in a first processing order and a second processing order, the data in the multimodal fusion encoded data can be more effectively combined for feature representation, thereby outputting more accurate state classification results and emotion classification results.

[0128] The state-emotion classification method in this invention introduces a multi-branch graph-Transformer architecture, enabling the classification method to output more accurate results. The processing of cross-attention fusion and gating data can correct features in the multimodal fusion encoded data, achieving dynamic coupling between EEG and physiological modal data.

[0129] Figure 3 A block diagram of a state emotion classification device according to an embodiment of the present invention is shown.

[0130] like Figure 3 The state emotion classification device 300 may include a dynamic graph construction module 310, a graph-Transformer dual-branch encoding module 320, a cross-modal cross-attention and gating fusion module 330, and a temporal state output module 340.

[0131] The dynamic graph construction module 310 can be used to construct physiological feature maps and EEG feature maps. The Graph-Transformer dual-branch encoding module 320 can use a Graph-Transformer-Based Multi-Branch Network (GT-MBN) to encode the physiological feature maps and EEG feature maps respectively, obtaining physiological encoded data and EEG encoded data. The cross-modal cross-attention and gating fusion module 330 can perform cross-attention fusion on the physiological encoded data and EEG encoded data to obtain intermediate EEG-physiological fusion data. Based on the feature dimensions of the intermediate EEG-physiological fusion data and the feature dimensions of the EEG encoded data, gating data based on feature dimensions is obtained. The gating data is used to perform weighted fusion of the intermediate EEG-physiological fusion data and the EEG encoded data to obtain multimodal fusion encoded data. The time-series state output module 340 can obtain first multimodal fusion processing data based on a first processing order and second multimodal fusion processing data based on a second processing order based on the multimodal fusion encoded data; connect the first multimodal fusion processing data and the second multimodal fusion processing data to obtain multimodal fusion connection data; and obtain the state classification result and emotion classification result of the target object based on the multimodal fusion connection data.

[0132] The state and emotion classification device can be integrated into an MCU-level chip and completed in milliseconds. The modules of the state and emotion classification device can be regarded as a four-level hardware unit of "sensing-computing-fusion-decision", which is suitable for classifying the state and emotion of drivers in various driving scenarios such as highways, cities, nights, and long distances.

[0133] Taking a driving scenario as an example, EEG signal data can be acquired by using a low-channel EEG signal acquisition module to collect the driver's EEG signals. Alternatively, a wearable physiological signal acquisition module can collect signals such as the driver's heart rate and blood pressure as physiological signal data. The acquired EEG and physiological signal data are preprocessed, and feature extraction, feature fusion, and recognition are achieved through a state and emotion classification device that integrates and connects multimodal data. A graph-transformer architecture is used to capture dynamic relationships within and between modalities, enabling high-precision, personalized state and emotion classification and state monitoring.

[0134] The state emotion classification method of this invention can be implemented by a trained deep learning model, and the training method of the deep learning model is as follows.

[0135] Figure 4 A flowchart of a state emotion classification model training method according to an embodiment of the present invention is shown.

[0136] like Figure 4As shown, the training method for the state emotion classification model may include operations S410 to S450.

[0137] In operation S410, the physiological feature map of the sample is encoded to obtain the physiological encoded data of the sample.

[0138] The sample physiological feature map is constructed based on the sample physiological feature data extracted from the sample physiological signal data of the sample target object in multiple sample physiological channels. The sample physiological feature nodes in the sample physiological feature map represent the sample physiological feature data of the sample physiological channel, and the sample physiological feature edge weights of the sample physiological feature map represent the degree of correlation between the sample physiological feature data corresponding to the two connected sample physiological feature nodes.

[0139] In operation S420, the sample EEG feature map is encoded to obtain sample EEG encoded data.

[0140] The sample EEG feature map is constructed based on the sample EEG feature data extracted from the sample EEG signal data of the sample target object in multiple sample EEG channels. The sample EEG feature nodes of the sample EEG feature map represent the sample EEG feature data of the sample EEG channel, and the sample EEG feature edge weights of the sample EEG feature map represent the degree of correlation between the sample EEG feature data corresponding to the two sample EEG feature nodes connected to each other.

[0141] In operation S430, the physiological coding data and EEG coding data of the sample are fused to obtain multimodal fused coding data of the sample.

[0142] In operation S440, based on the multimodal fusion coding data of the samples, the sample state classification results and sample emotion classification results of the target objects are obtained.

[0143] In operating the S450, a deep learning model is trained based on the objective loss function, the sample state classification results, the sample emotion classification results, the sample multimodal fusion encoded data, the state emotion category features, the sample physiological encoded data, the sample EEG encoded data, and the sample labels.

[0144] The state emotion category features are obtained by multimodal fusion encoding data of samples with the same sample label.

[0145] The processing procedures of operations S410 to S440 in the state emotion classification model training method of this embodiment are similar to the application process of the state emotion classification method of this embodiment. Please refer to the description of the state emotion classification method of this embodiment.

[0146] A deep learning model can be trained based on a combination of various datasets. As an example, in the initial training process, the learning rate lr = 1 × 10^{-3}, which can be decayed to 1 × 10^{-6} using cosine annealing. The sample batch can be 64, and the training epochs can be 15. Based on this, a deep learning model is trained to obtain a state-emotion classification model, which can be used to execute the state-emotion classification method of this invention.

[0147] According to embodiments of the present invention, training a deep learning model based on a target loss function, according to sample state classification results, sample emotion classification results, sample multimodal fusion encoded data, state emotion category features, sample physiological encoded data, sample EEG encoded data, and sample labels, may include: obtaining a target loss value based on the target loss function, according to sample state classification results, sample emotion classification results, sample multimodal fusion encoded data, state emotion category features, and sample labels, wherein the target loss function is determined based on at least one of a state-emotion center contrast loss term or a cross-modal mutual information regularization loss term, and a state classification cross-entropy loss term and an emotion classification cross-entropy loss term, and the target loss value is based on the state-emotion center contrast loss value of the state-emotion center contrast loss term or the cross-modal mutual information loss term. The loss is determined by at least one of the cross-modal mutual information regularization loss values ​​in the regularization loss term, and the state classification cross-entropy loss value of the state classification cross-entropy loss term and the emotion classification cross-entropy loss value of the emotion classification cross-entropy loss term. The state classification cross-entropy loss value is determined based on the state classification cross-entropy loss term and the sample state classification result. The emotion classification cross-entropy loss value is determined based on the emotion classification cross-entropy loss term and the sample emotion classification result. The state emotion center contrast loss value is determined based on the state emotion center contrast loss term and the sample multimodal fusion encoding data and state emotion category features. The cross-modal mutual information regularization loss value is determined based on the cross-modal mutual information regularization loss term and the sample physiological encoding data and sample EEG encoding data. The deep learning model is trained based on the target loss value.

[0148] The target loss function is expressed as follows.

[0149] (19);

[0150] in, Represents the target loss function. The cross-entropy loss term represents the state classification. This represents the cross-entropy loss term for emotion classification. This represents the state-emotion center contrast loss term. This represents the cross-modal mutual information regularization loss term. Represents the regularization loss term. , and represents the coefficients of the state-emotion center contrast loss term, the cross-modal mutual information regularization loss term, and the regularization loss term, respectively.

[0151] Among them, the cross-modal mutual information regularization loss term is This represents the mutual information between sample physiological coding data and sample EEG coding data across modalities. It can be set to ≤0.35 to prevent redundancy. , and These values ​​can be optimized using Bayesian methods and set to, for example, 0.30, 0.15, and 1^e. -4 .

[0152] The calculation of the state-emotion center contrastive loss term can be achieved by setting a learnable center for each classification category, i.e., a state-emotion category feature, to avoid classification deviation. The state-emotion category feature can be calculated based on multimodal fusion encoded data of samples with the same label.

[0153] Since the output includes two major categories, state and emotion, the introduction of cross-modal mutual information regularization loss term and state-emotion center contrast loss term reduces disordered drift during the training of deep learning models and reduces the problem of sample features of the same category gradually dispersing in the feature space.

[0154] Figure 5 A block diagram of an electronic device suitable for implementing the methods described above, according to an embodiment of the present invention, is shown. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0155] like Figure 5 As shown, an electronic device according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in ROM 502 or a program loaded from storage portion 508 into RAM 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0156] RAM 503 stores various programs and data required for the operation of the electronic device. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0157] According to embodiments of the present invention, the electronic device may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0158] According to embodiments of the present invention, the method flow according to embodiments of the present invention can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of the embodiments of the present invention. According to embodiments of the present invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0159] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0160] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0161] For example, according to embodiments of the present invention, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0162] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of the present invention. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of the present invention.

[0163] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this embodiment of the invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0164] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0165] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0166] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or pairings fall within the scope of this invention.

[0167] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A state-emotion classification method, characterized in that, include: The physiological feature map is encoded to obtain physiological encoded data. The physiological feature map is constructed based on the physiological feature data extracted from the physiological signal data of the target object in multiple physiological channels. The physiological feature nodes in the physiological feature map represent the physiological feature data of the physiological channel. The physiological feature edge weights of the physiological feature map represent the degree of correlation between the physiological feature data corresponding to the two connected physiological feature nodes. The EEG feature map is encoded to obtain EEG encoded data. The EEG feature map is constructed based on the EEG feature data extracted from the EEG signal data of the target object in multiple EEG channels. The EEG feature nodes of the EEG feature map represent the EEG feature data of the EEG channel, and the EEG feature edge weights of the EEG feature map represent the degree of correlation between the EEG feature data corresponding to the two connected EEG feature nodes. The physiological coding data and the EEG coding data are fused to obtain multimodal fused coding data; Based on the multimodal fusion coding data, the state classification result and emotion classification result of the target object are obtained; The step of obtaining the state classification result and emotion classification result of the target object based on the multimodal fusion encoding data includes: Based on the multimodal fusion encoded data, first multimodal fusion processed data based on a first processing order and second multimodal fusion processed data based on a second processing order are obtained; The first multimodal fusion processing data and the second multimodal fusion processing data are concatenated to obtain multimodal fusion concatenated data; Based on the multimodal fusion connection data, the state classification result and emotion classification result of the target object are obtained.

2. The method according to claim 1, characterized in that, The process of fusing the physiological coding data and the EEG coding data to obtain multimodal fused coding data includes: Cross-attention fusion is performed on the physiological coding data and the EEG coding data to obtain intermediate EEG-physiological fusion data; The intermediate fusion data of electroencephalography (EEG) and the EEG coding data are fused based on the feature dimensions of the intermediate fusion data of EEG and the feature dimensions of the EEG coding data to obtain the multimodal fusion coding data.

3. The method according to claim 2, characterized in that, The cross-attention fusion of the physiological coding data and the EEG coding data to obtain intermediate EEG-physiological fusion data includes: Cross-attention fusion is performed on the query matrix determined based on the EEG coding data and the key matrix determined based on the physiological coding data to obtain a cross-attention weight matrix; The context information of the query matrix is ​​obtained by using the cross-attention weight matrix to perform a weighted summation on the value matrix determined based on the physiological coding data; The intermediate electroencephalographic fusion data is obtained based on the context information of the query matrix and the EEG coding data.

4. The method according to claim 2 or 3, characterized in that, The process of fusing the intermediate EEG data and the EEG encoded data based on the feature dimensions of the intermediate EEG data and the feature dimensions of the EEG encoded data to obtain the multimodal fused encoded data includes: Based on the feature dimensions of the intermediate fusion data of electroencephalography and the feature dimensions of the electroencephalogram encoded data, gating data based on feature dimensions is obtained, wherein the gating data represents the components of the intermediate fusion data of electroencephalography and the electroencephalogram encoded data used for fusion in the corresponding feature dimensions; The gating data is used to perform weighted fusion of the intermediate EEG fusion data and the EEG coding data to obtain the multimodal fusion coding data.

5. The method according to claim 4, characterized in that, The step of obtaining feature-dimensional gating data based on the feature dimensions of the intermediate fusion data of electroencephalography and the feature dimensions of the electroencephalogram encoded data includes: The electroencephalogram (EEG) intermediate fusion data and the EEG encoded data are spliced ​​together to obtain spliced ​​EEG data. The gating data is obtained by performing a linear transformation on the spliced ​​EEG data.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the ordering of at least one of the said physiological feature edge weights, a first number of target physiological feature edge weights are determined; The physiological feature map is obtained based on the target physiological feature edge weights and the physiological feature nodes corresponding to the target physiological feature edge weights; and / or Based on the frequency domain EEG feature data and the graph domain EEG feature data included in the EEG feature data, the weight of the EEG feature edges between any two EEG channels is determined. The frequency domain EEG feature data represents the energy distribution of the EEG feature data in multiple predetermined physiological frequency bands, and the graph domain EEG feature data represents the connectivity of the EEG sampling times corresponding to each of the multiple EEG sampling values ​​of the EEG signal data. The connectivity of the EEG sampling times is the number of other EEG sampling times that can be connected, determined based on the visibility relationship within the EEG channel. A second number of target EEG feature edge weights are determined based on the ordering of at least one of the EEG feature edge weights. The EEG feature map is obtained based on the target EEG feature edge weights and the EEG feature nodes related to the target EEG feature edge weights.

7. A method for training a state-emotion classification model, characterized in that, include: The sample physiological feature map is encoded to obtain sample physiological encoded data. The sample physiological feature map is constructed based on the sample physiological feature data extracted from the sample physiological signal data of the sample target object in multiple sample physiological channels. The sample physiological feature nodes in the sample physiological feature map represent the sample physiological feature data of the sample physiological channel. The sample physiological feature edge weights of the sample physiological feature map represent the degree of correlation between the sample physiological feature data corresponding to the two connected sample physiological feature nodes. The sample EEG feature map is encoded to obtain sample EEG encoded data. The sample EEG feature map is constructed based on the sample EEG feature data extracted from the sample EEG signal data of the sample target object in each of the multiple sample EEG channels. The sample EEG feature nodes of the sample EEG feature map represent the sample EEG feature data of the sample EEG channel, and the sample EEG feature edge weights of the sample EEG feature map represent the degree of correlation between the sample EEG feature data corresponding to the two sample EEG feature nodes connected to each other. The physiological coding data and the EEG coding data of the sample are fused to obtain multimodal fused coding data of the sample; Based on the multimodal fusion coding data of the samples, the sample state classification result and sample emotion classification result of the target object of the samples are obtained; Based on the objective loss function, a deep learning model is trained according to the sample state classification results, the sample emotion classification results, the sample multimodal fusion encoded data, state emotion category features, the sample physiological encoded data, the sample EEG encoded data, and the sample labels. The state emotion category features are obtained based on the sample multimodal fusion encoded data corresponding to the same sample label.

8. The method according to claim 7, characterized in that, The deep learning model, based on the objective loss function and according to the sample state classification results, sample emotion classification results, sample multimodal fusion encoded data, state and emotion category features, sample physiological encoded data, sample EEG encoded data, and sample labels, is trained, including: Based on the target loss function, a target loss value is obtained according to the sample state classification result, the sample emotion classification result, the sample multimodal fusion encoded data, the state emotion category features, and the sample labels. The target loss function is determined based on at least one of the state-emotion center contrast loss term or the cross-modal mutual information regularization loss term, and the state classification cross-entropy loss term and the emotion classification cross-entropy loss term. The target loss value is based on at least one of the state-emotion center contrast loss term or the cross-modal mutual information regularization loss term, and the state classification cross-entropy loss term. The cross-entropy loss value and the emotion classification cross-entropy loss value of the emotion classification cross-entropy loss term are determined as follows: the state classification cross-entropy loss value is determined based on the state classification cross-entropy loss term and the sample state classification result; the emotion classification cross-entropy loss value is determined based on the emotion classification cross-entropy loss term and the sample emotion classification result; the state emotion center contrast loss value is determined based on the state emotion center contrast loss term and the sample multimodal fusion coding data and the state emotion category features; the cross-modal mutual information regularization loss value is determined based on the cross-modal mutual information regularization loss term and the sample physiological coding data and the sample EEG coding data. The deep learning model is trained based on the target loss value.

9. A state-emotion classification system, characterized in that, The system includes: The physiological signal data acquisition module is configured to acquire physiological signal data of the target object. The electroencephalogram (EEG) signal data acquisition module is configured to acquire EEG signal data of the target object. Electronic devices, including: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 8.

Images