Emotion Recognition Methods, Systems, and Terminals Based on Multi-Scale Brain Network Decoding Models

Abstract

This invention belongs to the field of emotion recognition technology, and relates to an emotion recognition method, system, and terminal based on a multi-scale brain network decoding model. The emotion recognition method includes: acquiring EEG data and segmenting it into segmented EEG data; calculating three types of brain effect connectivity matrices for each segmented EEG data; mapping the three types of brain effect connectivity matrices to RGB three-color channels using coarse-grained and fine-grained coding methods respectively, to obtain coarse-grained 3D brain network images and fine-grained 3D brain network images; constructing a multi-scale 3D convolutional neural network decoding model and decoding the coarse-grained and fine-grained 3D brain network images respectively, to obtain coarse-grained features and fine-grained features; fusing the coarse-grained and fine-grained features based on a multi-scale attention mechanism to obtain temporal decoding features; and classifying and mapping the temporal decoding features to obtain the probability distribution of different emotion categories. This invention can significantly improve the accuracy and reliability of emotion recognition based on EEG signals.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotion recognition, and in particular to an emotion recognition method and system based on a multi-scale brain network decoding model and a terminal. BACKGROUND

[0002] Emotion is a core component of human psychological and behavioral activities, and profoundly affects an individual's thinking, decision-making, and behavioral performance. With the continuous development of emotion recognition research, emotion recognition technology has expanded from traditional text, speech, facial expressions, and other signals to more diverse electrophysiological signals. Among them, electroencephalogram (EEG) signals can directly reflect the electrophysiological activity of the cerebral cortex, and are not easily affected by conscious control or emotional masking, and can accurately and objectively reflect the emotional state of an individual.

[0003] In recent years, a large number of studies have shown that brain effective connectivity has unique advantages in revealing emotion-driven brain network patterns. Brain effective connectivity goes beyond the simple cooperative activity between brain regions, and aims to reveal the direction and causal relationship of information flow between different brain regions, and further explore the directional information transmission between brain regions. However, the existing EEG emotion recognition methods based on brain effective connectivity still have the following problems:

[0004] First, there is a lack of effective brain network coding methods, although existing research has proposed various brain effective connectivity matrices, such as Granger causality (GC) which provides causal statistical characteristics in the time domain, and partial directed coherence (PDC) and directed transfer function (DTF) which reflect the local directionality and overall information flow strength in the frequency domain. However, there is currently a lack of a unified and efficient brain network coding method that can effectively capture the coarse-grained and fine-grained features of brain networks while incorporating multiple brain effective connections.

[0005] Second, emotional EEG signals are dynamically evolving, and brain network research needs to be dynamically modeled in the time domain to capture the dynamic brain network patterns and timing characteristics of information transmission over time. However, existing research generally calculates brain effective connectivity for the entire EEG signal, ignoring the dynamic changes in brain networks.

[0006] Third, most existing emotion recognition methods based on brain networks perform brain network modeling and analysis at the whole-brain scale, ignoring the hierarchical interaction mechanisms between macro-topology and key brain regions.

[0007] Therefore, an emotion recognition method capable of effectively capturing coarse and fine-grained features of brain networks, with high accuracy and strong reliability is urgently needed. SUMMARY

[0008] The present application aims to address the deficiencies of existing EEG emotion recognition methods based on brain effect connectivity, and proposes an emotion recognition method, system and terminal based on a multi-scale brain network decoding model. First, the collected electroencephalogram data is segmented by a sliding window, and three types of brain effect connectivity matrices, namely Granger causality, frequency domain partial directed coherence and directed transfer function, are calculated. Then, the present application innovatively uses coarse-grained encoding method and fine-grained encoding method to form a multi-scale 3D brain network image. By constructing a multi-scale 3D convolutional neural network decoding model with global branch networks and local branch networks, the global dynamic brain network pattern and the dynamic brain network pattern between partial brain regions are captured in parallel. Finally, based on the self-attention mechanism, the multi-scale brain network features are weighted in the channel to generate adaptive fusion weights, and the fused features are recognized by the full connection layer and Softmax.

[0009] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0010] In a first aspect, the present application provides an emotion recognition method based on a multi-scale brain network decoding model, comprising the following steps:

[0011] S1. Collecting electroencephalogram data, segmenting the electroencephalogram data by a sliding window to obtain segmented electroencephalogram data;

[0012] S2. Calculating three types of brain effect connectivity matrices of the segmented electroencephalogram data, namely Granger causality brain effect connectivity matrix, frequency domain partial directed coherence brain effect connectivity matrix and directed transfer function brain effect connectivity matrix;

[0013] S3. Mapping the three types of brain effect connectivity matrices to RGB three-color channels based on a coarse-grained encoding method to obtain a coarse-grained 3D brain network image; mapping the three types of brain effect connectivity matrices to RGB three-color channels based on a fine-grained encoding method to obtain a fine-grained 3D brain network image;

[0014] S4. Constructing a multi-scale 3D convolutional neural network decoding model, decoding the coarse-grained 3D brain network image based on the multi-scale 3D convolutional neural network decoding model to obtain coarse-grained features; decoding the fine-grained 3D brain network image based on the multi-scale 3D convolutional neural network decoding model to obtain fine-grained features;

[0015] S5. Fusing the coarse-grained features and the fine-grained features based on a multi-scale attention mechanism to obtain time domain decoding features;

[0016] S6. Classify and map the time domain decoding features to obtain the probability distribution of different emotional categories.

[0017] As a possible implementation, the coarse-grained encoding method maps the three-class brain effect connectivity matrix to the RGB three-color channel using the logistic function, including the following sub-steps:

[0018] Obtain the connectivity index value corresponding to each color channel in the RGB three-color channel, and calculate the mean and standard deviation of the connectivity index value;

[0019] Subtract the mean from the connectivity index value and divide by the standard deviation to obtain the normalized result;

[0020] Configure the slope control parameter, multiply the normalized result by the slope control parameter, and obtain the product as the input of the exponential function to calculate the output of the exponential function;

[0021] Substitute the output of the exponential function into the logistic function, specifically: take the negative value of the output of the exponential function as the power of the exponential function, take 1 plus the power of the exponential function as the denominator, and the numerator is fixed as 255. The obtained integer value is the pixel value mapped to the color channel.

[0022] As a possible implementation, the fine-grained encoding method amplifies the weak differences through periodic quantization, including:

[0023] Obtain the connectivity index value corresponding to each color channel in the RGB three-color channel, and calculate the mean and standard deviation of the connectivity index value;

[0024] Subtract the mean from the connectivity index value and divide by the standard deviation to obtain the normalized result;

[0025] Configure the sensitivity scaling factor, multiply the normalized result by the sensitivity scaling factor, and perform the modulo operation on the obtained product;

[0026] Take the modulo operation up to the integer, multiply the obtained integer value by 255, and divide by , The obtained result is the pixel value mapped to the color channel.

[0027] As a possible implementation, the multi-scale 3D convolutional neural network decoding model includes a global branch network and a local branch network; based on the multi-scale 3D convolutional neural network decoding model, the coarse-grained 3D brain network image is obtained. The coarse-grained features include:

[0028] The local branch network divides the coarse-grained 3D brain network image into a plurality of sub-regions, and performing local dynamic feature extraction on each sub-region based on 3D convolution operation;

[0029] concatenating the local dynamic features extracted from the plurality of sub-regions to obtain local features;

[0030] performing feature extraction on the fine-grained 3D brain network image by the global branch network to obtain global features, and fusing the global features and the local features to obtain fine-grained features.

[0031] As a possible implementation manner, the fine-grained 3D brain network image is decoded based on a multi-scale 3D convolutional neural network decoding model to obtain fine-grained features, including:

[0032] dividing the fine-grained 3D brain network image into a plurality of sub-regions in the vertical direction by the local branch network, and performing local dynamic feature extraction on each sub-region based on 3D convolution operation;

[0033] concatenating the local dynamic features extracted from the plurality of sub-regions to obtain local features;

[0034] performing feature extraction on the fine-grained 3D brain network image by the global branch network to obtain global features, and fusing the global features and the local features to obtain fine-grained features.

[0035] As a possible implementation manner, S5 includes the following sub-steps:

[0036] S50. performing broadcast addition on the coarse-grained features and the fine-grained features to obtain a preliminary fused feature map;

[0037] S51. respectively calculating global channel context and local channel context of the preliminary fused feature map;

[0038] S52. introducing a gating operator to enable nonlinear interaction between the global channel context and the local channel context;

[0039] S53. performing sigmoid mapping on the output of the gating operator to obtain weights of the coarse-grained features and the fine-grained features;

[0040] S54. performing weighted summation on the coarse-grained features and the fine-grained features to obtain time domain decoding features.

[0041] In a second aspect, the present application provides an emotion recognition system based on a multi-scale brain network decoding model, which is used to execute the emotion recognition method based on the multi-scale brain network decoding model provided in the first aspect, and the emotion recognition system comprises:

[0042] ​​​A multi-scale 3D brain network image encoding module is used to generate coarse-grained 3D brain network images and fine-grained 3D brain network images;

[0043] A multi-scale 3D convolutional neural network decoding module is used to obtain coarse-grained features from coarse-grained 3D brain network images and to decode fine-grained 3D brain network images to obtain fine-grained features.

[0044] The multi-scale feature fusion module fuses coarse-grained and fine-grained features based on a multi-scale attention mechanism to obtain temporal decoding features;

[0045] The emotion recognition module classifies and maps temporal decoded features based on a fully connected layer and Softmax to obtain the probability distribution of different emotion categories.

[0046] As one possible implementation, the multi-scale 3D convolutional neural network decoding module includes a global branch network and a local branch network;

[0047] Local branch networks perform the following operation on input coarse-grained or fine-grained 3D brain network images: They divide the input 3D brain network image vertically into... Each sub-region is used to extract local dynamic features based on 3D convolution operations. The local dynamic features extracted from each sub-region are spliced ​​together to obtain local features;

[0048] The global branch network extracts features from the input 3D brain network image to obtain global features. The global features are then fused with local features to obtain coarse-grained or fine-grained features.

[0049] As one possible implementation, the multi-scale feature fusion module includes a broadcast addition unit, a global channel context solving unit, a local channel context solving unit, a gated operator, a sigmoid mapping unit, and a fusion unit;

[0050] The broadcast addition unit is used to perform broadcast addition on coarse-grained features and fine-grained features to obtain a preliminary fused feature map;

[0051] The global channel context solving unit is used to calculate the global channel context of the preliminary fused feature map, which includes a global average pooling layer, a first fully connected layer, a first normalization layer, a first ReLU activation function layer, a second fully connected layer, and a second normalization layer connected in sequence.

[0052] The local channel context solving unit is used for calculating the local channel context of the preliminarily fused feature map, and includes sequentially connected first point-by-point convolutional layers, a third normalization layer, a second ReLU activation function layer, second point-by-point convolutional layers and a fourth normalization layer.

[0053] The gating operator is used for making the global channel context and the local channel context produce nonlinear interaction.

[0054] The sigmoid mapping unit is used for sigmoid mapping the output of the gating operator to obtain the weights of the coarse-grained features and the fine-grained features.

[0055] The fusion unit is used for weighted summing the coarse-grained features and the fine-grained features to obtain the time-domain decoding features.

[0056] In a third aspect, the present application provides a terminal, comprising a processor and a communication interface coupled with the processor, and the processor is used for running a computer program or instructions to realize the emotion recognition method based on the multi-scale brain network decoding model provided in the first aspect.

[0057] Compared with the prior art, the present application has the following beneficial effects:

[0058] 1. The emotion recognition method based on the multi-scale brain network decoding model simultaneously introduces the Granger causality brain effect connectivity matrix, the frequency domain partial directional coherence brain effect connectivity matrix and the directional transfer function brain effect connectivity matrix, and utilizes the complementarity of the three matrices in the time domain dynamic capability, the frequency domain directional intensity and the network topology description, so as to comprehensively map the information of different dimensions into a unified 3D brain network image, effectively improve the feature coverage, and fully reveal the multi-dimensional information flow mechanism of the brain.

[0059] 2. The emotion recognition method based on the multi-scale brain network decoding model innovatively adopts the coarse-grained encoding method and the fine-grained encoding method to form a multi-scale 3D brain network image, the coarse-grained encoding method highlights the macro connection mode of the brain network connection strength, and the fine-grained encoding method amplifies the weak differences through period quantization, so as to highlight the subtle changes of the local connection strength, provide a more discriminative fine-grained representation for the emotion recognition task, and form a multi-scale feature which is complementary to the coarse-grained encoding result.

[0060] 3. The emotion recognition method based on the multi-scale brain network decoding model constructs a multi-scale 3D convolutional neural network decoding model including a global branch network and a local branch network, so that the present application can jointly capture the global and local dynamic brain network mode, and lay a foundation for the subsequent classification accuracy and reliability.

[0061] 4. In order to eliminate the differences between the coarse-grained features and the fine-grained features in the feature scale and semantic information, the application designs an adaptive fusion module based on an attention mechanism to effectively fuse the coarse-grained features and the fine-grained features, so as to obtain more accurate emotion recognition results. BRIEF DESCRIPTION OF DRAWINGS

[0062] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0063] Figure 1 A flow chart of an emotion recognition method based on a multi-scale brain network decoding model in an embodiment of the application;

[0064] Figure 2 A schematic diagram of a 3D brain network image encoding process and coarse / fine-grained encoding visualization results in an embodiment of the application;

[0065] Figure 3 A structural schematic diagram of a multi-scale 3D convolutional neural network decoding model in an embodiment of the application;

[0066] Figure 4 A schematic diagram of fusing coarse-grained features and fine-grained features based on a multi-scale attention mechanism in an embodiment of the application;

[0067] Figure 5 A structural schematic diagram of an emotion recognition system based on a multi-scale brain network decoding model in an embodiment of the application. DETAILED DESCRIPTION

[0068] In order to clearly describe the technical solutions of the embodiments of the application, in the embodiments of the application, the same items or similar items with basically the same functions and effects are distinguished by using the words “first”, “second”, etc. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and the order is not limited. Those skilled in the art can understand that the words “first”, “second”, etc. do not limit the quantity and execution order, and the words “first”, “second”, etc. also do not necessarily mean different.

[0069] It should be noted that in the application, the words “exemplary” or “for example” are used to mean serving as an example, an instance, or an illustration. Any embodiment or design solution described as “exemplary” or “for example” in the application should not be interpreted as being more preferred or having more advantages than other embodiments or design solutions. Rather, the words “exemplary” or “for example” are used in the specific manner to present the relevant concept.

[0070] In the present application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. The following at least one or similar expressions mean any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b and c can be single or multiple.

[0071] The embodiments of the present application aim to provide an emotion recognition method, system and terminal based on a multi-scale brain network decoding model, so as to solve the problems of the existing EEG emotion recognition method based on brain effect connectivity, such as lack of brain network coding mode, insufficient attention to dynamic changes of brain network, and ignoring hierarchical interaction mechanism between macroscopic topology and key brain areas.

[0072] In a first aspect, the embodiments of the present application provide an emotion recognition method based on a multi-scale brain network decoding model, referring to Figure 1 , comprising the following steps:

[0073] S1. Collecting electroencephalogram (EEG) data, segmenting the EEG data by using a sliding window to obtain segmented EEG data;

[0074] As an example, after collecting the EEG data, the collected EEG data is segmented based on a 1-second sliding window (step size 0.2 seconds) to obtain a plurality of segmented EEG data.

[0075] S2. Calculating three types of brain effect connectivity matrices of the segmented EEG data, namely Granger causality (GC) brain effect connectivity matrix, frequency domain partial directed coherence (PDC) brain effect connectivity matrix, and directed transfer function (DTF) brain effect connectivity matrix;

[0076] In studies of emotion-related brain networks, any single brain effect connectivity index is insufficient to fully reveal the multidimensional information flow mechanisms of the brain. To address this limitation and improve feature coverage, this embodiment simultaneously introduces Granger causality brain effect connectivity matrix, frequency domain partially directed coherent brain effect connectivity matrix, and directed transfer function brain effect connectivity matrix. Leveraging their complementarity in temporal dynamics, frequency domain orientation strength, and network topology characterization, information from different dimensions is comprehensively mapped into a unified 3D brain network image.

[0077] Specifically, Granger causality focuses on the predictive improvement effect between time series. For any two generalized stationary brain region time series... and The causal direction can be characterized by the following bivariate autoregressive model:

[0078] (1)

[0079] (2)

[0080] in, For model order, and Representing the cross-regression coefficients, respectively measuring... and Granger's role; and For each of them, there are their respective autoregressive coefficients; This is an exogenous interference vector (such as physiological artifacts). and Its weight; and The residuals are zero-mean white noise. If all... ,but History The predictions made no contribution, indicating that there is no [contribution]. Granger causality; conversely, if at least one Then it is believed yes The reason. Similarly, through testing Whether it is significant can be determined The causal effect. This modeling method can meet the quantitative requirements of this invention for the temporal information flow direction between brain regions. However, since GC is inherently linear and sensitive to Gaussian noise, this invention subsequently combines frequency domain orientation measurements of PDC and DTF to form a complementary multi-scale brain network representation.

[0081] Partially directed coherence reveals the directionality between signals in the frequency domain and can suppress common-mode effects. This invention uses... - VAR model is used to model the brain region signals. The VAR model is transformed such that:

[0082] (3)

[0083] where is the frequency, is the sampling period, is the identity matrix, is the order index of the VAR model, is the total order value of the VAR, is the imaginary unit. Note that then the PDC from brain region to at frequency is defined as:

[0084] (4)

[0085] where is the element of , is the index of the EEG channel, is the total number of EEG channels. The denominator normalization ensures that the relative directional contribution is characterized rather than the total power, which has good resolution for band features. The DTF is based on the transfer function matrix of the VAR as follows:

[0086]

[0087] (5) The DTF from brain region

[0088] to brain region directly measures the transfer magnitude between signals, and is defined as:

[0089] (6) where

[0090] is the frequency response magnitude from brain region to brain region . The DTF emphasizes the energy distribution from the output to the input, which can reveal the overall network pathway, but has limited detection capability for weak connections. Since , there is theoretically a complementary relationship between PDC and DTF, and the coupling of the two can form a more robust network directionality estimation.

[0091] ​​​In summary, GC provides causal statistical features in time domain, PDC and DTF highlight local directional contribution and overall information flow strength in frequency domain respectively. The three can jointly encode brain effect connectivity in different scales. Therefore, after calculating the three types of brain effect connectivity matrices, the present application maps them to RGB three color channels, and designs coarse and fine granularity coding methods to represent brain network features in different scales, and defines them as 3D brain network images.

[0092] S3. Mapping the three types of brain effect connectivity matrices to RGB three color channels based on the coarse granularity coding method to obtain coarse granularity 3D brain network images; mapping the three types of brain effect connectivity matrices to RGB three color channels based on the fine granularity coding method to obtain fine granularity 3D brain network images;

[0093] Referring to Figure 2 , the encoding process of 3D brain network images and the visualization results of the coarse granularity coding method and the fine granularity coding method are shown.

[0094] As a possible implementation, the coarse granularity coding method maps the three types of brain effect connectivity matrices to RGB three color channels using a Logistic function, including the following sub-steps:

[0095] Obtain the connectivity index value corresponding to each color channel in the RGB three color channels, and calculate the mean and standard deviation of the connectivity index value;

[0096] Subtract the mean from the connectivity index value and divide by the standard deviation to obtain the standardization result;

[0097] Configure a slope control parameter, multiply the standardization result by the slope control parameter, take the obtained product as the input of the exponential function, and calculate the output of the exponential function;

[0098] Substitute the output of the exponential function into the Logistic function, specifically: take the negative value of the output of the exponential function as the power of the exponential function, take 1 plus the power of the exponential function as the denominator, and the numerator is fixed as 255. The obtained integer value is the pixel value mapped to the color channel.

[0099] The coarse granularity coding method focuses on the global distribution of brain effect connectivity strength, and uses the Logistic function for mapping, the expression is as follows:

[0100] (7)

[0101] In the formula, is the pixel value on the color channel of the color channel ; and Take the values ​​of GC, PDC, and DTF respectively, and map them to the R, G, and B channels respectively; Control the mapping slope; It is a passage All The mean and standard deviation are used for normalization and to improve robustness. The double-saturation region of the logistic function compresses extreme values ​​while preserving intermediate gradients, allowing 3D brain network images to better capture the overall connectivity trends of the brain network. See also Figure 2 In the left column of (b), the 3D brain network image under coarse-grained encoding does not show pixel-level abrupt changes, but rather large areas of gradual color changes. This coarse-grained encoding method highlights the macroscopic connectivity patterns of brain network strength, providing an intuitive encoding method for understanding the global characteristics of brain connectivity under emotional states.

[0102] Fine-grained 3D brain network images are decoded based on a multi-scale 3D convolutional neural network decoding model to obtain fine-grained features;

[0103] As one possible implementation, fine-grained coding methods amplify subtle differences through periodic quantization, including:

[0104] Obtain the connectivity index value corresponding to each color channel in the RGB three-color channels, and calculate the mean and standard deviation of the connectivity index value;

[0105] The standardized result is obtained by subtracting the mean from the connectivity index value and then dividing by the standard deviation.

[0106] Configure the sensitivity scaling factor, multiply the standardized result by the sensitivity scaling factor, and perform a modulo operation on the resulting product;

[0107] Round up the modulo operation, multiply the resulting integer value by 255, and then divide by... , This indicates the preset number of segments during the modulo operation, and the result is the pixel value mapped to that color channel.

[0108] Fine-grained coding methods amplify subtle differences through periodic quantization, thereby highlighting minute variations in local connectivity strength. The expression is as follows:

[0109] (8)

[0110] in, These are fine-grained pixel values; This is the sensitivity scaling factor (taken as 0.05). This represents the number of discrete segments (taken as 10). First, for... Perform standardization and scaling, then... Folded into length is the cycle interval, and finally linearly extended to [0, 255]. The values of GC, PDC and DTF are taken respectively. The above fine-grained encoding is essentially a nonlinear, periodic mapping function that uses non-uniform sampling to significantly improve the Manhattan distance of similar brain connectivity, making subtle connectivity differences more distinguishable. See Figure 2 In the right column of (b), the 3D brain network image under fine-grained encoding presents pixel-level mutations, which segment and stretch the continuous brain connectivity strength to [0, 255], making adjacent pixels present high-saturation small square textures. This periodic quantization is equivalent to injecting high-frequency components into the image domain, greatly expanding the gray scale difference near the edges, thereby amplifying the subtle changes in local brain connections to provide more distinguishable fine-grained representation for emotion recognition tasks, and forming a multi-scale feature that complements the coarse-grained encoding result.

[0111] S4. Constructing a multi-scale 3D convolutional neural network decoding model (MultiScale 3D Convolutional Neural Network, M3DCNN), based on the multi-scale 3D convolutional neural network decoding model, obtaining coarse-grained features from the coarse-grained 3D brain network image;

[0112] Referring to Figure 3 As a possible implementation, the multi-scale 3D convolutional neural network decoding model includes a global branch network and a local branch network;

[0113] Based on the multi-scale 3D convolutional neural network decoding model, the coarse-grained 3D brain network image is obtained, and the coarse-grained features are obtained, including:

[0114] The local branch network divides the coarse-grained 3D brain network image into sub-regions in the vertical direction, and extracts local dynamic features from each sub-region based on 3D convolution operation;

[0115] The local dynamic features extracted from the sub-regions are spliced to obtain local features;

[0116] The global branch network extracts features from the coarse-grained 3D brain network image to obtain global features, and the global features and the local features are fused to obtain coarse-grained features.

[0117] Based on the multi-scale 3D convolutional neural network decoding model, the fine-grained 3D brain network image is decoded to obtain fine-grained features, including:

[0118] The local branch network divides the fine-grained 3D brain network image into sub-regions in the vertical direction, and extracts local dynamic features from each sub-region based on 3D convolution operation;

[0119] The local dynamic features extracted from the sub-regions are spliced to obtain local features;

[0120] The global branch network extracts features from the fine-grained 3D brain network image to obtain global features, and the fine-grained features are obtained by fusing the global features and the local features.

[0121] Referring to Figure 3 , as an example, given an input 3D brain network image , where denotes the number of RGB channels, denotes the number of time frames of the electroencephalogram data, and are parameters of the 3D brain network image in the vertical and horizontal directions, respectively. To jointly capture the global and local dynamic brain network patterns, the decoding model sets a global branch network and a local branch network at the feature extraction level, and uses multi-scale convolution kernels to capture brain network changes at different scales. Referring to Figure 3 (a), in the local branch network, the input is first divided into sub-regions in the vertical direction to emphasize the role of local brain connectivity in emotion representation. The first sub-region is denoted as . On this basis, local dynamic feature extraction is performed on each sub-region through 3D convolution operation. If denotes the learnable parameters (including convolution kernels and bias terms, etc.) related to the local convolution kernel, the mapping form of the local branch can be written as:

[0122] (9)

[0123] wherein denotes 3D convolution operation in the time, vertical, and horizontal directions, is a nonlinear activation function (such as ReLU), and are the local convolution kernel and bias term, respectively, and the convolution kernel size can be denoted by to control the receptive field in the time domain, vertical domain, and horizontal direction. After completing the local feature extraction, the output results of all local branch networks can be spliced in the channel or feature dimension, thereby obtaining:

[0124] (10)

[0125] Referring to Figure 3 ​In the middle (b), in order to capture long-term global brain connectivity in the global branch network, the patent does not perform any vertical direction splitting, but directly uses a 3D convolutional network to extract features from the image. Let be the corresponding learnable parameters, then the mapping of the global branch network can be represented as:

[0126] (11)

[0127] wherein, and are the 3D convolutional kernels and bias terms of the global branch. When the global branch and the local branch complete the convolutional mapping of in turn, the model obtains the final feature output through a multi-scale fusion strategy. For ease of expression, the fusion process can be denoted as:

[0128] (12)

[0129] wherein the symbol represents broadcasting addition, that is, performing the corresponding addition operation on the element level or channel level feature components without changing the consistency of the feature dimensions. Through this multi-scale convolutional structure, the present application can jointly capture global and local dynamic brain network patterns.

[0130] S5. Based on the multi-scale attention mechanism, the coarse-grained features and the fine-grained features are fused to obtain time domain decoding features;

[0131] As a possible implementation manner, S5 includes the following sub-steps:

[0132] S50. The coarse-grained features and the fine-grained features are broadcasted and added to obtain a preliminary fused feature mapping;

[0133] S51. The global channel context and the local channel context of the preliminary fused feature mapping are calculated respectively;

[0134] S52. A gating operator is introduced to enable nonlinear interaction between the global channel context and the local channel context;

[0135] S53. The output of the gating operator is sigmoid mapped to obtain the weights of the coarse-grained features and the fine-grained features;

[0136] S54. The coarse-grained features and the fine-grained features are weighted and summed to obtain the time domain decoding features.

[0137] As an example, the coarse-grained features and the fine-grained features​ However There are significant differences in feature scales and semantic information. Therefore, the application proposes to fuse coarse-grained features and fine-grained features based on a multi-scale attention mechanism.

[0138] Referring to Figure 4 , first, the coarse-grained features and the fine-grained features are broadcasted and added to obtain a preliminary fused feature map:

[0139] (13)

[0140] wherein, represents the preliminary fused feature map, is a broadcast addition, which is used to perform element-wise addition of high-dimensional tensors in the channel and spatial dimensions. Subsequently, the multi-scale attention mechanism is used to fuse the features The global channel context uses a global average pooling (GAP) combined with a fully connected layer (FC) to extract macro dependencies as follows:

[0141] (14)

[0142] wherein, represents the GAP operation, and are the dimension reduction and dimension increase fully connected layers, respectively, represents a batch normalization operation, is a ReLU activation function.

[0143] Correspondingly, the local channel context uses a point-wise convolution (PWConv) to extract fine-grained spatial information as follows:

[0144] (15)

[0145] wherein, and are 3D point-wise convolution kernels of , which are used to capture local channel relationships. In order to further enhance feature interaction, the application introduces a gating operator, so that global information and local information can produce nonlinear interaction in the feature fusion process:

[0146] ​(16)

[0147] in, and These represent the global and local channel contexts, respectively. and For learnable weight tensors, This represents element-wise multiplication. Represents the Hadamard product. This is used to model nonlinear interaction relationships. Subsequently, applying a sigmoid mapping to the gated output yields the attention weight allocation function:

[0148] (17)

[0149] Its output values ​​are distributed in The interval can be considered as an adaptive fusion ratio. Finally, the global and local features are weighted to obtain the final fusion mapping:

[0150] (18)

[0151] in For element-wise multiplication, That is, the time-domain decoding features after fusion.

[0152] S6. Based on the fully connected layer and Softmax, the temporal decoding features are classified and mapped to obtain the probability distribution of different emotion categories.

[0153] Secondly, embodiments of the present invention provide an emotion recognition system based on a multi-scale brain network decoding model, used to execute the emotion recognition method based on a multi-scale brain network decoding model provided in the first aspect, see [link to previous section]. Figure 5 Emotion recognition systems include:

[0154] A multi-scale 3D brain network image encoding module is used to generate coarse-grained 3D brain network images and fine-grained 3D brain network images;

[0155] A multi-scale 3D convolutional neural network decoding module is used to obtain coarse-grained features from coarse-grained 3D brain network images and to decode fine-grained 3D brain network images to obtain fine-grained features.

[0156] As one possible implementation, the multi-scale 3D convolutional neural network decoding module includes a global branch network and a local branch network;

[0157] Local branch networks perform the following operation on input coarse-grained or fine-grained 3D brain network images: They divide the input 3D brain network image vertically into... a plurality of sub-regions, performing local dynamic feature extraction on each sub-region based on a 3D convolution operation, splicing the local dynamic features extracted from the plurality of sub-regions to obtain local features;

[0158] The global branch network performs feature extraction on the input 3D brain network image to obtain global features, and the global features and the local features are fused to obtain coarse-grained features or fine-grained features.

[0159] The multi-scale feature fusion module fuses the coarse-grained features and the fine-grained features based on a multi-scale attention mechanism to obtain time domain decoding features.

[0160] As a possible implementation manner, the multi-scale feature fusion module includes a broadcast addition operation unit, a global channel context solving unit, a local channel context solving unit, a gating operator, a sigmoid mapping unit, and a fusion unit.

[0161] The broadcast addition operation unit is configured to perform broadcast addition on the coarse-grained features and the fine-grained features to obtain a preliminary fused feature mapping.

[0162] The global channel context solving unit is configured to calculate the global channel context of the preliminary fused feature mapping, and includes a global average pooling layer, a first full connection layer, a first normalization layer, a first ReLU activation function layer, a second full connection layer, and a second normalization layer connected in sequence.

[0163] The local channel context solving unit is configured to calculate the local channel context of the preliminary fused feature mapping, and includes a first point-wise convolution layer, a third normalization layer, a second ReLU activation function layer, a second point-wise convolution layer, and a fourth normalization layer connected in sequence.

[0164] The gating operator is configured to cause the global channel context and the local channel context to interact nonlinearly.

[0165] The sigmoid mapping unit is configured to perform sigmoid mapping on the output of the gating operator to obtain the weights of the coarse-grained features and the fine-grained features.

[0166] The fusion unit is configured to perform weighted summation on the coarse-grained features and the fine-grained features to obtain the time domain decoding features.

[0167] The emotion recognition module performs classification mapping on the time domain decoding features based on a full connection layer and a Softmax to obtain a probability distribution of different emotion categories.

[0168] As Figure 5 ​As shown in (a), in the multi-scale 3D brain network image encoding module, the original EEG data is first segmented based on a 1-second sliding time window (step size 0.2 seconds), and coarse-grained 3D brain network images and fine-grained 3D brain network images are generated respectively. Figure 5 As shown in (b), during the M3DCNN processing stage, multi-scale 3D brain network images undergo cross-global / local feature decoding to jointly capture global and local dynamic brain network patterns. Specifically, for global feature extraction, M3DCNN first extracts global dynamic brain network patterns through a global branch network and then uses multi-layer 3D convolutions to extract global features. In the local feature extraction process, the 3D brain network image is first divided into multiple sub-regions, and dynamic brain network patterns in some brain regions are extracted through a local branch network, ultimately forming local features. Figure 5 As shown in (c), in the multi-scale feature fusion module, in order to adaptively fuse multi-scale 3D brain network images, this application first calculates preliminary fusion features through broadcast addition. Subsequently, a multi-scale channel attention mechanism is employed to extract global and local channel context information. Global channel information is extracted using global average pooling combined with a fully connected layer, while local channel information is modeled through two pointwise convolutional layers to strengthen channel dependencies at the local scale. Finally, a sigmoid normalization mechanism is used to calculate attention weights and apply them to coarse-grained features. With fine-grained features Weighted fusion is performed to obtain the final brain network features. .like Figure 5 As shown in (d), in the emotion recognition module, this application fuses the temporal domain decoding features. As the final input, a classification network is used to perform EEG emotion recognition. This module uses a standard fully connected layer mapped to a Softmax layer to calculate the probability distribution of different emotion categories, and employs cross-validation to train the model.

[0169] Thirdly, embodiments of the present invention provide a terminal, including a processor and a communication interface coupled to the processor, wherein the processor is used to run computer programs or instructions to implement the emotion recognition method based on a multi-scale brain network decoding model provided in the first aspect.

[0170] Although the present application has been described in connection with various embodiments thereof, it will be understood that the application is capable of further modifications and that this application is intended to cover any and all such variations, using the scope of the application, which is defined by the appended claims. The use of the terms "including", "comprising" or "having" and the like, are used herein to mean inclusion of elements, steps, and the like, that are recited in the description and / or claims that are not necessarily limited to those elements, steps and the like specifically recited. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover the modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

[0171] Although the present application has been described in connection with specific features thereof, it will be evident to those skilled in the art that various modifications and changes can be made to the application without departing from the spirit and scope of the application. Accordingly, it is intended that all possible modifications and changes be within the scope of the application as set forth in the appended claims. Clearly, the application is intended to cover all changes and modifications of the application which are reasonably to be expected within the scope of the application and the equivalent thereof.

Claims

1. An emotion recognition method based on a multi-scale brain network decoding model, characterized in that, The method comprises the following steps: S1. Collecting electroencephalogram data, segmenting the electroencephalogram data by using a sliding window to obtain segmented electroencephalogram data; S2. Calculating three types of brain effect connectivity matrices of the segmented electroencephalogram data, namely, a Granger causality brain effect connectivity matrix, a frequency domain partial directed coherence brain effect connectivity matrix and a directed transfer function brain effect connectivity matrix; S3. Mapping the three types of brain effect connectivity matrices to RGB three color channels based on a coarse-grained encoding method to obtain a coarse-grained 3D brain network image; Mapping the three types of brain effect connectivity matrices to RGB three color channels based on a fine-grained encoding method to obtain a fine-grained 3D brain network image; S4. Constructing a multi-scale 3D convolutional neural network decoding model, decoding the coarse-grained 3D brain network image based on the multi-scale 3D convolutional neural network decoding model to obtain coarse-grained features, and decoding the fine-grained 3D brain network image based on the multi-scale 3D convolutional neural network decoding model to obtain fine-grained features; S5. Fusing the coarse-grained features and the fine-grained features based on a multi-scale attention mechanism to obtain time domain decoding features; S6. Classifying and mapping the time domain decoding features to obtain probability distributions of different emotional categories.

2. The emotion recognition method based on a multi-scale brain network decoding model according to claim 1, characterized in that, The coarse-grained encoding method maps the three types of brain effect connectivity matrices to RGB three color channels by using a logistic function, comprising the following sub-steps: Obtaining a connectivity index value corresponding to each color channel in the RGB three color channels, calculating the mean and standard deviation of the connectivity index value; Subtracting the mean from the connectivity index value and dividing by the standard deviation to obtain a standardized result; Configuring a slope control parameter, multiplying the standardized result by the slope control parameter, taking the obtained product as the input of an exponential function, and calculating the output of the exponential function; Substituting the output of the exponential function into the logistic function, specifically: taking the negative value of the output of the exponential function as the power of the exponential function, taking 1 added to the power of the exponential function as the denominator, and taking the integer part of the obtained fractional value to obtain the pixel value mapped to the color channel.

3. The emotion recognition method based on a multi-scale brain network decoding model according to claim 1, characterized in that, The fine-grained encoding method amplifies weak differences through periodic quantization, comprising: Obtaining a connectivity index value corresponding to each color channel in the RGB three color channels, calculating the mean and standard deviation of the connectivity index value; Subtracting the mean from the connectivity index value and dividing by the standard deviation to obtain a standardized result; Configuring a sensitivity scaling factor, multiplying the standardized result by the sensitivity scaling factor, and performing a modulo operation on the obtained product; The integer value obtained by rounding up the modulo operation is multiplied by 255 and then divided by , The preset number of segments when the modulo operation is performed, and the result obtained is the pixel value mapped to the color channel.

4. The emotion recognition method based on a multi-scale brain network decoding model according to claim 1, characterized in that, The multi-scale 3D convolutional neural network decoding model comprises a global branch network and a local branch network; based on the multi-scale 3D convolutional neural network decoding model, the coarse-grained 3D brain network image is obtained, comprising: The local branch network divides the coarse-grained 3D brain network image into sub-regions in the vertical direction, and extracts local dynamic features of each sub-region based on 3D convolution operation; The local dynamic features extracted from the sub-regions are spliced to obtain local features. the local dynamic features extracted from the sub-regions are spliced to obtain local features. The global branch network extracts features from the coarse-grained 3D brain network image to obtain global features, and the global features and local features are fused to obtain coarse-grained features.

5. The emotion recognition method based on a multi-scale brain network decoding model according to claim 4, characterized in that, Based on the multi-scale 3D convolutional neural network decoding model, the fine-grained 3D brain network image is decoded to obtain fine-grained features, comprising: The local branch network divides the fine-grained 3D brain network image into sub-regions in the vertical direction, and performs local dynamic feature extraction on each sub-region based on 3D convolution operation; The local dynamic features extracted from the sub-regions are spliced to obtain local features. the local dynamic features extracted from the sub-regions are spliced to obtain local features. The global branch network extracts features from the fine-grained 3D brain network image to obtain global features, and the global features and local features are fused to obtain fine-grained features.

6. The emotion recognition method based on a multi-scale brain network decoding model according to claim 1, characterized in that, S5 comprises the following sub-steps: S50. Broadcast addition is performed on the coarse-grained features and the fine-grained features to obtain a preliminary fused feature map; S51. Global channel context and local channel context of the preliminary fused feature map are calculated respectively; S52. A gating operator is introduced to enable nonlinear interaction between the global channel context and the local channel context; S53. Sigmoid mapping is performed on the output of the gating operator to obtain the weights of the coarse-grained features and the fine-grained features; S54. Weighted summation is performed on the coarse-grained features and the fine-grained features to obtain the time-domain decoding features.

7. An emotion recognition system based on a multi-scale brain network decoding model, characterized in that, The emotion recognition system for performing the emotion recognition method based on the multi-scale brain network decoding model according to any one of claims 1 to 6 comprises: a multi-scale 3D brain network image encoding module configured to generate coarse-grained 3D brain network images and fine-grained 3D brain network images; a multi-scale 3D convolutional neural network decoding module configured to decode the coarse-grained 3D brain network images to obtain coarse-grained features and decode the fine-grained 3D brain network images to obtain fine-grained features; a multi-scale feature fusion module configured to fuse the coarse-grained features and the fine-grained features based on a multi-scale attention mechanism to obtain time-domain decoding features; an emotion recognition module configured to classify and map the time-domain decoding features based on a fully connected layer and a Softmax to obtain probability distributions of different emotion categories.

8. The emotion recognition system based on a multi-scale brain network decoding model according to claim 7, characterized in that, The multi-scale 3D convolutional neural network decoding module comprises a global branch network and a local branch network; The local branch network performs the following operations on the input coarse-grained 3D brain network image or fine-grained 3D brain network image: dividing the input 3D brain network image into M sub-regions in the vertical direction, performing local dynamic feature extraction on each sub-region based on a 3D convolution operation, splicing the local dynamic features extracted from the M sub-regions to obtain a local feature. the local feature.​ The global branch network is configured to extract features from the input 3D brain network images to obtain global features, and the global features are fused with local features to obtain coarse-grained features or fine-grained features.

9. The emotion recognition system based on a multi-scale brain network decoding model according to claim 7, characterized in that, The multi-scale feature fusion module comprises a broadcast addition operation unit, a global channel context solving unit, a local channel context solving unit, a gating operator, a sigmoid mapping unit, and a fusion unit; The broadcast addition operation unit is configured to perform broadcast addition on the coarse-grained features and the fine-grained features to obtain a preliminary fused feature map; The global channel context solving unit is configured to calculate the global channel context of the preliminary fused feature map, and comprises a global average pooling layer, a first fully connected layer, a first normalization layer, a first ReLU activation function layer, a second fully connected layer, and a second normalization layer connected in sequence; The local channel context solving unit is configured to calculate the local channel context of the preliminary fused feature map, and comprises a first point-wise convolution layer, a third normalization layer, a second ReLU activation function layer, a second point-wise convolution layer, and a fourth normalization layer connected in sequence; The gating operator is configured to enable nonlinear interaction between the global channel context and the local channel context; The sigmoid mapping unit is configured to perform sigmoid mapping on the output of the gating operator to obtain the weights of the coarse-grained features and the fine-grained features; The fusion unit is configured to perform weighted summation on the coarse-grained features and the fine-grained features to obtain the time-domain decoding features.

10. A terminal comprising a processor and a communication interface coupled to the processor, the processor being configured to execute a computer program or instructions to implement the emotion recognition method based on the multi-scale brain network decoding model according to any one of claims 1 to 6.

Images