Man-machine interaction multi-scale neural mechanism interpretation method for emotion calculation

Abstract

The invention relates to the technical field of emotion calculation, neuroscience and man-machine interaction crossing, in particular to an emotion calculation-oriented man-machine interaction multi-scale neural mechanism interpretation method, which comprises the following steps of: (1) in a man-machine interaction process, synchronously acquiring multi-channel electroencephalogram and electrocardio physiological data, and acquiring a multi-dimensional subjective emotion score as a label; (2) calculating dynamic function connection between brain intervals, and quantifying a brain collaborative activity mode; meanwhile, heart rate variability characteristics are extracted, and autonomic nervous system response is quantified; and (3) establishing a mapping relation from multi-scale physiological features to multi-dimensional emotion tags by adopting a partial least squares regression model, and identifying key heart-brain interaction indexes through variable projection importance analysis and load analysis so as to realize visualization and deep interpretation of an emotional neural mechanism. According to the method, the core challenges of difficulty in emotion state decoding and opaque internal biological mechanism of an existing emotion calculation technology in a complex human-computer interaction scene can be effectively solved.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of affective computing, neuroscience, and human-computer interaction, and in particular to a method for explaining the multi-scale neural mechanisms of human-computer interaction for affective computing. Background Technology

[0002] Affective computing, a key branch of artificial intelligence, aims to endow machines with the ability to understand and respond to human emotions. Physiological signals such as electroencephalograms (EEGs) and electrocardiograms (ECGs), due to their objectivity, have become the core basis for emotion decoding. However, existing affective computing technologies based on physiological signals face multiple bottlenecks in practical applications.

[0003] Most studies use single-dimensional emotion labels (such as "positive / negative") induced by standard laboratory stimuli (e.g., pictures, sounds). This approach severely simplifies the complexity of human emotions. In deep conversations with advanced AI such as large language models, participants' emotional experiences are often a continuous, multifaceted, and highly personalized mixture. Therefore, simple labels predefined by eliciting conditions cannot truly reflect the emotional changes during the interaction.

[0004] Emotion is a high-level cognitive activity involving the overall physical and mental state. Existing research often relies on single-modal signals, or analyzes only brain activity (EEG) or only autonomic nervous system responses (ECG). This approach ignores the essence of emotion—the complex bidirectional regulation and information interaction between the central nervous system (brain) and the peripheral autonomic nervous system (heart), namely, "brain-heart interaction." The lack of fusion analysis of this multi-scale interaction makes the representation of emotion one-sided and incomplete.

[0005] While algorithms, particularly deep learning, have achieved high accuracy in emotion classification tasks, their "black box" nature makes the decision-making process opaque. Models can identify a specific physiological pattern corresponding to a particular emotion, but they cannot explain the underlying biological mechanisms. This severely hinders a deeper understanding of the neural mechanisms of human emotions and limits the application of technology in fields with high reliability requirements, such as mental health and medical diagnosis.

[0006] With the rapid development of large-scale language models, deep collaboration between humans and artificial intelligence has become a new mode of interaction. In such complex, dynamic, and highly personalized interaction scenarios, how to effectively perform emotional and physiological computation and reveal its unique neural mechanisms is a new and urgent challenge facing the current technological field.

[0007] Therefore, a new technical solution is urgently needed to solve the above-mentioned technical problems. Summary of the Invention

[0008] The purpose of this invention is to overcome the problems of the prior art and provide a multi-scale neural mechanism interpretation method for human-computer interaction in affective computing. This method addresses the core challenges of existing affective computing technologies, such as the difficulty in decoding emotional states in complex human-computer interaction scenarios and the lack of transparency in their underlying biological mechanisms.

[0009] The above objectives are achieved through the following technical solutions: A multi-scale neural mechanism explanation method for human-computer interaction in affective computing includes: (1) Data collection and annotation: Design multiple rounds of human-computer interaction tasks to obtain the subjective scores of users' emotional experiences in the human-computer interaction process on the multi-dimensional emotional coordinate system, forming an N-dimensional subjective emotional coordinate; during the task process, collect users' multi-channel electroencephalogram (EEG) and electrocardiogram (ECG) physiological data synchronously and continuously, and use event tagging to ensure strict time alignment between the data and the interaction task; (2) Multi-scale physiological feature quantification based on human-computer interaction tasks: The collected EEG signals are processed to quantify the dynamic functional connections within and between different brain functional networks and to construct an EEG feature set; The collected ECG signals are processed to extract the RR interval sequence and calculate the time and frequency domain features of heart rate variability (HRV) to quantify the response pattern of the autonomic nervous system and to construct an ECG feature set. (3) Multiscale neural mechanism explanation: Construct a correlation matrix between physiological characteristics and multidimensional emotional labels, and use a partial least squares regression (PLSR) model to fit the correlation matrix to establish a mapping model from objective physiological characteristics to subjective emotional experience; through variable projection importance (VIP) analysis and loading analysis, identify key heart-brain interaction indicators to achieve visualization and in-depth explanation of emotional neural mechanisms.

[0010] Furthermore, the emotional dimensions of the N-dimensional subjective emotion coordinates mentioned in step (1) include anger, disgust, sadness, fear, happiness, surprise and neutrality. After each round of human-computer interaction task, the user scores 1-7 points on a Likert scale containing the above 7 core emotions. The scores together constitute the N-dimensional subjective emotion coordinates.

[0011] Further, in step (2), when constructing the EEG feature set, the brain regions are first defined, specifically: according to the international 10-20 standard electrode cap layout, the collected EEG channels are divided into predefined brain regions, including the prefrontal lobe (F), parietal lobe (P), temporal lobe (T), and default mode network (DMN); among them, the EEG channels corresponding to the prefrontal lobe are Fp1, Fp2, F3, F4, F7, and F8, the EEG channels corresponding to the parietal lobe are P3, P4, P7, and P8, the EEG channels corresponding to the temporal lobe are T7 and T8, and the EEG channels corresponding to the default mode network are FC5 and FC6; the brain region interaction pairs of interest include prefrontal-parietal lobe (FP), prefrontal-temporal lobe (FT), and default network intra (DMN-intra).

[0012] Furthermore, in step (2), when constructing the EEG feature set, the original EEG signal is preprocessed. The preprocessing steps include: (2a) Rereference the original EEG signal to a whole-brain average reference or REST (Reference Electrode Standardization Technique) reference to reduce the bias caused by the reference electrode position; (2b) Apply a 1-45Hz finite impulse response (FIR) filter to remove DC drift and high-frequency noise; (2c) A frequency band power threshold removal method based on short-time Fourier analysis (STFT) is used to detect and remove possible electromyographic artifacts in the 20-40Hz frequency band. (2d) According to the start and end event markers of the human-computer interaction task, the preprocessed clean EEG data is divided into EEG data segments corresponding to each human-computer interaction task.

[0013] Furthermore, in step (2), when calculating the EEG dynamic functional connectivity time series, a time window length (WindowLength, W) and a step size (StepSize, S) are defined; the process of calculating functional connectivity within each time window includes: (2i) Extract EEG data for all channels within this time window; (2ii) For the preset brain region interaction pairs, calculate the average value of all channel signals in the brain region to obtain the signal representing the overall activity of each brain region; (2iii) The correlation between representative signals from two brain regions within the time window is calculated using the Pearson correlation coefficient, as follows: ,in, Represented as a correlation value, it represents the degree of coordination between the prefrontal and parietal lobes within the i-th time window. The average value of the signals from all prefrontal channels. The average value of the signals from all apical leaf channels. The standard deviation of the prefrontal cortex signal within the time window W is used to measure the fluctuation amplitude of the prefrontal cortex signal. The standard deviation of the apical leaf signal within the time window W is used to measure the fluctuation amplitude of the apical leaf signal. (2iv) Move the sliding time window backward continuously from the beginning of the human-computer interaction task with a step size S until it covers the entire task. Arrange the correlation values ​​of each time window in order to form a dynamic connection time series.

[0014] Furthermore, in step (2), the process of extracting the RR interval sequence when constructing the ECG feature set includes: (2A) A 45Hz low-pass filter is used to filter out high-frequency noise interference from the original ECG signal, and wavelet decomposition and reconstruction method is used to remove the baseline drift of the ECG signal. (2B) The Pan-Tompkins algorithm was used to locate the R peaks, and the precise occurrence time of each R peak on the time axis was recorded. , where k is the heartbeat number; (2C) The RR interval series is generated by calculating the time difference between adjacent R peaks, using the following formula: The unit of RR(k) is milliseconds (ms); (2D) Correction of RR interval series: If the value of a certain RR interval deviates from the local mean of its neighboring values ​​by more than 20%, it is marked as an outlier and replaced by cubic spline interpolation of the neighboring normal values.

[0015] Further, in step (2), the time-domain and frequency-domain characteristics of heart rate variability (HRV) are calculated, where: The time-domain features include: (2a1) SDNN (Standard deviation of the NN intervals): The standard deviation of the normal RR interval, in milliseconds, reflects the combined effects of the sympathetic and parasympathetic nervous systems on the heart; (2a2) RMSSD (Root mean square of successive differences): The root mean square of the difference between adjacent RR intervals, in milliseconds, reflects the vagal nerve regulatory function of the heart; The frequency domain features include: (2b1) First, the RR interval sequence is resampled using cubic spline interpolation at a frequency of 4Hz; (2b2) ​​The power spectral density is estimated by using Fast Fourier Transform on the resampled sequence; (2b3) Calculate low frequency (LF): power in the 0.04-0.15Hz frequency band, which is affected by both the sympathetic and parasympathetic nervous systems; (2b4) Calculate high frequency power (HF): power in the 0.15-0.4Hz frequency band, reflecting cardiac vagal tone; (2b5) Calculation The ratio quantifies the balance between the sympathetic and vagus nerves.

[0016] Furthermore, the construction of the correlation matrix in step (3) includes the following sub-steps: (3a) Constructing a representation sample set: integrate the normalized EEG and ECG features of all subjects, as well as the N-dimensional emotion scores, into a data sample. Each row represents a "user-task" observation sample, and each column represents a variable. (3b) Data cleaning: Perform missing value removal operation on the representative sample set; (3c) Construct the physiological feature matrix X: Filter all EEG and ECG features from the sample set; (3d) Construct a multidimensional emotion label matrix Y: Extract the corresponding columns from the sample set according to the preset emotion label list; (3e) Data Standardization: The physiological feature matrix X is subjected to Z-score standardization. The standardized matrix is ​​denoted as... .

[0017] Further, the variable projection importance (VIP) analysis in step (3) is performed as follows: the cumulative contribution of each physiological feature to the variance of the emotion label matrix Y is calculated to obtain the VIP score, and the formula for calculating the VIP score is: Where p is the total number of independent variables and g is the total number of latent components. It is the weight vector W h The j-th element, The sum of squares of Y explained by the h-th latent component represents the sum of squares of Y; physiological characteristics with VIP scores greater than 1.0 are identified as key heart-brain interaction indicators that make significant contributions to the model.

[0018] Further, the loading analysis process in step (3) is as follows: extract the X loading matrix P and Y loading matrix Q from the fitted PLSR model, project the column vectors of P and Q onto a two-dimensional coordinate system composed of the two most important latent components to generate a PLSR loading map; in the loading map, physiological characteristics and emotional labels are presented in vector form, variables with similar directions are positively correlated, and variables with opposite directions are negatively correlated. By analyzing the clustering and relative positions of variables, the intrinsic connection between the "mind-brain interaction mode" and the "emotional experience cluster" is identified, thereby realizing the visualization and in-depth explanation of the emotional neural mechanism.

[0019] This invention provides a multi-scale neural mechanism interpretation method for human-computer interaction in affective computing. By simultaneously collecting multi-channel EEG and ECG physiological data and fusing information from the central nervous system and autonomic nervous system, it avoids the one-sidedness of single-modal signals in emotional representation and fully captures the dynamic pattern of "heart-brain interaction." Combined with multi-dimensional subjective emotion scoring, it breaks through the limitations of traditional single-dimensional emotion labels and more realistically reflects the continuous and diverse emotional experiences during the interaction process. With the help of partial least squares regression models and variable projection importance and loading analysis, it achieves accurate mapping between physiological characteristics and emotional experiences, making the decision-making process of affective computing models transparent. This not only deepens the understanding of the neural mechanisms of human emotions but also provides interpretable technical support for fields with high reliability requirements such as mental health and medical diagnosis. At the same time, it is adapted to new deep human-computer interaction modes with large-scale language models, enhancing the practical application value of affective computing technology. Attached Figure Description

[0020] Figure 1 This is a flowchart of a multi-scale neural mechanism explanation method for human-computer interaction oriented towards emotion computing, as described in this invention. Figure 2 This is a schematic diagram of the EEG brain region definition and the brain region interaction pairs of interest in the multi-scale neural mechanism explanation method for human-computer interaction in the present invention for emotion computing. Figure 3 This is a schematic diagram of the EEG signal processing and functional connectivity calculation process in the multi-scale neural mechanism interpretation method for human-computer interaction oriented towards emotion computing described in this invention. Figure 4 This is a schematic diagram of the ECG signal processing and HRV feature extraction process in the multi-scale neural mechanism explanation method for human-computer interaction oriented towards emotion computing described in this invention. Figure 5 This is a schematic diagram of the PLSR model and its interpretability analysis framework constructed in the multi-scale neural mechanism explanation method for human-computer interaction oriented towards emotion computing described in this invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. The described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0022] This solution provides a multi-scale neural mechanism explanation method for human-computer interaction in affective computing, and fully demonstrates the implementation process of an end-to-end interpretable affective computing framework. This framework covers three core modules: data acquisition, multimodal feature engineering, and interpretable modeling based on statistical learning, including the following steps: like Figure 1 As shown, this embodiment fully demonstrates the implementation process of an end-to-end interpretable sentiment computing framework. This framework covers three core modules: data acquisition, multimodal feature engineering, and interpretable modeling based on statistical learning. Specifically, it includes the following steps: (1) N-dimensional emotion annotation and synchronous physiological data acquisition based on human-computer interaction (11) In this example, users are required to perform several independent "existential question" interaction tasks with a large language model. After each round of interaction tasks, users rate their emotions on a Likert scale of 1-7, which includes 7 core emotions (anger, disgust, sadness, fear, happiness, surprise, and neutrality). These ratings together constitute an "N-dimensional subjective emotion coordinate" representing the overall emotional experience of that round of dialogue.

[0023] (12) Throughout each round of the task, the user’s multi-channel EEG and single-channel ECG data are collected synchronously and continuously using high-precision equipment, and event tagging is used to ensure that the data is strictly time-aligned with the dialogue task.

[0024] (2) Quantification of multi-scale physiological features based on human-computer interaction tasks The goal of this step is to extract computational features from raw physiological signals that can effectively characterize emotional states, and it is one of the core components of affective computing. These include: quantification of brain region coordination patterns in EEG and quantification of autonomic neural response features in ECG.

[0025] (21) Quantification of brain region coordination patterns in EEG: The whole process is divided into three sub-steps: brain region definition and data preprocessing, calculation of dynamic functional connectivity time series, and extraction of overall pattern statistical features.

[0026] (211) Brain region definition and data preprocessing (2111) Based on the international 10-20 standard electrode cap layout, the collected EEG channels (14 channels) were divided into predefined brain regions of interest (ROI) related to higher cognition and emotional processing.

[0027] like Figure 2 As shown, the specific brain region pathways in the examples of this invention are: Frontal Lobe (F): Fp1, Fp2, F3, F4, F7, F8; Parietal Lobe (P): P3, P4, P7, P8; Temporal lobe (T): T7, T8; Key nodes (approximate representation) of the Default Mode Network (DMN): FC5, FC6.

[0028] The brain region interaction pairs (ROI Pairs) that this invention focuses on include, but are not limited to: Prefrontal-parietal (FP): Associated with executive control and attention networks.

[0029] Prefrontal-temporal lobe (FT): Associated with emotion regulation and episodic memory retrieval.

[0030] The default network inside (DMN-intra) is associated with self-reflection and inner thoughts.

[0031] (2112) For example Figure 3 As shown, the specific steps of EEG preprocessing in this example are as follows: The original EEG signal is rereferenced to a whole-brain average reference or REST reference to reduce the bias caused by the reference electrode position. A 1-45 Hz finite impulse response (FIR) filter is applied to remove DC drift and high-frequency noise; A frequency band power threshold removal method based on short-time Fourier analysis (STFT) is used to detect and remove possible electromyographic artifacts in the 20–40 Hz frequency band. The preprocessed and cleaned EEG data is divided into EEG data segments corresponding to each dialogue task according to the start and end event markers of the dialogue task. (212) Calculation of dynamic functional connection time series Define a time window length (W) and a step size (S). In this invention, the time window W is selected as 2s, and the sliding step size S is selected as 0.5s.

[0032] Calculation function connections within each time window: For the i-th time window: a. Extract EEG data for all channels within the given time window.

[0033] b. For a pre-defined brain region interaction pair, taking the prefrontal-parietal (FP) lobe as an example: (i) Intra-brain signal averaging: The average value of signals from all prefrontal channels is calculated to obtain a signal representing the overall activity of the prefrontal cortex. ; Similarly, the average value of all apical leaf channel signals is calculated to obtain... .

[0034] (ii) Calculate using Pearson correlation coefficient and The correlation within this 2-second time window.

[0035] (1), where, Represented as a correlation value, it represents the degree of coordination between the prefrontal and parietal lobes within the i-th time window. The average value of the signals from all prefrontal channels. The average value of the signals from all apical leaf channels. The standard deviation of the prefrontal cortex signal within the time window W is used to measure the fluctuation amplitude of the prefrontal cortex signal. The standard deviation of the apical leaf signal within the time window W is used to measure the fluctuation amplitude of the apical leaf signal.

[0036] (iii) Move the sliding time window W backward from the beginning of the human-computer interaction task in steps S until it covers the entire module.

[0037] The correlation values ​​calculated for each time window Arranged in sequence, they form a dynamic time series of brain network connections.

[0038] Repeat this process for all brain region interaction pairs of interest to obtain multiple parallel dynamic connection time series.

[0039] (213) Extraction of statistical features of the overall pattern For each dynamically connected time series generated in the previous sub-step, extract the statistical features of the overall pattern: (i) Average connection strength (taking FP as an example, denoted as ): Calculate time series The arithmetic mean of all points.

[0040] (2), where N is the total number of points in the time series. This is expressed as a correlation value.

[0041] (ii) Standard deviation of connection strength (taking FP as an example, denoted as ): Calculate time series The standard deviation of all points.

[0042] (3) Where N is the total number of points in the time series, Represented as a correlation value, This represents the average connection strength.

[0043] (iii) Slope of connection strength trend (taking FP as an example, denoted as ): For time series Perform a univariate linear regression fit.

[0044] Using time point indices (1, 2, ..., N) as the independent variable x, the connection strength is... As the dependent variable y, the fitted equation is... The slope 'a' of the equation is extracted as a feature quantity.

[0045] Positive slope (a>0): This indicates that the synergistic relationship gradually strengthens as the dialogue deepens, which may correspond to the gradual involvement of learning, adaptation, or emotions. Negative slope (a<0): indicates that the synergy is gradually weakening, which may correspond to fatigue, habituation, or problem-solving. A slope close to zero indicates that the cooperative relationship does not have a clear unidirectional evolutionary trend.

[0046] Ultimately, all these statistical features calculated for each human-computer interaction task are combined to form the EEG feature set, which is a representation of the overall brain activity patterns during dialogue.

[0047] (22) Quantification of autonomic nervous system response characteristics of ECG like Figure 4 As shown, the process mainly includes three steps: ECG preprocessing and RR interval sequence extraction, heart rate variability (HRV) feature calculation, and extraction of statistical features of the overall response pattern.

[0048] (221) ECG preprocessing and RR interval sequence extraction a. Apply a 45Hz low-pass filter to filter out high-frequency noise interference in the ECG.

[0049] b. Wavelet decomposition and reconstruction method is used to remove baseline drift of ECG signal.

[0050] CR peak detection and RR interval sequence generation (i) The Pan-Tompkins algorithm is used to locate the R peak.

[0051] (ii) Record the precise occurrence time of each detected R peak on the time axis. , where k is the heartbeat number.

[0052] (iii) Generate a heartbeat interval (i.e. RR interval) sequence by calculating the time difference between adjacent R peaks.

[0053] (4) d. Sequence correction (i) Examine the generated RR interval sequence to identify and correct abnormal values ​​caused by artifacts (such as missed or false R peaks) or physiological arrhythmias (such as ectopic beats).

[0054] (ii) If the value of a certain RR interval deviates from the local mean of its neighboring values ​​by more than 20%, it is marked as an outlier and replaced with an interpolation of the neighboring normal values ​​(such as cubic spline interpolation).

[0055] (222) Calculation of Heart Rate Variability (HRV) Characteristics After calibration, the purified RR interval sequence was used to calculate a series of standardized HRV parameters to quantify the regulatory function of the cardiac autonomic nervous system from different dimensions.

[0056] a. Calculation of time-domain features: Calculated based on the statistical distribution of the RR interval sequence, reflecting the overall level and short-term variability of autonomic nervous regulation.

[0057] (i) SDNN (Standard deviation of the NN intervals): The standard deviation of the normal RR interval, in milliseconds. This indicator is the gold standard for overall HRV variability, reflecting the combined effects of the sympathetic and parasympathetic nervous systems on the heart.

[0058] (ii) RMSSD (Root mean square of successive differences): The root mean square of the difference between adjacent RR intervals, in milliseconds (ms). This indicator mainly reflects beat-by-beat heart rate variability and is a primary time-domain indicator for assessing cardiac vagal (parasympathetic) regulatory function.

[0059] b. Frequency domain feature calculation: The variation of the RR interval sequence is decomposed into different frequency components, which are associated with specific physiological regulatory mechanisms.

[0060] (i) Since the time intervals of the RR interval series are not uniform, they need to be resampled first to generate an equally spaced time series. Cubic spline interpolation is usually used, and the resampling frequency is 4 Hz.

[0061] (ii) For the resampled sequence, the power spectral density is estimated by using the fast Fourier transform.

[0062] (iii) According to the standard definition, the power spectrum is mainly integrated over the following two frequency bands: Low Frequency (LF): Power in the 0.04 - 0.15 Hz frequency band. The LF component is influenced by both the sympathetic and parasympathetic nervous systems and is usually related to the blood pressure reflex (baroreceptor reflex).

[0063] High Frequency (HF) power: Power within the 0.15 - 0.4 Hz frequency band. The HF component is synchronized with respiratory sinus rhythm arrhythmia and is a reliable indicator of cardiac vagal (parasympathetic) tone.

[0064] Ratio: This ratio is widely used to quantify the balance or relative innervation of the sympathetic and vagus nerves. A high LF / HF ratio is usually associated with increased sympathetic activation or decreased vagal activity, and vice versa.

[0065] (223) Extraction of statistical features of overall response pattern In order to extract stable quantitative indicators that can represent the overall physiological arousal and autonomic nervous system regulation state at this stage from the dynamic HRV changes of the entire human-computer interaction task, this invention does not only focus on the instantaneous HRV value, but also calculates its statistics throughout the entire module.

[0066] (i) Overall variability index ( ): Calculate the SDNN value once for the complete RR interval sequence of the entire human-computer interaction task.

[0067] (ii) Mean vagal nerve regulation level ( ): Calculate the RMSSD value once for the complete RR interval sequence of the entire human-computer interaction task.

[0068] (iii) Average low-frequency power ( After performing frequency domain analysis on the RR interval sequence of the entire human-computer interaction task, the power value of the LF band is obtained.

[0069] (iv) Average high-frequency power ( After performing frequency domain analysis on the RR interval sequence of the entire human-computer interaction task, the power value of the HF band is obtained.

[0070] (v) Average sympathetic-vagal balance ( ): Using the entire human-computer interaction task and Value calculation ratio: .

[0071] Ultimately, all these statistical features calculated for each human-computer interaction task are combined to form the ECG feature representation, which reflects the autonomic nervous system response pattern.

[0072] (3) Explanation of Multi-Scale Affective Neural Mechanisms: Through systematic data integration and advanced statistical modeling methods, a correlation model between objective physiological characteristics and multi-dimensional subjective emotion scores is established, and explanations with clear biological significance are extracted from this model. This step is the core innovation of this invention; it breaks down the "black box" between physiological characteristics and subjective emotions, realizing transparency and in-depth explanation of affective computing. (See also...) Figure 1 The specific implementation steps are as follows: a. Construct a correlation matrix between features and labels (i) Constructing a representation sample set: The normalized EEG and ECG features of all subjects, as well as the seven-dimensional emotion scores, are integrated into a data sample. Each row in the sample represents a unique “user-task” observation sample, and each column represents a variable (identifier, physiological feature, or emotion score).

[0073] (ii) Data cleaning: Perform missing value removal on the generated representation sample set to ensure the integrity of the data used for modeling.

[0074] (iii) Matrix construction: Select all samples with EEG or ECG features from the sample set and construct the physiological feature matrix X.

[0075] (iv) Based on the preset list of emotion labels The corresponding columns are extracted from the sample set to construct a multidimensional emotion label matrix Y.

[0076] (v) Data Standardization: The physiological feature matrix X is subjected to Z-score standardization. The standardized matrix is ​​denoted as... .

[0077] b. Fitting the Partial Least Squares Regression (PLSR) Statistical Model The PLSR model was fitted using a standardized physiological feature matrix X (dimension N * p, where N is the number of samples and p is the number of physiological features) as the independent variable and a multidimensional emotion label matrix Y (dimension N * q, where q is the number of emotion labels) as the dependent variable.

[0078] like Figure 5 As shown, the core idea of ​​PLSR is to simultaneously extract a series of latent components from X and Y. For the h-th component (h = 1, 2, ..., g), the model aims to find a pair of score vectors. (From X) and (From Y), so that their covariance is maximized.

[0079] (5) (6) in: (i) and It is the residual matrix after regression of the first h-1 components. Initially, = X, =Y; (ii) It is the weight vector of X. These are the weight vectors of Y. They are obtained by maximizing Cov( Under constraints The following is obtained; After extracting the latent components, X and Y are associated with these latent components through a regression model.

[0080] (7) (8) in: (i) It is the score matrix of X; (ii) The load matrix of X is given by the following formula: (9), (iii) The loading matrix of Y is given by the following formula: (10), (iv) E and F are the residual matrices of the model.

[0081] Ultimately, the regression relationship between Y and X is established through the score matrix T, and its final regression coefficient matrix B can be expressed as: (11) c. Interpretability Analysis and Load Analysis After the model is fitted, the results are interpreted using the following two methods to reveal the neural mechanisms behind the emotions induced by human-computer interaction tasks: (i) Variable Importance in Projection (VIP) Analysis: The VIP score is a comprehensive indicator that measures the cumulative explanatory power of the j-th independent variable (physiological characteristics) for all dependent variables (emotional labels).

[0082] (12), where: p is the total number of independent variables; g is the total number of potential components; It is a weight vector The j-th element; , representing the sum of squares of Y explained by the h-th latent component.

[0083] The VIP score is obtained by calculating the cumulative contribution of each physiological characteristic to explaining the variance of the Y matrix. The VIP score is a key indicator of the explanatory power of independent variables. Generally, features with a VIP score greater than 1.0 are considered to be variables that make significant contributions to the model.

[0084] This invention sorts and visualizes the VIP scores of all features to identify the most critical core brain-heart interaction indicators in human-computer interaction.

[0085] (ii) Loading analysis: Extract the X loading matrix (representing the relationship between the original physiological characteristics and the latent components) P ​​and the Y loading matrix (representing the relationship between the original emotion labels and the latent components) Q from the fitted model.

[0086] Each column of matrix P This indicates how primitive physiological characteristics are linearly combined to form the h-th latent component. of.

[0087] Each column of the Q matrix This represents the h-th latent component. The relationship between the original emotion label and the emotion label.

[0088] The PLSR load map is generated by projecting the column vectors of P and Q into a two-dimensional coordinate system consisting of the two most important latent components.

[0089] In this diagram, physiological characteristics and emotional labels are represented as vectors. The direction and length of the vectors visually reveal the correlation patterns between them: variables with similar directions are positively correlated, while those with opposite directions are negatively correlated.

[0090] like Figure 5As shown, the variable projection importance (VIP) analysis results indicate that the mean electrocardiogram ratio (i.e., the mean LF / HF ratio) dominates the VIP score rankings by a significant margin. This suggests that the overall physiological arousal state of the subjects is the most powerful and fundamental background factor determining their emotional experience. Furthermore, the subjects' emotional experience is closely related to the efficiency and stability of the coordinated work of the left and right hemispheres of the brain. This result indicates that the subjects' subjective emotions are not determined by a single physiological indicator, but rather by the overall arousal state of the autonomic nervous system and the integrative ability of the brain hemispheres to process higher cognitive tasks.

[0091] like Figure 5 As shown, the horizontal axis of the PLSR loading plot represents the "cognitive processing mode" axis, with the left side of this axis representing an introspective, analytical, and stable cognitive engagement. The vertical axis represents the "physiological arousal and regulation" axis, with the top of this axis representing a highly adaptive and flexible state of physiological regulation. In the "existential question" interaction task with artificial intelligence, the participants showed high mean values ​​for left-brain functional connectivity and frontal lobe functional connectivity, indicating that they experienced a range of negative emotions (anger, disgust, fear, sadness). The physiological basis of these emotions was not due to intense physical stress, but rather to specific cognitive activity patterns in the brain.

[0092] By analyzing the clustering and relative positions of variables in the graph, the intrinsic connection between specific “mind-brain interaction patterns” and specific “emotional experience clusters” can be intuitively identified, thereby enabling visualization and in-depth explanation of the neural mechanisms of emotion.

[0093] The above description is merely illustrative of the embodiments of the present invention and is not intended to limit the present invention. For those skilled in the art, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for explaining the multi-scale neural mechanisms of human-computer interaction for affective computing, characterized in that, include: (1) Data collection and annotation: Design multiple rounds of human-computer interaction tasks to obtain the subjective scores of users' emotional experiences in the human-computer interaction process on the multi-dimensional emotional coordinate system, forming an N-dimensional subjective emotional coordinate; during the task process, collect users' multi-channel electroencephalogram (EEG) and electrocardiogram (ECG) physiological data synchronously and continuously, and use event tagging to ensure strict time alignment between the data and the interaction task; (2) Multi-scale physiological feature quantification based on human-computer interaction tasks: The collected EEG signals are processed to quantify the dynamic functional connections within and between different brain functional networks and to construct an EEG feature set; The collected ECG signals are processed to extract the RR interval sequence and calculate the time and frequency domain features of heart rate variability (HRV) to quantify the response pattern of the autonomic nervous system and to construct an ECG feature set. (3) Multiscale neural mechanism explanation: Construct a correlation matrix between physiological characteristics and multidimensional emotional labels, and use a partial least squares regression (PLSR) model to fit the correlation matrix to establish a mapping model from objective physiological characteristics to subjective emotional experience; through variable projection importance (VIP) analysis and loading analysis, identify key heart-brain interaction indicators to achieve visualization and in-depth explanation of emotional neural mechanisms.

2. The method for explaining the multi-scale neural mechanisms of human-computer interaction for emotion computing according to claim 1, characterized in that, The emotional dimensions of the N-dimensional subjective emotion coordinates mentioned in step (1) include anger, disgust, sadness, fear, happiness, surprise and neutrality. After each round of human-computer interaction task, the user scores 1-7 points on a Likert scale containing the above 7 core emotions. The scores together constitute the N-dimensional subjective emotion coordinates.

3. The method for explaining the multi-scale neural mechanisms of human-computer interaction for emotion computing according to claim 1, characterized in that, In step (2), when constructing the EEG feature set, the brain regions are first defined. Specifically, according to the international 10-20 standard electrode cap layout, the collected EEG channels are divided into predefined brain regions, including the frontal lobe (F), parietal lobe (P), temporal lobe (T), and default mode network (DMN). Among them, the EEG channels corresponding to the frontal lobe are Fp1, Fp2, F3, F4, F7, and F8; the EEG channels corresponding to the parietal lobe are P3, P4, P7, and P8; the EEG channels corresponding to the temporal lobe are T7 and T8; and the EEG channels corresponding to the default mode network are FC5 and FC6. The brain region interaction pairs of interest include frontal lobe-parietal lobe (FP), frontal lobe-temporal lobe (FT), and default network intra-DMN (DMN-intra).

4. A method for explaining the multi-scale neural mechanisms of human-computer interaction for emotion computing according to claim 1 or 3, characterized in that, In step (2), when constructing the EEG feature set, the original EEG signal is preprocessed. The preprocessing steps include: (2a) Rereference the original EEG signal to a whole-brain average reference or REST (Reference Electrode Standardization Technique) reference to reduce the bias caused by the reference electrode position; (2b) Apply a 1-45Hz finite impulse response (FIR) filter to remove DC drift and high-frequency noise; (2c) A frequency band power threshold removal method based on short-time Fourier analysis (STFT) is used to detect and remove possible electromyographic artifacts in the 20-40Hz frequency band. (2d) According to the start and end event markers of the human-computer interaction task, the preprocessed clean EEG data is divided into EEG data segments corresponding to each human-computer interaction task.

5. A method for explaining the multi-scale neural mechanisms of human-computer interaction for emotion computing according to claim 1 or 3, characterized in that, In step (2), when calculating the EEG dynamic functional connectivity time series, a time window length (WindowLength, W) and a step size (StepSize, S) are defined; the process of calculating functional connectivity within each time window includes: (2i) Extract EEG data for all channels within this time window; (2ii) For the preset brain region interaction pairs, calculate the average value of all channel signals in the brain region to obtain the signal representing the overall activity of each brain region; (2iii) The correlation between representative signals from two brain regions within the time window is calculated using the Pearson correlation coefficient, as follows: in, Represented as a correlation value, it represents the degree of coordination between the prefrontal and parietal lobes within the i-th time window. The average value of the signals from all prefrontal channels. The average value of the signals from all apical leaf channels. The standard deviation of the prefrontal cortex signal within the time window W is used to measure the fluctuation amplitude of the prefrontal cortex signal. The standard deviation of the apical leaf signal within the time window W is used to measure the fluctuation amplitude of the apical leaf signal. (2iv) Move the sliding time window backward continuously from the beginning of the human-computer interaction task with a step size S until it covers the entire task. Arrange the correlation values ​​of each time window in order to form a dynamic connection time series.

6. The method for explaining the multi-scale neural mechanisms of human-computer interaction for emotion computing according to claim 1, characterized in that, In step (2), the process of extracting the RR interval sequence when constructing the ECG feature set includes: (2A) A 45Hz low-pass filter is used to filter out high-frequency noise interference from the original ECG signal, and wavelet decomposition and reconstruction method is used to remove the baseline drift of the ECG signal. (2B) The Pan-Tompkins algorithm was used to locate the R peaks, and the precise occurrence time of each R peak on the time axis was recorded. , where k is the heartbeat number; (2C) The RR interval series is generated by calculating the time difference between adjacent R peaks, using the following formula: The unit of RR(k) is milliseconds (ms); (2D) Correction of RR interval series: If the value of a certain RR interval deviates from the local mean of its neighboring values ​​by more than 20%, it is marked as an outlier and replaced by cubic spline interpolation of the neighboring normal values.

7. The method for explaining the multi-scale neural mechanisms of human-computer interaction for emotion computing according to claim 6, characterized in that, In step (2), the time-domain and frequency-domain characteristics of heart rate variability (HRV) are calculated, where: The time-domain features include: (2a1) SDNN (Standard deviation of the NN intervals): The standard deviation of the normal RR interval, in milliseconds, reflects the combined effects of the sympathetic and parasympathetic nervous systems on the heart; (2a2) RMSSD (Root mean square of successive differences): The root mean square of the difference between adjacent RR intervals, in milliseconds, reflects the vagal nerve regulatory function of the heart; The frequency domain features include: (2b1) First, the RR interval sequence is resampled using cubic spline interpolation at a frequency of 4Hz; (2b2) ​​The power spectral density is estimated by using Fast Fourier Transform on the resampled sequence; (2b3) Calculate low frequency (LF): power in the 0.04-0.15Hz frequency band, which is affected by both the sympathetic and parasympathetic nervous systems; (2b4) Calculate high frequency power (HF): power in the 0.15-0.4Hz frequency band, reflecting cardiac vagal tone; (2b5) Calculation The ratio quantifies the balance between the sympathetic and vagus nerves.

8. The method for explaining the multi-scale neural mechanisms of human-computer interaction for emotion computing according to claim 1, characterized in that, When constructing the correlation matrix in step (3), the following sub-steps are included: (3a) Constructing a representation sample set: integrate the normalized EEG and ECG features of all subjects, as well as the N-dimensional emotion scores, into a data sample. Each row represents a "user-task" observation sample, and each column represents a variable. (3b) Data cleaning: Perform missing value removal operation on the representative sample set; (3c) Construct the physiological feature matrix X: Filter all EEG and ECG features from the sample set; (3d) Construct a multidimensional emotion label matrix Y: Extract the corresponding columns from the sample set according to the preset emotion label list; (3e) Data Standardization: The physiological feature matrix X is subjected to Z-score standardization. The standardized matrix is ​​denoted as... .

9. The method for explaining the multi-scale neural mechanisms of human-computer interaction for emotion computing according to claim 1, characterized in that, The process of variable projection importance (VIP) analysis in step (3) is as follows: calculate the cumulative contribution of each physiological feature to the variance of the emotion label matrix Y, and obtain the VIP score. The formula for calculating the VIP score is: Where p is the total number of independent variables and g is the total number of latent components. It is a weight vector The j-th element, The sum of squares of Y explained by the h-th latent component represents the sum of squares of Y; physiological characteristics with VIP scores greater than 1.0 are identified as key heart-brain interaction indicators that make significant contributions to the model.

10. The method for explaining the multi-scale neural mechanisms of human-computer interaction for emotion computing according to claim 1, characterized in that, The load analysis process in step (3) is as follows: extract the X load matrix P and Y load matrix Q from the fitted PLSR model, project the column vectors of P and Q onto the two-dimensional coordinate system composed of the two most important latent components, and generate the PLSR load map. In the load diagram, Physiological characteristics and emotional labels are presented in vector form. Variables with similar directions are positively correlated, while variables with opposite directions are negatively correlated. By analyzing the clustering and relative positions of variables, the intrinsic connection between the "mind-brain interaction pattern" and the "emotional experience cluster" is identified, thereby achieving visualization and in-depth explanation of the neural mechanisms of emotion.

Images