Cognitive emotion collaborative rehabilitation method based on brain-computer interface and ai

Abstract

This invention relates to a cognitive-emotional collaborative rehabilitation method based on brain-computer interface and AI, comprising: acquiring and preprocessing the patient's electroencephalogram (EEG) signals to obtain clean signals; inputting the clean signals into a pre-trained multimodal state assessment model, simultaneously decoding and outputting a cognitive index quantifying the cognitive state and an emotion vector representing the emotional state; generating a personalized initial music therapy plan based on the cognitive index and emotion vector using a dynamic music mapping algorithm; executing the plan, and monitoring changes in the cognitive index and emotion vector in real time during the intervention; when the changes exceed a preset threshold, dynamically adjusting at least one music parameter in the treatment plan to form a closed-loop rehabilitation intervention. This invention achieves real-time monitoring of the patient's cognitive and emotional states and adaptive adjustment of personalized music intervention, improving the accuracy and effectiveness of rehabilitation treatment.

Description

Technical Field

[0001] This invention relates to the field of medical and health information technology and artificial intelligence, and in particular to a cognitive and emotional collaborative rehabilitation method based on brain-computer interface and AI. Background Technology

[0002] Alzheimer's disease (AD) is a complex neurodegenerative disease. Traditional cognitive training and music therapy are often separate and mostly static, general programs that cannot dynamically adjust to the patient's fluctuating neuropsychological state in real time. While emerging brain-computer interface (BCI) technology can monitor brain activity, it is mostly used for single-function communication or control, lacking a closed-loop system that combines multi-dimensional EEG characteristics with personalized, adaptive interventions to achieve synergistic cognitive and emotional rehabilitation. Therefore, there is an urgent need for an intelligent rehabilitation method that can perceive, accurately decode, and respond instantly to changes in the patient's internal state. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by providing a cognitive-emotional collaborative rehabilitation method based on brain-computer interfaces and AI, thereby solving the problems existing in the prior art.

[0004] To achieve the above objectives, this invention provides a cognitive-emotional collaborative rehabilitation method based on brain-computer interface and AI, the method comprising: The patient's raw EEG signals were collected and preprocessed to obtain clean EEG signals; wherein, the raw EEG signals refer to the bioelectric signal sequences recorded from the scalp surface, including delta wave, theta wave, alpha wave, beta wave and gamma wave frequency band components; The cleaned EEG signals are input into a pre-trained multimodal state assessment model, which simultaneously decodes and outputs a cognitive index that quantifies the patient's current cognitive state and an emotion vector that represents the patient's current emotional state. The cognitive state is obtained by quantifying features in the EEG signals related to attention, working memory, and information processing speed. The cognitive index is a normalized scalar value that integrates the theta / alpha wave power ratio, P300 event-related potential latency, and brain network connectivity complexity. Its value ranges from 0 to 1; a higher value indicates a stronger overall cognitive ability (including attention maintenance, working memory capacity, and information processing speed), while a lower value indicates a lower cognitive function or a state of distraction. The emotional state is obtained by identifying features in the EEG signals related to emotional arousal and pleasure. The emotion vector is a multidimensional feature that includes at least components representing emotional valence and / or emotional arousal. Based on the cognitive index and emotion vector, a personalized initial music therapy plan is generated using a dynamic music mapping algorithm, including music type, rhythm, volume and timbre parameters. The initial music therapy program is implemented, and the changes in the cognitive index and the emotion vector are monitored in real time during the intervention. When the changes exceed a preset threshold, at least one music parameter in the treatment program is dynamically adjusted to form a closed-loop rehabilitation intervention.

[0005] In one possible implementation, the acquisition and preprocessing of the patient's raw EEG signals to obtain clean EEG signals specifically includes: Using an EEG cap containing at least 64 recording electrodes, raw EEG signals from multiple brain regions on the patient's scalp are simultaneously acquired at a sampling rate of not less than 512 Hz. The raw EEG signals are a mixed time series of bioelectrical activity in the frequency bands of delta waves, theta waves, alpha waves, beta waves, and gamma waves. The raw EEG signal was processed sequentially as follows to obtain a clean EEG signal: A finite impulse response filter with cutoff frequencies of 0.5 Hz and 45 Hz is used to preserve the effective neural oscillation component and suppress low-frequency drift and high-frequency noise. Apply a 50Hz or 60Hz notch filter to eliminate ambient AC interference; An independent component analysis method is used to automatically identify and remove artifacts generated by eye movements, blinking, and muscle activity; The signals from all electrodes are converted into EEG signals with reference to the average potential of the whole brain; and the clean EEG signals are segmented according to fixed time windows to generate sample segments for model input.

[0006] In one possible implementation, the multimodal state assessment model is a multi-branch fusion assessment network with a deep neural network as its core, including: a multi-dimensional EEG feature extraction module, a feature fusion and state encoding module, and a dual-task collaborative output module; The multi-dimensional EEG feature extraction module is used to extract features of different attribute dimensions in parallel from the clean EEG signal, including: time-frequency feature extraction branch, spatiotemporal dynamics feature extraction branch and event-related potential feature extraction branch; The time-frequency feature extraction branch consists of a short-time Fourier transform layer and a convolutional layer. The input is an EEG signal within a time window, and the output is a time-frequency feature map representing the power and spatial distribution pattern of each frequency band (δ, θ, α, β, γ). The spatiotemporal dynamics feature extraction branch is composed of a graph convolutional network. The input is the EEG signal of the time window and its predefined brain region connection topology, and the output is a spatiotemporal feature vector characterizing the functional connectivity strength and information transmission efficiency of the brain network. The event-related potential feature extraction branch is composed of a temporal convolutional network. Its input is a segment of EEG signal induced by a specific cognitive task, and its output is an ERP feature vector representing the latency and amplitude of the P300 component. The feature fusion and state encoding module is used to integrate multi-dimensional features and generate a unified context-aware state representation, including: a cross-dimensional attention fusion submodule and a state encoder; The cross-dimensional attention fusion submodule takes the time-frequency feature map, spatiotemporal feature vector, and ERP feature vector as input, calculates the contribution weight of different feature dimensions to the current state evaluation, performs weighted fusion, and outputs a unified fusion feature representation. The state encoder is composed of a multi-head self-attention mechanism and a feedforward neural network. Its input is the fused feature representation. Its function is to capture long-range dependencies between features and perform deep encoding. Its output is an encoded feature vector that includes global context information. The dual-task collaborative output module is used to simultaneously generate quantitative assessments of cognitive and emotional states based on encoded features, including a cognitive regression head and an emotion classification head. The input to the cognitive regression head is the encoded feature vector, the structure is a fully connected layer, and the output is the cognitive index, which is a normalized scalar value that comprehensively reflects attention and information processing ability. The input to the emotion classification head is the encoded feature vector, the structure is a fully connected layer and a softmax layer, and the output is the emotion vector, which is a probability distribution vector representing the emotion category in a multi-dimensional emotion space.

[0007] In one possible implementation, the training of the multimodal state evaluation model includes: A multi-task training dataset is constructed, which includes a set of labeled EEG data samples from multiple subjects. Each sample includes: raw EEG signals for a time window; a corresponding standardized cognitive function score obtained from a professional neuropsychological assessment scale, serving as the ground truth label for the cognitive state; and a corresponding multidimensional emotion category label recorded from standard emotion-inducing experiments or clinical observations, serving as the ground truth label for the emotion state. Construct a weighted joint loss function L_total to simultaneously optimize cognitive regression and emotion classification tasks, its expression is: L_total = α * L_cognitive + β * L_emotion + γ * L_regularization Wherein, L_cognitive is the mean squared error loss used for cognitive regression, which calculates the difference between the cognitive index output by the multimodal state assessment model and the true value of the standardized cognitive function score; L_emotion is the cross-entropy loss used for emotion classification, which calculates the difference between the emotion vector output by the multimodal state assessment model and the true value of the multidimensional emotion category label; L_regularization is the L2 weight regularization term, used to prevent the model from overfitting; α, β, γ are preset task balance weight hyperparameters; Using the multi-task training dataset, with the multimodal state assessment model as the training object and the joint loss function L_total as the optimization objective, the backpropagation algorithm and the adaptive moment estimation algorithm are used to iteratively update all trainable parameters in the multimodal state assessment model until the loss function converges, thus obtaining the pre-trained multimodal state assessment model.

[0008] In one possible implementation, the generation of a personalized initial music therapy plan based on cognitive index and emotion vector through a dynamic music mapping algorithm specifically includes: Using the cognitive index and emotion vector as input, a predefined music mapping knowledge base is applied to generate basic music parameters through music type selection rules, rhythm parameter calculation rules, and volume and timbre adjustment rules. Specifically, the music type selection rule maps the cognitive index to a first music type set if it is below a first cognitive threshold, which includes classical music fragments with a brisk rhythm and clear structure; if the cognitive index is above or equal to the first cognitive threshold, it maps to a second music type set, which includes natural sounds or ambient music with a soothing rhythm and simple melody. The rhythm parameter calculation rule maps the cognitive index linearly or non-linearly to a preset rhythm value range, where a lower cognitive index corresponds to a higher base beat value. The volume and timbre adjustment rules determine the base volume gain value and timbre filtering parameters based on the emotional valence and emotional arousal components in the emotion vector using a two-dimensional lookup table, where a lower valence combined with a higher arousal corresponds to a reduced volume gain and an applied low-pass filter parameter. Retrieve the patient's pre-stored personal music preference profile, which includes a list of the patient's historical preferred music genres, artists, or specific tracks; The music type set output by the music type selection rule is matched and weighted with the personal music preference profile, and music entries that meet both the music type selection rule and the personal music preference profile are selected first. The final music type, calculated rhythm parameters, and adjusted volume and timbre parameters are encapsulated and combined to generate the personalized initial music therapy plan.

[0009] In one possible implementation, the execution of the initial music therapy program, and the real-time monitoring of changes in the cognitive index and the emotion vector during the intervention, wherein when the changes exceed a preset threshold, at least one music parameter in the treatment program is dynamically adjusted to form a closed-loop rehabilitation intervention, specifically includes: Simultaneously with the commencement of the personalized initial music therapy program, a real-time EEG signal acquisition and analysis process is initiated. The cognitive index and emotion vector acquired at the initial moment of program execution are recorded as the baseline values ​​for this intervention. The baseline values ​​include the baseline cognitive index and the baseline emotion vector. During the music intervention, signal acquisition, preprocessing, and state decoding were repeated at fixed time intervals to obtain real-time cognitive indices and emotion vectors. Calculate the change in cognitive index ΔC and the change in emotion vector ΔE; the change in cognitive index ΔC is the difference between the real-time cognitive index and the baseline cognitive index; the change in emotion vector ΔE is the Euclidean distance or cosine similarity between the real-time emotion vector and the baseline emotion vector in the multidimensional emotion space. The changes ΔC and ΔE are compared with the preset cognitive adjustment threshold θ_C and emotion adjustment threshold θ_E, respectively; If the conditions ΔC>θ_C or ΔE>θ_E are met, the patient's condition is determined to have changed significantly, triggering a dynamic adjustment to the music therapy plan; Based on the specific state dimension that triggered the adjustment, perform at least one of the following adjustment operations: If the event is triggered by ΔC > θ_C and ΔC is increasing in a positive direction, then the focus is considered to be scattered. The adjustment plan is to gradually increase the music rhythm parameter by 15%-30% within 10 seconds and switch to a music type with a more distinct rhythm. If triggered by ΔE>θ_E, and ΔE points to a high-arousal negative emotion, then anxiety or agitation is determined to occur. The adjustment plan is as follows: gradually reduce the volume parameter to below 50% of the baseline value within 5 seconds, and immediately switch to the preset soothing intervention music segment, which is mainly composed of continuous low-frequency natural sounds. During the implementation of the adjusted treatment plan, the changes in the cognitive index ΔC and the changes in the emotion vector ΔE were continuously calculated. If, within three consecutive monitoring periods after adjustment, the change in cognitive index ΔC and the change in emotion vector ΔE both fall back to within the corresponding cognitive adjustment threshold and emotion adjustment threshold, then the intervention is deemed effective, and the current adjusted parameters are maintained until the end of this intervention session.

[0010] In one possible implementation, before generating a personalized initial music therapy plan for the patient, the method further includes the step of constructing a patient-specific predefined music mapping knowledge base and a personal music preference profile, specifically including: During the initial assessment phase, patients are played various types of standard test music clips; While playing each test music segment, the patient's EEG signals were collected and analyzed simultaneously, and their corresponding cognitive index and emotion vector were recorded. Based on the records, an initial music-state response mapping relationship is established for the patient, serving as the basis for their exclusive predefined music mapping knowledge base; The patients' personal music preference profiles were collected and established through questionnaires or analysis of historical playback records.

[0011] In one possible implementation, before applying the multimodal state assessment model to the target patient, a step of personalized fine-tuning of the model is included, specifically including: Acquire historical EEG data of the target patient in a quiet resting state and under standard cognitive tasks; Using the historical EEG data and corresponding clinical assessment records, small-scale incremental training or parameter adaptation is performed on the feature fusion and state encoding module and the dual-task collaborative output module in the pre-trained multimodal state assessment model, so that the model output is more consistent with the patient's individual physiological and psychological baseline.

[0012] In one possible implementation, after a complete closed-loop rehabilitation intervention session has concluded, the method further includes: Calculate and generate a quantitative report on the effectiveness of this intervention. The report shall include at least: the overall trend curve of the cognitive index during the intervention, the evolution trajectory of the emotion vector, the historical record of music parameter adjustments, and the comprehensive improvement score based on the baseline and end values ​​before and after the intervention. The effectiveness quantification report is stored in association with all process data of this intervention and updated in the patient's long-term rehabilitation record.

[0013] In one possible implementation, a system initialization and verification step is included before acquiring the patient's raw EEG signals: Verify operator permissions and patient identity; Load a personal profile uniquely corresponding to the patient's identity, which includes their historical baseline data, personalized model parameters, music mapping knowledge base, and preference profile; Perform self-testing and synchronous calibration of the brain-computer interface device and the audio playback device to ensure the real-time performance and reliability of signal acquisition and intervention output.

[0014] By applying the cognitive-emotional collaborative rehabilitation method based on brain-computer interface and AI provided in this invention, a multimodal state assessment model is used to synchronously and accurately decode EEG features related to cognitive function and emotional state, overcoming the limitations of separate assessment in traditional methods and providing precise input for collaborative intervention. Furthermore, based on the decoded state index, a dynamic music mapping algorithm automatically generates music therapy plans that integrate neuroscience rules and personal preferences, such as type, rhythm, volume, and timbre, making the intervention starting point highly personalized. Further, state changes are monitored in real time during intervention, and dynamic adjustments to music parameters are automatically triggered when changes exceed thresholds, forming a closed loop of monitoring-decision-intervention-remonitoring. This allows treatment to be dynamically optimized according to the patient's state, significantly improving the accuracy and effectiveness of rehabilitation. Furthermore, this application ensures the individual adaptability, safety, and execution reliability of the entire method from analysis to intervention through pre-treatment personalized model fine-tuning, file construction, and synchronous hardware calibration, laying the foundation for clinical translation. Attached Figure Description

[0015] Figure 1 The flowchart of the cognitive-emotional collaborative rehabilitation method based on brain-computer interface and AI provided by the present invention; Figure 2 for Figure 1 Flowchart for step 110; Figure 3 This is a structural diagram of a multimodal state assessment model; Figure 4 for Figure 1 Flowchart for step 130; Figure 5 for Figure 1 The flowchart for step 140. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Figure 1 This is a flowchart of a cognitive-emotional collaborative rehabilitation method based on brain-computer interface and AI, such as... Figure 1 As shown, the method includes the following steps: Step 110: Collect the patient's raw EEG signal and preprocess it to obtain a clean EEG signal; wherein, the raw EEG signal refers to the bioelectric signal sequence recorded from the scalp surface, including delta wave, theta wave, alpha wave, beta wave and gamma wave frequency band components.

[0018] The clean EEG signal refers to the effective neural oscillation component retained after the original EEG signal has undergone preprocessing steps such as filtering, notch filtering, artifact removal, and rereference. Its characteristics include the near elimination of low-frequency drift, power frequency interference, and electrooculography / electromyography artifacts, and it uses the whole-brain average potential as a reference, making it directly usable by subsequent analysis modules. For ease of description, the EEG signal processed through steps 1102 to 1106 is referred to as the clean EEG signal in this application.

[0019] like Figure 2 As shown, step 110 includes the following steps: Step 1101: Using an EEG cap containing at least 64 recording electrodes, raw EEG signals from multiple brain regions on the patient's scalp are simultaneously acquired at a sampling rate of not less than 512 Hz. The raw EEG signals are a mixed time series of bioelectrical activity in the frequency bands of delta waves, theta waves, alpha waves, beta waves, and gamma waves. Specifically, to achieve precise analysis of brain spatial activity patterns, an EEG cap with at least 64 recording electrodes is used, arranged according to the international 10-10 system or a higher precision 10-5 system. This ensures comprehensive coverage of key brain regions such as the prefrontal cortex (related to executive function and emotion regulation), parietal lobe (related to sensory integration and spatial attention), temporal lobe (related to hearing and memory), and occipital lobe (related to visual processing). The sampling rate is set to no less than 512Hz to satisfy the Nyquist sampling theorem, ensuring the lossless capture of high-frequency gamma wave components (up to 45Hz) and providing sufficient temporal resolution for subsequent time-frequency analysis and event-related potential extraction. This step simultaneously acquires raw signals from multiple brain regions on the patient's scalp, forming a multi-channel mixed time series.

[0020] Step 1102: The raw EEG signal is processed sequentially as follows to obtain a clean EEG signal: Specifically, the raw signals collected need to undergo a series of rigorous signal processing procedures to gradually remove various interferences and obtain clean, analyzable EEG signals.

[0021] Step 1103: Use a finite impulse response filter with cutoff frequencies of 0.5 Hz and 45 Hz to preserve effective neural oscillation components and suppress low-frequency drift and high-frequency noise; First, a finite impulse response filter with a cutoff frequency of 0.5 Hz (high-pass) and 45 Hz (low-pass) is used. The scientific purpose is: High-pass filters eliminate extremely low-frequency signal drift caused by breathing, sweating, and slow electrode drift, typically <0.5 Hz. These drifts do not include neural information and can drown out low-frequency delta waves. Low-pass filters, while preserving all meaningful neural oscillation frequencies (δ, θ, α, β, γ), suppress high-frequency harmonics above 50 / 60 Hz power line noise, electromyographic interference, typically >50 Hz, and environmental radio frequency noise, preparing for subsequent power line notch filtering.

[0022] Step 1104: Apply a 50Hz or 60Hz notch filter to eliminate ambient AC interference. Specifically, although sampling rate and low-pass filtering help suppress interference, AC power supply interference (50Hz or 60Hz, depending on the region) in the environment can still strongly couple into the signal due to problems such as grounding loops. Therefore, applying a highly targeted narrowband notch filter can almost completely eliminate this fixed-frequency interference without affecting neural signals in adjacent frequency bands, such as 48-52Hz or 58-62Hz.

[0023] Step 1105: Using independent component analysis, artifacts generated by eye movements, blinking, and muscle activity are automatically identified and removed. Specifically, the electrical signal amplitudes generated by eye movements, blinking, and subtle head muscle activities are much higher than those of cortical EEG, and must be effectively identified and eliminated. This system employs Independent Component Analysis (ICA). ICA is a blind source separation algorithm that assumes that the multi-channel recorded EEG signals are a linear mixture of several statistically independent source signals (including neural sources and various artifact sources). By performing matrix decomposition on the mixed signal, ICA can separate the ocular artifact components and the electromyographic artifact components. The ocular artifact components exhibit a typical temporal morphology of maximum frontal amplitude and high synchronization across all channels. The electromyographic artifact components manifest as irregular high-frequency bursts. The system then automatically identifies and eliminates these independent components associated with physiological artifacts, and reconstructs the remaining components, mainly neural activity signals, back into the electrode space. Compared to simple regression or thresholding methods, this method can remove artifacts more accurately.

[0024] Step 1106: Convert the signals from all electrodes into EEG signals with reference to the average potential of the whole brain; and segment the clean EEG signals into segments according to fixed time windows to generate sample segments for model input.

[0025] Specifically, the raw EEG recording requires a voltage reference point. Initial reference points, such as those behind the ear or mastoid process, may be unstable or introduce noise. To obtain a more robust and interpretable signal, this step converts the signals from all electrodes to a whole-brain average potential. That is, the potential value of each electrode at each time point is subtracted from the average potential of all electrodes at that time point. This method helps reduce common noise and makes the recorded potentials closer to the relative intensity of local neural activity in the brain, and is a widely accepted standard preprocessing step in EEG analysis.

[0026] Specifically, the clean EEG signal can be segmented according to a fixed-duration time window to generate model input samples. A sliding window of 2 seconds with a 50% overlap is used to divide the continuous EEG signal into multiple analysis sample segments. Each sample segment contains the synchronization signals of all electrodes within that time window, serving as the basic input unit for the subsequent multimodal state assessment model.

[0027] Step 120: Input the cleaned EEG signal into the pre-trained multimodal state assessment model, and simultaneously decode and output the cognitive index that quantifies the patient's current cognitive state and the emotion vector that represents the patient's current emotional state.

[0028] The cognitive state is obtained by quantifying features related to attention, working memory, and information processing speed in EEG signals. The cognitive index is a normalized scalar value that integrates the theta / alpha wave power ratio, P300 event-related potential latency, and brain network connectivity complexity. Its value ranges from 0 to 1. A higher value indicates a stronger overall cognitive ability (including attention maintenance, working memory capacity, and information processing speed) for the patient, while a lower value indicates a lower cognitive function or a state of distraction. The emotional state is obtained by identifying features related to emotional arousal and pleasure in EEG signals. The emotional vector is a multidimensional feature that includes at least components representing emotional valence and / or emotional arousal.

[0029] This step is the core analysis and decision-making process of this invention, aiming to transform the pre-processed clean EEG signals into quantifiable and interpretable indicators of cognitive and emotional states. Traditional methods typically analyze cognition and emotion separately and sequentially, which is not only inefficient but also ignores the close interaction between the two in terms of neural mechanisms. For example, anxiety can significantly impair working memory. This invention employs a pre-trained multimodal state assessment model to achieve end-to-end, synchronous decoding from EEG signals to the cognitive-emotional joint state.

[0030] The quantitative goal of cognitive status assessment is to evaluate core areas closely related to Alzheimer's disease rehabilitation: attention maintenance, working memory capacity, and information processing speed. These functions are closely related to the synchronicity of neural oscillations in the prefrontal and parietal networks of the brain. This model integrates these into a Cognitive Index (CI), which is a normalized scalar value, for example, between 0 and 1, preferably 0.4. Its calculation incorporates the following multidimensional EEG characteristics: The theta / alpha wave power ratio, in the prefrontal cortex, shows that a relative increase in theta waves and a decrease in alpha waves is generally associated with increased cognitive load, inattention, and cognitive decline related to Alzheimer's disease (AD). This ratio is a sensitive indicator of cognitive effort and efficiency.

[0031] The latency and amplitude of P300 event-related potentials (ERPs) are P300 components induced in auditory or visual oddball paradigms. Their latency directly reflects the processing speed of information classification and memory updating (a longer latency indicates slower processing); their amplitude is related to the degree of attention resource allocation.

[0032] Brain network connectivity complexity is analyzed using graph theory to assess the synchronicity between different EEG channels, such as phase lock values, and to construct functional connectivity networks. A decrease in topological indicators such as global efficiency or small-world properties signifies a decline in the brain's ability to integrate and separate information, representing a network-level marker of cognitive impairment.

[0033] The goal of emotional state identification is to capture the patient's internal emotional experience, particularly pleasure and arousal, which directly impact the rehabilitation process. This model represents these as an emotion vector, output as a probability distribution or continuous value within a two-dimensional or three-dimensional emotion space, including valence and arousal, or valence, arousal, and dominance. This definition is based on the consensus in affective neuroscience that different emotions exhibit specific EEG patterns in areas such as asymmetrical activity in the frontotemporal lobes and visceral motor cortex.

[0034] like Figure 3 As shown, the multimodal state assessment model is a multi-branch fusion assessment network with a deep neural network as its core, including: a multi-dimensional EEG feature extraction module, a feature fusion and state encoding module, and a dual-task collaborative output module.

[0035] The multi-dimensional EEG feature extraction module is used to extract features of different attribute dimensions in parallel from the clean EEG signal, including: time-frequency feature extraction branch, spatiotemporal dynamics feature extraction branch and event-related potential feature extraction branch; The time-frequency feature extraction branch consists of a short-time Fourier transform layer and a convolutional layer. The input is an EEG signal within a time window, and the output is a time-frequency feature map representing the power and spatial distribution pattern of each frequency band (δ, θ, α, β, γ). The spatiotemporal dynamics feature extraction branch is composed of a graph convolutional network. The input is the EEG signal of the time window and its predefined brain region connection topology, and the output is a spatiotemporal feature vector characterizing the functional connectivity strength and information transmission efficiency of the brain network. The event-related potential feature extraction branch is composed of a temporal convolutional network. Its input is a fragment of EEG signal induced by a specific cognitive task and collected periodically or on demand during the rehabilitation intervention. Its output is an ERP feature vector that characterizes the latency and amplitude of the P300 component. The feature fusion and state encoding module is used to integrate multi-dimensional features and generate a unified context-aware state representation, including: a cross-dimensional attention fusion submodule and a state encoder; The cross-dimensional attention fusion submodule takes the time-frequency feature map, spatiotemporal feature vector, and ERP feature vector as input, calculates the contribution weight of different feature dimensions to the current state evaluation, performs weighted fusion, and outputs a unified fusion feature representation. The state encoder is composed of a multi-head self-attention mechanism and a feedforward neural network. Its input is the fused feature representation. Its function is to capture long-range dependencies between features and perform deep encoding. Its output is an encoded feature vector that includes global context information. The dual-task collaborative output module is used to simultaneously generate quantitative assessments of cognitive and emotional states based on encoded features, including a cognitive regression head and an emotion classification head. The input to the cognitive regression head is the encoded feature vector, the structure is a fully connected layer, and the output is the cognitive index, which is a normalized scalar value that comprehensively reflects attention and information processing ability. The input to the emotion classification head is the encoded feature vector, the structure is a fully connected layer and a softmax layer, and the output is the emotion vector, which is a probability distribution vector representing the emotion category in a multi-dimensional emotion space.

[0036] like Figure 3 As shown, the multimodal state assessment model is not a single network, but a multi-branch fusion assessment network. First, dedicated branches extract EEG features of different attributes; then, a high-level fusion mechanism captures their interaction relationships; and finally, collaborative learning completes the dual-task output. This module addresses the multifaceted nature of EEG signals by establishing three parallel and complementary feature extraction pathways to replace traditional manual feature engineering, achieving automated and deep feature learning.

[0037] The time-frequency feature extraction branch first transforms the time-domain signal into a time-spectrum map through a Short-time Fourier Transform (STFT) layer, capturing the fine-grained patterns of power evolution over time in each frequency band (δ, θ, α, β, γ). Subsequently, a convolutional layer (Conv1D) performs convolutions in both the time and frequency dimensions, automatically learning discriminative time-frequency patterns, such as rhythmic bursts of α waves or continuous slow activity of θ waves, outputting a time-frequency feature map.

[0038] The spatiotemporal dynamics feature extraction branch explores how cognition and emotion arise from the collaborative work of large-scale brain networks. This branch employs Graph Convolutional Networks (GCNs), treating each electrode as a graph node and defining brain region connectivity topologies as edges based on prior neuroanatomical knowledge, such as AAL maps or functional connectivity matrices. GCNs operate directly on the graph structure, effectively modeling the dynamic changes in functional connectivity between brain regions and extracting spatiotemporal feature vectors characterizing the strength of functional connectivity and information transmission efficiency in brain networks. This is crucial for assessing cognitive integration capabilities and emotion regulation networks.

[0039] The ERP feature extraction branch addresses the challenge of capturing precise waveforms of transient cognitive potentials such as P300 in noisy environments. This branch utilizes a Temporal Convolutional Network (TCN), which features a causal dilated convolution structure and excels at capturing long-term dependencies in long sequences. It can accurately extract ERP feature vectors such as latency and amplitude of P300 components from single or a small number of superimposed EEG segments, significantly improving the feasibility of assessing cognitive processing speed in natural conditions, i.e., without multiple averaging.

[0040] The cross-dimensional attention fusion submodule receives the three feature vectors / graphs mentioned above. Through an attention mechanism, it dynamically calculates the contribution weights of time-frequency features, network connectivity features, and ERP features to the final state judgment under the current input. For example, when judging attention level, it may rely more on ERP features and frontal θ / α time-frequency features; when judging emotional valence, it may rely more on frontotemporal asymmetric time-frequency features and network features involving the limbic system. This module performs weighted fusion, outputting a unified and focused fusion feature representation.

[0041] The state encoder is composed of multiple multi-head self-attention mechanisms and feedforward neural network layers stacked together. This structure can perform global context modeling on the fused features, capture complex, long-range nonlinear interactions between features, and finally output a highly abstract, information-rich context-aware encoded feature vector that encapsulates comprehensive information for inferring cognitive and emotional states.

[0042] The neural foundations of cognition and emotion overlap but also differ. Joint learning allows models to share underlying feature representations, mutually regularize, and improve generalization ability. This module includes two parallel heads that partially share parameters. The cognitive regression head is essentially a fully connected layer that maps the encoded feature vector to a continuous cognitive exponential scalar value to complete the regression task. The emotion classification / regression head typically consists of a fully connected layer followed by a Softmax layer, mapping the encoded feature vector to a probability distribution of discrete emotion categories (emotion vector) to complete the classification task; or directly outputting continuous values ​​of valence and arousal.

[0043] Furthermore, the training of the multimodal state assessment model includes: A multi-task training dataset is constructed, which includes a set of labeled EEG data samples from multiple subjects. Each sample includes: raw EEG signals for a time window; a corresponding standardized cognitive function score obtained from a professional neuropsychological assessment scale, serving as the ground truth label for the cognitive state; and a corresponding multidimensional emotion category label recorded from standard emotion-inducing experiments or clinical observations, serving as the ground truth label for the emotion state. Construct a weighted joint loss function L_total to simultaneously optimize cognitive regression and emotion classification tasks, its expression is: L_total = α * L_cognitive + β * L_emotion + γ * L_regularization Wherein, L_cognitive is the mean squared error loss used for cognitive regression, which calculates the difference between the cognitive index output by the multimodal state assessment model and the true value of the standardized cognitive function score; L_emotion is the cross-entropy loss used for emotion classification, which calculates the difference between the emotion vector output by the multimodal state assessment model and the true value of the multidimensional emotion category label; L_regularization is the L2 weight regularization term, used to prevent the model from overfitting; α, β, γ are preset task balance weight hyperparameters; Using the multi-task training dataset, with the multimodal state assessment model as the training object and the joint loss function L_total as the optimization objective, the backpropagation algorithm and the adaptive moment estimation algorithm are used to iteratively update all trainable parameters in the multimodal state assessment model until the loss function converges, thus obtaining the pre-trained multimodal state assessment model.

[0044] Specifically, a large number of multi-task training datasets were collected, with each sample consisting of a triplet of EEG signal-cognitive label-emotion label. Cognitive labels were derived from specific sub-items of standardized neuropsychological scales, such as MoCA and MMSE; emotion labels were derived from standard emotion-evoking experiments, such as IAPS image viewing or clinical observation records, and assessed using standardized tools such as PANAS.

[0045] Define the weighted joint loss function: L_total = α * L_cognitive (MSE) + β * L_emotion(Cross-Entropy) + γ * L_regularization (L2). Through error backpropagation and an adaptive optimizer, such as Adam, both the cognitive index prediction error and the emotion classification error are minimized, forcing the model to learn robust feature representations that can explain both states simultaneously.

[0046] Given the heterogeneity of Alzheimer's patients, before applying a general pre-trained model to a specific patient, the model's feature fusion and state encoding modules and output head can be fine-tuned using the patient's previous treatment data. This allows the model to quickly adapt to the patient's individualized EEG feature patterns, thereby improving the personalized accuracy of state decoding and the targeted nature of rehabilitation interventions.

[0047] Step 130: Based on the cognitive index and emotion vector, a personalized initial music therapy plan including music type, rhythm, volume and timbre parameters is generated through a dynamic music mapping algorithm.

[0048] For ease of description, the dynamic music mapping algorithm described in this application refers to a decision-making method that dynamically generates music therapy parameters based on cognitive index and emotion vector, according to preset rules and patient preferences.

[0049] like Figure 4 As shown, step 130 includes the following: Step 1301: Using the cognitive index and emotion vector as input, a predefined music mapping knowledge base is applied to generate basic music parameters through music type selection rules, rhythm parameter calculation rules, and volume and timbre adjustment rules. Specifically, the music type selection rule maps the cognitive index to a first music type set if it is below a first cognitive threshold, which includes upbeat, clearly structured classical music excerpts; if the cognitive index is above or equal to the first cognitive threshold, it maps to a second music type set, which includes soothing, simple melodies, natural sounds, or ambient music. The rhythm parameter calculation rule maps the cognitive index linearly or non-linearly to a preset rhythm value range, such as 60-120. BPM, where a low cognitive index, indicating poor or scattered cognitive function, is mapped to a higher base beat value to enhance the vitality of internal neural oscillations through external rhythmic stimulation; and a high cognitive index, indicating a good cognitive state, is mapped to a lower base beat value to maintain a relaxed but focused state. The volume and timbre adjustment rules are based on the emotional valence and emotional arousal components in the emotional vector, using a two-dimensional lookup table to determine the base volume gain value and timbre filtering parameters, where a lower valence and a higher arousal combination corresponds to a reduced volume gain and an applied low-pass filter parameter.

[0050] Specifically, this step converts abstract neural indicators into specific musical acoustic parameters. The system has a built-in predefined music mapping knowledge base, whose rules are derived from research findings in music neuroscience and clinical rehabilitation experience. It is a structured data storage and rule set used to map the decoded cognitive index and emotion vector to specific music therapy parameters. The knowledge base includes at least: (1) a music type selection rule table, which stores the correspondence between cognitive index thresholds and music type sets; (2) a rhythm parameter mapping function, which defines the linear or nonlinear conversion formula from cognitive index to beats per minute (BPM); and (3) a volume and timbre parameter lookup table, which uses emotional valence and arousal components as indices to output volume gain values ​​and filter coefficients. The initial version of this knowledge base is built based on publicly available literature and clinical experience in neuromusic therapy and can be iteratively updated using subsequent personalized response data from patients.

[0051] Its mapping logic is as follows: The scientific principle behind the music genre selection rule is that the structure and rhythm of music have different effects on the brain's cognitive network. Classical music with a brisk rhythm and clear structure (such as Mozart's K.448, Baroque music), with its regular periodic rhythms and clear harmonic progressions, helps promote the activation and neural synchronization of the prefrontal cortex, thereby improving attention and working memory encoding. This is especially suitable for patients in a state of cognitive low arousal, such as apathy or slow reaction. The mapping rule sets a first cognitive threshold, for example, a cognitive index of 0.4. A value below this indicates significant cognitive low load or inattention. If the real-time cognitive index is below this threshold, the algorithm automatically maps to the first music genre set, prioritizing classical music fragments with cognitive enhancement potential. Conversely, if the cognitive index is above or equal to the threshold, indicating that the patient may be in a normal or tense state, the algorithm maps to a second music genre set. This set includes soothing, simple melodies of natural sounds, such as streams, birdsong, or ambient music, designed to reduce cognitive load, induce relaxation, and provide rest for overactive cognitive networks.

[0052] The rhythm parameter calculation rule is based on the scientific principle of musical rhythm, or BPM, which exhibits a correlation between beats per minute and heart rate and brainwave oscillation frequency. Faster rhythms, such as 100-120 BPM, may promote beta wave activity, associated with alertness and attention; while slower rhythms, such as 60-80 BPM, help guide brainwave activity towards relaxed alpha waves. The mapping rule maps the cognitive index to a specific rhythm range, for example, 60-120 BPM, using a pre-defined linear or non-linear function. Lower cognitive indices, indicating impaired cognitive function, correspond to higher baseline beat values, aiming to enhance the activity of internal neural oscillations through external rhythmic stimulation. The specific values ​​for "lower" and "impaired" can be empirical values ​​from multiple experiments, and this application does not limit this.

[0053] The scientific principle behind volume and timbre adjustment rules is that volume directly affects the intensity of stimulation to the auditory system, while timbre, through filtering and adjusting brightness / softness, influences emotional perception. High-arousal negative emotions, such as anxiety and agitation, require reduced stimulation intensity and sensory load. The mapping rule involves querying a two-dimensional lookup table based on two core components of the emotion vector—emotional valence (e.g., pleasant / unpleasant) and emotional arousal (e.g., calm / excited). For example, when the system identifies a combination of "lower valence (unpleasant) and higher arousal (excited)," it determines there is a risk of anxiety or agitation. The lookup table will output instructions to reduce the base volume gain, such as setting it to 70% of normal, and apply low-pass filtering parameters to the music, such as attenuating high-frequency components, making the timbre softer and more muted. This aims to quickly reduce sensory input intensity, create a safe and inclusive auditory environment, and calm emotional fluctuations.

[0054] As an optional exemplary implementation, the music mapping knowledge base may include, but is not limited to, the following mapping rule table, as shown in Table 1: Table 1 It should be noted that the thresholds, parameters, and mapping formulas in the table above are merely illustrative examples and can be optimized and adjusted based on clinical data, patient population characteristics, and individualized feedback in practical applications. The scope of protection of this invention is not limited to the specific example values.

[0055] Step 1302: Retrieve the pre-stored personal music preference file of the patient, which includes a list of music genres, artists or specific tracks that the patient has historically preferred; Specifically, rule mapping ensures the scientific rigor of the intervention, but adherence to rehabilitation and emotional resonance are crucial. The system retrieves a personalized music preference profile, independently created and maintained for each patient. This profile is dynamically built and updated in the following ways: Initial interviews and questionnaires were conducted in the early stages of treatment to collect information from the patient, their family, or caregivers about their preferred music genres, artists, specific pieces of music, or fond memories related to music at different stages of their lives.

[0056] Implicit learning involves recording the patient's physiological responses to different music segments during subsequent treatment, such as the music being played when the emotional vector tends to change positively, and updating the preference weights accordingly.

[0057] Nostalgia therapy integrates music that was popular during the patient's youth or middle age, usually a period when memories are relatively intact. This approach utilizes the nostalgia effect to stimulate positive emotions and enhance self-identity, and has proven to have unique value in Alzheimer's disease care.

[0058] Step 1303: Match and weight the music type set output by the music type selection rule with the personal music preference profile, and prioritize the selection of music entries that simultaneously meet the music type selection rule and the personal music preference profile.

[0059] Specifically, the algorithm does not simply choose between the rule output and personal preference, but rather performs intelligent matching and weighted fusion. The system cross-matches the music genre set output by the rule in step 1301, such as upbeat classical music, with the tracks in the personal preference profile, and filters out a list of candidate tracks that meet the requirements of both.

[0060] If no perfect match is found, the system will dynamically weight rule compliance and personal preference based on the current treatment goal, such as whether the focus is on cognitive improvement or emotional stability. For example, rule compliance has a higher weight in acute emotional crisis intervention; in maintenance rehabilitation, the weight of personal preference can be appropriately increased to improve patient acceptance. This process ensures that the final selected music is not only theoretically effective but also as appealing and acceptable to the patient as possible.

[0061] Step 1304: The final determined music type, the calculated rhythm parameters, and the adjusted volume and timbre parameters are encapsulated and combined to generate the personalized initial music therapy plan.

[0062] Specifically, the results of the above decision-making process—the final determined music type (specific track), the calculated precise beat per minute (BPM), and the adjusted volume and timbre parameters, such as gain dB and filter coefficients—are encapsulated according to a predetermined data protocol and combined to generate a structured, machine-readable, personalized initial music therapy plan. This plan is then sent to the audio rendering engine for execution. This marks the official start of a personalized intervention cycle based on the patient's real-time brain state, combining scientific precision with humanistic care.

[0063] Step 140: Execute the initial music therapy plan and monitor the changes in the cognitive index and the emotion vector in real time during the intervention. When the changes exceed a preset threshold, dynamically adjust at least one music parameter in the treatment plan to form a closed-loop rehabilitation intervention.

[0064] like Figure 5 As shown, step 140 includes the following: Step 1401: While starting the personalized initial music therapy plan, initiate the real-time EEG signal acquisition and analysis process, and record the cognitive index and emotion vector obtained at the initial moment of plan execution as the state baseline value of this intervention; the state baseline value includes the baseline cognitive index and the baseline emotion vector.

[0065] The baseline state value refers to the initial reference value of the patient's cognitive and emotional state at the start of this intervention session, specifically including the baseline cognitive index C_baseline and the baseline emotional vector E_baseline. All subsequent state changes, such as ΔC and ΔE, are calculated based on this baseline value to eliminate the influence of differences in the patient's baseline state across different treatment days.

[0066] Specifically, as the personalized initial music therapy plan begins playing, the system simultaneously initiates a real-time EEG acquisition and analysis thread. The cognitive index and emotion vector decoded at the initial moment of plan execution, approximately 5-10 seconds after the music starts (to exclude the attention shift effect at the start), are recorded as the baseline values ​​for this intervention session, including the baseline cognitive index C_baseline and the baseline emotion vector E_baseline. This step is crucial because it provides an individualized, session-specific reference point for subsequent assessments of state changes, avoiding misjudgments caused by differences in the patient's underlying physiological and psychological state at different dates or times.

[0067] Step 1402: During the duration of the music intervention, signal acquisition, preprocessing, and state decoding are repeated at fixed time intervals to obtain real-time cognitive index and emotion vector. Specifically, during the music intervention, the system repeatedly executes the signal acquisition, preprocessing, and state decoding process at fixed time intervals, for example, every 30 seconds as a monitoring cycle, namely steps 110 and 120, to obtain the real-time cognitive index C_t and the real-time emotion vector E_t. This high-frequency and continuous monitoring constitutes a continuous monitoring of the patient's internal neuropsychological state's "vital signs".

[0068] Step 1403: Calculate the change in cognitive index ΔC and the change in emotion vector ΔE; the change in cognitive index ΔC is the difference between the real-time cognitive index and the baseline cognitive index; the change in emotion vector ΔE is the Euclidean distance or cosine similarity between the real-time emotion vector and the baseline emotion vector in the multidimensional emotion space.

[0069] Specifically, to accurately measure state changes, the system calculates two core offsets.

[0070] The change in the cognitive index, ΔC, is calculated as ΔC = C_t - C_baseline. This difference directly reflects the change in the patient's attention level relative to the initial stage of intervention. A positive increase indicates potential inattention, while a negative decrease may indicate increased attention or a state of excessive relaxation.

[0071] The change in the emotion vector, ΔE, is calculated using Euclidean distance, ΔE = ||E_t - E_baseline||, since emotion is multidimensional. This scalar value comprehensively reflects the overall shift of the emotional state in multidimensional spaces such as valence and arousal. The larger the distance, the further the emotional state deviates from the initial baseline.

[0072] It should be noted that an increase in the cognitive index in this method does not always represent an improvement in cognitive ability. In the context of real-time monitoring, when ΔC exceeds the preset threshold θ_C, it indicates that the patient's neurophysiological state has deviated from the optimal working range set for this intervention. This deviation may manifest as EEG characteristics related to inattention, such as a surge in the prefrontal θ / α power ratio. Therefore, a positive change in ΔC in this specific context is interpreted as a negative state change requiring intervention.

[0073] Step 1404: Compare the changes ΔC and ΔE with preset cognitive adjustment thresholds θ_C and θ_E, respectively; Step 1405: If the conditions ΔC>θ_C or ΔE>θ_E are met, it is determined that the patient's condition has changed significantly, triggering a dynamic adjustment of the music therapy plan; Specifically, the system compares the calculated ΔC and ΔE with pre-set cognitive adjustment thresholds θ_C and θ_E, respectively. These thresholds can be clinically calibrated based on the patient's disease stage and individual responsiveness. Once the condition ΔC > θ_C or ΔE > θ_E is met, indicating a clinically significant change in the patient's current state that exceeds the normal fluctuation range, the system immediately triggers a dynamic adjustment decision. This signifies a switch from a "implement a predetermined plan" mode to an adaptive adjustment mode.

[0074] Step 1406: Based on the specific state dimension that triggered the adjustment, perform at least one of the following adjustment operations: If triggered by ΔC > θ_C, and ΔC is positively increasing (i.e., the cognitive index value increases, which, according to the definition of this method, means that the patient has shifted from the baseline state to a state of high arousal, high load of inattention or tension, manifested as a neurophysiological pattern such as an abnormally increased θ / α power ratio), then inattention is determined, and the adjustment plan is: gradually increase the music rhythm parameter by 15%-30% within 10 seconds, and switch to a music type with a more distinct rhythm; If triggered by ΔE>θ_E, and ΔE points to a high-arousal negative emotion, then anxiety or agitation is determined to occur. The adjustment plan is as follows: gradually reduce the volume parameter to below 50% of the baseline value within 5 seconds, and immediately switch to the preset "soothing intervention" music segment, which is mainly composed of continuous low-frequency natural sounds. Specifically, the adjustments are not made blindly, but rather through precise and targeted interventions based on the triggering source and the direction of change.

[0075] Scenario 1: Cognitive Distraction Intervention. If the intervention is triggered by ΔC > θ_C and ΔC being positive (i.e., an increase in the cognitive index indicating a decline in attention level), the system determines that the patient is experiencing inattention. The adjustment strategy is as follows: The rhythm is intensified by gradually increasing the music's beat rate (BPM) by 15%-30% over 10 seconds. By enhancing the driving force of the external rhythm, it attempts to create a "carrying" effect on internal neural oscillations, thus pulling scattered attention back.

[0076] Type switching involves simultaneously switching to music genres in the music library that have more distinct rhythms and more regular structures (such as marches or Baroque polyphonic music), utilizing their strong sense of rhythm and predictability to provide clear temporal structure cues, and helping the brain reorganize attentional resources.

[0077] Scenario 2: Negative Emotion Intervention. If triggered by ΔE > θ_E and the direction of ΔE pointing towards high-arousal negative emotions such as anxiety or irritability, the system determines that the patient is experiencing emotional agitation. The adjustment strategy is as follows: Stimulus reduction involves gradually lowering the volume parameter to below 50% of the baseline value within 5 seconds. This rapidly reduces the input intensity through the auditory channel, lowers sensory load, and cools down the overexcited nervous system.

[0078] The content transitions immediately, without gradual transition, to a preset soothing intervention music clip. This clip specifically refers to an acoustically designed soundscape dominated by continuous low-frequency natural sounds, such as the surging of the deep sea, steady rain, or a single instrument's sustained note. It is characterized by a lack of pronounced rhythm, gentle melodic changes, and a spectral energy concentrated in the low frequencies. This sound environment is designed to induce a relaxation response and counteract a state of hyperarousal.

[0079] Step 1407: During the implementation of the adjusted treatment plan, continuously calculate the change in cognitive index ΔC and the change in emotion vector ΔE; Step 1408: If, within three consecutive monitoring periods after adjustment, the change in cognitive index ΔC and the change in emotion vector ΔE both fall back to within the corresponding cognitive adjustment threshold and emotion adjustment threshold, then the intervention is deemed effective, and the current adjusted parameters are maintained until the end of this intervention session.

[0080] Specifically, during the implementation of the adjusted protocol, the system continuously calculates new ΔC and ΔE. The system sets an effectiveness judgment window, for example, three consecutive monitoring cycles. If, after adjustment, ΔC and ΔE continuously decrease and stabilize within their respective thresholds, i.e., within θ_C and θ_E, then the dynamic adjustment intervention is considered effective, successfully guiding the patient's state back to the desired range. Once deemed effective, the system maintains the current adjusted music parameter combination until the intervention session ends as planned. This ensures that the treatment environment remains in an optimal steady state for the patient for the remaining time.

[0081] Furthermore, before generating a personalized initial music therapy plan for the patient, the method also includes the step of constructing a predefined music mapping knowledge base and a personal music preference profile specific to the patient, including: During the initial assessment phase, patients are played various types of standard test music clips. While each test music clip is played, the patient's electroencephalogram (EEG) signals are simultaneously collected and analyzed, and their corresponding cognitive indices and emotion vectors are recorded. Based on these records, an initial music-state response mapping relationship is established for the patient, serving as the foundation for their exclusive predefined music mapping knowledge base. Through questionnaires or analysis of historical playback records, the patient's personal music preference profile is collected and established.

[0082] Specifically, before generating an initial personalized music therapy plan for a patient, the system must complete the construction of a personalized knowledge base for that patient. This process is the cornerstone of all subsequent personalized interventions. This step is completed during the initial assessment phase and includes: The music-neural response baseline test involves the system sequentially playing a library of standard test music clips to the patient. This library is scientifically designed to cover different music genres, such as classical, natural sound, ambient electronic, rhythms (slow, medium, fast), modes (major, minor), and emotional tones (cheerful, calm, sad). While playing each test music clip, the system simultaneously collects and analyzes the patient's electroencephalogram (EEG) signals, calculating and recording the corresponding cognitive index and emotional vector in real time. Based on these records, the system constructs an initial music-state response mapping database for the patient. For example, the records show that when playing a Baroque piece, the patient's cognitive index significantly increased and their emotional vector pointed to calm-pleasure; while when playing a dissonant modern piece, their emotional vector pointed to unpleasant-excitement. This data constitutes the prototype of a predefined music mapping knowledge base specific to the patient, providing initial parameters and validation basis for subsequent rules based on the patient's empirical data.

[0083] A personal music preference profile is established through standardized questionnaires, such as those covering music genres, eras, and cultural background preferences, as well as interviews with patients and their families. This process collects information on the patient's musical memories, cultural background, favorite artists, and specific pieces. The system then structures this information to create a personal music preference profile. This profile not only includes a static list of preferences but can also be dynamically updated and its weights adjusted based on implicit learning of patient responses during subsequent treatment, such as changes in more positive emotional vectors.

[0084] This step combines general neuroscience rules with patients' individual physiological responses and subjective preferences, realizing a shift from a general approach to a personalized approach. It provides a precise starting point for subsequent highly personalized dynamic interventions and is key to improving patient acceptance and intervention adherence.

[0085] Furthermore, before applying the multimodal state assessment model to the target patient, a step of personalized fine-tuning of the model is included, specifically including: Acquire historical EEG data of the target patient in a quiet resting state and under standard cognitive tasks; use the historical EEG data and corresponding clinical assessment records to perform small-scale incremental training or parameter adaptation on the feature fusion and state encoding module and the dual-task collaborative output module in the pre-trained multimodal state assessment model, so that the output of the model is more consistent with the individual physiological and psychological baseline of the patient.

[0086] Specifically, while pre-trained multimodal state assessment models possess powerful general decoding capabilities, the EEG signal characteristics of each Alzheimer's patient vary due to individual differences in pathological severity, brain atrophy patterns, and medication use. Therefore, lightweight, personalized fine-tuning is necessary before applying the model to specific target patients. Individual baseline data acquisition involves collecting EEG data from the target patient in a quiet, undisturbed environment while they are resting with their eyes closed, as well as EEG data while performing standard cognitive tasks, such as the simple auditory oddball task.

[0087] Combining these historical EEG data with their corresponding brief clinical assessment records, such as the scores on the mental and behavioral scales at the time, and starting with the aforementioned pre-trained model, only small-scale incremental training or transfer learning adaptation was performed on the parameters of the feature fusion and state encoding modules and the dual-task collaborative output module within the model. The goal of this fine-tuning is to ensure that the model's internal feature representations and final output more accurately reflect the correlation between the patient's unique baseline EEG physiological characteristics and behavioral performance. This is equivalent to calibrating a general model to the current patient, thereby significantly improving the individualized accuracy and reliability of subsequent real-time state decoding and reducing misjudgments.

[0088] This step addresses a key challenge in adaptability of AI models for medical applications. It avoids the impracticality of training a massive model from scratch for each patient and overcomes the potential accuracy issues that may arise from directly using general-purpose models.

[0089] Furthermore, after a complete closed-loop rehabilitation intervention session, the method also includes: Calculate and generate a quantitative report on the effectiveness of this intervention. The report shall include at least: the overall trend curve of the cognitive index during the intervention, the evolution trajectory of the emotion vector, the historical record of music parameter adjustments, and the comprehensive improvement score based on the baseline and end values ​​before and after the intervention. The quantitative report on the effectiveness shall be stored in association with all process data of this intervention and updated in the patient's long-term rehabilitation record.

[0090] Specifically, in order to scientifically evaluate efficacy, optimize treatment plans, and provide data support for clinical decision-making, the system automatically performs quantitative analysis of effects after a complete closed-loop rehabilitation intervention session, including generating an efficacy quantitative report and scoring the overall improvement.

[0091] The system automatically analyzes all data from the entire session, generating a structured quantitative report on the effectiveness, including: a cognitive index trend curve, an emotion vector evolution trajectory, and a music parameter adjustment log. The cognitive index trend curve, with time as the horizontal axis, visually displays the patient's changing attention levels throughout the intervention. The emotion vector evolution trajectory plots the dynamic transition path of emotional states in a two-dimensional or multi-dimensional emotion space. The music parameter adjustment log comprehensively records all automatically triggered music parameter adjustment events and their corresponding triggering reasons.

[0092] The algorithm comprehensively calculates the state data before intervention (baseline value), during intervention (mean or best value), and after intervention (stable value before termination) to obtain a quantitative improvement score, such as (S_post - S_baseline) / S_baseline * 100%.

[0093] The aforementioned efficacy quantification report is linked and encrypted with all process data, including the original EEG signal fragments, all decoded state sequences, and the executed treatment plan. This data is integrated and updated into the patient's long-term rehabilitation record, forming a efficacy database that evolves over time. This step transforms subjective rehabilitation experiences into objective, continuous, and traceable data assets. This not only provides clear efficacy feedback for patients and doctors, but more importantly, it provides a valuable data foundation for long-term longitudinal studies of efficacy, discovery of optimal intervention models, and future optimization of algorithms, achieving a closed loop between treatment and research.

[0094] Furthermore, before acquiring the patient's raw EEG signals, a system initialization and verification step is included: Verify operator permissions and patient identity; load a personal configuration file uniquely corresponding to the patient's identity, which includes their historical baseline data, personalized model parameters, music mapping knowledge base, and preference profile; perform self-testing and synchronization calibration of the brain-computer interface device and audio playback device to ensure the real-time performance and reliability of signal acquisition and intervention output.

[0095] Specifically, to ensure the safety, reliability, and accuracy of each treatment process, the system must complete a rigorous initialization and verification process before executing the core rehabilitation steps.

[0096] Through biometric authentication, such as facial recognition, or device identification, such as RFID wristband technology, the system dual-verifies the operator's identity, including the therapist's authorization and the patient's identity, ensuring accurate treatment relationships, correct data attribution, and recording operation logs to meet medical compliance requirements. Based on successfully verified patient identity, the system loads a unique personal profile from a secure server. This file is a data package integrating all the patient's personalized settings, including their historical baseline, fine-tuned personalized model parameters, a dedicated music mapping knowledge base, and a personal music preference profile. The system automatically performs self-diagnostic checks on brain-computer interface devices, such as checking electrode impedance, signal quality, and high-fidelity audio playback devices. Subsequently, time-domain synchronization calibration is performed to ensure strict alignment between the timestamps of EEG signal acquisition and the event timestamps of music playback and parameter adjustments. This is a technical prerequisite for achieving real-time closed-loop intervention and subsequent accurate data analysis.

[0097] By applying the cognitive-emotional collaborative rehabilitation method based on brain-computer interface and AI provided in this invention, a multimodal state assessment model is used to synchronously and accurately decode EEG features related to cognitive function and emotional state, overcoming the limitations of separate assessment in traditional methods and providing precise input for collaborative intervention. Furthermore, based on the decoded state index, a dynamic music mapping algorithm automatically generates music therapy plans that integrate neuroscience rules and personal preferences, such as type, rhythm, volume, and timbre, making the intervention starting point highly personalized. Further, state changes are monitored in real time during intervention, and dynamic adjustments to music parameters are automatically triggered when changes exceed thresholds, forming a closed loop of monitoring-decision-intervention-remonitoring. This allows treatment to be dynamically optimized according to the patient's state, significantly improving the accuracy and effectiveness of rehabilitation. Furthermore, this application ensures the individual adaptability, safety, and execution reliability of the entire method from analysis to intervention through pre-treatment personalized model fine-tuning, file construction, and synchronous hardware calibration, laying the foundation for clinical translation.

[0098] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0099] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0100] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cognitive-emotional collaborative rehabilitation method based on brain-computer interface and AI, characterized in that, The method includes: The patient's raw EEG signals were collected and preprocessed to obtain clean EEG signals; wherein, the raw EEG signals refer to the bioelectric signal sequences recorded from the scalp surface, including delta wave, theta wave, alpha wave, beta wave and gamma wave frequency band components; The cleaned EEG signals are input into a pre-trained multimodal state assessment model, which simultaneously decodes and outputs a cognitive index that quantifies the patient's current cognitive state and an emotion vector that represents the patient's current emotional state. The cognitive state is obtained by quantifying features in the EEG signals related to attention, working memory, and information processing speed. The cognitive index is a normalized scalar value that integrates the theta / alpha wave power ratio, P300 event-related potential latency, and brain network connectivity complexity, ranging from 0 to 1, with higher values ​​representing the patient's current comprehensive cognitive ability. The emotional state is obtained by identifying features in the EEG signals related to emotional arousal and pleasure. The emotion vector is a multidimensional feature that includes at least components representing emotional valence and / or emotional arousal. Based on the cognitive index and emotion vector, a personalized initial music therapy plan is generated using a dynamic music mapping algorithm, including music type, rhythm, volume and timbre parameters. The initial music therapy program is implemented, and the changes in the cognitive index and the emotion vector are monitored in real time during the intervention. When the changes exceed a preset threshold, at least one music parameter in the treatment program is dynamically adjusted to form a closed-loop rehabilitation intervention.

2. The method according to claim 1, characterized in that, The process of collecting and preprocessing the patient's raw EEG signals to obtain clean EEG signals specifically includes: Using an EEG cap containing at least 64 recording electrodes, raw EEG signals from multiple brain regions on the patient's scalp are simultaneously acquired at a sampling rate of not less than 512 Hz. The raw EEG signals are a mixed time series of bioelectrical activity in the frequency bands of delta waves, theta waves, alpha waves, beta waves, and gamma waves. The raw EEG signal was processed sequentially as follows to obtain a clean EEG signal: A finite impulse response filter with cutoff frequencies of 0.5 Hz and 45 Hz is used to preserve the effective neural oscillation component and suppress low-frequency drift and high-frequency noise. Apply a 50Hz or 60Hz notch filter to eliminate ambient AC interference; An independent component analysis method is used to automatically identify and remove artifacts generated by eye movements, blinking, and muscle activity; The signals from all electrodes are converted into EEG signals with reference to the average potential of the whole brain; and the clean EEG signals are segmented according to fixed time windows to generate sample segments for model input.

3. The method according to claim 1, characterized in that, The multimodal state assessment model is a multi-branch fusion assessment network with deep neural networks as its core, including: a multi-dimensional EEG feature extraction module, a feature fusion and state encoding module, and a dual-task collaborative output module; The multi-dimensional EEG feature extraction module is used to extract features of different attribute dimensions in parallel from the clean EEG signal, including: time-frequency feature extraction branch, spatiotemporal dynamics feature extraction branch and event-related potential feature extraction branch; The time-frequency feature extraction branch consists of a short-time Fourier transform layer and a convolutional layer. The input is an EEG signal within a time window, and the output is a time-frequency feature map representing the power and spatial distribution pattern of each frequency band (δ, θ, α, β, γ). The spatiotemporal dynamics feature extraction branch is composed of a graph convolutional network. The input is the EEG signal of the time window and its predefined brain region connection topology, and the output is a spatiotemporal feature vector characterizing the functional connectivity strength and information transmission efficiency of the brain network. The event-related potential feature extraction branch is composed of a temporal convolutional network. Its input is a segment of EEG signal induced by a specific cognitive task, and its output is an ERP feature vector representing the latency and amplitude of the P300 component. The feature fusion and state encoding module is used to integrate multi-dimensional features and generate a unified context-aware state representation, including: a cross-dimensional attention fusion submodule and a state encoder; The cross-dimensional attention fusion submodule takes the time-frequency feature map, spatiotemporal feature vector, and ERP feature vector as input, calculates the contribution weight of different feature dimensions to the current state evaluation, performs weighted fusion, and outputs a unified fusion feature representation. The state encoder is composed of a multi-head self-attention mechanism and a feedforward neural network. Its input is the fused feature representation. Its function is to capture long-range dependencies between features and perform deep encoding. Its output is an encoded feature vector that includes global context information. The dual-task collaborative output module is used to simultaneously generate quantitative assessments of cognitive and emotional states based on encoded features, including a cognitive regression head and an emotion classification head. The input to the cognitive regression head is the encoded feature vector, the structure is a fully connected layer, and the output is the cognitive index, which is a normalized scalar value that comprehensively reflects attention and information processing ability. The input to the emotion classification head is the encoded feature vector, the structure is a fully connected layer and a softmax layer, and the output is the emotion vector, which is a probability distribution vector representing the emotion category in a multi-dimensional emotion space.

4. The method according to claim 3, characterized in that, The training of the multimodal state assessment model includes: A multi-task training dataset is constructed, which includes a set of labeled EEG data samples from multiple subjects. Each sample includes: raw EEG signals for a time window; a corresponding standardized cognitive function score obtained from a professional neuropsychological assessment scale, serving as the ground truth label for the cognitive state; and a corresponding multidimensional emotion category label recorded from standard emotion-inducing experiments or clinical observations, serving as the ground truth label for the emotion state. Construct a weighted joint loss function L_total to simultaneously optimize cognitive regression and emotion classification tasks. Its expression is: L_total = α * L_cognitive + β * L_emotion + γ * L_regularization Wherein, L_cognitive is the mean squared error loss used for cognitive regression, which calculates the difference between the cognitive index output by the multimodal state assessment model and the true value of the standardized cognitive function score; L_emotion is the cross-entropy loss used for emotion classification, which calculates the difference between the emotion vector output by the multimodal state assessment model and the true value of the multidimensional emotion category label; L_regularization is the L2 weight regularization term, used to prevent the model from overfitting; α, β, γ are preset task balance weight hyperparameters; Using the multi-task training dataset, with the multimodal state assessment model as the training object and the joint loss function L_total as the optimization objective, the backpropagation algorithm and the adaptive moment estimation algorithm are used to iteratively update all trainable parameters in the multimodal state assessment model until the loss function converges, thus obtaining the pre-trained multimodal state assessment model.

5. The method according to claim 1, characterized in that, The process of generating a personalized initial music therapy plan based on cognitive index and emotion vector through a dynamic music mapping algorithm specifically includes: Using the cognitive index and emotion vector as input, a predefined music mapping knowledge base is applied to generate basic music parameters through music type selection rules, rhythm parameter calculation rules, and volume and timbre adjustment rules. Specifically, the music type selection rule maps the cognitive index to a first music type set if it is below a first cognitive threshold, which includes classical music fragments with a brisk rhythm and clear structure; if the cognitive index is above or equal to the first cognitive threshold, it maps to a second music type set, which includes natural sounds or ambient music with a soothing rhythm and simple melody. The rhythm parameter calculation rule maps the cognitive index linearly or non-linearly to a preset rhythm value range, where a lower cognitive index corresponds to a higher base beat value. The volume and timbre adjustment rules determine the base volume gain value and timbre filtering parameters based on the emotional valence and emotional arousal components in the emotion vector using a two-dimensional lookup table, where a lower valence combined with a higher arousal corresponds to a reduced volume gain and an applied low-pass filter parameter. Retrieve the patient's pre-stored personal music preference profile, which includes a list of the patient's historical preferred music genres, artists, or specific tracks; The music type set output by the music type selection rule is matched and weighted with the personal music preference profile, and music entries that meet both the music type selection rule and the personal music preference profile are selected first. The final music type, calculated rhythm parameters, and adjusted volume and timbre parameters are encapsulated and combined to generate the personalized initial music therapy plan.

6. The method according to claim 1, characterized in that, The execution of the initial music therapy plan, and the real-time monitoring of changes in the cognitive index and the emotion vector during the intervention, wherein at least one music parameter in the treatment plan is dynamically adjusted when the changes exceed a preset threshold, forming a closed-loop rehabilitation intervention, specifically includes: Simultaneously with the commencement of the personalized initial music therapy program, a real-time EEG signal acquisition and analysis process is initiated. The cognitive index and emotion vector acquired at the initial moment of program execution are recorded as the baseline values ​​for this intervention. The baseline values ​​include the baseline cognitive index and the baseline emotion vector. During the music intervention, signal acquisition, preprocessing, and state decoding were repeated at fixed time intervals to obtain real-time cognitive indices and emotion vectors. Calculate the change in cognitive index ΔC and the change in emotion vector ΔE; the change in cognitive index ΔC is the difference between the real-time cognitive index and the baseline cognitive index; the change in emotion vector ΔE is the Euclidean distance or cosine similarity between the real-time emotion vector and the baseline emotion vector in the multidimensional emotion space. The changes ΔC and ΔE are compared with the preset cognitive adjustment threshold θ_C and emotion adjustment threshold θ_E, respectively; If the conditions ΔC>θ_C or ΔE>θ_E are met, the patient's condition is determined to have changed significantly, triggering a dynamic adjustment to the music therapy plan; Based on the specific state dimension that triggered the adjustment, perform at least one of the following adjustment operations: If the event is triggered by ΔC > θ_C and ΔC is increasing in a positive direction, then the focus is considered to be scattered. The adjustment plan is to gradually increase the music rhythm parameter by 15%-30% within 10 seconds and switch to a music type with a more distinct rhythm. If triggered by ΔE>θ_E, and ΔE points to a high-arousal negative emotion, then anxiety or agitation is determined to occur. The adjustment plan is as follows: gradually reduce the volume parameter to below 50% of the baseline value within 5 seconds, and immediately switch to the preset soothing intervention music segment, which is mainly composed of continuous low-frequency natural sounds. During the implementation of the adjusted treatment plan, the changes in the cognitive index ΔC and the changes in the emotion vector ΔE were continuously calculated. If, within three consecutive monitoring periods after adjustment, the change in cognitive index ΔC and the change in emotion vector ΔE both fall back to within the corresponding cognitive adjustment threshold and emotion adjustment threshold, then the intervention is deemed effective, and the current adjusted parameters are maintained until the end of this intervention session.

7. The method according to any one of claims 1 or 5, characterized in that, Before generating a personalized initial music therapy plan for the patient, the method also includes the step of building a predefined music mapping knowledge base and a personal music preference profile specific to the patient, including: During the initial assessment phase, patients are played various types of standard test music clips; While playing each test music segment, the patient's EEG signals were collected and analyzed simultaneously, and their corresponding cognitive index and emotion vector were recorded. Based on the records, an initial music-state response mapping relationship is established for the patient, serving as the basis for their exclusive predefined music mapping knowledge base; The patients' personal music preference profiles were collected and established through questionnaires or analysis of historical playback records.

8. The method according to any one of claims 3 or 4, characterized in that, Before applying the multimodal state assessment model to the target patient, the method further includes a step of personalized fine-tuning of the model, specifically including: Acquire historical EEG data of the target patient in a quiet resting state and under standard cognitive tasks; Using the historical EEG data and corresponding clinical assessment records, small-scale incremental training or parameter adaptation is performed on the feature fusion and state encoding module and the dual-task collaborative output module in the pre-trained multimodal state assessment model, so that the model output is more consistent with the patient's individual physiological and psychological baseline.

9. The method according to any one of claims 1 or 6, characterized in that, The method further includes the following after a complete closed-loop rehabilitation intervention session: Calculate and generate a quantitative report on the effectiveness of this intervention. The report shall include at least: the overall trend curve of the cognitive index during the intervention, the evolution trajectory of the emotion vector, the historical record of music parameter adjustments, and the comprehensive improvement score based on the baseline and end values ​​before and after the intervention. The effectiveness quantification report is linked and stored with all process data of this intervention, and updated in the patient's long-term rehabilitation record.

10. The method according to claim 1, characterized in that, Before acquiring the patient's raw EEG signals, system initialization and verification steps are also included: Verify operator permissions and patient identity; Load a personal profile that uniquely corresponds to the patient's identity. The profile includes the patient's historical baseline data, personalized model parameters, music mapping knowledge base, and preference profile. Perform self-testing and synchronous calibration of the brain-computer interface device and audio playback device to ensure the real-time performance and reliability of signal acquisition and intervention output.

Images