Multi-mode subconsciousness obstacle degree dynamic evaluation method

Abstract

The invention discloses a method for dynamically evaluating the degree of disturbance of consciousness under multiple modes, and the method comprises the steps: S1, obtaining a physiological signal of a detected person under multiple modes; s2, preprocessing the physiological signal to obtain a preprocessed signal; s3, calculating features of the physiological signals based on the preprocessed signals to obtain multi-dimensional features; s4, preprocessing the multi-dimensional features to obtain preprocessed data; s5, fusing the preprocessed data to obtain fusion features; and S6, inputting the fusion features into a mode recognition output module to obtain a disturbance of consciousness assessment score of the tested person. According to the method, the degree of disturbance of consciousness is evaluated in multiple modes and multiple dimensions, so that the result is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of dynamic assessment of consciousness disorders, and in particular to a multimodal dynamic assessment method for the degree of consciousness disorders. Background Art

[0002] Chronic disorder of consciousness (CD) is a pathological process in which a person loses consciousness due to severe brain damage or cerebrovascular disease. Patients are unable to correctly understand their own state or the environment, and are unable to respond correctly to external stimuli. The main types of CD include vegetative state and minimally conscious state.

[0003] Accurate classification is crucial for subsequent treatment strategies for patients with chronic disorders of consciousness. The current clinical diagnostic standard is the Coma Recovery Scale-Revised (CRS-R). The CRS-R is the only standardized neuropsychological assessment scale, providing detailed scoring instructions for six major and 23 sub-items. However, due to challenges such as the diversity of clinical symptoms of chronic disorders of consciousness, the instability of subtle behavioral differences between minimally conscious and vegetative states, and variability in physician clinical experience, behavioral assessments have a high rate of misdiagnosis. Summary of the Invention

[0004] The purpose of the present invention is to provide a multimodal method for dynamic assessment of the degree of consciousness disorder, aiming to solve the problem of dynamic assessment of the degree of consciousness disorder.

[0005] The present invention provides a multimodal subconscious disorder degree dynamic assessment method, comprising: S1. Acquire the physiological signals of the person being measured under multimodal conditions; S2. preprocessing the physiological signal to obtain a preprocessed signal; S3, calculating the features of the physiological signal based on the preprocessed signal to obtain a multi-dimensional feature; S4. Preprocessing the multidimensional features to obtain preprocessed data; S5, fusing the preprocessed data to obtain fusion features; S6. Input the fused features into the pattern recognition output module to obtain the consciousness disorder assessment score of the person being tested.

[0006] By adopting the embodiment of the present invention, dynamic physiological signals in multimodal conditions are acquired to obtain multidimensional features, and the consciousness disorder assessment of the subject is judged based on the multidimensional features, so that the multidimensional assessment is more accurate.

[0007] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it is implemented in accordance with the contents of the specification, and in order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0009] Figure 1 4 is a flowchart of a multimodal subconscious disorder degree dynamic assessment method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0010] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0011] Method Example According to an embodiment of the present invention, a multimodal subconscious disorder degree dynamic assessment method is provided. Figure 1 FIG. 1 is a flow chart of a multimodal subconsciousness disorder degree dynamic assessment method according to an embodiment of the present invention. Figure 1 As shown, specifically including: S1. Acquire the physiological signals of the person being measured under multimodal conditions; The S1 specifically includes: collecting the subject's electroencephalogram (EEG) signal, electrocutaneous signal, electrooculogram (EOG) signal, electrocardiogram (ECG) signal and blood oxygenation signal in real time in a music state and a quiet state respectively.

[0012] In this embodiment of the present invention, five sensors—electroencephalography, electrodermal conduction, electrocardiography, electrooculography, and blood oxygenation—are installed on the patient, enabling simultaneous dynamic assessment of the patient's neural activity, autonomic nervous system, cardiovascular, visual, and metabolic status. In the ward, selected music is played for 30 minutes every three hours from 8:00 AM to 5:00 PM, four times daily, while the patient's physiological indicators are recorded in real time. The music is intended to be the patient's favorite music or a symphony with a wide range of rhythmic variations.

[0013] S2. preprocessing the physiological signal to obtain a preprocessed signal; In a static state, the original EEG signals are subjected to preprocessing operations such as downsampling, filtering, interpolation, artifact removal, re-referencing, and segmentation. The electrocutaneous signals are subjected to preprocessing operations such as low-pass filtering, baseline correction, and artifact removal. The blood oxygen and ECG signals are subjected to preprocessing operations mainly including filtering. The electrooculogram signals are subjected to preprocessing operations such as filtering, endpoint detection, and normalization. The original EEG signals are collected by 64 electrodes, representing different brain regions.

[0014] S3, calculating the features of the physiological signal based on the preprocessed signal to obtain a multi-dimensional feature; The S3 specifically includes: In a quiet state, EEG power spectrum density features and EEG Hilbert Huang entropy features are calculated based on EEG signals; Eigenvalue 1: Power Spectral Density The method of short-time Fourier transform is to perform Fourier transform on the EEG signal, namely: (1) in, is the angular frequency, t is the time variable, is the integral variable, which is used to perform integration operations on the EEG signal on the time axis. For EEG signals at time The value of . is the window function, is a complex exponential function and is the core component of the Fourier transform.

[0015] The power spectral density calculation formula is as follows: (2) is the EEG power spectral density, is the amplitude obtained after short-time Fourier transform.

[0016] Eigenvalue 2: Hilbert-Huang entropy Signal entropy is a very good measure of signal complexity. The Hilbert transform formula of the EEG signal x(t) is as follows: (3) Where t is the time variable, is the integration variable, For brain electrical signals.

[0017] In the music state, the phase locking value feature and event-related desynchronization or synchronization index feature are calculated based on the EEG signal. Different values ​​reflect the synchronization or desynchronization of neuronal discharges. In the music state, the frequency following response is extracted through time-frequency analysis, and its phase locking value with the music and the event-related desynchronization or synchronization index are calculated, which are recorded as eigenvalues ​​P1 and P2 respectively. The frequency following response is the steady-state response wave of the EEG to the periodic frequency components in the music, which is used to evaluate the synchronization and coupling characteristics of the brain to audio stimulation. The core of its analysis lies in the extraction of two types of time-frequency features: ① Phase locking value (PLV): The EEG signal and the music signal are resampled to obtain the sampled EEG signal and the sampled music signal. The time-frequency features of the EEG signal and the music signal with the same sampling rate are extracted based on the short-time Fourier transform. The phase values ​​of the time-frequency features of the two are extracted using the Hilbert transform. The difference between the phase values ​​of the two time-frequency features is calculated to obtain the phase difference value. The correlation value between the music and EEG is analyzed for the phase difference value of each frequency to obtain the phase locking value feature; Feature 1: Phase lock value Method: The short-time Fourier transform (STFT) formula is: (4) Where t is the time variable, is the integration variable, For sampling EEG signals or music signals at time h(τ−t) is the window function (such as Hamming window), Suitable for analyzing steady-state frequency components (such as the fundamental frequency of music), but the resolution is limited by the window length. = , is the time-frequency characteristics of the EEG signal or the music signal, that is, the amplitude.

[0018] The amplitude extracted by the short-time Fourier transform is converted into a complex form through the Hilbert transform. The formula is as follows: (5) t is the time variable, is the frequency; Calculated It can be expressed as: (6) Among them A and B They represent the real and imaginary parts of the transformed sampled EEG signal or sampled music signal respectively.

[0019] The calculated phase is: (7) The phases of the sampled EEG signal and the sampled music signal at the same time t and the same frequency f are respectively recorded as and , then the phase difference calculation formula is: (8) Calculate the phase locking value (PLV), which reflects the time domain phase synchronization between the EEG signal and the music stimulus. The formula is: (9) in, is the phase difference, and N is the sampling point in the selected time window.

[0020] PLV is suitable for steady-state frequency tracking (such as the fundamental frequency of music) and directly characterizes neural synchronization, but its time and frequency resolution is limited by the window length and it has low sensitivity to transient changes.

[0021] ② Event-related desynchronization or synchronization (ERD / ERS) index: After quantifying the dynamic changes in frequency band energy using the quadratic time-frequency analysis method (ZAM distribution), the event-related desynchronization / synchronization index is calculated for the frequency band of interest within the time window.

[0022] Feature 2: Event-related desynchronization or synchronization index (based on quadratic time-frequency analysis) Methods: The time-frequency energy was extracted based on the ZAM distribution, and the event-related desynchronization or synchronization index characteristics were calculated according to the time-frequency energy.

[0023] ZhaoAtlasMarks (ZAM) distribution, the formula is: (10) Where t is the time variable, f is the frequency variable, and s is the integral variable for calculating the local autocorrelation. τ is the delay variable that determines the upper and lower limits of the integral.

[0024] Points limit is the window function, X(t) is the EEG signal, represents the complex conjugate of the signal.

[0025] The ZAM distribution is used to reduce cross-term interference and enhance the time-frequency resolution of transient features (such as music dynamic changes and consonant onsets).

[0026] The ERD / ERS index is calculated based on the time-frequency energy extracted from the ZAM distribution. The formula is: (11) Among them, the time-frequency energy It is calculated by averaging the amplitude of the time-frequency distribution over frequency and time, that is: (12) in, and Represents each time window separately and each frequency band The number of sampling points within the time window is 3s, the sliding overlap ratio is 50%, and the frequency bands are beta band (13-30Hz) and gamma band (31-45Hz). is the time-frequency distribution calculated by the ZAM method. In a similar way, the numerical The calculation is as follows: (13) is the time-frequency distribution estimated from the EEG segments recorded in the reference time interval before the stimulus onset, is the number of samples in the latter interval.

[0027] It is suitable for non-stationary signal analysis and quantifies cortical activation patterns through dynamic changes in energy.

[0028] The phase-locking value captures the temporal synchronization of the brainstem cortex, and the ERD / ERS index reflects the energy modulation of the cortical frequency band. The two complement each other to construct a model of the auditory perception emotion processing pathway.

[0029] In quiet and music states, the skin electrode power spectrum density characteristics are calculated based on the skin electrode signal, the eye electrode frequency characteristics are calculated based on the eye electrode signal, the heart electrode power spectrum density characteristics are extracted based on the heart electrode signal, and the blood oxygen saturation is calculated based on the blood oxygen signal.

[0030] Skin electrical feature extraction: Short-time Fourier transform was used to calculate the power spectral density of skin electrical activity in the range of 0.03 to 0.5 Hz, which was used as the characteristic of this parameter and recorded as P3.

[0031] Blood oxygen feature extraction: The blood oxygen saturation SpO2 is calculated by detecting light of multiple wavelengths and is recorded as P4.

[0032] SpO2=110-25R; (14) R=(AC660 / DC660) / (AC940 / DC940); (15) Among them, AC660 / DC660 is the ratio of AC to DC of the received red light part; AC940 / DC940 is the ratio of AC to DC of the received infrared light part.

[0033] ECG feature extraction: Heart rate variability was extracted, and the RR interval series was converted into frequency domain signals by fast Fourier transform. The power spectral density of the LF low frequency (0.04-0.15 Hz) and the HF high frequency (0.15-0.40 Hz) was calculated. The LF / HF ratio was calculated to quantify the sympathetic-vagal balance coefficient, which was recorded as P5.

[0034] (16) P LE is the power spectral density of the low-frequency LE, P HE is the power spectral density of high-frequency HE.

[0035] Electrooculogram feature extraction; The vertical and horizontal electrooculogram signals were extracted, and then continuous wavelet transform was used to extract blinks and saccades in the eye movement data. The mean of the frequencies of the two was used as the electrooculogram features, which were recorded as P6 and P7 respectively.

[0036] Spike potential signals appear during blinking or scanning. Wavelet transform can detect spike potential signals, such as the Mexican hat wavelet transform, whose function is as follows: (17) Where t is the time variable. After performing the Mexican hat wavelet transform, the trough signal is inverted to convert all troughs into peaks. The findpeaks peak detection function is then used to detect all peaks and troughs, obtaining the electrooculographic features detected. For saccade detection, an appropriate threshold amplitude is set based on the characteristics of human eye saccades, generally between 20 and 200 ms. This needs to be adjusted based on the actual signal sampling rate. All peaks within the set threshold range are marked, and then the peak and trough values ​​are encoded, with peak values ​​encoded as "1" and trough values ​​as "0." Sequences of "01" or "10" within the time window threshold of 20 to 200 ms are considered candidate saccade segments. The slope between the start and end points of the candidate segments is then calculated, and the correlation coefficient between this slope and the feature vector is calculated. A corresponding threshold is set. If the calculated coefficient value falls within the threshold range, the candidate segment is considered the desired saccade segment. Blink detection uses the same encoding method, verifying signal polarity. If a positive peak corresponds to a blink (closed), and a negative peak corresponds to an open eye, the "010" sequence is selected as a blink candidate. A corresponding maximum segment length threshold is set, typically between 100 and 400 milliseconds. The slope and correlation coefficient of the candidate blink segments are then calculated. When this coefficient reaches a pre-collected threshold, the candidate blink segment is identified as a blink.

[0037] The mean frequencies of saccades and blinks were then calculated for each type of chronic disorder of consciousness.

[0038] S4, preprocessing the multi-dimensional features and inputting them into the neural network fusion model to obtain fusion features; In this embodiment of the present invention, seven characteristic variables are extracted: the power spectral density of EEG (P1 in a static state), the Hilbert-Huang entropy of EEG (P2 in a static state), the phase locking value of the frequency following response (P1 in a music state), the event-related desynchronization / synchronization index (P2 in a music state), the power spectral density of electrodermal activity at 0.03-0.5 Hz (P3), the HbO concentration (P4), the sympathetic-vagal balance coefficient in heart rate variability (P5), the blinking frequency (P6) and the scanning frequency (P7), among which the P1 and P2 features are selected differently according to the different modes of quiet and music.

[0039] The preprocessing of the multidimensional features specifically includes: The multidimensional features are normalized, and the corresponding two-dimensional spatial coordinate positions of the phase-locking value features, event-related desynchronization or synchronization index features, EEG power spectral density features, and EEG Hilbert Huang entropy features of each electrode after normalization are encoded to obtain single-electrode coding features, wherein the single-electrode coding features include: phase-locking value coding features, event-related desynchronization or synchronization index coding features, EEG power spectral density coding features, and EEG Hilbert Huang entropy coding features.

[0040] In this embodiment of the present invention, a minimum-maximum normalization method is used to uniformly map eigenvalues ​​to the [0, 1] interval, and the two-dimensional spatial coordinate position of each electrode is used for position encoding. In position embedding, the present invention uses the two-dimensional spatial coordinate position of each electrode because electrode location helps estimate correlations between brain regions. The position of each electrode is defined using the well-known 10-5 system.

[0041] The 10-5 system is a system published in 2001 that describes the placement of EEG electrodes. Because this system uses a proportional distance of 5% of the total length along the contour lines between skull landmarks, while the 10-20 and 10-10 systems use 20% and 10% of the distances, respectively, it is called the 5% system or the 10-5 system.

[0042] The neural network fusion model specifically includes: Multidimensional feature fusion module, used to fuse the preprocessed multidimensional features to obtain multidimensional fusion features; The multi-dimensional feature fusion module specifically includes: The single electrode encoding features are input into the convolution layer for preliminary fusion to obtain preliminary fusion features; The preliminary fusion features and the remaining normalized multidimensional features are input into the depth-separable convolution for feature fusion to obtain the deep fusion features, and the deep fusion features are input into the batch normalization and GeLU activation layer to obtain the multidimensional single electrode features; The self-attention channel spatial fusion features are input into a feedforward network layer to obtain enhanced nonlinear features, wherein the feedforward network layer includes: a fully connected layer and a Gaussian error linear unit activation function; and the enhanced nonlinear features are weighted averaged to obtain fusion features.

[0043] The channel-space feature fusion module is used to perform linear transformation on multi-dimensional single-electrode features to generate queries, keys, and values. The queries, keys, and values ​​are split into multiple heads, and the attention of multiple heads is calculated and then spliced. The attention scores are calculated to obtain channel weights, and the weights are assigned to the values. Then, the dropout operation randomly discards part of the attention weights to obtain a weighted representation of each head. The weighted sum of multiple heads is linearly transformed to obtain the transformed features. The transformed features are subjected to dropout operation, residual connection and layer normalization to obtain the self-attention channel-space fusion features. The self-attention channel spatial fusion features are input into a feedforward network layer to obtain enhanced nonlinear features, wherein the feedforward network layer includes: a fully connected layer and a Gaussian error linear unit activation function; and the enhanced nonlinear features are weighted averaged to obtain fusion features.

[0044] The channel space feature fusion module is specifically used to split the single-layer self-attention of each head into three levels of attention, where the three levels of attention include local attention, mid-range attention, and global attention, and calculate the attention of each head based on the three levels of attention.

[0045] In the embodiment of the present invention, the local attention of the three-level attention structure is to perform a 3×3 depthwise convolution (Depthwise Conv) on the Q and K of each head, forcing the attention to only calculate the relationship between adjacent electrodes.

[0046] Medium-range attention is to apply 7×7 dilated convolution to Q and K (dilated convolution is to introduce intervals in the convolution block to expand the range), and (dilation=2) to expand the receptive field to the medium-range range.

[0047] Global attention: A traditional self-attention mechanism that directly computes the interactions of all electrode pairs without convolution operations.

[0048] Finally, the three attentions are merged to get the attention of each head.

[0049] In this embodiment of the present invention, based on a traditional multi-head self-attention mechanism, the fused features are linearly transformed to generate queries (Q), keys (K), and values ​​(V), which are d-dimensional. Q represents each channel used for comparison, K represents all comparisons with channels in Q, and V represents the representation in the high-level feature space. Q, K, and V are split into h heads, and attention is calculated for each of them before being concatenated. The formula is as follows: (18) in, is the weight matrix of different heads, and the final output is obtained by dot multiplication with the spliced ​​multi-head attention matrix. The single-layer self-attention of each head is split into three levels of attention: local, medium-range, and global. The inter-electrode dependency is modeled from the local to the global level. Local attention targets a 3×3 area and only focuses on the electrodes adjacent to each channel; medium-range attention targets a 7×7 area and sequentially expands the receptive field; global attention performs fully connected interactions to obtain long-distance channel spatial dependencies across brain regions. The exponential decay of the channel (electrode) distance is introduced in the attention score calculation, and the formula is as follows: (19) in, refers to the transposed matrix of K, is the distance between lead i and lead j, λ is the distance attenuation coefficient, and d represents is d-dimensional.

[0050] For each head, the product is divided by the square root of d as data normalization to ensure the effectiveness of the Softmax function. This method obtains the weight of each channel and assigns it to V. A dropout operation is then performed to randomly discard some attention weights to prevent overfitting, obtaining a weighted representation for each head. Finally, the weighted sum of multiple heads is applied through a linear transformation to keep the output dimension consistent with the module input dimension.

[0051] Next, dropout operations, residual connections, and layer normalization are performed to help the network learn and converge adaptive weights faster, alleviating the gradient vanishing problem caused by large convolution kernels.

[0052] The self-attention channel spatial fusion features are input into a feedforward network layer to obtain enhanced nonlinear features, wherein the feedforward network layer includes: a fully connected layer and a Gaussian error linear unit activation function; and the enhanced nonlinear features are weighted averaged to obtain fusion features.

[0053] In this embodiment of the present invention, the features obtained by self-attention are input into the feed-forward network layer to enhance nonlinearity. Two fully connected layers and the Gaussian error linear unit (GeLU) activation function are used to form the FFN, and the formula is: (20) Among them, x is the input feature, (x) refers to the Sigmoid function 、 is the weight matrix and bias term of the first fully connected layer, 、 is the weight matrix and bias term of the second fully connected layer.

[0054] Then, the present invention adjusts the dropout rate to 0.1 and performs the operation again to obtain a better training effect. After repeating the above operation multiple times, the present invention performs a weighted average on the spatial fusion output to obtain a global high-level representation that considers all channels and features.

[0055] S5. Input the fused features into the pattern recognition output module to obtain the consciousness disorder assessment score of the person being tested.

[0056] The S5 specifically includes: The fused features are input into the pattern recognition output module for processing to obtain the probabilities of the subject's minimal consciousness state and vegetative state, and the consciousness level score is calculated based on the probabilities.

[0057] The fusion features are input into the pattern recognition output module for processing to obtain the probabilities of the minimum consciousness state and the vegetative state of the person being tested, and the consciousness level score is calculated based on the probabilities, specifically including: performing global average pooling on the fusion features to obtain key features, inputting the key features into the fully connected layer with the Softmax function to obtain the minimum consciousness state probability and the vegetative state probability of the consciousness level, and multiplying the minimum consciousness state probability and the vegetative state probability by 100 to obtain the consciousness disorder assessment score.

[0058] In this embodiment of the present invention, global average pooling is applied to the model output layer to compress channel dimensions while preserving key features. A fully connected layer with a softmax function maps the output values ​​to the [0, 1] interval, outputting two neurons to represent the probabilities of detecting the MCS (minimally conscious state) and VS (vegetative state) levels of consciousness. The SIOU loss function is used for optimization.

[0059] The output value is then multiplied by 100 to obtain the final level of consciousness score, which ranges from 0 to 100. The formula is as follows: ; (twenty one) in, is the Softmax function, and z is the weighted sum of the output layer.

[0060] In this embodiment of the present invention, to ensure that the model results are independent of the dataset partitioning, the present invention uses a k-fold cross-validation method. The dataset used in model training is evenly divided into k parts (in this embodiment, k=5). Training and validation are performed k times. In each validation, one part is used as the validation set, and the other k-1 parts are used as the training set. Each time, a different part is used as the validation set. After k times, the entire dataset is used in both the training and validation sets. Finally, when calculating the model accuracy, the different validation data are averaged to obtain the final model accuracy.

[0061] In this embodiment of the present invention, multiple physiological signal sensors, memory, a processor, and a computer program capable of executing the overall technical solution are used. The system assesses consciousness levels in real time in both quiet and music modes, assigning scores. Dynamic physiological signals and real-time assessment scores are stored over a 72-hour period, and daily average assessment scores are stored year-round.

[0062] Beneficial effects: 1. This invention uses personalized music stimulation (music that the patient likes) or symphonies with large melodic variations to assess the patient's condition in real time based on music therapy and indicate the effectiveness of the treatment strategy; 2. Existing technologies typically analyze the cross-band coupling and brain region connectivity of EEG in patients with impaired consciousness under musical stimulation based on frequency domain and connectivity. These assessments are intermittent and fail to provide real-time feedback on EEG signal changes in response to the music melody. This technology uses time-frequency analysis to extract EEG features. It extracts the frequency-following response of the patient's EEG within a smaller time window and calculates its phase-locking value and event correlation / decorrelation index with the music melody. This allows for dynamic calculation of music-EEG coupling and captures the dynamic changes in the patient's state of consciousness. It also integrates other physiological signals to overcome the one-sidedness of a single signal.

[0063] 3. A bimodal model is trained for both quiet and music scenarios, innovatively employing large-kernel deep separable convolution to fuse single-channel EEG features with multiple physiological signals. Hierarchical expanded attention optimizes spatial feature fusion, enabling accurate and efficient assessment and classification of impaired consciousness, covering subtle changes in a patient's level of consciousness throughout the day. The system outputs a consciousness score for impaired consciousness patients in the corresponding modality in that scenario, providing quantifiable numbers that reduce the comprehension burden for doctors and patients' families, thereby improving medical efficiency.

[0064] 4. Continuously collect multiple physiological signals (EEG, GSR, Seven features are extracted from the brain (SpO2, EOG, ECG) respectively, and the features are fused through large-kernel deep separable convolution to train a bimodal model based on hierarchical expanded attention. This invention realizes the effective dynamic assessment of the degree of consciousness disorder of patients with consciousness disorder in a bimodal environment of music and silence.

[0065] 5. In the music state, the phase locking value of the frequency following response and the music and the event-related synchronization / desynchronization index are calculated. The frequency following response is the steady-state response wave of the EEG to the periodic frequency components in the music, which is used to evaluate the synchronization and coupling characteristics of the brain to audio stimulation. In order to achieve dynamic matching with the music, two types of frequency following response features are extracted based on two time-frequency analysis methods: short-time Fourier transform and ZAM distribution. The frequency characteristics of music and EEG are extracted in a relatively short time window, and the patient's following response to the frequency characteristics of the musical melody is reflected from the perspective of phase-phase coupling and energy distribution, and the patient's state of consciousness is dynamically evaluated.

[0066] 6. A dual-modal deep learning model trained under quiet and musical stimulation based on large-kernel convolution and hierarchical dilated attention: Two different modal models are trained with different input and output data for applications in two different scenarios, music and quiet. By adopting large-kernel deep separable convolution, it can cover more local relationships between feature dimensions when applied to multimodal physiological signals; at the same time, hierarchical dilated attention is used to achieve cross-channel spatial fusion, and the receptive field is gradually expanded with the three-level attention of "local-mid-range-global". The spatial attenuation factor is introduced to focus on key brain areas and avoid global computational redundancy. The scene is judged and the evaluation model is switched through the music playback device, which is suitable for daily assessment of patients with impaired consciousness and evaluation of the effectiveness of music therapy.

[0067] 7. This method fuses multiple physiological signals to extract the frequency-following response characteristics of the melody-EEG coupling in music (phase locking value and event correlation / decorrelation index are extracted after time-frequency analysis). Based on the adaptive attention mechanism, it can assess the consciousness level of patients with consciousness disorders in both quiet and music modes, which can help to objectively feedback the treatment effect.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. These modifications or replacements of the technical solutions of the embodiments of the present invention do not cause the essence of the corresponding technical solutions to deviate from the scope of this solution.

Claims

1. A multimodal subconscious disorder degree dynamic assessment method, characterized in that: include, S1. Acquire the physiological signals of the person being measured under multimodal conditions; S2. preprocessing the physiological signal to obtain a preprocessed signal; S3, calculating the features of the physiological signal based on the preprocessed signal to obtain a multi-dimensional feature; S4. Preprocessing the multidimensional features to obtain preprocessed data; S5, fusing the preprocessed data to obtain fusion features; S6. Input the fused features into the pattern recognition output module to obtain the consciousness disorder assessment score of the person being tested.

2. The method according to claim 1, characterized in that The S1 specifically includes: real-time collection of multi-electrode EEG signals, electrocutaneous signals, electrooculographic signals, electrocardiographic signals and blood oxygen signals of the person being measured in a music state and a quiet state respectively.

3. The method according to claim 2, characterized in that The S3 specifically includes: In a quiet state, the EEG power spectrum density feature and EEG Hilbert Huang entropy feature are calculated based on the preprocessed EEG signals of each electrode; In the music state, the phase locking value feature and event-related desynchronization or synchronization index feature are calculated based on the preprocessed EEG signal of each electrode; In the quiet state and music state, the skin electrode power spectrum density characteristics are calculated based on the preprocessed skin electrode signal, the electrooculogram frequency characteristics are calculated based on the preprocessed eyeball signal, the electrocardiogram power spectrum density characteristics are obtained by extracting the electrocardiogram characteristics based on the preprocessed electrocardiogram signal, and the blood oxygen saturation characteristics are calculated based on the preprocessed blood oxygen signal.

4. The method according to claim 3, characterized in that The calculation of the phase locking value feature and the event-related desynchronization or synchronization index feature based on the EEG signal of each electrode specifically includes: Resampling the EEG signal and the music signal to obtain a sampled EEG signal and a sampled music signal, extracting the time-frequency features of the EEG signal and the music signal with the same sampling rate according to the short-time Fourier transform, extracting the phase values ​​of the time-frequency features of the two using the Hilbert transform, calculating the difference between the phase values ​​of the two time-frequency features to obtain the phase difference value, and performing correlation analysis between the music and EEG on the phase difference value of each frequency to obtain the phase locking value feature; The time-frequency energy is extracted based on the ZAM distribution, and the event-related desynchronization or synchronization index characteristics are calculated according to the time-frequency energy.

5. The method according to claim 4, characterized in that The S4 specifically includes: The multidimensional features are normalized, and the corresponding two-dimensional spatial coordinate positions of the phase-locking value features, event-related desynchronization or synchronization index features, EEG power spectral density features, and EEG Hilbert Huang entropy features of each electrode after normalization are encoded to obtain single-electrode coding features, wherein the single-electrode coding features include: phase-locking value coding features, event-related desynchronization or synchronization index coding features, EEG power spectral density coding features, and EEG Hilbert Huang entropy coding features.

6. The method according to claim 5, characterized in that The S5 specifically includes: The single electrode encoding features are input into the convolution layer for preliminary fusion to obtain preliminary fusion features; The preliminary fusion features and the remaining normalized multidimensional features are input into the depth-separable convolution for feature fusion to obtain the deep fusion features, and the deep fusion features are input into the batch normalization and GeLU activation layer to obtain the multidimensional single electrode features; Multidimensional single-electrode features are linearly transformed to generate queries, keys, and values. The queries, keys, and values ​​are split into multiple heads, and the attention of multiple heads is calculated separately and then concatenated. The attention scores are calculated to obtain channel weights, and the weights are assigned to the values. Then, a dropout operation is performed to randomly discard some of the attention weights to obtain a weighted representation of each head. The weighted sum of multiple heads is linearly transformed to obtain the transformed features. The transformed features are subjected to dropout, residual connection, and layer normalization to obtain the self-attention channel space fusion features. The self-attention channel spatial fusion features are input into a feedforward network layer to obtain enhanced nonlinear features, wherein the feedforward network layer includes: a fully connected layer and a Gaussian error linear unit activation function; and the enhanced nonlinear features are weighted averaged to obtain fusion features.

7. The method according to claim 6, characterized in that The S6 specifically includes: The fused features are input into the pattern recognition output module for processing to obtain the probabilities of the subject's minimal consciousness state and vegetative state, and the consciousness level score is calculated based on the probabilities.

8. The method according to claim 7, characterized in that The method of calculating the attention of multiple heads separately specifically includes: splitting the single-layer self-attention of each head into three levels of attention, wherein the three levels of attention include: local attention, mid-range attention and global attention, and calculating the attention of each head based on the three levels of attention.

9. The method according to claim 8, characterized in that: The fusion features are input into the pattern recognition output module for processing to obtain the probabilities of the minimum consciousness state and the vegetative state of the person being tested, and the consciousness level score is calculated based on the probabilities, specifically including: performing global average pooling on the fusion features to obtain key features, inputting the key features into the fully connected layer with the Softmax function to obtain the minimum consciousness state probability and the vegetative state probability of the consciousness level, and multiplying the minimum consciousness state probability and the vegetative state probability by 100 to obtain the consciousness disorder assessment score.

Images