Photoelectric bimodal depression crowd judgment system combining functional near infrared spectrum and electroencephalogram

Abstract

The invention belongs to the technical field of auxiliary diagnosis of depressive disorder, and particularly relates to a functional near infrared spectrum and electroencephalogram combined photoelectric bimodal depressive population judgment system. The system comprises a functional near infrared spectrum and electroencephalogram synchronous acquisition module, a signal preprocessing module, a feature extraction and fusion module and an audio experiment paradigm stimulation module. The functional near infrared spectrum and electroencephalogram synchronous acquisition module is used for acquiring a prefrontal lobe region functional near infrared spectrum signal and a prefrontal lobe region electroencephalogram signal of the crowd under the audio stimulation played by the audio experiment paradigm stimulation module; the signal preprocessing module is used for removing interference in the near infrared spectrum signal and the electroencephalogram signal; the feature extraction and fusion module is used for performing feature extraction and fusion on the near infrared spectrum and electroencephalogram data subjected to artifact noise removal, and performing depression population judgment on fusion features by adopting a machine learning classifier; and the audio experiment paradigm stimulation module is used for playing audio clips containing different stimulation attributes.

Description

Technical Field

[0001] The present invention belongs to the technical field of auxiliary diagnosis of depression disorders, and specifically relates to a photoelectric dual-modality depression population determination system combining functional near-infrared spectroscopy and electroencephalography. Background Art

[0002] The clinical diagnosis of depressive disorders mainly relies on scale screening and the subjective judgment of doctors. There is no objective biological diagnostic indicator, and there are problems such as low diagnostic efficiency and lack of unified objective evaluation criteria. According to statistics, the accuracy rate of general practitioners in correctly diagnosing depression is only 47.3% (Lancet, 2009), which shows that the lack of objective and universal diagnostic tools makes it difficult for patients with depression to be diagnosed and receive intervention treatment in a timely manner.

[0003] Therefore, how to overcome the problems of low efficiency, strong subjectivity, and low clinical recognition rate (47.3%, Lancet, 2009) of traditional clinical scales and doctor interviews, break the traditional subjective "symptom description" mental disorder diagnosis and treatment framework, and achieve objective "data-driven" accurate identification of depressive disorders and achieve the effect of "preventive treatment" is of great significance.

[0004] Relevant studies have shown that the pathogenesis of depressive disorders is closely related to the caudate nucleus of the brain. When patients with depressive disorders have mood swings, the prefrontal cortex connected to the caudate nucleus shows abnormalities, causing a decrease in the level of certain excitatory neurotransmitters in the prefrontal lobe and a reduction in neuronal synapses. Wearable brain imaging technology can specify abnormalities in brain areas, which is more scientific and quick, and thus assists doctors in diagnosing depressive disorders. Common brain imaging technologies include electroencephalogram (EEG), magnetoencephalogram (MEG), functional near-infrared spectroscopy (fNIRS), positron emission tomography (PET) and functional magnetic resonance imaging (fMRI). EEG has the advantages of high temporal resolution and rapid response to stimuli, so it is widely popular in medical auxiliary diagnosis, but EEG is easily affected by various artifacts and environmental noise such as movement, electrooculography, and electricity. Functional near-infrared spectroscopy has strong robustness to artifacts and noise, and has high spatial resolution. By collecting changes in the patient's cerebral blood oxygen, it can show changes in the spatial state of the brain, but the temporal resolution of functional near-infrared spectroscopy is poor. Therefore, the combination of functional near-infrared spectroscopy and EEG can construct a dual-modal physiological indicator system, which takes advantage of the high temporal resolution of EEG and the high spatial resolution of functional near-infrared spectroscopy, and is expected to make medical auxiliary diagnosis results more reliable and accurate.

[0005] The basis of the combination of functional near-infrared spectroscopy and EEG signals is the phenomenon of neurovascular coupling. Neural activity is accompanied by fluctuations in cerebral blood flow, so the two complement each other. EEG has been regarded as the most active research tool used in brain-computer interfaces due to its advantage of high temporal resolution. However, low spatial resolution limits the spatial information of brain activity recorded by EEG and cannot accurately locate the relevant cortical sources. Functional near-infrared spectroscopy uses near-infrared light to measure concentration changes of oxyhemoglobin concentration and deoxyhemoglobin concentration, which is not affected by artifacts and noise. However, the main limitation of functional near-infrared spectroscopy is the long response lag, because the hemodynamic response takes time to reach the maximum amplitude, which means that the measurement cannot be used for real-time applications, so combining the complementary characteristics of high temporal resolution of EEG and high spatial resolution of functional near-infrared spectroscopy helps to make up for the shortcomings of both.

[0006] The frontal lobe of the brain is closely related to human emotions and feelings. The functional near-infrared spectrum and EEG signals of the frontal lobe can be used to effectively assist in the identification of patients with depression and healthy people. Since there is no hair blocking the forehead of the brain, it is conducive to the acquisition of high-quality (high signal-to-noise ratio) functional near-infrared spectrum and EEG signals. In addition, external audio and video stimulation will have different effects on the frontal lobe of the human brain. The generation of positive emotions, negative emotions and neutral emotions is closely related to the left and right frontal lobe brain areas. Compared with healthy people, patients with depression have a large difference in their performance in response to positive and negative emotional stimulation. The effect of positive emotional stimulation is weaker, and the effect of negative emotional stimulation is stronger.

[0007] Traditional clinical scales and doctor interviews are inefficient, highly subjective, and have low clinical recognition rates (47.3%, Lancet, 2009). In order to break the traditional subjective "symptom description" mental disorder diagnosis and treatment framework, realize the objective "data-driven" accurate judgment of depressed people, and reduce the limitations of functional near-infrared spectroscopy and EEG, the present invention provides a photoelectric dual-modal depression population judgment system combining functional near-infrared spectroscopy and EEG, extracts multidimensional feature vectors strongly related to the depressive state through 24-lead functional near-infrared spectroscopy in the prefrontal region and 5-lead EEG dual-modal signals in the prefrontal region (AF7, Fp1, Fpz, Fp2, AF8), constructs an objective physiological indicator system for judging depression population by dual-modal functional near-infrared spectroscopy and EEG, gives full play to the advantages of high temporal resolution of EEG and high spatial resolution of functional near-infrared spectroscopy, uses the complementary information of the two modalities to overcome the limitations of individual modalities, and improves the reliability and robustness of the judgment results of depressed people through feature layer fusion. Summary of the invention

[0008] In view of this, the present invention provides a photoelectric dual-modality depression population determination system combining functional near-infrared spectroscopy and electroencephalography. The system can utilize near-infrared spectroscopy and electroencephalography signals to complement each other and accurately and objectively identify and determine depressed people.

[0009] The technical solution for implementing the present invention is as follows:

[0010] A photoelectric dual-modal depression population determination system combining functional near-infrared spectroscopy and electroencephalogram, comprising: a functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module, a signal preprocessing module, a feature extraction and fusion module, and an audio experimental paradigm stimulation module;

[0011] The functional near-infrared spectroscopy and EEG synchronous acquisition module is used to collect functional near-infrared spectroscopy signals and EEG signals of the prefrontal region of the crowd under the audio stimulation played by the audio experimental paradigm stimulation module;

[0012] The signal preprocessing module is used to remove interference from near-infrared spectrum signals and EEG signals;

[0013] The feature extraction and fusion module is used to extract and fuse features from the near-infrared spectrum and EEG data after artifact noise removal, and use a machine learning classifier to determine the depressed population based on the fused features; wherein the feature extraction and fusion module uses a heterogeneous modality decoupling interactive fusion model and a fusion model based on spatiotemporal attention guidance to perform feature fusion;

[0014] The audio experiment paradigm stimulation module is used to play audio clips containing different stimulation attributes.

[0015] Furthermore, the heterogeneous modality decoupling interaction fusion model of the present invention includes a near infrared decoupling feature extraction network, an electroencephalogram decoupling feature extraction network and a multi-layer cross attention network, wherein the near infrared decoupling feature extraction network and the electroencephalogram decoupling feature extraction network have the same structure; wherein,

[0016] The near-infrared decoupling feature extraction network and the electroencephalogram decoupling feature extraction network are used to realize the alignment of the feature realization modal time dimension and obtain the modal low-level features; decouple the modal low-level features into common features and specific features, extract the common features and output them;

[0017] A multi-layer cross-attention network is used to alternately use the common features and specific features of two different modalities as query Q, key K and value V. In the calculation of each head, the features from one modality are converted into a set of key-value pairs, and the other modality is used as a query to achieve interaction between the two modalities, outputting additional complementary information provided by NIR features to EEG features and vice versa.

[0018] The output of the heterogeneous modality decoupling interactive fusion model is: the superposition of the multi-layer cross attention network output, the near-infrared decoupling feature extraction network output and the EEG decoupling feature extraction network output.

[0019] Furthermore, the present invention uses the common features and specific features of two different modalities alternately as the query Q, key K and value V, specifically:

[0020]

[0021]

[0022] in, and They represent the decoupled near-infrared specific features and EEG near-infrared specific features respectively; W is the weight matrix;

[0023] The additional supplementary information provided by the output near-infrared feature to the EEG feature and the additional supplementary information provided by the EEG feature to the near-infrared feature are:

[0024]

[0025]

[0026] in, It indicates that the near-infrared features provide additional supplementary information to the EEG features. It represents the additional supplementary information provided by EEG features to near-infrared features, and d represents the dimension of Q and K.

[0027] Furthermore, the present invention gradually changes heterogeneous queries into homogeneous queries, that is, first using specific features of one modality as queries and specific features of another modality as keys and values. After initial cross-attention, simple features with higher similarity are captured and then used as new queries to find more complex complementary features. The process can be expressed as:

[0028]

[0029]

[0030] in, and Represents a homogeneous query.

[0031] Furthermore, when training the heterogeneous modality decoupling interaction fusion model of the present invention, the decoupling network task loss is defined as:

[0032]

[0033] in, Represents the cosine similarity of common features and specific features, Represents the difference between the decoupled features and the original coupled features;

[0034] During training, the relational knowledge distillation method is also used to learn the relationship between features from the two dimensions of distance and angle.

[0035] Furthermore, the fusion model based on spatiotemporal attention guidance of the present invention includes:

[0036] Align the collected EEG signals and functional near-infrared spectroscopy signals

[0037] For input features Perform global average pooling and global maximum pooling in the channel dimension to obtain two two-dimensional feature maps. and

[0038] These two-dimensional feature maps and Concatenate them together and then pass them through a convolutional layer to get the spatial attention weight M;

[0039] Multiply the weight M by the input feature to get the adjusted feature

[0040] Furthermore, the functional near-infrared spectrum and EEG synchronous acquisition module of the present invention is mainly composed of four parts: a sensor mainboard, a flexible Ag / AgCl EEG acquisition electrode, a functional near-infrared spectrum light source and detector, and a silicone headband; wherein,

[0041] Flexible Ag / AgCl EEG acquisition electrodes are used to collect EEG signals from 5 leads at the frontal lobe AF7, Fp1, Fpz, Fp2, and AF8. Two additional Ag / AgCl EEG acquisition electrodes are used as reference electrodes and feedback electrodes to collect signals from the mastoid processes behind the left and right ears.

[0042] The functional near-infrared spectroscopy light source and detector include 6 emission light sources and 14 detectors, which are used to collect 24-lead functional near-infrared spectroscopy signals in the prefrontal area.

[0043] Furthermore, the audio stimulation paradigm in the audio experimental paradigm stimulation module of the present invention contains 4 groups of stimulation paradigms, each group contains 4 types of audio, which are played in Latin square order, and the playing time of each audio segment is set, and a rest time is set between audio playbacks.

[0044] Furthermore, the interference described in the present invention includes baseline drift, power frequency interference, electrooculogram artifacts, heart rate signal interference, respiratory signal interference and blood pressure signal interference; the signal preprocessing module uses a 500-order finite impulse response filter with a passband of 0.1 to 45 Hz to remove baseline drift and power frequency interference, uses a model combining discrete wavelet transform and Kalman filtering to remove electrooculogram artifacts, and uses a 500-order finite impulse response filter with a passband of 0.01 to 0.1 Hz to remove interference in heart rate, respiratory and blood pressure signals.

[0045] Furthermore, the system of the present invention also includes a PC host computer display system module, which is used to call the audio experiment paradigm stimulation module to realize the audio experiment paradigm playback, and to display the functional near-infrared spectrum and electroencephalogram signals collected under the audio experiment paradigm stimulation in real time; the signal preprocessing module, feature extraction and fusion module and audio experiment paradigm stimulation module are integrated in the PC host computer display system module.

[0046] Beneficial effects:

[0047] First, the present invention focuses on solving a new method for objectively identifying depressive disorders. It obtains brain activity information through prefrontal functional near-infrared spectroscopy and electroencephalogram signals, and realizes objective identification and judgment of depressed people through feature extraction, feature fusion and artificial intelligence technology.

[0048] Second, the present invention uses near infrared spectroscopy and EEG bimodal signals as objective indicators for judging depressed people, constructs a functional near infrared spectroscopy and EEG bimodal physiological indicator system, combines the advantages of high temporal resolution and high spatial resolution of the two, and uses the complementary information of the two modalities to overcome the limitations of individual modalities. And through feature layer fusion, the reliability and robustness of the judgment results of the artificial intelligence classifier are improved.

[0049] Third, it helps to improve the problems of low efficiency, strong subjectivity and low clinical recognition rate of traditional clinical scales and doctor interviews, and provides doctors with an objective and effective auxiliary diagnostic method for judging depressive disorders. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A schematic block diagram of the principle of a photoelectric dual-modality depression crowd determination system combining functional near-infrared spectroscopy and electroencephalography provided in an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of the positions of the functional near-infrared spectroscopy emitter, detector, and EEG acquisition electrodes of a photoelectric dual-modality depression population determination system combining functional near-infrared spectroscopy and EEG, and its overall rendering, provided in an embodiment of the present invention;

[0053] Figure 3 A functional near infrared spectroscopy and EEG signal acquisition sensor mainboard principle block diagram of a photoelectric dual-modality depression crowd determination system combining functional near infrared spectroscopy and EEG provided in an embodiment of the present invention;

[0054] Figure 4 An optimal selection strategy for heterogeneous sensors in a photoelectric dual-modality depression population determination system combining functional near-infrared spectroscopy and electroencephalography provided in an embodiment of the present invention;

[0055] Figure 5 A data transmission strategy based on an event trigger mechanism for a photoelectric dual-modality depression population determination system combining functional near-infrared spectroscopy and electroencephalography provided in an embodiment of the present invention;

[0056] Figure 6 A block diagram of an audio stimulation test paradigm of an optoelectronic dual-modality depression population determination system combining functional near-infrared spectroscopy and electroencephalography provided in an embodiment of the present invention;

[0057] Figure 7 An overall flow chart of data collection and algorithm analysis of a photoelectric dual-modality depression population determination system combining functional near-infrared spectroscopy and electroencephalography provided in an embodiment of the present invention;

[0058] Figure 8 The embodiment of the present invention adopts the feature layer fusion process of the heterogeneous modality decoupling interactive fusion model;

[0059] Fig. 9 The feature layer fusion process of the fusion model based on spatiotemporal attention guidance in an embodiment of the present invention;

[0060] Among them: power management module-1, functional near-infrared spectroscopy and EEG synchronous acquisition module-2, Bluetooth module-3, signal preprocessing module-4, feature extraction and fusion module-5, PC and host computer display system module-6, audio experimental paradigm stimulation module-7. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] The embodiment of the present application is a photoelectric dual-modal depression population determination system combining functional near-infrared spectroscopy and electroencephalography, comprising: a functional near-infrared spectroscopy and electroencephalography synchronous acquisition module, a signal preprocessing module, a feature extraction and fusion module, and an audio experimental paradigm stimulation module;

[0064] The functional near-infrared spectrum and EEG synchronous acquisition module is used to collect functional near-infrared spectrum signals and EEG signals of the prefrontal region of the crowd under the audio stimulation played by the audio experimental paradigm stimulation module; the signal preprocessing module is used to remove interference in the near-infrared spectrum signals and EEG signals; the feature extraction and fusion module is used to extract and fuse features of the near-infrared spectrum and EEG data with artifact noise removed, and use a machine learning classifier to determine the depressed population based on the fused features; wherein the feature extraction and fusion module uses a heterogeneous modal decoupling interactive fusion model and a fusion model based on spatiotemporal attention guidance to perform feature fusion; the audio experimental paradigm stimulation module is used to play audio clips containing different stimulation attributes.

[0065] like Figure 1 As shown, another embodiment of the present invention provides a photoelectric dual-modality depression population determination system combining functional near-infrared spectroscopy and electroencephalography, comprising:

[0066] Power management module 1, functional near-infrared spectroscopy and EEG synchronous acquisition module 2, Bluetooth module 3, signal preprocessing module 4, feature extraction and fusion module 5, PC and host computer display system module 6 and audio experimental paradigm stimulation module 7.

[0067] The power management module 1 is used to power the functional near-infrared spectroscopy and EEG synchronous acquisition module 2 and the Bluetooth module 3, providing high-quality power for efficient and stable operation;

[0068] The functional near-infrared spectroscopy and EEG synchronous acquisition module 2 is used to synchronously acquire the patient's high signal-to-noise ratio prefrontal area 24-lead functional near-infrared spectroscopy signals and prefrontal area 5-lead (AF7, Fp1, Fpz, Fp2, AF8) EEG signals under the audio stimulation played by the audio experimental paradigm stimulation module 7.

[0069] The Bluetooth module 3 is used to transmit the high signal-to-noise ratio prefrontal area 24-lead functional near-infrared spectroscopy signal and the prefrontal area 5-lead (AF7, Fp1, Fpz, Fp2, AF8) EEG signal collected by the functional near-infrared spectroscopy and EEG synchronous acquisition module 2 to the PC host computer display system module 6;

[0070] Signal preprocessing module 4 is used to remove artifact noises such as body movement, eye movement, and electric power in functional near-infrared spectroscopy and EEG signals, and improve the signal-to-noise ratio of the signal;

[0071] Feature extraction and fusion module 5 is used to extract features from the collected functional near infrared spectrum and EEG data, then perform feature layer fusion on the extracted functional near infrared spectrum and EEG features, and finally use a machine learning classifier to determine the depressed population;

[0072] The PC host computer display system module 6 is used to call the audio experiment paradigm stimulation module to realize the audio experiment paradigm playback, and to display the functional near infrared spectrum and EEG signals collected under the audio experiment paradigm stimulation in real time. In addition, the PC host computer display system module performs functional near infrared spectrum and EEG bimodal data preprocessing, feature extraction, fusion and model classification through its integrated signal preprocessing module, feature extraction and fusion module, and finally outputs the judgment result;

[0073] The audio experiment paradigm stimulation module 7 is used to play audio clips containing different stimulation attributes.

[0074] Embodiments of the present application: First, a 24-lead functional near-infrared spectroscopy signal of the prefrontal area with a high signal-to-noise ratio and a 5-lead (AF7, Fp1, Fpz, Fp2, AF8) electroencephalogram (EEG) signal of the prefrontal area under audio stimulation are synchronously collected. Secondly, a depression recognition technology with dual-modal fusion of functional near-infrared spectroscopy and electroencephalogram (EEG) signals is proposed. This technology improves the reliability and robustness of the determination of depressed people by fusing the functional near-infrared spectroscopy and the multi-dimensional features of EEG signals. Finally, based on the proposed sensors, algorithms and models, an auxiliary diagnosis and treatment system suitable for the determination of depressed people is developed. The present invention focuses on solving a new method for the objective identification of depressive disorders, obtains brain activity information through prefrontal functional near-infrared spectroscopy and EEG signals, and realizes objective identification and determination of depressed people through feature extraction, feature fusion and artificial intelligence technology.

[0075] The embodiment of the present application uses the functional near infrared spectroscopy and EEG synchronous acquisition module 2 to collect the high signal-to-noise ratio prefrontal area 24-lead functional near infrared spectroscopy signals and prefrontal area 5-lead (AF7, Fp1, Fpz, Fp2, AF8) EEG signals under audio stimulation played by the audio experimental paradigm stimulation module 7. The functional near infrared spectroscopy signals and EEG signals were collected for 628 seconds, with the sampling rate of the functional near infrared spectroscopy signals being 35.7SPS and the EEG sampling rate being 250SPS.

[0076] In another embodiment of the present application, the functional near-infrared spectrum and EEG synchronous acquisition module 2 is composed of four parts: a functional near-infrared spectrum and EEG signal acquisition sensor mainboard, a flexible Ag / AgCl EEG acquisition electrode, a functional near-infrared spectrum light source and detector, and a medical-grade silicone headband that can fix the Ag / AgCl EEG acquisition electrode and the functional near-infrared spectrum light source and detector;

[0077] In the above embodiment, the flexible Ag / AgCl EEG acquisition electrodes are located in the frontal lobe (AF7, Fp1, Fpz, Fp2, AF8) for collecting three-lead EEG signals; two additional Ag / AgCl EEG acquisition electrodes (A1, A2) are placed at the left and right mastoids behind the ears, respectively, and used as reference electrodes and feedback electrodes; the 6 emitting light sources and 14 detectors of the functional near-infrared spectroscopy acquisition electrode are placed in the frontal brain area; in the hardware design of the functional near-infrared spectroscopy and EEG synchronous acquisition module 2, an optimal selection strategy for heterogeneous sensors and a data transmission strategy based on an event trigger mechanism are proposed.

[0078] In the above embodiment, the optimal selection strategy for heterogeneous sensors: In order to better guide the reasonable layout of EEG electrodes, near-infrared light sources, and detectors, an optimal combination strategy of light sources, detectors, and EEG electrodes is proposed. According to the international 10-20 system, EEG electrodes are deployed in the forehead area, and filtering, wrapping, and embedded methods are used for feature selection to guide the layout of heterogeneous sensors. Figure 4 As shown in the figure, when designing the sensor layout, each channel is regarded as a feature, and the purpose of feature selection is to select the most representative part from a large number of features. After data collection and necessary preprocessing, a filtering method is used to preliminarily screen each channel to exclude obviously irrelevant or redundant channels; based on the preliminarily screened channels, a wrapper method is used to train the model on each feature subset to evaluate its recognition performance and find a better combination of channels; an embedded method is used to further refine the selection and determine which channels contribute more to the model recognition performance. Finally, the optimal layout of EEG and functional near-infrared spectroscopy sensors is determined based on the identified key channels.

[0079] In the above embodiment, the data transmission strategy based on the event trigger mechanism: the uncertain interference caused by the environment, personnel, etc. during the collection process may cause high noise in the data within a certain period of time, which will bring difficulties to the subsequent modeling. Therefore, it is proposed to build a data transmission scheme based on the event trigger mechanism, which performs normal data transmission when the predefined event trigger conditions are met, and shuts down the transmission when they are not met, thereby reducing unnecessary noise data transmission and saving bandwidth and storage resources. Figure 5As shown in the figure, the event-triggered transmission process of a single channel is as follows: to address noise interference, a noise threshold is set in advance, and the signal is checked in each sampling cycle to see if it exceeds the threshold. When the signal of a channel exceeds the threshold continuously, it is considered that it has generated high noise. At this time, the data transmission of the channel will be closed. When the noise is continuously lower than the threshold, the transmission will be reopened.

[0080] In another embodiment of the present application, the audio stimulation paradigm in the audio experimental paradigm stimulation module 7 contains 4 groups of stimulation paradigms, each group contains 4 types of audio (happy, calm, fear, white noise), which are played in Latin square order, each audio segment is 18 seconds long, and there is a 20-second rest time between audio playbacks. The total duration of the audio stimulation experimental paradigm is 628 seconds.

[0081] In another embodiment of the present application, the Bluetooth module 3 is used to transmit the high signal-to-noise ratio prefrontal area 24-lead functional near infrared spectroscopy signal and the prefrontal area 5-lead (AF7, Fp1, Fpz, Fp2, AF8) EEG signal collected by the functional near infrared spectroscopy and EEG synchronous acquisition module 2 and played by the audio experimental paradigm stimulation module 7 to the PC host computer display system module 6. The Bluetooth module 3 uses the WH-BLE105 low-power Bluetooth module to transmit the functional near infrared spectroscopy signal and EEG signal collected by the functional near infrared spectroscopy and EEG synchronous acquisition module 2 to the PC host computer display system module 6.

[0082] In another embodiment of the present application, the signal preprocessing module 4 is used to eliminate the baseline drift and artifact interference in the original functional near-infrared spectroscopy signal and the EEG signal. The signal preprocessing module 4 uses a 500-order finite impulse response (FIR) filter with a passband of 0.1 to 45 Hz to remove baseline drift and power frequency interference. At the same time, since the EEG acquisition points (AF7, Fp1, Fpz, Fp2, AF8) are close to the eyes, electrooculographic artifacts will inevitably be collected. Therefore, the signal preprocessing module 4 uses a model combining discrete wavelet transform and Kalman filtering to remove electrooculographic artifacts. The original functional near-infrared spectroscopy signal is mainly interfered by the heart rate signal in the 1-2 Hz frequency band, the respiratory signal in the 0.15-0.6 Hz frequency band, and the blood pressure signal of about 0.1 Hz, because the functional near-infrared spectroscopy signal is mainly located in the 0.02-0.06 Hz frequency band. Therefore, the signal preprocessing module is a 500th order finite impulse response (FIR) filter with a frequency of 0.01 to 0.1 Hz to remove interference from the heart rate, respiration and blood pressure signals.

[0083] In another embodiment of the present application, the feature extraction and fusion module 5 processes and extracts features from the functional near-infrared spectroscopy signal and the EEG signal collected under the 628-second audio stimulation. The feature extraction and fusion module 5 mainly includes two parts: a heterogeneous modality decoupling interaction fusion strategy and a fusion strategy based on spatiotemporal attention guidance.

[0084] In the above embodiment, the heterogeneous modality decoupling interactive fusion strategy is: EEG signals reflect the changes in electrical signals of brain neuron activity, while near-infrared spectral signals reflect the changes in brain hemodynamics. The inherent heterogeneity of the two signals poses a challenge to fusion recognition performance. In order to obtain higher quality fusion features and more abundant complementary knowledge between modalities, an interactive fusion model based on feature decoupling is established. Its overall framework is as follows: Figure 8 As shown, it includes a near-infrared decoupling feature extraction network, an EEG decoupling feature extraction network and a cross attention network, wherein the near-infrared decoupling feature extraction network and the EEG decoupling feature extraction network have the same structure, including a shallow encoding network, a decoupling network, a feature extraction network and a coupled encoding network; specifically:

[0085] The working process of the near infrared decoupling feature extraction network and the EEG decoupling feature extraction network is: the modal time dimension is aligned through the shallow encoding network to obtain the modal low-level feature X m Considering the inherent heterogeneity of physiological signals and the inherent commonality when targeting the same emotional state, each modality representation X is decoupled by sharing and specific feature encoders in the network. m Decoupling as a common feature and specific features Among the common features Represents the consistent information contained in the two modal data, specific features Represents the unique expression of each modality, m∈{fNIRS,EEG}; the feature extraction network extracts the common features after decoupling.

[0086] At the same time, in order to avoid the problem of decoupled features ambiguity, an autoregressive approach is used for supervision, and the decoupled features of each modality are connected and passed through a dedicated encoder D m Generate Coupling Features Then, with the original coupling feature (modal low-level feature X m ) for comparison, it can be formulated as:

[0087]

[0088] In addition, in order to maintain common characteristics and specific features Difference, using cosine similarity to monitor the difference between two features, which can be expressed as:

[0089]

[0090] Then, the decoupled network task loss is defined as:

[0091]

[0092] The working process of the criss-cross attention network is as follows: decoupled modality-specific features When facing the same emotional state, there is complementary knowledge, and effective cross-modal knowledge transfer helps improve model performance. It is proposed to extract complementary knowledge between different modalities through a progressive cross-attention network, and alternately use the common features and specific features of two different modalities, near-infrared fNIRS (f) and EEG (E), as queries (Q), keys (K), and values ​​(V), which can be expressed as:

[0093]

[0094]

[0095] Where W is the weight matrix. Subsequently, each modality is passed through a multi-head attention module to achieve the interaction of information from different modalities. In the calculation of each head, the features from one modality are converted into a set of key-value pairs, and the other modality is used as a query to achieve the interaction between the two modalities. Multi-head attention uses the dot product to calculate the similarity between the query and the key to obtain the attention weight distribution vector.

[0096] The output of the i-th attention head can be expressed as follows:

[0097]

[0098]

[0099] represents the additional complementary information that fNIRS provides to EEG. represents the additional complementary information provided by EEG to fNIRS, and d represents the dimensions of Q and K.

[0100] It is proposed to gradually change heterogeneous queries into homogeneous queries, that is, first use specific features of one modality as queries, and specific features of another modality as keys and values. After the initial cross-attention, simple features with high similarity are captured, which are then used as new queries to find more complex complementary features. This process can be expressed as:

[0101]

[0102] in, and Represents a homogeneous query.

[0103] The output of the interactive fusion model based on feature decoupling is the superposition of the common features of EEG data, the common features of infrared data, and the output of the attention heads.

[0104] Furthermore, considering that the intra-modal feature relationship may be lost due to the decoupling operation, the relational knowledge distillation method is proposed to learn the relationship between features from the two dimensions of distance and angle. Relational knowledge learning consists of two branches: internal aggregation (sa) and external distillation (ke). The internal aggregation concatenates the common features and specific features, performs a channel rearrangement operation, and then uses the self-attention mechanism to achieve internal attention; the external distillation obtains the undecoupled features from the original feature map; the distillation operation is performed from the two dimensions of distance and angle to reduce the gap between the two branch features, thereby supplementing the relational knowledge. Concatenate common features and specific features For the features after splicing and the original undecoupled features Calculate the Euclidean distance between two sets of features, which can be expressed as:

[0105]

[0106] Among them, d euc is the Euclidean distance, v i and v j is a sample in the branch feature, μ is the distance normalization factor, defined as the average Euclidean distance of all sample pairs in a batch, ensuring that there are no outliers and reflecting the average structural characteristics of the entire batch:

[0107]

[0108] Where N is the number of mini-batches.

[0109] hope Able to learn fully The relationship between the feature structures contained in d in euc (sa i ,sa j )close enough Medium euc (ke i ,ke j ), which can be expressed as:

[0110]

[0111] in, represents the Huber loss, d euc (sa i ,sa j ) represents the splicing feature The Euclidean distance between two samples in euc (ke i ,ke j ) represents the original feature that is not decoupled The Euclidean distance between two samples in .

[0112] On the other hand, changes in angle can reveal complex dependencies between features that may not be fully captured by distance measurement alone. The angle calculation is defined as the cosine similarity between three sample points:

[0113] d ang (v i ,v j ,v k )=cos∠v i v j v k = <e ij ,e kj >

[0114] In the formula, e ij is the branch feature v i to v j The unit vector, e kj is the branch feature v k to v j In order to minimize the difference in sample angles, the angle loss is defined as follows:

[0115]

[0116] In the formula, ∠sa i s j s k and ∠ke i ke j ke k Represent the cosine similarity between the three sample points in the inner and outer branches respectively.

[0117] Angular distillation loss can help the decoupled features learn similar angular structures as uncoupled features in the feature space, so as to better understand the relative relationship between features. In addition, cross entropy loss is introduced for non-decoupled features. To avoid ambiguous knowledge transfer.

[0118] Finally, combined with the above constraints, the learning loss of relational knowledge is formed:

[0119]

[0120] In the decoupled interaction model, the common, specific and complementary features of different modalities are fully considered, and the relational knowledge is supplemented for the final fusion feature, which can effectively realize the cross-modal interactive fusion recognition of depressive states. The final constraint is formed by combining decoupling, relational learning and emotion classification loss:

[0121]

[0122] in, is the loss of emotion classification, which consists of multi-classification cross entropy, and λ1 and λ2 are weight factors.

[0123] In the above embodiment, the fusion strategy based on spatiotemporal attention guidance is:

[0124] EEG can capture the brain's electrical activity at the millisecond level and reflect the brain's dynamic changes in real time; functional near-infrared spectroscopy can measure the level of changes in blood oxygen in the cerebral cortex, which has a high spatial resolution. In order to make full use of the advantages of the two modalities in time and space, the data of the two are effectively integrated. It is planned to establish a depression detection model based on the spatiotemporal attention mechanism to achieve a more comprehensive analysis of brain activity. The attention mechanism is a technology widely used in deep learning. Its main purpose is to enable the model to selectively focus on different parts of the input data, so as to process information more effectively. In the temporal dimension, the attention mechanism can focus on the changes in the electrical activity of EEG data within a specific time period, thereby giving these key time periods a higher weight and highlighting their importance. In the spatial dimension, the attention mechanism can use the high spatial resolution information of the changes in blood oxygen in the cerebral cortex provided by the functional near-infrared spectroscopy to focus on specific brain areas. In addition, the near-infrared information can be used to guide the attention distribution of EEG signals in the spatial dimension, so that the EEG signals are more focused on important brain areas, thereby improving the ability to capture brain activity patterns.

[0125] The overall structure of the depression recognition model based on spatiotemporal attention mechanism is as follows Fig. 9 shown.

[0126] First, align the collected EEG signals and functional near-infrared spectroscopy signals:

[0127]

[0128] in, is the modal feature after alignment, is the alignment operation, X m Represents the original modality features, m∈{fNIRS,EEG}. Through the alignment operation, the EEG signal and the functional near-infrared spectroscopy signal can be synchronized at the time point, ensuring that the corresponding events or states captured by the two modalities can be accurately compared and analyzed. This is crucial for studying the relationship between brain electrical activity and hemodynamic changes, and it can help the model capture neurovascular coupling, that is, the association between neural activity and blood flow changes.

[0129] Secondly, the temporal attention weight W(t) is applied to EEG signals and functional near-infrared spectroscopy to adaptively learn which time points are more important and assign higher weights to them accordingly.

[0130] Finally, in view of the high spatial resolution of near-infrared spectroscopy, spatial attention is applied to guide the spatial attention distribution of EEG signals, and then weight information is applied to EEG signals in the spatial dimension to enhance the spatial features related to EEG and tasks. Specifically, spatial attention has an impact on the input features. Global average pooling and global maximum pooling are performed on the channel dimension to obtain two two-dimensional feature maps:

[0131]

[0132]

[0133] Where C is the number of feature map channels. These two 2D feature maps are concatenated and then passed through a convolutional layer to obtain the spatial attention weights:

[0134]

[0135] Among them, σ is the Sigmoid function and f is the convolution operation.

[0136] Finally, the weights are multiplied by the input features to obtain the adjusted features:

[0137]

[0138] In another embodiment of the present application, the PC host computer display system module 6 is used to call the audio experiment paradigm stimulation module 7 to play the audio experiment paradigm, and can display the functional near infrared spectrum and EEG collected by the functional near infrared spectrum and EEG synchronization acquisition module 2 in real time. At the same time, the PC host computer display system module 6 performs functional near infrared spectrum and EEG data preprocessing, feature extraction, fusion and classification through its integrated signal preprocessing module 4 and feature extraction and fusion module 5, and finally outputs the judgment result.

[0139] The photoelectric dual-modal depression population determination system combining functional near-infrared spectroscopy and EEG in the embodiment of the present application obtains brain activity information through 24-lead functional near-infrared spectroscopy of the frontal lobe and 5-lead EEG signals of the frontal lobe, and realizes objective identification and determination of depression population through feature extraction, feature fusion and artificial intelligence technology, which can effectively reduce the cost and subjectivity of depression disorder identification, improve the timeliness and accuracy of depression disorder identification, and effectively promote the intervention and treatment of depression disorder population. According to experimental data, the efficiency of depression population determination by this method is more than 90%.

[0140] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A photoelectric dual-modality depression crowd determination system combining functional near-infrared spectroscopy and electroencephalography, characterized in that: include: Functional near-infrared spectroscopy and EEG synchronous acquisition module, signal preprocessing module, feature extraction and fusion module, and audio experimental paradigm stimulation module; The functional near-infrared spectroscopy and EEG synchronous acquisition module is used to collect functional near-infrared spectroscopy signals and EEG signals of the prefrontal region of the crowd under the audio stimulation played by the audio experimental paradigm stimulation module; The signal preprocessing module is used to remove interference from near-infrared spectrum signals and EEG signals; The feature extraction and fusion module is used to extract and fuse features from the near-infrared spectrum and EEG data after artifact noise removal, and use a machine learning classifier to determine the depressed population based on the fused features; wherein the feature extraction and fusion module uses a heterogeneous modality decoupling interactive fusion model and a fusion model based on spatiotemporal attention guidance to perform feature fusion; The audio experiment paradigm stimulation module is used to play audio clips containing different stimulation attributes.

2. The photoelectric dual-modality depression crowd determination system combining functional near infrared spectroscopy and electroencephalography according to claim 1 is characterized in that: The heterogeneous modality decoupling interaction fusion model includes a near infrared decoupling feature extraction network, an electroencephalogram decoupling feature extraction network and a multi-layer cross attention network, wherein the near infrared decoupling feature extraction network and the electroencephalogram decoupling feature extraction network have the same structure; wherein, The near-infrared decoupling feature extraction network and the electroencephalogram decoupling feature extraction network are used to realize the alignment of the feature realization modal time dimension and obtain the modal low-level features; decouple the modal low-level features into common features and specific features, extract the common features and output them; A multi-layer cross-attention network is used to alternately use the common features and specific features of two different modalities as query Q, key K and value V. In the calculation of each head, the features from one modality are converted into a set of key-value pairs, and the other modality is used as a query to achieve interaction between the two modalities, outputting additional complementary information provided by NIR features to EEG features and vice versa. The output of the heterogeneous modality decoupling interactive fusion model is: the superposition of the multi-layer cross attention network output, the near-infrared decoupling feature extraction network output and the EEG decoupling feature extraction network output.

3. The photoelectric dual-modality depression crowd determination system combining functional near infrared spectroscopy and electroencephalography according to claim 2 is characterized in that: The method for alternately using the common features and specific features of two different modalities as query Q, key K and value V is specifically: in, and They represent the decoupled near-infrared specific features and EEG near-infrared specific features respectively; W is the weight matrix; The additional supplementary information provided by the output near-infrared feature to the EEG feature and the additional supplementary information provided by the EEG feature to the near-infrared feature are: in, It indicates that the near-infrared features provide additional supplementary information to the EEG features. It represents the additional supplementary information provided by EEG features to near-infrared features, and d represents the dimension of Q and K.

4. The photoelectric dual-modality depression crowd determination system combining functional near infrared spectroscopy and electroencephalography according to claim 2 is characterized in that: The heterogeneous query is gradually changed to a homogeneous query, that is, the specific features of one modality are first used as the query, and the specific features of the other modality are used as the key and value. After the initial cross attention, the simple features with higher similarity are captured, and then used as new queries to find more complex complementary features. The process can be expressed as: in, and Represents a homogeneous query.

5. The photoelectric dual-modality depression crowd determination system combining functional near infrared spectroscopy and electroencephalography according to claim 2 is characterized in that: When training the heterogeneous modality decoupling interaction fusion model, the decoupling network task loss is defined as: in, Represents the cosine similarity of common features and specific features, Represents the difference between the decoupled features and the original coupled features; During training, the relational knowledge distillation method is also used to learn the relationship between features from the two dimensions of distance and angle.

6. The photoelectric dual-modality depression crowd determination system combining functional near infrared spectroscopy and electroencephalography according to claim 1 is characterized in that: The fusion model based on spatiotemporal attention guidance includes: Align the collected EEG signals and functional near-infrared spectroscopy signals For input features Perform global average pooling and global maximum pooling in the channel dimension to obtain two two-dimensional feature maps. and These two-dimensional feature maps and Concatenate them together and then pass them through a convolutional layer to get the spatial attention weight M; Multiply the weight M by the input feature to get the adjusted feature 7. The photoelectric dual-modality depression crowd determination system combining functional near infrared spectroscopy and electroencephalography according to claim 1 is characterized in that: The functional near-infrared spectrum and EEG synchronous acquisition module is mainly composed of four parts: sensor mainboard, flexible Ag / AgCl EEG acquisition electrode, functional near-infrared spectrum light source and detector, and silicone headband; Flexible Ag / AgCl EEG acquisition electrodes are used to collect EEG signals from 5 leads at the frontal lobe AF7, Fp1, Fpz, Fp2, and AF8. Two additional Ag / AgCl EEG acquisition electrodes are used as reference electrodes and feedback electrodes to collect signals from the mastoid processes behind the left and right ears. The functional near-infrared spectroscopy light source and detector include 6 emission light sources and 14 detectors, which are used to collect 24-lead functional near-infrared spectroscopy signals in the prefrontal area.

8. The photoelectric dual-modality depression crowd determination system combining functional near infrared spectroscopy and electroencephalography according to claim 1 is characterized in that: The audio stimulation paradigm in the audio experimental paradigm stimulation module contains 4 groups of stimulation paradigms, each group contains 4 types of audio, which are played in Latin square order, with a set duration for each audio playback and a resting time between audio playbacks.

9. The photoelectric dual-modality depression crowd determination system combining functional near infrared spectroscopy and electroencephalography according to claim 1 is characterized in that: The interference includes baseline drift, power frequency interference, electrooculogram artifacts, heart rate signal interference, respiratory signal interference and blood pressure signal interference; the signal preprocessing module uses a 500-order finite impulse response filter with a passband of 0.1 to 45 Hz to remove baseline drift and power frequency interference, uses a model combining discrete wavelet transform and Kalman filtering to remove electrooculogram artifacts, and uses a 500-order finite impulse response filter with a passband of 0.01 to 0.1 Hz to remove interference from heart rate, respiratory and blood pressure signals.

10. The photoelectric dual-modality depression crowd determination system combining functional near infrared spectroscopy and electroencephalography according to claim 1 is characterized in that: The system also includes a PC host computer display system module, which is used to call the audio experiment paradigm stimulation module to realize the audio experiment paradigm playback, and to display the functional near-infrared spectrum and electroencephalogram signals collected under the audio experiment paradigm stimulation in real time; the signal preprocessing module, feature extraction and fusion module and audio experiment paradigm stimulation module are integrated in the PC host computer display system module.

Images