A functional near-infrared spectroscopy and electroencephalogram combined photoelectric dual-mode depression population judgment system

Abstract

The present application belongs to the technical field of auxiliary diagnosis of depressive disorders, and particularly relates to a photoelectric dual-mode depressive population judgment system combining functional near-infrared spectroscopy and electroencephalogram. The system comprises a functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module, a signal preprocessing module, a feature extraction and fusion module, and an audio experiment paradigm stimulation module. The functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module is used for acquiring functional near-infrared spectroscopy signals and electroencephalogram signals of the frontal lobe of a population under audio stimulation played by the audio experiment paradigm stimulation module. The signal preprocessing module is used for removing interference in the near-infrared spectroscopy signals and the electroencephalogram signals. The feature extraction and fusion module is used for extracting and fusing features of the near-infrared spectroscopy signals and the electroencephalogram signals after removing artifacts and noise. A machine learning classifier is used for judging depressive population according to the fused features. The audio experiment paradigm stimulation module is used for playing audio segments containing different stimulation attributes.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of auxiliary diagnosis of depressive disorders, and particularly relates to a photoelectric dual-mode depressive population judgment system combining functional near-infrared spectroscopy and electroencephalogram. BACKGROUND

[0002] The clinical diagnosis of depressive disorders mainly relies on scale screening and subjective judgment of doctors, and there is no objective biological diagnostic index, which has problems such as low diagnostic efficiency and lack of unified objective evaluation standard. According to statistics, the accuracy rate of general practitioners in correctly diagnosing depressive disorders is only 47.3% (Lancet, 2009), which shows that there is a lack of objective and universal diagnostic tools, resulting in difficulties in timely diagnosis and intervention treatment of depressive patients.

[0003] Therefore, how to overcome the problems of low efficiency, strong subjectivity and low clinical recognition rate (47.3%, Lancet, 2009) of traditional clinical scales and doctor interviews, break the traditional subjective "symptom description type" mental disorder diagnosis and treatment framework, realize the accurate identification of depressive disorders in an objective "data-driven" manner, and achieve the effect of "preventing disease" are of great significance.

[0004] Research shows that the pathogenesis of depressive disorders is closely related to the caudate nucleus of the brain. When depressive patients experience emotional fluctuations, the prefrontal cortex connected to the caudate nucleus shows abnormal performance, causing a decrease in the level of some excitatory neurotransmitters in the prefrontal cortex and a decrease in neuronal synapses. Wearable brain imaging technology can visualize brain abnormalities more scientifically and quickly, thereby assisting doctors in diagnosing depressive disorders. Common brain imaging technologies include electroencephalogram (EEG), magnetoencephalogram (MEG), functional near-infrared spectroscopy (fNIRS), positron emission tomography (PET), and functional magnetic resonance imaging (fMRI), etc. Electroencephalogram has the advantages of high temporal resolution and rapid response to stimulation, so it is widely welcomed in medical auxiliary diagnosis. However, electroencephalogram is easily affected by various artifacts such as motion, eye movement, electricity, and environmental noise. Functional near-infrared spectroscopy has strong robustness to artifacts and noise, and has high spatial resolution. By collecting changes in brain oxygenation of patients, it can show the spatial state changes of the brain. However, functional near-infrared spectroscopy has poor temporal resolution. Therefore, the combination of functional near-infrared spectroscopy and electroencephalogram can construct a dual-mode physiological index system, taking advantage of the high temporal resolution of electroencephalogram and the high spatial resolution of functional near-infrared spectroscopy, which is expected to make the results of medical auxiliary diagnosis more reliable and accurate.

[0005] The basis of the combination of functional near-infrared spectroscopy and electroencephalogram signals is the neurovascular coupling phenomenon, that is, neural activity is accompanied by fluctuations in cerebral blood flow, and thus the two complement each other. Electroencephalogram has always been regarded as the most active research tool in brain-computer interface due to its high time resolution. However, the low spatial resolution limits the electroencephalogram to record the spatial information of brain activity, and cannot accurately locate the relevant cortical source. Functional near-infrared spectroscopy uses near-infrared light to measure the concentration changes of oxyhemoglobin and deoxyhemoglobin, and is not affected by artifacts and noise. However, the main limitation of functional near-infrared spectroscopy is the long response lag, because the hemodynamic response needs time to reach the maximum amplitude, which means that the measurement cannot be used for real-time applications, so the complementary characteristics of the high time resolution of electroencephalogram and the high spatial resolution of functional near-infrared spectroscopy help to make up for the shortcomings of the two.

[0006] The prefrontal lobe of the brain is closely related to human emotions, moods and the like, and the functional near-infrared spectroscopy and electroencephalogram signals of the prefrontal lobe brain area can effectively assist in identifying depressive patients and healthy people. Since the prefrontal lobe of the brain is not covered by hair, it is beneficial for high-quality (high signal-to-noise ratio) functional near-infrared spectroscopy and electroencephalogram signal acquisition. In addition, external audio and video stimuli will have different effects on the prefrontal lobe of the human brain, and the generation of positive emotions, negative emotions and neutral emotions is closely related to the left and right prefrontal lobe brain areas. Compared with healthy people, depressive patients have a larger difference in performance for positive and negative emotional stimuli, and the effect of positive emotional stimuli is weaker, and the effect of negative emotional stimuli is stronger.

[0007] Traditional clinical scales and doctor interviews are low in efficiency, strong in subjectivity and low in clinical recognition rate (47.3%, Lancet, 2009). In order to break the traditional subjective "symptom description type" mental disorder diagnosis and treatment framework, realize the objective "data driven type" precise determination of depressive population, and at the same time reduce the respective limitations of functional near-infrared spectroscopy and electroencephalogram, the present application provides a photovoltaic dual-mode depressive population determination system combining functional near-infrared spectroscopy and electroencephalogram, extracts a multi-dimensional feature vector strongly related to depressive state through 24-lead functional near-infrared spectroscopy of the prefrontal lobe area and 5-lead (AF7, Fp1, Fpz, Fp2, AF8) electroencephalogram dual-mode signal of the prefrontal lobe area, constructs an objective physiological index system for determining depressive population in the dual-mode of functional near-infrared spectroscopy and electroencephalogram, takes advantage of the high time resolution of electroencephalogram and the high spatial resolution of functional near-infrared spectroscopy, uses the complementary information of the two modalities to overcome the limitations of individual modalities, and through feature layer fusion, improves the reliability and robustness of the determination result of depressive population. SUMMARY

[0008] Therefore, the application provides a photoelectric dual-mode depression population judgment system combining functional near-infrared spectroscopy and electroencephalogram, which can complementarily utilize near-infrared spectroscopy and electroencephalogram signals to accurately and objectively identify and judge depression population.

[0009] The technical solutions of the application are as follows:

[0010] A photoelectric dual-mode depression population judgment system combining functional near-infrared spectroscopy and electroencephalogram comprises a functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module, a signal preprocessing module, a feature extraction and fusion module, and an audio experiment paradigm stimulation module.

[0011] The functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module is used to acquire functional near-infrared spectroscopy signals and electroencephalogram signals of a frontal lobe region of a population under audio stimulation of the audio experiment paradigm stimulation module.

[0012] The signal preprocessing module is used to remove interference in the near-infrared spectroscopy signals and the electroencephalogram signals.

[0013] The feature extraction and fusion module is used to extract and fuse features of the near-infrared spectroscopy signals and the electroencephalogram signals after removing artifacts and noise, and a machine learning classifier is used to judge depression population according to the fused features.

[0014] The audio experiment paradigm stimulation module is used to play audio clips containing different stimulation attributes.

[0015] Further, the heterogeneous modal decoupling interactive fusion model comprises a near-infrared decoupling feature extraction network, an electroencephalogram decoupling feature extraction network, and a multi-layer cross-attention network, wherein the near-infrared decoupling feature extraction network and the electroencephalogram decoupling feature extraction network have the same structure.

[0016] The near-infrared decoupling feature extraction network and the electroencephalogram decoupling feature extraction network are used to realize modal time dimension alignment to obtain modal low-level features, and to decouple the modal low-level features into common features and specific features, and output the common features.

[0017] The multi-layer cross-attention network is used to alternately use the common features and the specific features of the two different modalities as queries Q, keys K and values V, and in the calculation of each head, the features from one modality are converted into a set of key-value pairs, and the other modality is used as a query to realize interaction between the two modalities, and to output additional supplementary information provided by the near-infrared features to the electroencephalogram features and additional supplementary information provided by the electroencephalogram features to the near-infrared features.

[0018] The output of the heterogeneous modal decoupling interactive fusion model is: the superposition of the multi-layer cross-attention network output, the near-infrared decoupling feature extraction network output and the electroencephalogram decoupling feature extraction network output.

[0019] Further, the common features and specific features of two different modalities are alternately used as query Q, key K and value V in the application, specifically:

[0020]

[0021]

[0022] wherein, and respectively represent the decoupled near-infrared specific features and electroencephalogram near-infrared specific features; W is a weight matrix.

[0023] The additional supplementary information provided by the output near-infrared features to the electroencephalogram features and the additional supplementary information provided by the electroencephalogram features to the near-infrared features are:

[0024]

[0025]

[0026] wherein, represents the additional supplementary information provided by the near-infrared features to the electroencephalogram features, represents the additional supplementary information provided by the electroencephalogram features to the near-infrared features, and d represents the dimension of Q and K.

[0027] Further, the heterogeneous query is gradually changed into a homogeneous query in the application, that is, the specific features of one modality are used as the query first, and the specific features of another modality are used as the key and value, after the initial cross-attention, the simple features with higher similarity are captured, and then they are used as new queries to find more complex complementary features, and the process can be represented as:

[0028]

[0029]

[0030] wherein, and represent the homogeneous query.

[0031] Further, the heterogeneous modal decoupling interactive fusion model in the application defines the decoupling network task loss as:

[0032]

[0033] wherein, Cosine similarity representing common features and specific features, Indicating the difference between the decoupling feature and the original coupling feature;

[0034] During training, a relationship knowledge distillation method is also used to learn the relationship between features from two dimensions of distance and angle.

[0035] Further, the fusion model based on space-time attention guidance comprises:

[0036] The collected electroencephalogram signals and functional near-infrared spectroscopy signals are aligned

[0037] The input features are Global average pooling and global maximum pooling are respectively performed on the channel dimension to obtain two two-dimensional feature maps And

[0038] The two two-dimensional feature maps are spliced And Then, a convolution layer is used to obtain spatial attention weight M;

[0039] The weight M is multiplied by the input feature to obtain the adjusted feature

[0040] Further, the functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module mainly comprises a sensor mainboard, a flexible Ag / AgCl electroencephalogram acquisition electrode, a functional near-infrared spectroscopy light source and a detector, and a silica gel headband; wherein,

[0041] The flexible Ag / AgCl electroencephalogram acquisition electrode is used to acquire 5-channel electroencephalogram signals at frontal lobe AF7, Fp1, Fpz, Fp2 and AF8 positions; two additional Ag / AgCl electroencephalogram acquisition electrodes are used as reference electrodes and feedback electrodes to acquire signals at the left and right mastoid processes.

[0042] The functional near-infrared spectroscopy light source and the detector comprise 6 emitting light sources and 14 detectors, which are used to acquire 24-channel functional near-infrared spectroscopy signals in the frontal lobe region.

[0043] Further, the audio stimulus paradigm in the audio experiment paradigm stimulation module comprises 4 groups of stimulus paradigms, each group comprising 4 types of audio, played in Latin square order, each audio played for a set duration, and a resting state rest time set between audio plays.

[0044] Further, the interference of the present application includes baseline drift, power frequency interference, eye movement artifact, heart rate signal interference, respiratory signal interference and blood pressure signal interference; the signal preprocessing module is used to remove baseline drift and power frequency interference by using a 500-order finite impulse response filter with a passband of 0.1-45Hz, remove eye movement artifacts by using a model combining discrete wavelet transform and Kalman filtering, and remove heart rate, respiratory and blood pressure signal interference by using a 500-order finite impulse response filter with a passband of 0.01-0.1Hz.

[0045] Further, the system of the present application further comprises a PC host display system module for calling the audio experimental paradigm stimulation module to realize audio experimental paradigm playing, and real-time display of functional near-infrared spectroscopy and electroencephalogram signals collected under audio experimental paradigm stimulation; the signal preprocessing module, the feature extraction and fusion module and the audio experimental paradigm stimulation module are integrated in the PC host display system module.

[0046] Beneficial effects:

[0047] First, the present application focuses on a new method for objective recognition of depressive disorder, brain activity information is obtained through frontal lobe functional near-infrared spectroscopy and electroencephalogram signals, and objective recognition and determination of depressive population are realized through feature extraction, feature fusion and artificial intelligence technology.

[0048] Second, the present application uses near-infrared spectroscopy and electroencephalogram dual-mode signals as objective indicators for judging depressive population, constructs a functional near-infrared spectroscopy and electroencephalogram dual-mode physiological index system, comprehensively utilizes the advantages of high time resolution and high spatial resolution of the two, uses complementary information of the two modes to overcome the limitations of individual mode, and improves the reliability and robustness of the determination result of the artificial intelligence classifier through feature layer fusion.

[0049] Third, it helps to improve the problems of low efficiency, strong subjectivity and low clinical recognition rate of the past traditional clinical scale and doctor interview, and provides an objective and effective auxiliary diagnosis method for doctors to judge depressive disorder. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0051] Figure 1 A functional near-infrared spectroscopy and electroencephalogram combined optoelectronic dual-mode depressive population determination system provided by the present application is shown in the principle schematic block diagram.

[0052] Figure 2 A schematic diagram of the location of the functional near-infrared spectroscopy emitter, detector, and EEG acquisition electrodes, as well as an overall rendering, of a photoelectric dual-modal depression diagnosis system combining functional near-infrared spectroscopy and EEG, provided for an embodiment of the present invention.

[0053] Figure 3 A schematic diagram of the mainboard of the functional near-infrared spectroscopy and electroencephalography (EEG) signal acquisition sensor for a photoelectric dual-modal depression diagnosis system that combines functional near-infrared spectroscopy and EEG, provided in an embodiment of the present invention;

[0054] Figure 4 The present invention provides an optimal selection strategy for heterogeneous sensors in a photoelectric dual-modal depression diagnosis system that combines functional near-infrared spectroscopy and electroencephalography.

[0055] Figure 5 This invention provides a data transmission strategy based on an event-triggered mechanism for a photoelectric bimodal depression assessment system that combines functional near-infrared spectroscopy and electroencephalography (EEG).

[0056] Figure 6 A block diagram of an audio stimulation test paradigm for a photoelectric bimodal depression assessment system combining functional near-infrared spectroscopy and electroencephalography (EEG) provided in an embodiment of the present invention;

[0057] Figure 7 The overall flowchart of data acquisition and algorithm analysis of a photoelectric bimodal depression diagnosis system combining functional near-infrared spectroscopy and electroencephalography is provided in an embodiment of the present invention.

[0058] Figure 8 This is the feature layer fusion process of the heterogeneous modal decoupling interactive fusion model used in this embodiment of the invention;

[0059] Figure 9 This describes the feature layer fusion process of a spatiotemporal attention-guided fusion model in an embodiment of the present invention.

[0060] The module includes: power management module-1, functional near-infrared spectroscopy and EEG synchronous acquisition module-2, Bluetooth module-3, signal preprocessing module-4, feature extraction and fusion module-5, PC and host computer display system module-6, and audio experimental paradigm stimulation module-7. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] In order to make the above objectives, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0063] The application provides a functional near-infrared spectroscopy and electroencephalogram combined optoelectronic dual-mode depression population determination system, which comprises a functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module, a signal preprocessing module, a feature extraction and fusion module and an audio experimental paradigm stimulation module.

[0064] The functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module is used for acquiring functional near-infrared spectroscopy signals and electroencephalogram signals of a frontal lobe region of a population under audio stimulation of the audio experimental paradigm stimulation module.

[0065] As shown in Figure 1 The application further provides a functional near-infrared spectroscopy and electroencephalogram combined optoelectronic dual-mode depression population determination system, which comprises:

[0066] A power management module 1, a functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module 2, a Bluetooth module 3, a signal preprocessing module 4, a feature extraction and fusion module 5, a PC and an upper computer display system module 6 and an audio experimental paradigm stimulation module 7.

[0067] The power management module 1 is used for supplying power for the functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module 2 and the Bluetooth module 3, and providing high-quality power for efficient and stable work.

[0068] The functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module 2 is used for synchronously acquiring high-signal-to-noise ratio 24-lead functional near-infrared spectroscopy signals and 5-lead (AF7, Fp1, Fpz, Fp2, AF8) electroencephalogram signals of a frontal lobe region of a patient under audio stimulation of the audio experimental paradigm stimulation module 7.

[0069] The Bluetooth module 3 is used for transmitting the high-signal-to-noise ratio 24-lead functional near-infrared spectroscopy signals and 5-lead (AF7, Fp1, Fpz, Fp2, AF8) electroencephalogram signals of a frontal lobe region of a patient acquired by the functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module 2 to the PC upper computer display system module 6.

[0070] The signal preprocessing module 4 is used for removing body movement, eye movement, power and other artifact noise in the functional near-infrared spectroscopy and electroencephalogram signals, and improving the signal-to-noise ratio of the signals.

[0071] The feature extraction and fusion module 5 is used for extracting features from the collected functional near-infrared spectroscopy and electroencephalogram data, then fusing the extracted functional near-infrared spectroscopy and electroencephalogram features, and finally determining the depression population by using a machine learning classifier.

[0072] The PC host display system module 6 is used for calling the audio experimental paradigm stimulation module to realize audio experimental paradigm playing, and displaying the functional near-infrared spectroscopy and electroencephalogram signals collected under the audio experimental paradigm stimulation in real time. In addition, the PC host display system module performs functional near-infrared spectroscopy and electroencephalogram dual-modal data preprocessing, feature extraction, fusion and model classification through the integrated signal preprocessing module and feature extraction and fusion module, and finally outputs the determination result.

[0073] The audio experimental paradigm stimulation module 7 is used for playing audio segments containing different stimulation attributes.

[0074] The embodiment of the present application: first, the functional near-infrared spectroscopy signal of 24 channels in the frontal lobe region and the electroencephalogram signal of 5 channels (AF7, Fp1, Fpz, Fp2, AF8) in the frontal lobe region under audio stimulation are synchronously collected. Second, a depression recognition technology based on the fusion of functional near-infrared spectroscopy and electroencephalogram signals is proposed. The technology improves the reliability and robustness of depression population determination by fusing the multi-dimensional features of functional near-infrared spectroscopy and electroencephalogram signals. Finally, based on the proposed sensor, algorithm and model, an auxiliary diagnosis and treatment system suitable for depression population determination is developed. The present application focuses on solving the new method of objective recognition of depression disorder, and obtains the brain activity information through the functional near-infrared spectroscopy and electroencephalogram signals of the frontal lobe, and realizes the objective recognition and determination of the depression population through feature extraction, feature fusion and artificial intelligence technology.

[0075] The embodiment of the present application uses the functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module 2 to collect the functional near-infrared spectroscopy signal of 24 channels in the frontal lobe region and the electroencephalogram signal of 5 channels (AF7, Fp1, Fpz, Fp2, AF8) in the frontal lobe region under the audio stimulation of the audio experimental paradigm stimulation module 7. The functional near-infrared spectroscopy signal and the electroencephalogram signal are collected for 628 seconds, the sampling rate of the functional near-infrared spectroscopy signal is 35.7 SPS, and the sampling rate of the electroencephalogram is 250 SPS.

[0076] In another embodiment of this application, the functional near-infrared spectroscopy and electroencephalogram (EEG) synchronous acquisition module 2 consists of four parts: a functional near-infrared spectroscopy and EEG signal acquisition sensor motherboard, a flexible Ag / AgCl EEG acquisition electrode, a functional near-infrared spectroscopy light source and detector, and a medical-grade silicone headband that can fix the Ag / AgCl EEG acquisition electrode and the functional near-infrared spectroscopy light source and detector.

[0077] In the above embodiments, flexible Ag / AgCl EEG acquisition electrodes are located in the prefrontal cortex (AF7, Fp1, Fp2, Fp2, AF8) for acquiring three-lead EEG signals; two additional Ag / AgCl EEG acquisition electrodes (A1, A2) are placed at the mastoid processes behind the left and right ears, respectively, as reference electrodes and feedback electrodes; the six emission sources and 14 detectors of the functional near-infrared spectroscopy acquisition electrode are placed in the prefrontal cortex; in the hardware design of the functional near-infrared spectroscopy and EEG synchronous acquisition module 2, a heterogeneous sensor optimal selection strategy and an event-triggered data transmission strategy are proposed.

[0078] In the above embodiments, the optimal selection strategy for heterogeneous sensors is proposed: to better guide the rational layout of EEG electrodes, near-infrared light sources, and detectors, an optimal combination strategy for light sources, detectors, and EEG electrodes is suggested. Following the international 10-20 system, EEG electrodes are deployed in the forehead region, and filtering, wrap-around, and embedded methods are used for feature selection to guide the layout of heterogeneous sensors. For example... Figure 4 As shown, when designing the sensor layout, each channel is treated as a feature, and the purpose of feature selection is to choose the most representative parts from a large number of features. After data acquisition and necessary preprocessing, a filtering method is used to initially screen each channel, eliminating obviously irrelevant or redundant channels. Based on the initially screened channels, a wrapping method is used to train the model on each feature subset, evaluate its recognition performance, and thus find better combination channels. An embedded method is used to further refine the selection, determining which channels contribute more to the model's recognition performance. Finally, based on the identified key channels, the optimal layout of the EEG and functional near-infrared spectroscopy sensors is determined.

[0079] In the above embodiments, the data transmission strategy based on an event-triggered mechanism addresses the issue that uncertainties arising during the data acquisition process, such as those caused by the environment and personnel, may lead to high noise levels in the data within a certain period, which can hinder subsequent modeling. Therefore, a data transmission scheme based on an event-triggered mechanism is proposed. Normal data transmission occurs when predefined event triggering conditions are met, and transmission is shut down when these conditions are not met, thereby reducing unnecessary noise data transmission and saving bandwidth and storage resources. Figure 5As shown, the event-triggered transmission process of a single channel is as follows: a noise threshold is set in advance, and in each sampling period, it is checked whether the signal exceeds the threshold. When the signal of a channel continuously exceeds the threshold, it is considered that high noise is generated, and at this time, the data transmission of the channel is closed. When the noise continuously falls below the threshold, the transmission is restarted.

[0080] In another embodiment of the present application, the audio stimulus paradigm in the audio stimulus paradigm stimulation module 7 contains 4 groups of stimulation paradigms, each group containing 4 types (happy, calm, fear, white noise) of audio, played in Latin square order, each audio lasting 18 seconds, with 20 seconds of resting state rest time between audio playing, and the total duration of the audio stimulus experiment paradigm being 628 seconds.

[0081] In another embodiment of the present application, the high signal-to-noise ratio prefrontal lobe 24 lead functional near-infrared spectroscopy signal and the prefrontal lobe 5 lead (AF7, Fp1, Fpz, Fp2, AF8) electroencephalogram signal under the audio stimulus played by the audio stimulus paradigm stimulation module 7 are transmitted to the PC host computer display system module 6 using the Bluetooth module 3. The Bluetooth module 3 uses a WH-BLE105 low-power Bluetooth module to transmit the functional near-infrared spectroscopy signal and the electroencephalogram signal collected by the functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module 2 to the PC host computer display system module 6.

[0082] In another embodiment of the present application, the signal preprocessing module 4 is used to eliminate baseline drift and artifact interference in the original functional near-infrared spectroscopy signal and the electroencephalogram signal. The signal preprocessing module 4 uses a 500-order finite impulse response (FIR) filter with a passband of 0.1-45 Hz to remove baseline drift and power frequency interference. At the same time, since the electroencephalogram acquisition points (AF7, Fp1, Fpz, Fp2, AF8) are close to the eyes, it is inevitable to collect eye movement artifacts. Therefore, the signal preprocessing module 4 uses a model combining discrete wavelet transform and Kalman filter to remove eye movement artifacts. The original functional near-infrared spectroscopy signal is mainly disturbed by the heart rate signal in the 1-2 Hz frequency band, the respiration signal in the 0.15-0.6 Hz frequency band, and the blood pressure signal around 0.1 Hz. Since the functional near-infrared spectroscopy signal is mainly located in the 0.02-0.06 Hz frequency band. Therefore, the signal preprocessing module uses a 500-order finite impulse response (FIR) filter for 0.01-0.1 Hz to remove the interference of heart rate, respiration and blood pressure signals.

[0083] In another embodiment of the present application, the feature extraction and fusion module 5 processes and extracts features from the 628-second audio stimulus collected functional near-infrared spectroscopy signal and electroencephalogram signal. The feature extraction and fusion module 5 mainly includes two parts: a heterogeneous modal decoupling interaction fusion strategy and a fusion strategy based on spatiotemporal attention guidance.

[0084] In the above embodiments, the heterogeneous modal decoupling interactive fusion strategy is as follows: EEG signals reflect changes in the electrical signals of brain neuron activity, while near-infrared spectral signals reflect changes in brain hemodynamics. The inherent heterogeneity of these two signals poses a challenge to the fusion recognition performance. To obtain higher quality fusion features and richer intermodal complementary knowledge, an interactive fusion model based on feature decoupling is established, the overall framework of which is as follows: Figure 8 As shown, it includes a near-infrared decoupling feature extraction network, an EEG decoupling feature extraction network, and a cross-attention network. The near-infrared decoupling feature extraction network and the EEG decoupling feature extraction network have the same structure, containing a shallow coding network, a decoupling network, a feature extraction network, and a coupled coding network; specifically:

[0085] The working process of the near-infrared decoupling feature extraction network and the EEG decoupling feature extraction network is as follows: Modal temporal dimension alignment is achieved through a shallow coding network to obtain low-level modal features X. m Considering the inherent heterogeneity of physiological signals and the inherent commonalities when dealing with the same emotional state, each modality representation X is decoupled from the shared and specific feature encoders in the network. m Decoupling is a common feature and specific features Common features This represents the consistency information contained in the two modalities of data, specific features. Each modality has its own unique expression, m∈{fNIRS,EEG}; the feature extraction network extracts the common features after decoupling.

[0086] Meanwhile, to avoid ambiguity in decoupled features, an autoregressive approach is used for supervision. The decoupled features of each modality are concatenated and passed through a dedicated encoder D. m Generate coupling features Then coupled with the original features (low-level modal features X) m The comparison can be formalized as follows:

[0087]

[0088] In addition, in order to maintain common characteristics and specific features Differences, using cosine similarity to monitor the differences between two features, can be represented as:

[0089]

[0090] Therefore, the loss of decoupling network tasks is defined as:

[0091]

[0092] The working process of cross attention network is: the decoupled modal specific features In the face of the same emotional state, the complementary knowledge has the effective cross-modal knowledge transfer, which helps to improve the model performance. The progressive cross-attention network is used to realize the complementary knowledge extraction between different modalities, and the common features and specific features of two different modalities of near-infrared fNIRS (f) and electroencephalogram EEG (E) are alternately used as query (Q), key (K) and value (V), that is:

[0093]

[0094]

[0095] Wherein, W is a weight matrix. Then, each modality realizes the interaction of information from different modalities through a multi-head attention module. In the calculation of each head, the features from one modality are converted into a set of key-value pairs, and the other modality is used as a query to realize the interaction between the two modalities. The multi-head attention uses the dot product to calculate the similarity between the query and the key, and obtains an attention weight distribution vector.

[0096] The output of the i-th attention head can be represented as follows:

[0097]

[0098]

[0099] fNIRS provides additional complementary information to EEG, EEG provides additional complementary information to fNIRS, and d represents the dimension of Q and K.

[0100] It is proposed to change the heterogeneous query to homogeneous query gradually, that is, first use the specific features of one modality as the query, and use the specific features of the other modality as the key and value. After the initial cross-attention, the simple features with higher similarity are captured, and then they are used as new queries to find more complex complementary features. This process can be represented as:

[0101]

[0102] Wherein, and represent homogeneous query.

[0103] The output of the interactive fusion model based on feature decoupling is the superposition of the common features of electroencephalogram data, the common features of infrared data and the attention head output.

[0104] Further, considering that the intra-modal feature relationship may be lost due to the decoupling operation, a relationship knowledge distillation method is adopted to learn the relationship between the features from the two dimensions of distance and angle. The relationship knowledge learning consists of two branches: internal aggregation (sa) and external distillation (ke). The internal aggregation concatenates the common features and the specific features, then performs a channel rearrangement operation, and then uses a self-attention mechanism to realize internal attention. The external distillation obtains the undecoupled features from the original feature map; and performs distillation operations from the two dimensions of distance and angle, to reduce the gap between the features of the two branches, and thus to supplement the relationship knowledge. The common features and the specific features are concatenated for the concatenated features and the undecoupled original features , the Euclidean distance between the two groups of features is calculated, which can be expressed as:

[0105]

[0106] where d euc is the Euclidean distance, v i and v j are a certain sample in the branch feature, and μ is the distance normalization factor, which is defined as the average Euclidean distance of all sample pairs in a batch, to ensure that there is no abnormal value, and to reflect the average structural features of the entire batch:

[0107]

[0108] where N is the number of small batch processing.

[0109] It is hoped that can fully learn the relationship between the feature structures contained in , so that d euc (sa i ,sa j ) in is close enough to d euc (ke i ,ke j ) in , which can be expressed as:

[0110]

[0111] where represents the Huber loss, d euc (sa i ,sa j ) represents the Euclidean distance of two samples in the concatenated feature , and d euc (ke i ,ke j ) represents the Euclidean distance of two samples in the undecoupled original feature Euclidean distance between any two samples in.

[0112] On the other hand, the change of angle can reveal the complex dependence between features that may not be fully captured by distance measurement alone. The angle calculation is defined as the cosine similarity between three sample points:

[0113] d ang (v i ,v j ,v k )=cos∠v i v j v k =<e ij ,e kj >

[0114] where e ij is the unit vector of branch feature v i to v j , and e kj is the unit vector of branch feature v k to v j . To minimize the difference of sample angles, the angle loss is defined as follows:

[0115]

[0116] where ∠sa i sa j sa k and ∠ke i ke j ke k represent the cosine similarity between three sample points in the inner and outer branches, respectively.

[0117] The angle distillation loss can help decoupled post-features learn similar angle structures in feature space as non-coupled features, so that the relative relationship between features can be better understood. In addition, cross-entropy loss is also introduced for non-decoupled features to avoid ambiguous knowledge transfer.

[0118] Finally, combined with the above constraints, the learning loss of relational knowledge is formed:

[0119]

[0120] In the decoupled interaction model, different modal common, specific and complementary features are fully considered, and relationship knowledge is supplemented for the final fusion features, which can effectively realize the cross-modal interaction fusion recognition of depression state. The joint decoupling, relationship learning and emotion classification loss form the final constraint:

[0121]

[0122] where, is the loss for emotion classification, consisting of multi-class cross-entropy, and and are weight factors.

[0123] The fusion strategy based on spatio-temporal attention guidance in the above embodiment is:

[0124] Electroencephalogram (EEG) can capture brain electrical activity at the millisecond level, reflecting the dynamic changes of the brain in real time. Functional near-infrared spectroscopy (fNIRS) can measure the level of blood oxygen changes in the cerebral cortex, which has high spatial resolution. In order to make full use of the advantages of the two modalities in time and space, and effectively fuse their data, a depression detection model based on spatio-temporal attention mechanism is proposed to achieve a more comprehensive analysis of brain activity. Attention mechanism is a widely used technique in deep learning, and its main purpose is to enable the model to selectively focus on different parts of the input data, thereby more effectively processing information. In the time dimension, attention mechanism can focus on the electrical activity changes of EEG data in a specific time period, and then give these key time periods higher weights to highlight their importance. In the spatial dimension, attention mechanism can focus on specific brain regions using the high spatial resolution information of blood oxygen changes in the cerebral cortex provided by fNIRS. In addition, it can use fNIRS information to guide the attention distribution of EEG signals in the spatial dimension, making EEG signals more focused on important brain regions, thereby improving the ability to capture brain activity patterns.

[0125] The overall structure of the depression recognition model based on spatio-temporal attention mechanism is shown in Figure 9 .

[0126] First, the collected EEG signals and fNIRS signals are aligned:

[0127]

[0128] where, is the aligned modal feature, is the alignment operation, X m represents the original modal feature, m e {fNIRS, EEG}. Through the alignment operation, EEG signals and fNIRS signals can be synchronized in time points, ensuring that the corresponding events or states captured by the two modalities can be accurately compared and analyzed. This is crucial for studying the relationship between brain electrical activity and blood dynamics, as it can help the model capture neurovascular coupling, the relationship between neural activity and blood flow changes.

[0129] Second, temporal attention weights W(t) are applied to EEG signals and fNIRS to adaptively learn which time points are more important and give them higher weights.

[0130] Finally, for the high spatial resolution characteristics of near-infrared spectroscopy, spatial attention is applied, and its guidance of the spatial attention distribution of electroencephalogram signals is used to further impose weight information on the electroencephalogram signals in the spatial dimension, enhancing the spatial characteristics of electroencephalogram signals related to tasks. Specifically, spatial attention imposes weight information on the input features Global average pooling and global maximum pooling are respectively performed in the channel dimension to obtain two two-dimensional feature maps:

[0131]

[0132]

[0133] wherein C is the number of feature map channels. The two two-dimensional feature maps are spliced together, and then a convolution layer is used to obtain spatial attention weights:

[0134]

[0135] wherein σ is a Sigmoid function, and f is a convolution operation.

[0136] Finally, the weight is multiplied by the input feature to obtain the adjusted feature:

[0137]

[0138] In another embodiment of the present application, the PC host display system module 6 is used to call the audio experimental paradigm stimulation module 7 to play the audio experimental paradigm, and can display the functional near-infrared spectroscopy and electroencephalogram collected by the functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module 2 in real time. At the same time, the PC host display system module 6 performs functional near-infrared spectroscopy and electroencephalogram data preprocessing, feature extraction, fusion and classification through the signal preprocessing module 4 and the feature extraction and fusion module 5 integrated therein, and finally outputs the judgment result.

[0139] The functional near-infrared spectroscopy and electroencephalogram combined optoelectronic dual modality depression population judgment system of the embodiment of the present application acquires brain activity information through 24-channel functional near-infrared spectroscopy of the prefrontal lobe and 5-channel electroencephalogram of the prefrontal lobe, and realizes objective recognition and judgment of the depression population through feature extraction, feature fusion and artificial intelligence technology, which can effectively reduce the cost and subjectivity of depression disorder recognition, improve the timeliness and accuracy of depression disorder recognition, and effectively promote the intervention and treatment of depression disorder population. According to experimental data, the efficiency of depression population judgment of this method is more than 90%.

[0140] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A photoelectric dual-mode system for judging depression population by combining functional near-infrared spectroscopy and electroencephalogram, characterized in that, include: Functional near-infrared spectroscopy and EEG synchronous acquisition module, signal preprocessing module, feature extraction and fusion module, and audio experimental paradigm stimulation module; The functional near-infrared spectroscopy and electroencephalogram (EEG) synchronous acquisition module is used to acquire functional near-infrared spectral signals and EEG signals of the prefrontal cortex of the population under the audio stimulation played by the audio experimental paradigm stimulation module. The signal preprocessing module is used to remove interference from near-infrared spectral signals and electroencephalogram (EEG) signals; The feature extraction and fusion module is used to extract and fuse features from near-infrared spectral and electroencephalogram (EEG) data after removing artifact noise, and to use a machine learning classifier to determine the depressed population based on the fused features; wherein the feature extraction and fusion module uses a heterogeneous modal decoupled interactive fusion model and a spatiotemporal attention-guided fusion model for feature fusion; The audio experimental paradigm stimulation module is used to play audio segments containing different stimulus attributes. The heterogeneous modal decoupling and interaction fusion model includes a near-infrared decoupling feature extraction network, an EEG decoupling feature extraction network, and a multi-layer cross-attention network, wherein the near-infrared decoupling feature extraction network and the EEG decoupling feature extraction network have the same structure; Near-infrared decoupling feature extraction network and EEG decoupling feature extraction network are used to achieve modal temporal dimension alignment to obtain low-level modal features; the low-level modal features are decoupled into common features and specific features, and the common features are extracted and output; A multi-layer cross-attention network is used to alternately use common features and specific features from two different modalities as queries Q, keys K and values ​​V. In the computation of each head, features from one modality are converted into a set of key-value pairs, while the other modality is used as a query to achieve interaction between the two modalities. The output is the additional supplementary information provided by near-infrared features to EEG features and the additional supplementary information provided by EEG features to near-infrared features. The output of the heterogeneous modal decoupling interaction fusion model is the superposition of the output of the multi-layer cross-attention network, the output of the near-infrared decoupling feature extraction network, and the output of the EEG decoupling feature extraction network. 2.The system of claim 1, wherein, The method for alternately using common features and specific features of two different modalities as query Q, key K, and value V is as follows: wherein, and respectively represent decoupling to obtain near-infrared specific features and electroencephalogram specific features; is a weight matrix; The additional supplementary information provided by the output near-infrared features to the EEG features and the additional supplementary information provided by the EEG features to the near-infrared features are as follows: wherein, represents additional complementary information provided by the near-infrared features to the electroencephalographic features, represents additional complementary information provided by the electroencephalographic features to the near-infrared features, represents and dimensions. 3.The system of claim 1, wherein, The heterogeneous query is progressively changed to a homogeneous query. This involves first using specific features from one modality as the query and specific features from another modality as the key and value. After initial cross-attention, simple features with high similarity are captured and then used as new queries to find more complex complementary features. This can be represented as: wherein and represent homogenous queries. 4.The system of claim 1, wherein, During training, the heterogeneous modal decoupling interaction fusion model defines the decoupling network task loss as: wherein, represents a cosine similarity of common features and specific features, represents a difference of decoupled features from the original coupled features; During training, a relational knowledge distillation method is also used to learn the relationships between features from two dimensions: distance and angle.

5. The functional near-infrared spectroscopy and electroencephalogram combined photoelectric dual-mode system for judging depression population according to claim 1, characterized in that, The spatiotemporal attention-guided fusion model includes: The collected electroencephalogram signals and functional near-infrared spectrum signals are aligned: wherein, is the aligned modal feature, is the alignment operation, represents the original modal feature, ; On the input features Global average pooling and global max pooling are respectively performed on the channel dimension to obtain two two-dimensional feature maps And ; Map these two two-dimensional features And Concatenate them, then pass through a convolutional layer to get spatial attention weights ; The weights are multiplied with the input features to get the adjusted features .​ 6.The system of claim 1, wherein, The functional near-infrared spectroscopy and electroencephalogram synchronous acquisition module mainly comprises a sensor mainboard, a flexible Ag / AgCl electroencephalogram acquisition electrode, a functional near-infrared spectroscopy light source and a detector, and a silica gel headband, wherein The flexible Ag / AgCl electroencephalogram acquisition electrode is used for acquiring electroencephalogram signals of five leads at positions AF7, Fp1, Fpz, Fp2 and AF8 of a frontal lobe; two additional Ag / AgCl electroencephalogram acquisition electrodes are used as reference electrodes and feedback electrodes for acquiring signals at left and right mastoid processes behind ears. The functional near-infrared spectroscopy light source and the detector comprise six emitting light sources and 14 detectors, and are used for acquiring 24-lead functional near-infrared spectroscopy signals of the frontal lobe region. 7.The system of claim 1, wherein, The audio stimulus paradigm in the audio experiment paradigm stimulation module comprises four groups of stimulus paradigms, each group comprising four types of audio, played in Latin square order, each audio played for a set time length, and a resting state rest time set between audio playing. 8.The system of claim 1, wherein, The interference includes baseline drift, power frequency interference, electrooculogram artifact, heart rate signal interference, respiration signal interference and blood pressure signal interference; the signal preprocessing module adopts a 500-order finite impulse response filter with a passband of 0.1-45 Hz to remove baseline drift and power frequency interference, adopts a model combining discrete wavelet transform and Kalman filtering to remove electrooculogram artifacts, and adopts a 500-order finite impulse response filter with a passband of 0.01-0.1 Hz to remove heart rate, respiration and blood pressure signal interference. 9.The system of claim 1, wherein, The system further comprises a PC host display system module for calling the audio experiment paradigm stimulation module to realize audio experiment paradigm playing, and for real-time displaying functional near-infrared spectroscopy and electroencephalogram signals acquired under audio experiment paradigm stimulation; the signal preprocessing module, the feature extraction and fusion module and the audio experiment paradigm stimulation module are integrated in the PC host display system module.

Images