A multi-modal signal acquisition device and system fusing near-infrared and ear electroencephalogram

Abstract

The application discloses a kind of multi-modal signal acquisition device and system of fusion near infrared and ear brain electricity, for brain electricity signal acquisition, including earphone main body, earplug ear electricity collector, near infrared emitter, detection device;Wherein, the earphone main body is ear cover type structure;The earplug ear electricity collector is fixedly connected with the inner wall of ear cover of the earphone main body, and the earplug ear electricity collector is compatible with human ear;N the near infrared emitter is installed in the ear cover and scalp contact place of the earphone main body;M the detection device is installed in the ear cover and scalp contact place of the earphone main body.The electrode of the present application is fully contacted with human body, and is stably contacted in a large area, and is not easy to slide or fall off, and the concealment is considered.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of signal acquisition, in particular to a multi-modal signal acquisition device and system fusing near-infrared and ear electroencephalogram. BACKGROUND

[0002] Electroencephalogram is the overall reflection of the electrical physiological activity of brain nerve cell groups, and the different states and causes of brain will cause the changes of electroencephalogram in different regions or frequency bands of brain, so that the electroencephalogram shows different modes. Therefore, electroencephalogram examination plays an important role in the clinical diagnosis of neurology and psychiatry, and it has been found that many brain diseases and mental illnesses are related to electroencephalogram abnormalities. With the continuous development of computer technology, computer-aided electroencephalogram analysis and automatic electroencephalogram analysis system are developed and researched, and many systems for analyzing electroencephalogram are proposed from the engineering point of view, and some progress and achievements have been made. However, the common electroencephalogram acquisition system is complicated to use, high in cost, and usually needs to wear an information recording device, which is difficult to realize concealed acquisition and acquisition under the condition of user's free movement. SUMMARY

[0003] The present application aims to provide a multi-modal signal acquisition device and system fusing near-infrared and ear electroencephalogram, to solve the problems existing in the prior art.

[0004] In order to achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0005] A multi-modal signal acquisition device fusing near-infrared and ear electroencephalogram, comprising an earphone main body, an in-ear ear electroacquisition device, a near-infrared emitter, and a detection device.

[0006] The earphone main body is of an ear cover type structure.

[0007] The in-ear ear electroacquisition device is fixedly connected to the inner wall of the ear cover of the earphone main body and is adapted to the shape of human ear.

[0008] N near-infrared emitters are installed at the contact part between the ear cover of the earphone main body and the scalp.

[0009] M detection devices are installed at the contact part between the ear cover of the earphone main body and the scalp.

[0010] Preferably, the in-ear ear electroacquisition device comprises a deformable support and a silica gel electrode head.

[0011] One end of the deformable support is fixedly connected to the earphone main body.

[0012] The other end is movably connected to the silica gel conductive electrode head.

[0013] Preferably, the near-infrared emitters and the detection devices are arranged alternately on the earphone body; wherein the alternately arranged near-infrared emitters and the detection devices form an arc; adjacent near-infrared emitters and the detection devices on the arc are at a distance fcm.

[0014] Preferably, the two ear covers of the earphone body have the same structure.

[0015] Preferably, the earphone further comprises a wireless communication device.

[0016] The wireless communication device is embedded in the earphone body and electrically connected with the earphone body.

[0017] A multi-modal signal acquisition system fusing near-infrared and ear electroencephalogram, comprising: the multi-modal signal acquisition device fusing near-infrared and ear electroencephalogram according to claim 1, and further comprising:

[0018] A cloud server receiving signals sent by the multi-modal signal acquisition device fusing near-infrared and ear electroencephalogram according to claim 1.

[0019] The data preprocessing module attenuates alternating current and its harmonics.

[0020] The feature extraction module extracts electroencephalogram features based on linear and nonlinear electroencephalogram signals and extracts near-infrared features based on light intensity data.

[0021] The feature integration module fuses and aligns all features extracted by the feature extraction module to obtain target multi-modal data.

[0022] Preferably, the data processing module attenuates the alternating current and its harmonics by performing i-order Butterworth notch on the alternating current.

[0023] Preferably, the feature extraction module comprises an in-ear ear electroacquisition device, a near-infrared emitter and a detection device.

[0024] The in-ear ear electroacquisition device acquires ear electroencephalogram based on the similarity between ear electroencephalogram and brain electroencephalogram to obtain brain electroencephalogram features.

[0025] The near-infrared emitter emits near-infrared light, which is absorbed and scattered by the head, and then the scattered near-infrared light is detected by the detection device and converted into an electrical signal to obtain near-infrared features.

[0026] Preferably, the feature integration module fuses and aligns all features extracted by the feature extraction module.

[0027] As can be known from the above, compared with the prior art, the present application has the following beneficial effects:

[0028] The electrode is in full, large-area and stable contact with the human body, is not easy to slide or fall off, and gives consideration to concealment, and the ear canal electroencephalogram acquisition is made of flexible material. The advantages of collecting electroencephalogram in the ear include: 1) because the ear canal electrode is inserted into the ear, the contact is relatively close, and the ear canal electrode is not easy to slide or fall off, thereby reducing the problem that the conventional electrode slides or falls off due to vibration caused by movement; 2) the earplug-type electrode base can support the electrode and the skin of the ear canal to be in close contact, and the contact area is large, so that the contact resistance can easily meet the measurement requirement; 3) compared with the conventional hard electrode such as silver chloride electrode and silver electrode, the flexible material has better comfort in use, and is not easy to cause skin irritation and abrasion; 4) the earplug-type electroencephalogram electrode also gives consideration to concealment, because the appearance and use are similar to earphones, the user is not different from ordinary people when the user is free to move, and the privacy of the user is protected. In the project, the earplug-type electrode is designed by using PDMS mixed with carbon nanotubes, the PDMS is a flexible material, has excellent ductility, flexibility and low dielectric constant, is non-toxic and odorless, has good biocompatibility, can be fixed and shaped at room temperature, the carbon nanotube has high conductivity, a mold is designed according to the ear canal model and size, and the ear canal flexible electrode is formed by sufficient mixing, fusion and solidification. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of the provided drawings.

[0030] Figure 1 It is a front view of the device of the present application;

[0031] Figure 2 It is a perspective view of the device of the present application;

[0032] Fig. 3(a)-(b) is a comparison diagram of ear electroencephalogram acquisition signal and industry-recognized electroencephalogram equipment;

[0033] Figure 4 It is a system block diagram of the present application;

[0034] Figure 5 Feature fusion alignment algorithm flow chart.

[0035] In the figure: 1 earphone main body, 2 in-ear ear electroacquisition, 3 detection device, 4 near-infrared emitter. DETAILED DESCRIPTION

[0036] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of the present application.

[0037] Embodiment 1

[0038] The embodiment discloses a multi-modal signal acquisition device fusing near-infrared and ear electroencephalogram, comprising an earphone main body 1, an in-ear ear electroacquisition device 2, a near-infrared emitter 4, a detection device 3;

[0039] The earphone main body 1 is of a circumaural structure.

[0040] The in-ear ear electroacquisition device 2 is fixedly connected to the inner wall of the ear cover of the earphone main body 1 and is adapted to the shape of a human ear.

[0041] N near-infrared emitters 4 are installed at the contact position between the ear cover of the earphone main body 1 and the scalp.

[0042] M detection devices 3 are installed at the contact position between the ear cover of the earphone main body 1 and the scalp.

[0043] Specifically,

[0044] The in-ear ear electroacquisition device 2 in the embodiment 1 comprises a deformable support and a silica gel electrode head.

[0045] One end of the deformable support is fixedly connected to the earphone main body 1, and the other end is movably connected to the silica gel conductive electrode head. In the embodiment, the in-ear ear electroacquisition device 2 is a soft conductive silica gel, which can also play a role in fixing the wearing of the ear cover when collecting electroencephalogram, and can reduce the device vibration caused by the user's head movement and limb movement.

[0046] The three near-infrared emitters 4 and the three detection devices 3 are alternately arranged on the earphone main body 1, wherein the alternately arranged near-infrared emitters 4 and detection devices 3 form an arc line, and the adjacent near-infrared emitters 4 and detection devices 3 on the arc line are at a distance fcm, f=3. The near-infrared emitter 4 adopts an smt760 / 850nm dual-wavelength integrated module of Epitex Company. The detection device 3 adopts a photoelectric detection module ADPD2211.

[0047] The ear cover structures on both sides of the earphone main body 1 are the same.

[0048] Further, the wireless communication device is further included.

[0049] The wireless communication device is embedded in the earphone main body 1 and electrically connected to the earphone main body 1.

[0050] Embodiment 2:

[0051] The embodiment further discloses a multi-modal signal acquisition system fusing near-infrared and ear electroencephalogram on the basis of Embodiment 1, comprising: the multi-modal signal acquisition device fusing near-infrared and ear electroencephalogram of Embodiment 1, further comprising:

[0052] a cloud server receiving signals sent by the multi-modal signal acquisition device fusing near-infrared and ear electroencephalogram;

[0053] a data preprocessing module attenuating alternating current and harmonics thereof;

[0054] a feature extraction module extracting electroencephalogram features based on linear and nonlinear electroencephalogram signals and extracting near-infrared features based on light intensity data;

[0055] a feature integration module fusing and aligning all features extracted by the feature extraction module to obtain target multi-modal data.

[0056] Specifically:

[0057] Since the designed system does not limit the activities of users, the electroencephalogram signals are greatly affected by muscle electricity, electrooculogram, baseline drift and power frequency interference, and the near-infrared signals are relatively greatly affected by ambient light and baseline drift. Therefore, the data processing module attenuates the alternating current and harmonics thereof by 5-order Butterworth notch to alternating current. The alternating current refers to alternating current causing power frequency interference; the power frequency interference is mainly caused by 50Hz alternating current in the room, wherein i=5 in the embodiment;

[0058] Furthermore, the feature extraction module comprises: an in-ear ear electroacquisition device 2, a near-infrared emitter and a detection device.

[0059] The earphone ear electro-acquirer 2 acquires ear electro based on the similarity of ear electro and brain electro to obtain brain electro characteristics; in the embodiment, a professional 40-lead brain electro device produced by NeuroScan Company is used as a control, wherein the electrodes of the NeuroScan device are placed on the head of the user according to the standard 10-20 electrode placement, and conductive paste is applied for signal acquisition; after the ear canal is cleaned, the ear-brain electro electrode is inserted into the ear to acquire signals, and part of the signal comparison is shown in Fig. 3, wherein Fig. (a) is the layout of ear-brain electro and NeuroScan electrode, and Fig. (b) is a part of the signal comparison chart. As can be seen, the signal (Neuroscan acquisition) acquired by the channel T8 closest to the right ear canal has great similarity with the ear-brain electro acquired by the right ear canal, and the similarity of the two signals is more than 95.3% on average through calculation, proving that the ear-brain electro has the brain electro signal acquisition effect close to the international standard brain electro, and has effectiveness. The ear-brain electro is proved to have high correlation with the brain electro of the temporal lobe, and the temporal lobe is closely related to emotions and spirits.

[0060] The near-infrared emitter emits near-infrared light, which is absorbed and scattered by the head, and the scattered near-infrared light is detected by the detection device and converted into an electrical signal to obtain near-infrared characteristics.

[0061] The specific steps of the brain electro feature extraction include:

[0062] Extracting energy of ear electro signals in multiple frequency bands: linear and nonlinear ear-brain electro signals are x(n), and the sampling frequency is 250 Hz. A 5th order Butterworth filter bank is applied to x(n) for frequency band filtering, and the frequency bands are B1-B6: [0.5-4 Hz], [4-8 Hz], [8-12 Hz], [12-18 Hz], [16-28 Hz], and [21-48 Hz], and all frequency bands B0 are [0.5-48 Hz], so that 7 ear-brain electro signal energy features in 7 frequency bands are obtained, and there are 7 features;

[0063] Sample entropy features of each channel and each frequency band are obtained based on an algorithm, wherein the sequence u(1), u(2)…u(n) is the data of the brain electro signal divided into the above-mentioned 7 frequency bands. Sample entropy represents the chaos degree between the front and rear parts of a time sequence, and the greater the sample entropy, the more chaotic the data and the more complex the pattern. The specific algorithm is as follows:

[0064] ①Suppose there is an original time sequence u(1), u(2)…u(N) with a length of N, and a m-dimensional vector X m (i) is constructed by using the sequence u(i)

[0065] ②The distance d[X m (i),X m (j)] between any two m-dimensional vectors is defined as

[0066] d[X m (i),X m (j)] = max{u(i+k)-u(j+k)}, where 0≤k≤m-1, 1≤i≤N-m+1, 1≤j≤N-m+1, i≠j.

[0067] ③ Given a distance tolerance r (r>0), when d[X m (i),X m (j)] < r, the two vectors are considered similar to each other, at this time, the number of each calculated distance d[X m (i),X m (j)] is counted and represented by N(i), and the ratio

[0068]

[0069] ④ The average of is calculated;

[0070]

[0071] ⑤ The dimension m is increased by 1, that is, the steps of formulas (8)-(11) are repeated for m+1-dimensional vectors to obtain B m+1 (r), thus the sample entropy of the original sequence under the finite value of the data length is obtained:

[0072]

[0073] The Fourier transform result of each channel signal is calculated, and the spectral entropy is obtained; the spectral entropy can depict the chaos degree of the spectral distribution of data, and the greater the spectral entropy represents the more irregular the spectral distribution of data.

[0074] After the signal x(n) is decomposed into 7 frequency bands B0-B6 by a filter bank, the three parameters of Hjorth of the multi-frequency band signal are extracted;

[0075] The mean and variance of the signal are calculated based on the Teager-Kaiser energy operator:

[0076] The Teager-Kaiser Operator (TKO) is a non-linear energy operator that can determine the instantaneous energy of a non-stationary signal. In addition to energy, TKEO can also track instantaneous amplitude and signal frequency. Hilbert transform (HT) can be used to separate instantaneous frequency and amplitude, but HT is applied in a global range, while local amplitude and frequency fluctuations can be effectively separated by TEKO. For non-stationary signals such as electroencephalogram and near-infrared, TEKO has better tracking ability.

[0077] For continuous signal x(t), TKO is defined as: ψ(x(t)) = x(t) 2 - x'(t) - x"(t);

[0078] For discrete signal x(n) corresponding to x(t), TKO operator is calculated as follows:

[0079] ψ(x(n)) = x 2 (n) - x(n+1) - x(n-1)

[0080] In specific calculation, first, x(n) is subjected to Discrete wavelet transform (DWT) as follows, After obtaining corresponding wavelet transform coefficients, each layer wavelet transform coefficient is processed according to the above discrete TKO operator, and finally the corresponding mean and variance are obtained.

[0081] Among them, the specific steps of near-infrared feature extraction include:

[0082] Removing the influence of physiological activity on near-infrared data, wherein the physiological activity is as shown in the following table;

[0083] Table 1. Physiological activity meaning of each frequency band

[0084]

[0085] The light intensity data item corresponding to the wavelength in the above table is subjected to band-pass filtering of 0.01 Hz-0.3 Hz to reduce physiological noise caused by breathing and heart activity, then the concentration changes of oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) are extracted from the light intensity data using the modified Beer-Lambert law and the average baseline is subtracted. The conversion method is to calculate the blood oxygen concentration based on the modified Lambert-Beer law, and to convert the wavelength data into blood oxygen data by applying the relative amount formula of hemodynamic parameters of Hb and HbO2. At a wavelength of 760 nm, DPF is 7.25. At a wavelength of 850 nm, DPF is 6.38. d is the distance between the detector and the light source, which is 3 cm in this project.

[0086] After obtaining the blood oxygen and deoxygenation data, the peak value, mean value, kurtosis and Hjorth mobility and complexity of HbO2 and Hb are extracted as characteristic values. At the same time, according to the following formula, the correlation coefficient between HbO2 and Hb is calculated as a characteristic value. Wavelet transform is performed, and 4-layer decomposition is performed. Similarly, the peak value, mean value and kurtosis of the wavelet coefficients of each frequency band are calculated as characteristic values. All characteristic values are normalized by subtracting the mean value and dividing by the standard deviation of each characteristic vector (z-score). Outliers are trimmed by setting all values that differ from the mean value by more than three characteristic standard deviations to values that differ from the mean value by only three standard deviations. This is done in order to eliminate any degrading effects of the range of characteristic values on the feature selection process.

[0087] Furthermore, the feature integration module integrates and aligns all the features extracted by the feature extraction module. As shown in Figure 5 the algorithm flowchart for feature integration and alignment of the electroencephalogram features and near-infrared features in the present embodiment.

[0088] For the device disclosed in the embodiments, since it corresponds to the disclosure of the embodiments, the description is relatively simple, and the relevant part can be referred to the part of the description.

[0089] The above description of the disclosed embodiments enables those skilled in the art to carry out or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal signal acquisition system integrating near-infrared and electroencephalography (EEG) signals, characterized in that, include: The multimodal signal acquisition device also includes: The cloud server receives signals sent by the multimodal signal acquisition device; The data preprocessing module attenuates alternating current and its harmonics; The feature extraction module performs EEG feature extraction based on linear and nonlinear EEG signals; and extracts near-infrared features based on light intensity data. The feature integration module fuses and aligns all features extracted by the feature extraction module to obtain target multimodal data; The steps of EEG feature extraction include: extracting the energy of ear signals in multiple frequency bands: linear and nonlinear ear-brain signals are represented by x(n), with a sampling frequency of 250Hz. A 5th-order Butterworth filter bank is applied to filter x(n) by frequency bands, which are divided into B1 to B6: [0.5-4Hz], [4-8Hz], [8-12Hz], [12-18Hz], [16-28Hz], [21-48Hz], and the entire frequency band B0 [0.5-48Hz], resulting in the energy features of ear-brain signals in 7 frequency bands; Calculate the sample entropy features for each channel and frequency band, where the sequences u(1), u(2)...u(n) are the data of the EEG signal divided into the above 7 frequency bands, and the sample entropy represents the degree of disorder between the preceding and following parts of the time series; Calculate the Fourier transform of each channel signal and obtain the spectral entropy; the spectral entropy can characterize the disorder of the spectral distribution of the data, and the larger the entropy, the more irregular the spectral distribution of the data. After decomposing the signal x(n) into 7 frequency bands B0 to B6 using a filter bank, the three parameters of Hjorth of the signal multi-band are extracted. The mean and variance of the signal x(n) are obtained based on the Teager-Kaiser energy operator; The multimodal signal acquisition device includes an earphone body (1), an in-ear electroencephalogram (EEG) collector (2), a near-infrared transmitter (4), and a detection device (3); The in-ear electroencephalogram (EEG) acquisition device collects EEG data based on the similarity between EEG and EEG to obtain EEG characteristics; the near-infrared emitter emits near-infrared light, which is absorbed and scattered by the head, and the scattered near-infrared light is detected by the detection device and converted into an electrical signal to obtain near-infrared characteristics.

2. The multimodal signal acquisition system integrating near-infrared and electroencephalogram (EEG) signals according to claim 1, characterized in that, The headphone body (1) is an earcup structure; The in-ear electroacoustic sensor (2) is fixedly connected to the inner wall of the earcup of the headphone body (1), and the in-ear electroacoustic sensor (2) is adapted to the human ear; N near-infrared emitters (4) are installed at the contact point between the earcups and the scalp of the headphone body (1); m of the detection devices (3) are installed at the contact point between the earcups and the scalp of the headphone body (1).

3. The multimodal signal acquisition system integrating near-infrared and electroencephalogram (EEG) signals according to claim 1, characterized in that, The in-ear electrocoagulation (ECG) device (2) includes: a deformable support and a silicone electrode head; One end of the deformable support is fixedly connected to the headphone body (1); The other end is movably connected to the silicone conductive electrode head.

4. The multimodal signal acquisition system integrating near-infrared and electroencephalogram (EEG) signals according to claim 3, characterized in that, The near-infrared emitter (4) and the detection device (3) are arranged alternately on the headphone body (1); wherein the alternatingly arranged near-infrared emitter (4) and the detection device (3) form an arc; the distance between adjacent near-infrared emitters (4) and the detection device (3) on the arc is fcm.

5. A multimodal signal acquisition system integrating near-infrared and electroencephalogram (EEG) signals according to claim 1, characterized in that, The two earcups of the headphone body (1) have the same structure.

6. The multimodal signal acquisition system integrating near-infrared and electroencephalogram (EEG) signals according to claim 1, characterized in that, It also includes wireless communication devices; The wireless communication device is embedded in the headphone body (1) and is electrically connected to the headphone body (1).

7. A multimodal signal acquisition system integrating near-infrared and electroencephalogram (EEG) signals according to claim 1, characterized in that, The data preprocessing module attenuates the alternating current and its harmonics by applying an i-th order Butterworth notch filter to the alternating current.

Images