Multi-view collaborative ensemble classification method and system for electroencephalogram signals

Abstract

The invention discloses an electroencephalogram signal multi-view collaborative ensemble classification method and system, mainly relates to the technical field of electroencephalogram signals, and is used for solving the problems that when an EEG signal of a sensor domain is mapped to a source space in an existing source imaging technology, the data volume and redundant information are increased, and the imaging quality is poor. And the problems that the online processing time of a brain signal processing system is prolonged and the decoding precision of a source signal is reduced are solved. Comprising the following steps: calculating a central dipole from a source dipole to obtain a circular sub-region; taking the mapped signal in the circular sub-region as a to-be-processed source signal; calculating to obtain a three-dimensional geometric feature corresponding to the source signal; extracting a feature vector corresponding to each to-be-processed source signal; obtaining a first classification result of the geometric features; obtaining a second classification result of the feature vectors; carrying out weighted fusion on the three-dimensional geometric features and the feature vectors, and obtaining a third classification result of the fused data; and inputting the three classification results into a majority voting mechanism to obtain a final classification result.

Description

Technical Field

[0001] The present application relates to the field of electroencephalogram (EEG) signal technology, and in particular to a method and system for collaborative integration and classification of EEG signals from multiple perspectives. Background Art

[0002] Brain Computer Interface (BCI) is an interactive communication system that directly connects the brain to external devices. BCI systems usually consist of three parts: brain signal acquisition, signal decoding, and output control. Common brain signal acquisition methods for BCI systems include electroencephalogram (EEG), magnetoencephalography (MEG), electrocorticography (ECoG), and magnetic resonance imaging (MRI). Among them, EEG has attracted widespread attention due to its advantages such as easy mobility, low cost, and non-invasiveness. EEG-based BCI systems have achieved significant applications in many fields, especially in continuous control of robotic arms, drone target search, and smart wheelchairs, providing practical help for people with special needs.

[0003] Since EEG is detected through the scalp, it records the aliased potential formed outside the skull after the activity of the intracranial nerve cluster diffuses through the brain tissue, cerebrospinal fluid, skull, scalp and other body volume conductor tissues. At the same time, the spatial resolution of EEG is at the centimeter level. Compared with MRI, MEG, etc., its spatial resolution is poor and it is impossible to accurately locate every activated neuron. These factors limit the development of EEG-BCI systems. Electrophysiological source imaging (ESI) has been proven to effectively solve the problems of EEG's volume conduction effect and poor spatial resolution.

[0004] However, when ESI technology maps EEG signals in the sensor domain to the source space, the amount of data increases significantly from 32 to 256 scalp electrode signals to 5,000 to 10,000 source signals. At the same time, ESI's inverse problem-solving algorithm relies on mathematical models and assumptions, which often produce multiple possible solutions, resulting in redundant information in the source signal. The challenges brought by ESI technology, such as the huge amount of data and redundant information, have led to an increase in the online processing time of the brain signal processing system and a decrease in the decoding accuracy of the source signal. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present application provides a multi-perspective collaborative integrated classification method and system for EEG signals to solve the problem that the existing ESI technology causes an increase in data volume when mapping EEG signals in the sensor domain to the source space. In addition, the inverse problem solving algorithm of ESI will produce multiple possible solutions, which in turn leads to an increase in the online processing time of the brain signal processing system and a decrease in the decoding accuracy of the source signal.

[0006] In a first aspect, the present application provides a method for multi-perspective collaborative integration classification of EEG signals, the method comprising: Collect EEG signals generated by the same subject performing a running imagination task and head MRI signals in a resting state; process the MRI signals into two-dimensional slice images; obtain the pre-processed EEG signals through common average reference and frequency domain filtering technology; The head model is obtained by using the forward problem processing mechanism and two-dimensional slice diagram of ESI technology; wherein the head model includes a three-layer structure, and the three-layer structure is scalp, skull and cortex from outside to inside; The cortex of the head model is divided to obtain a source model containing a number of grid points, wherein each grid point corresponds to a source dipole; the potential generated by each dipole at the scalp electrode in three directions of space is calculated, and then the lead field matrix is ​​obtained; Using the inverse problem processing mechanism and lead field matrix of the ESI technology, the pre-processed EEG signal of the current subject's cerebral scalp is mapped to the cortex of the head model as a cortical mapping signal; Based on the coordinates of the subject's scalp electrodes, the central dipole is calculated from the source dipole to obtain a circular sub-region; the cortical mapping signal in the circular sub-region is used as the source signal to be processed; The source signal to be processed is converted into a 3D SODP graph, and then the three-dimensional geometric features of the 3D SODP graph are calculated; the feature vector corresponding to each source signal to be processed is extracted using the CSP method; The three-dimensional geometric features are input into the first classifier corresponding to 3D SODP to obtain the first classification result; the feature vector is input into the second classifier corresponding to the CSP method to obtain the second classification result; the three-dimensional geometric features and feature vectors are weighted fused, and the fused data is input into the third classifier to obtain the third classification result; the first classification result, the second classification result and the third classification result are input into the majority voting mechanism to obtain the final classification result.

[0007] In one implementation of the present application, the number of preset sampling points of the EEG signal is n, and the number of acquisition channels is N; Collect EEG signals generated by the same subject performing a running imagination task, including: Collection i The EEG signal of the channel;i The EEG signal of each channel is expressed as , i ∈[1,N]; Process the MRI signal into a two-dimensional slice image, including: Acquire MRI signals; wherein the MRI signals are three-dimensional MRI images in the format of A*B*C, where A represents pixels of the image in the left-right direction, B represents pixels of the image in the front-back direction, and C represents pixels of the image in the up-down direction. The sagittal plane, coronal plane, and transverse plane of the three-dimensional MRI image correspond to three directions perpendicular to the X-axis, the Y-axis, and the Z-axis, respectively; By slicing along the X-axis, Y-axis and Z-axis directions, a two-dimensional slice image corresponding to the MRI signal is obtained.

[0008] In one implementation of the present application, the pre-processed EEG signal is obtained by using common mean reference and frequency domain filtering technology, specifically including: The formula corresponding to the common average reference is: , filter out the common mode components in the EEG signal; in, Indicates i The signal after the average reference of the channels is i ∈[1,N], N represents the number of acquisition channels, Indicates i The signal of each channel; Then, the common average reference signal was band-pass filtered in the 8-30 Hz frequency band through a fourth-order Butterworth band-pass filter to obtain the preprocessed EEG signal.

[0009] In one implementation of the present application, the cortex of the head model is divided to obtain a source model containing several grid points, specifically including: using FreeSurfer software and MNE-python library to process the two-dimensional slice image, and using the boundary element model to generate a three-shell volume conduction model with a conductivity ratio of 1:1 / 50:1 as the head model; wherein the head model includes a three-layer structure, and the three-layer structure is scalp, skull and cortex from the outside to the inside; the cortical surface of the three-layer structure is evenly divided to obtain a source model containing 20484 grid points; wherein each grid point corresponds to a source dipole.

[0010] In one implementation of the present application, the inverse problem processing mechanism and lead field matrix of the ESI technology are used to map the preprocessed EEG signal of the current subject's cerebral scalp to the cortex of the head model, specifically including: using the sLORETA method to map the preprocessed EEG signal of the current subject to the cortex of the head model; wherein the regularization parameter is set to 1.

[0011] In one implementation of the present application, based on the coordinates of the subject's scalp electrodes, the center dipole is calculated from the source dipole to obtain the circular sub-area, specifically including: The coordinates of the scalp electrodes are obtained; the source dipole closest to the scalp electrodes is calculated as the central dipole; and a circular sub-area is obtained based on a preset radius and centered on the central dipole.

[0012] In one implementation of the present application, the geometric features include at least: the sum of vector lengths, the sum of vector angles, the sum of distances to the origin, the sum of distances to the major axis, and the sum of distances to the minor axis; Converting the source signal to be processed into a 3D SODP map, and then calculating the three-dimensional geometric features of the 3D SODP map, specifically including: obtaining the central dipoles corresponding to the electrodes located in the left central area of ​​the cerebral scalp, the electrodes in the right central area of ​​the cerebral scalp, and the electrodes in the central area of ​​the midline of the cerebral scalp, and drawing a 3D SODP map for the source signal of the circular sub-area corresponding to the central dipole to obtain the three-dimensional geometric features of the source signal; By the vector length sum formula: , calculate the sum of vector lengths SVL; in, , and Indicates 3D SODP j The three-dimensional coordinate values ​​of the sample points, T is the total number of 3D SODP sample points; The total formula for the vector angle is: , calculate the sum of the vector angles SA; in,( , , )=( , , ), ( , , )=( , , ); By the formula of the sum of distances to the origin: , calculate the total distance to the origin SDTO; The formula for the sum of the distances to the major axis is: , Calculate the sum of distances to the major axis SDTMA; in,( , , )=[ MajorAxis , , ]* MajorAxis, , MajorAxis=( , , ),( , , ) represents the point with the largest Z coordinate in 3D SODP, ( , , ) represents the point with the smallest Z coordinate in 3D SODP, and the symbol · represents vector dot product; The sum of the distances to the minor axis is calculated by: , calculate the sum of the distances to the minor axis ; in,( , , )=[ MinorAxis , , ]* MinorAxis, , a, b, c are a, b, c in the plane equation ax+by+cz=0 of the fitting plane involved in the 3D SODP diagram.

[0013] In one implementation of the present application, the CSP method is used to extract the feature vectors corresponding to each source signal to be processed, specifically including: A CSP projection matrix is ​​obtained; a preset number of row vectors are selected from the CSP projection matrix to form a spatial filter, and then a matrix multiplication of the source signal to be processed and the spatial filter is performed to obtain a eigenvector; wherein the number of the eigenvectors is the same as the preset number.

[0014] In one implementation of the present application, before inputting the three-dimensional geometric features into a first classifier corresponding to the 3D SODP to obtain a first classification result; inputting the feature vector into a second classifier corresponding to the CSP method to obtain a second classification result; performing weighted fusion on the three-dimensional geometric features and the feature vector, and inputting the fused data into a third classifier to obtain a third classification result, the method further includes: Input the three-dimensional geometric features of the labeled imagination task into several classifiers, determine the classifier with the highest classification accuracy as the first classifier, and record the accuracy acc1 of the first classifier; input the feature vectors of the labeled imagination task into several classifiers, determine the classifier with the highest classification accuracy as the second classifier, and record the accuracy acc2 of the second classifier; By formula: , calculate the weight weight1 of the three-dimensional geometric features during weighted fusion; By formula: , calculate the weight weight2 of the feature vector during weighted fusion; Multiply the 3D geometric features by weight weight1, multiply the feature vector by weight2, and then serially concatenate the two to obtain fused data. Input the fused data with labeled imagination tasks into several classifiers, and determine the classifier with the highest classification accuracy as the third classifier.

[0015] In a second aspect, the present application provides a multi-perspective collaborative integrated classification system for EEG signals, the system comprising: A processing module is used to collect EEG signals generated by the same subject performing the running imagination task and MRI signals in a resting state; process the MRI signals into two-dimensional slice images; and obtain the pre-processed EEG signals through common average reference and frequency domain filtering technology; The ESI module is used to obtain a head model by using the forward problem processing mechanism and two-dimensional slice diagram of the ESI technology; wherein the head model includes a three-layer structure, and the three-layer structure is scalp, skull and cortex from the outside to the inside; the cortex of the head model is divided to obtain a source model including a number of grid points; wherein each grid point corresponds to a source dipole; the potential generated by each dipole at the scalp electrode in three directions of space is calculated, and then a lead field matrix is ​​obtained; the inverse problem processing mechanism and lead field matrix of the ESI technology are used to map the preprocessed EEG signal of the scalp of the current subject to the cortex of the head model as a cortical mapping signal; The data extraction module is used to calculate the central dipole from the source dipole based on the coordinates of the subject's scalp electrodes, thereby obtaining a circular sub-region; using the cortical mapping signal in the circular sub-region as the source signal to be processed; converting the source signal to be processed into a 3D SODP map, thereby calculating the three-dimensional geometric features of the 3D SODP map; and using the CSP method to extract the feature vector corresponding to each source signal to be processed; The classification module is used to input the three-dimensional geometric features into the first classifier corresponding to the 3D SODP to obtain the first classification result; input the feature vector into the second classifier corresponding to the CSP method to obtain the second classification result; perform weighted fusion of the three-dimensional geometric features and the feature vector, input the fused data into the third classifier to obtain the third classification result; input the first classification result, the second classification result and the third classification result into the majority voting mechanism to obtain the final classification result.

[0016] Those skilled in the art can understand that the present application has at least the following beneficial effects: This application extracts the features of the source signal through multi-view features (3D SODP+CSP+weighted fusion), and maximizes the use of features through collaborative integrated classification. While ensuring high accuracy and strong robustness of the classification results, the online processing time is reduced; the central dipole is calculated from the source dipole based on the coordinates of the subject's scalp electrodes, and then the circular sub-region is obtained; because the solution generated by the ESI inverse problem solving algorithm is limited to the circular sub-region, the data volume and redundant information of the source signal are reduced, and the problem of increased online processing time of the brain signal processing system and reduced decoding accuracy of the source signal is avoided. In addition, this application adopts a weighted fusion method, which fully integrates the 3D SODP dynamic change features (three-dimensional geometric features) and CSP spatial feature information (feature vectors), and comprehensively considers the dynamic change characteristics and spatial characteristics of the source signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0018] Figure 1 This is a flow chart of a multi-perspective collaborative integrated classification method for EEG signals provided in an embodiment of the present application.

[0019] Figure 2 It is a schematic diagram of a sagittal plane of a two-dimensional slice corresponding to a three-dimensional image involved in a multi-view collaborative integrated classification method for EEG signals provided in an embodiment of the present application.

[0020] Figure 3 It is a schematic diagram of a coronal plane of a two-dimensional slice corresponding to a three-dimensional image involved in a multi-view collaborative integrated classification method for EEG signals provided in an embodiment of the present application.

[0021] Figure 4 It is a schematic diagram of a two-dimensional slice cross-section corresponding to a three-dimensional image involved in a multi-view collaborative integrated classification method for EEG signals provided in an embodiment of the present application.

[0022] Figure 5 This is a schematic diagram of a vector length sum feature provided in an embodiment of the present application.

[0023] Figure 6 It is a schematic diagram of a vector angle sum feature provided in an embodiment of the present application.

[0024] Figure 7 This is a characteristic diagram of the sum of distances to the origin provided in an embodiment of the present application.

[0025] Figure 8 This is a characteristic schematic diagram of the sum of distances to the long axis provided in an embodiment of the present application.

[0026] Fig. 9 This is a characteristic schematic diagram of the sum of distances to the short axis provided in an embodiment of the present application.

[0027] Fig.10 It is a schematic diagram of an ellipse area characteristic provided in an embodiment of the present application.

[0028] Fig.11 This is a schematic diagram of standard deviation characteristics in two directions provided in an embodiment of the present application.

[0029] Fig.12 This is a characteristic diagram of the sum of distances to a 45-degree line provided in an embodiment of the present application.

[0030] Fig.13 This is a characteristic schematic diagram of the sum of distances to a 135-degree line provided in an embodiment of the present application.

[0031] Fig.14 It is a schematic diagram of a central tendency measurement feature provided in an embodiment of the present application.

[0032] Fig.15 It is a schematic diagram of a 3DS ODP diagram involved in a multi-view collaborative integrated classification method for EEG signals provided in an embodiment of the present application.

[0033] Fig.16 This is a schematic diagram of the projection of the fitting plane of the 3DS ODP image involved in the multi-view collaborative integrated classification method of EEG signals provided in the embodiment of the present application. Fig.17 It is a schematic diagram of a method for performing feature weighted fusion of three-dimensional geometric features and feature vectors provided in an embodiment of the present application.

[0034] Fig.18 It is a schematic diagram of a multi-perspective collaborative integrated classification method for left and right hand EEG signals provided in an embodiment of the present application.

[0035] Fig.19It is a schematic diagram of the internal structure of a multi-perspective collaborative integrated classification system for EEG signals provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] It should be understood by those skilled in the art that the embodiments described below are only preferred embodiments of the present disclosure, and do not mean that the present disclosure can only be implemented through the preferred embodiments. The preferred embodiments are only used to explain the technical principles of the present disclosure, and are not used to limit the protection scope of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should still fall within the protection scope of the present disclosure.

[0037] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0038] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0039] The embodiment provides a multi-view collaborative integration classification method for EEG signals, such as Figure 1 As shown, the method provided in the embodiment of the present application mainly includes the following steps: Step 110, collecting EEG signals generated by the same subject performing the running imagination task and the head MRI signals in the resting state; processing the MRI signals into two-dimensional slice images; and obtaining the preprocessed EEG signals through common mean reference and frequency domain filtering technology.

[0040] It should be noted that the running imagination task to be performed may be imagining left hand movement or right hand movement, the preset number of sampling points of the EEG signal is n, and the number of acquisition channels is N.

[0041] In some embodiments, the EEG signal generated by the same subject performing the running imagination task is collected, which can be specifically: Collection i The EEG signal of the channel; i The EEG signal of each channel is expressed as , i ∈[1,N].

[0042] The MRI signal is processed into a two-dimensional slice image, which can be specifically: MRI signals are acquired (each subject undergoes head magnetic resonance imaging on an MRI scanner in a medical institution to obtain a three-dimensional MRI image. During the acquisition process, the subject needs to close his eyes as much as possible and lie quietly, but cannot fall asleep); wherein the MRI signal is a three-dimensional MRI image in the format of A*B*C, where A represents the pixels of the image in the left-right direction, B represents the pixels of the image in the front-back direction, and C represents the pixels of the image in the up-down direction. The sagittal plane, coronal plane, and cross-section of the three-dimensional MRI image correspond to the three directions perpendicular to the X-axis, Y-axis, and Z-axis, respectively; by slicing along the X-axis, Y-axis, and Z-axis directions, a two-dimensional slice image corresponding to the MRI signal is obtained (including: sagittal plane, coronal plane, cross-section, respectively; specifically, as Figure 2 The schematic diagram of the sagittal plane composed of the X-axis and the Y-axis is shown in FIG. Figure 3 The schematic diagram of the coronal plane composed of the X-axis and the Z-axis is shown in FIG. Figure 4 Schematic diagram of the cross section consisting of the Y-axis and the Z-axis).

[0043] Among them, the pre-processed EEG signal is obtained through the common average reference and frequency domain filtering technology, which can be specifically: The formula corresponding to the common average reference is: , filter out the common mode components in the EEG signal; in, Indicates i The signal after the average reference of the channels is i ∈[1,N], N represents the number of acquisition channels, Indicates i The signal of each channel; Then, the common average reference signal was band-pass filtered in the 8-30 Hz frequency band through a fourth-order Butterworth band-pass filter to obtain the preprocessed EEG signal.

[0044] It should be noted that the event-related desynchronization (ERD) and event-related synchronization (ERS) phenomena related to motor imagery are mainly concentrated in the frequency band of 8-30 Hz. Therefore, in order to remove irrelevant information in other frequency bands, this application uses a fourth-order Butterworth bandpass filter to perform bandpass filtering on the signal after common average reference in the 8-30 Hz frequency band.

[0045] Step 120 , using the forward problem processing mechanism and two-dimensional slice diagram of the ESI technology to obtain a head model.

[0046] It should be noted that the head model includes a three-layer structure, and the three-layer structure is scalp, skull and cortex from outside to inside.

[0047] This application uses FreeSurfer software and MNE-python library to process two-dimensional slice images, and uses the boundary element model (BEM) to generate a three-shell volume conduction model (head model) with a conductivity ratio of 1:1 / 50:1. The three-layer structure of the model is scalp, skull and cortex from outside to inside.

[0048] Step 130: divide the cortex of the head model to obtain a source model including a number of grid points; calculate the potential generated by each dipole at the scalp electrode in three directions in space, and then obtain the lead field matrix.

[0049] It should be noted that each grid point corresponds to a source dipole.

[0050] The cortex of the head model is divided to obtain a source model containing a number of grid points, which can be specifically: This application evenly divides the cortical surface of the head model and obtains a source model containing 20484 grid points, each grid point corresponds to a source dipole. Finally, the potential generated by each dipole on the scalp electrode in three directions of space is calculated to obtain the lead field matrix , M represents the number of EEG electrodes, and D represents the number of dipoles.

[0051] Step 140: Using the inverse problem processing mechanism of the ESI technology and the lead field matrix, the preprocessed EEG signal of the scalp of the current subject is mapped to the cortex of the head model as a cortical mapping signal.

[0052] This step can be specifically as follows: The preprocessed EEG signal of the current subject is mapped to the cortex of the head model of the current subject using the sLORETA method; wherein the regularization parameter is set to 1.

[0053] Step 150: Based on the coordinates of the subject's scalp electrodes, a central dipole is calculated from the source dipole to obtain a circular sub-region; and the cortical mapping signal in the circular sub-region is used as a source signal to be processed.

[0054] It should be noted that the traditional area division method is to simply subdivide the existing brain partitions to create smaller sub-partitions. However, this method does not take into account the position of the scalp electrodes, and the aliasing relationship between the scalp electrode signals and the signals of many source dipoles, resulting in insufficient accuracy of the created regions. In order to solve these problems, this step adopts a regional division method that combines the electrode position and the dipole relationship. This method takes the dipole closest to the scalp electrode as the center and expands outward to form a circular area, thereby establishing a more accurate circular sub-region. In addition, it should be supplemented that there can be several circular sub-regions, and those skilled in the art can determine the specific number and the radius length of each circular sub-region according to actual needs. As an example, two circular sub-regions can be set, and the radius lengths can be 5 mm and 10 mm respectively.

[0055] Among them, based on the coordinates of the subject's scalp electrodes, the center dipole is calculated from the source dipole, and then the circular sub-area is obtained, which can be specifically: The coordinates of the scalp electrodes are obtained; the source dipole closest to the scalp electrodes is calculated as the central dipole; and a circular sub-area is obtained based on a preset radius and centered on the central dipole.

[0056] It should be noted that the coordinates of the scalp electrodes have been obtained in the initial stage, and the scheme for calculating the source dipole closest to the scalp electrode position is an existing scheme for calculating the distance between two coordinates.

[0057] Step 160: Convert the source signal to be processed into a 3D SODP graph, and then calculate and obtain the three-dimensional geometric features of the 3D SODP graph; and use the CSP method to extract the feature vector corresponding to each source signal to be processed.

[0058] It should be noted that feature extraction is a key step in successfully decoding the source signal. Therefore, the present application comprehensively extracts features from multiple perspectives (3D SODP+CSP+weighted fusion). After determining the circular sub-region, the present application first obtains the source signal in the region as the source signal to be processed, and then extracts features from three different perspectives: first, 3DSODP is used to capture the dynamic change characteristics (three-dimensional geometric features) of the source signal; second, the common space mode is used to extract the spatial features (feature vectors) of the source signal; third, the three-dimensional geometric features and feature vectors are fused to obtain the fusion process, see step 170.

[0059] The geometric features include at least: the sum of vector lengths (the feature presentation form is as follows Figure 5 As shown), the sum of the vector angles (the characteristic display form is as follows Figure 6 As shown in ), the sum of the distances to the origin (the feature display form is as Figure 7as shown), the sum of the distances to the major axis (the feature display form is as Figure 8 as shown), the sum of the distances to the short axis (the feature display form is as Fig. 9 as shown), ellipse area (characteristic display form as Fig.10 ), standard deviation in two directions (feature display form as Fig.11 As shown), the sum of the distances to the 45-degree line (the feature display form is as follows Fig.12 As shown), the sum of the distances to the 135 degree line (the feature display form is as follows Fig.13 as shown), measures of central tendency (characteristic presentation as shown Fig.14 as shown).

[0060] 3D SODP is used to capture the three-dimensional features of the source signal, and then the geometric features of the three-dimensional space corresponding to the source signal are calculated, which can be specifically: The central dipoles corresponding to the electrodes located in the central area of ​​the left side of the cerebral scalp, the electrodes in the central area of ​​the right side of the cerebral scalp, and the electrodes in the central area of ​​the midline of the cerebral scalp are obtained, and a 3D SODP diagram is drawn for the source signal of the circular sub-area corresponding to the central dipole to obtain the three-dimensional geometric characteristics of the source signal.

[0061] More specifically, By the vector length sum formula: , calculate the sum of vector lengths SVL; in, , and Indicates 3D SODP j The three-dimensional coordinate values ​​of the sample points, T is the total number of 3D SODP sample points.

[0062] The total formula for the vector angle is: , calculate the sum of the vector angles SA; in,( , , )=( , , ), ( , , )=( , , ).

[0063] By the formula of the sum of distances to the origin: , calculate the total distance to the origin SDTO.

[0064] The formula for the sum of the distances to the major axis is: .

[0065] Calculate the sum of distances to the major axis SDTMA; in,( , , )=[ MajorAxis , , ]* MajorAxis, , MajorAxis=( , , ),( , , ) represents the point with the largest Z coordinate in 3D SODP, ( , , ) represents the point with the smallest Z coordinate in 3D SODP, and the symbol · represents the vector dot product.

[0066] The sum of the distances to the minor axis is calculated by: , calculate the sum of the distances to the minor axis ; in,( , , )=[ MinorAxis , , ]* MinorAxis, , a, b, c are a, b, c in the plane equation ax+by+cz=0 of the fitting plane involved in the 3D SODP diagram; where the 3D SODP diagram, such as Fig.15 As shown (unit: microvolt); the projection diagram of the fitting plane corresponding to the 3D SODP diagram is as follows Fig.16 shown (in microvolts).

[0067] Using the ellipse area formula: , calculate the ellipse area EA; in, , , , , , , , It represents the j-th pixel point of the 3D SODP map corresponding to the fitting plane, and π is the circumference of a circle.

[0068] The formula for standard deviation in two directions is: , calculate the standard deviation TDSD in two directions; in, , , Represents the vector formed by the horizontal axis of all sample points, Represents the vector formed by the ordinate axes of all sample points, and Var represents the variance.

[0069] The formula for the sum of distances to the 45-degree line is: , calculate the sum of the distances from the data points to the 45-degree line SDT45 in 3D SODP.

[0070] The formula for the sum of distances to the 135 degree line is: , calculate the sum of the distances from the data points to the 135 degree line SDT135 in 3D SODP.

[0071] Through the central tendency measure formula: , calculate the central tendency measure; in, , , r represents the preset radius.

[0072] In the step, the CSP method is used to extract the feature vectors corresponding to each source signal to be processed, which can be specifically: Get the CSP projection matrix; A preset number of row vectors are selected from the CSP projection matrix to form a spatial filter, and then the source signal to be processed is matrix multiplied with the spatial filter to obtain a eigenvector; wherein the number of eigenvectors is the same as the preset number.

[0073] Step 170: input the three-dimensional geometric features into the first classifier corresponding to 3D SODP to obtain a first classification result; input the feature vector into the second classifier corresponding to the CSP method to obtain a second classification result; perform weighted fusion on the three-dimensional geometric features and the feature vector, input the fused data into the third classifier to obtain a third classification result; input the first classification result, the second classification result and the third classification result into the majority voting mechanism to obtain a final classification result.

[0074] It should be noted that the classifiers include Logistic Regression (LR), K-Nearest Neighbors (KNN), Decision Tree (DT), Support Vector Machine (SVM), Naïve Bayes (NB) and Multilayer Perceptron (MLP). In addition, the present application takes into account that 3D SODP is superior in capturing the dynamic change characteristics of source signals, while CSP can effectively extract the spatial characteristics of the source signals to be processed, and both have their own advantages. In order to further enhance the effectiveness of the features, the present application performs weighted fusion of 3DSODP features and CSP-based spatial features to obtain a new feature perspective.

[0075] The method further includes selecting a classifier before inputting the three-dimensional geometric features into a first classifier corresponding to 3D SODP to obtain a first classification result; inputting the feature vector into a second classifier corresponding to the CSP method to obtain a second classification result; and performing weighted fusion on the three-dimensional geometric features and the feature vector, and inputting the fused data into a third classifier to obtain a third classification result. The selection process may be: Input the three-dimensional geometric features of the labeled imagination task into several classifiers, determine the classifier with the highest classification accuracy as the first classifier, and record the accuracy acc1 of the first classifier; Input the feature vector of the labeled imagination task into several classifiers, determine the classifier with the highest classification accuracy as the second classifier, and record the accuracy acc2 of the second classifier; By formula: , calculate the weight weight1 of the three-dimensional geometric features during weighted fusion; By formula: , calculate the weight weight2 of the feature vector during weighted fusion; Weighted fusion schemes such as Fig.17 As shown, the three-dimensional geometric features are multiplied by weight weight1, the feature vector is multiplied by weight2, and then the two are serially spliced ​​to obtain fused data. The fused data with labeled imagination tasks are input into several classifiers, and the classifier with the highest classification accuracy is determined to be the third classifier.

[0076] It will be appreciated by those skilled in the art that there is a better fit between different feature vectors and specific classifiers. The present application performs feature extraction on the source signal from three different perspectives, wherein the fused data can effectively combine the advantages of 3D SODP features and spatial features, thereby enhancing the information content of the features. However, in the process of feature fusion, due to the high correlation between the sub-features, this fusion sometimes leads to redundancy, thereby covering up some subtle differences. Therefore, the fused data cannot completely replace the 3D SODP features and spatial features. The features of the above three perspectives each have their own advantages, and making full use of these features will help improve the classification effect of the signal. Based on this, the present application first selects the most suitable sub-classifier for each feature, and then combines the multi-perspective features with the most suitable classifier for each, and determines the final classification result through a voting mechanism.

[0077] Based on the above description, the embodiment of the present application provides a multi-view collaborative integrated classification method for left and right hand EEG signals, such as Fig.18 As shown, specifically: Signal acquisition: Signal acquisition is divided into EEG signal acquisition and MRI signal acquisition, and each acquisition must come from the same subject.

[0078] EEG signal acquisition is as follows: each subject needs to use an EEG signal acquisition device to collect EEG signals. During the process, the subject needs to perform different motor imagination tasks, such as imagining left hand movement or right hand movement. MRI signal acquisition is as follows: each subject undergoes head magnetic resonance imaging on an MRI scanner in a medical institution to obtain a three-dimensional MRI image, which is then processed into a two-dimensional slice image. During the acquisition process, the subject needs to close his eyes as much as possible and lie quietly, but cannot fall asleep.

[0079] EEG signal preprocessing: Due to its inherent randomness, EEG is easily interfered by external factors, thus affecting the accuracy of the analysis results. Therefore, in order to obtain more reliable analysis results, this application uses common average reference and frequency domain filtering (using a fourth-order Butterworth bandpass filter to perform bandpass filtering in the 8-30 Hz frequency band on the signal after common average reference), and it is particularly important to preprocess EEG.

[0080] ESI technology is processing the two-dimensional slice diagram of the problem: The structural MRI data (two-dimensional slices) were processed using FreeSurfer software and the MNE-python library, and a three-shell volume conduction model (head model) with a conductivity ratio of 1:1 / 50:1 was generated using the boundary element model (BEM). The cortical surface was evenly divided to obtain a source model containing 20,484 grid points. The potential generated by each dipole at the scalp electrode in three directions of space was calculated to obtain the lead field matrix (the calculation process can be implemented by existing technologies and is not limited in this application).

[0081] ESI technology processes EEG signals through the inverse problem: Since the number of sources is much larger than the number of EEG electrodes, and there are countless in vivo source distributions that can produce the same in vitro potential, the solution for inferring the source signal from the scalp EEG signal is highly underdetermined and non-unique. This application uses the sLORETA method to reconstruct the source of the EEG signal; wherein the regularization parameter is set to 1.

[0082] Furthermore, in order to reduce the amount of calculation and the complexity of calculation, the present application uses a division method that combines the electrode position and the dipole relationship. The steps of the division method used can be specifically as follows: Obtain the coordinates of the scalp electrodes; The source dipole closest to the scalp electrode position was calculated as the central dipole; With the central dipole as the center, circular sub-regions with different radii are expanded outward.

[0083] Based on the circular sub-region, multi-view feature extraction is performed: On the one hand, 3D SODP is used to capture the dynamic change characteristics (three-dimensional geometric features) of the source signal (signal in the circular sub-area), and the three-dimensional geometric features are input into the first classifier to obtain the first classification result (left hand or right hand); on the other hand, the common space mode is used to extract the feature vector of the source signal, and the feature vector is input into the second classifier to obtain the second classification result (left hand or right hand). In order to further enhance the effectiveness of the features, the dynamic change characteristics and spatial characteristics are integrated, and the features obtained by these two methods are weighted fused, and the fused features are input into the third classifier to obtain the third classification result (left hand or right hand); the features of the third angle are obtained.

[0084] The three results are fed into a majority voting mechanism to obtain the final result (left hand or right hand).

[0085] In addition, this application Fig.19 A multi-view collaborative integrated classification system for EEG signals is provided in the embodiment of the present application. Fig.19 As shown, the system provided in the embodiment of the present application mainly includes: The processing module 210 is used to collect the EEG signal generated by the same subject performing the running imagination task and the MRI signal in the resting state; process the MRI signal into a two-dimensional slice image; and obtain the pre-processed EEG signal through the common average reference and frequency domain filtering technology.

[0086] The ESI module 220 is used to obtain a head model using the forward problem processing mechanism and two-dimensional slice diagram of the ESI technology; wherein the head model includes a three-layer structure, and the three-layer structure is scalp, skull and cortex from the outside to the inside; the cortex of the head model is divided to obtain a source model containing a plurality of grid points; wherein each grid point corresponds to a source dipole; the potential generated by each dipole at the scalp electrode in three directions in space is calculated, and then a lead field matrix is ​​obtained; and the inverse problem processing mechanism and lead field matrix of the ESI technology are used to map the preprocessed EEG signal of the scalp of the current subject to the cortex of the head model as a cortical mapping signal.

[0087] The data extraction module 230 is used to calculate the central dipole from the source dipole based on the coordinates of the subject's scalp electrodes, and then obtain the circular sub-area; use the cortical mapping signal in the circular sub-area as the source signal to be processed; convert the source signal to be processed into a 3D SODP map, and then calculate the three-dimensional geometric features of the 3D SODP map; use the CSP method to extract the feature vector corresponding to each source signal to be processed.

[0088] The classification module 240 is used to input the three-dimensional geometric features into the first classifier corresponding to the 3D SODP to obtain a first classification result; input the feature vector into the second classifier corresponding to the CSP method to obtain a second classification result; perform weighted fusion of the three-dimensional geometric features and the feature vector, and input the fused data into the third classifier to obtain a third classification result; input the first classification result, the second classification result and the third classification result into the majority voting mechanism to obtain a final classification result.

[0089] So far, the technical solutions of the present disclosure have been described in combination with the above multiple embodiments, but it is easy for those skilled in the art to understand that the protection scope of the present disclosure is not limited to these specific embodiments. Without departing from the technical principles of the present disclosure, those skilled in the art can split and combine the technical solutions in the above-mentioned various embodiments, and can also make equivalent changes or replacements to the relevant technical features. Any changes, equivalent replacements, improvements, etc. made within the technical concept and / or technical principle of the present disclosure will fall within the protection scope of the present disclosure.

Claims

1. A multi-view collaborative integrated classification method for EEG signals, characterized in that: The method comprises: Collect EEG signals generated by the same subject performing a running imagination task and head MRI signals in a resting state; process the MRI signals into two-dimensional slice images; obtain the pre-processed EEG signals through common average reference and frequency domain filtering technology; The head model is obtained by using the forward problem processing mechanism and two-dimensional slice diagram of ESI technology; wherein the head model includes a three-layer structure, and the three-layer structure is scalp, skull and cortex from outside to inside; The cortex of the head model is divided to obtain a source model containing a number of grid points, wherein each grid point corresponds to a source dipole; the potential generated by each dipole at the scalp electrode in three directions of space is calculated, and then the lead field matrix is ​​obtained; Using the inverse problem processing mechanism and lead field matrix of the ESI technology, the pre-processed EEG signal of the current subject's cerebral scalp is mapped to the cortex of the head model as a cortical mapping signal; Based on the coordinates of the subject's scalp electrodes, the central dipole is calculated from the source dipole to obtain a circular sub-region; the cortical mapping signal in the circular sub-region is used as the source signal to be processed; The source signal to be processed is converted into a 3D SODP graph, and then the three-dimensional geometric features of the 3D SODP graph are calculated; the feature vector corresponding to each source signal to be processed is extracted using the CSP method; The three-dimensional geometric features are input into the first classifier corresponding to 3D SODP to obtain the first classification result; the feature vector is input into the second classifier corresponding to the CSP method to obtain the second classification result; the three-dimensional geometric features and feature vectors are weighted fused, and the fused data is input into the third classifier to obtain the third classification result; the first classification result, the second classification result and the third classification result are input into the majority voting mechanism to obtain the final classification result.

2. The EEG signal multi-view collaborative integrated classification method according to claim 1, characterized in that: The preset number of sampling points for EEG signals is n, and the number of acquisition channels is N; Collect EEG signals generated by the same subject performing a running imagination task, including: Collection i The EEG signal of the channel; i The EEG signal of each channel is expressed as , i ∈[1,N]; Processing MRI signals into two-dimensional slice images includes: Acquire MRI signals; wherein the MRI signals are three-dimensional MRI images in the format of A*B*C, where A represents pixels of the image in the left-right direction, B represents pixels of the image in the front-back direction, and C represents pixels of the image in the up-down direction. The sagittal plane, coronal plane, and transverse plane of the three-dimensional MRI image correspond to three directions perpendicular to the X-axis, the Y-axis, and the Z-axis, respectively; By slicing along the X-axis, Y-axis and Z-axis directions, a two-dimensional slice image corresponding to the MRI signal is obtained.

3. The EEG signal multi-view collaborative integrated classification method according to claim 1, characterized in that: The pre-processed EEG signal is obtained by using common mean reference and frequency domain filtering technology, including: The formula corresponding to the common average reference is: , filter out the common mode components in the EEG signal; in, Indicates i The signal after the average reference of the channels is i ∈[1,N], N represents the number of acquisition channels, Indicates i The signal of each channel; Then, the common-average referenced signal was band-pass filtered in the 8-30 Hz frequency band through a fourth-order Butterworth band-pass filter to obtain the preprocessed EEG signal.

4. The method for multi-view collaborative integration classification of EEG signals according to claim 1, characterized in that: The cortex of the head model is divided to obtain a source model containing several grid points, including: FreeSurfer software and MNE-python library were used to process the two-dimensional slice images, and the boundary element model was used to generate a three-shell volume conduction model with a conductivity ratio of 1:1 / 50:1 as the head model; the head model includes a three-layer structure, and the three-layer structure is scalp, skull and cortex from the outside to the inside; The cortical surface of the three-layer structure is evenly divided to obtain a source model containing 20484 grid points, where each grid point corresponds to a source dipole.

5. The method for multi-view collaborative integration classification of EEG signals according to claim 1, characterized in that: Using the inverse problem processing mechanism and lead field matrix of ESI technology, the pre-processed EEG signal of the current subject's cerebral scalp is mapped to the cortex of the head model, including: The sLORETA method is used to map the preprocessed EEG signal of the current subject to the cortex of the head model; the regularization parameter is set to 1.

6. The EEG signal multi-view collaborative integrated classification method according to claim 1, characterized in that: Based on the coordinates of the subject's scalp electrodes, the center dipole is calculated from the source dipole to obtain the circular sub-area, including: Obtain the coordinates of the scalp electrodes; The source dipole closest to the scalp electrode position was calculated as the central dipole; Based on the preset radius, a circular sub-area is obtained with the central dipole as the center.

7. The method for multi-view collaborative integration classification of EEG signals according to claim 1, characterized in that: The geometric features include at least: the sum of vector lengths, the sum of vector angles, the sum of distances to the origin, the sum of distances to the major axis, and the sum of distances to the minor axis; The source signal to be processed is converted into a 3D SODP graph, and then the three-dimensional geometric features of the 3D SODP graph are calculated, including: Obtain the central dipoles corresponding to the electrodes located in the left central area of ​​the cerebral scalp, the electrodes located in the right central area of ​​the cerebral scalp, and the electrodes located in the midline central area of ​​the cerebral scalp, draw a 3D SODP diagram for the source signal of the circular sub-area corresponding to the central dipole, and obtain the three-dimensional geometric characteristics of the source signal; By the vector length sum formula: , calculate the sum of vector lengths SVL; in, , and Indicates 3D SODP j The three-dimensional coordinate values ​​of the sample points, T is the total number of 3D SODP sample points; The total formula for the vector angle is: , calculate the sum of the vector angles SA; in,( , , )=( , , ), ( , , )=( , , )； By the formula of the sum of distances to the origin: , calculate the total distance to the origin SDTO; The formula for the sum of the distances to the major axis is: ； Calculate the sum of distances to the major axis SDTMA; in,( , , )=[ MajorAxis , , ]* MajorAxis, , MajorAxis=( , , ),( , , ) represents the point with the largest Z coordinate in 3D SODP, ( , , ) represents the point with the smallest Z coordinate in 3D SODP, and the symbol · represents vector dot product; The sum of the distances to the minor axis is calculated by: , calculate the sum of the distances to the minor axis ; in,( , , )=[ MinorAxis , , ]* MinorAxis, , a, b, c are a, b, c in the plane equation ax+by+cz=0 of the fitting plane involved in the 3D SODP diagram.

8. The method for multi-view collaborative integration classification of EEG signals according to claim 1, characterized in that: Using the CSP method, the feature vectors corresponding to each source signal to be processed are extracted, including: Get the CSP projection matrix; A preset number of row vectors are selected from the CSP projection matrix to form a spatial filter, and then the source signal to be processed is matrix multiplied with the spatial filter to obtain a eigenvector; wherein the number of eigenvectors is the same as the preset number.

9. The method for multi-view collaborative integration classification of EEG signals according to claim 1, characterized in that: After inputting the three-dimensional geometric features into a first classifier corresponding to the 3D SODP, a first classification result is obtained; Input the feature vector into the second classifier corresponding to the CSP method to obtain a second classification result; Before performing weighted fusion of the three-dimensional geometric features and the feature vectors and inputting the fused data into a third classifier to obtain a third classification result, the method further includes: Input the three-dimensional geometric features of the labeled imagination task into several classifiers, determine the classifier with the highest classification accuracy as the first classifier, and record the accuracy acc1 of the first classifier; Input the feature vector of the labeled imagination task into several classifiers, determine the classifier with the highest classification accuracy as the second classifier, and record the accuracy acc2 of the second classifier; By formula: , calculate the weight weight1 of the three-dimensional geometric features during weighted fusion; By formula: , calculate the weight weight2 of the feature vector during weighted fusion; Multiply the 3D geometric features by weight weight1, multiply the feature vector by weight2, and then serially concatenate the two to obtain fused data. Input the fused data with labeled imagination tasks into several classifiers, and determine the classifier with the highest classification accuracy as the third classifier.

10. A multi-view collaborative integrated classification system for EEG signals, characterized in that: The system comprises: A processing module is used to collect EEG signals generated by the same subject performing the running imagination task and MRI signals in a resting state; process the MRI signals into two-dimensional slice images; and obtain the pre-processed EEG signals through common average reference and frequency domain filtering technology; The ESI module is used to obtain a head model by using the forward problem processing mechanism and two-dimensional slice diagram of the ESI technology; wherein the head model includes a three-layer structure, and the three-layer structure is scalp, skull and cortex from the outside to the inside; the cortex of the head model is divided to obtain a source model including a number of grid points; wherein each grid point corresponds to a source dipole; the potential generated by each dipole at the scalp electrode in three directions of space is calculated, and then a lead field matrix is ​​obtained; the inverse problem processing mechanism and lead field matrix of the ESI technology are used to map the preprocessed EEG signal of the scalp of the current subject to the cortex of the head model as a cortical mapping signal; The data extraction module is used to calculate the central dipole from the source dipole based on the coordinates of the subject's scalp electrodes, thereby obtaining a circular sub-region; using the cortical mapping signal in the circular sub-region as the source signal to be processed; converting the source signal to be processed into a 3D SODP map, thereby calculating the three-dimensional geometric features of the 3D SODP map; and using the CSP method to extract the feature vector corresponding to each source signal to be processed; The classification module is used to input the three-dimensional geometric features into the first classifier corresponding to the 3D SODP to obtain the first classification result; input the feature vector into the second classifier corresponding to the CSP method to obtain the second classification result; perform weighted fusion of the three-dimensional geometric features and the feature vector, input the fused data into the third classifier to obtain the third classification result; input the first classification result, the second classification result and the third classification result into the majority voting mechanism to obtain the final classification result.

Images