A multi-perspective collaborative integration classification method and system for electroencephalogram signals

Abstract

The present application discloses a multi-view collaborative integration classification method and system for electroencephalogram signals, mainly relating to the technical field of electroencephalogram signals, and is used to solve the problem that when the existing source imaging technology maps EEG signals in the sensor domain to the source space, it leads to an increase in data volume and redundant information, thereby resulting in an increase in the online processing time of the brain signal processing system and a decrease in the decoding accuracy of source signals. It includes: calculating a central dipole from source dipoles, and then obtaining a circular sub-region; using the mapped signals within the circular sub-region as source signals to be processed; calculating the corresponding three-dimensional geometric features of the source signals; extracting the feature vectors corresponding to each source signal to be processed; obtaining a first classification result of the geometric features; obtaining a second classification result of the feature vectors; performing weighted fusion on the three-dimensional geometric features and the feature vectors, and obtaining a third classification result of the fused data; inputting the three classification results into a majority voting mechanism to obtain a final classification result.

Description

Technical Field

[0001] The present application relates to the technical field of electroencephalogram signals, and particularly to a multi-view collaborative integration classification method and system for electroencephalogram signals. Background Technique

[0002] A brain-computer interface (BCI) is an interactive communication system that directly connects the brain to an external device. A BCI system generally consists of three parts: brain signal acquisition, signal decoding, and output control. Common brain signal acquisition methods for BCI systems include electroencephalogram (EEG), magnetoencephalography (MEG), electrocorticography (ECoG), and magnetic resonance imaging (MRI), etc. Among them, EEG has received extensive attention due to its advantages such as easy mobility, low cost, and non-invasiveness. BCI systems based on EEG have achieved remarkable applications in multiple fields, especially in aspects such as robotic arm continuous control, drone target search, and intelligent wheelchairs, providing practical help for people with special needs.

[0003] Since EEG is detected through the scalp, it records the aliased potential formed outside the skull after the intracranial nerve cluster activities are diffusely propagated through the human volume conductor tissues such as brain tissue, cerebrospinal fluid, skull, and scalp. At the same time, the spatial resolution of EEG is at the centimeter level, and compared with MRI, MEG, etc., its spatial resolution is poor and it is impossible to accurately locate each activated neuron. These factors limit the development of EEG-BCI systems. Electrical source imaging (ESI) has been proven to be able to effectively solve the problems of the volume conduction effect and poor spatial resolution of EEG.

[0004] However, when the ESI technology maps the EEG signals in the sensor domain to the source space, it will cause the data volume to increase significantly from 32 to 256 scalp electrode signals to 5,000 to 10,000 source signals. At the same time, the inverse problem solving algorithm of ESI depends on mathematical models and assumptions, and these assumptions often generate multiple possible solutions, resulting in redundant information in the source signals. Challenges such as the large data volume and a lot of redundant information brought by the ESI technology have led to an increase in the online processing time of the brain signal processing system and a decrease in the decoding accuracy of the source signals. Summary of the Invention

[0005] In view of the above deficiencies of the prior art, the present application provides a multi-perspective collaborative integration classification method and system for electroencephalogram (EEG) signals, aiming to solve the problems that when the existing ESI technology maps EEG signals in the sensor domain to the source space, the data volume increases. In addition, the inverse problem solving algorithm of ESI will generate multiple possible solutions, which in turn leads to an increase in the online processing time of the brain signal processing system and a decrease in the decoding accuracy of the source signal.

[0006] In the first aspect, the present application provides a multi-perspective collaborative integration classification method for EEG signals, and the method includes:

[0007] Collect EEG signals generated by the same subject performing a running imagination task and head MRI signals in the resting state; process the MRI signals into two-dimensional slice images; obtain preprocessed EEG signals through common average reference and frequency domain filtering techniques;

[0008] Utilize the forward problem processing mechanism of ESI technology and the two-dimensional slice images to obtain a head model; wherein, the head model includes a three-layer structure, and the three-layer structure from the outside to the inside is the scalp, skull, and cortex in sequence;

[0009] Divide the cortex of the head model to obtain a source model containing several grid points; wherein, each grid point corresponds to a source dipole; calculate the potential generated by each dipole at the scalp electrode in three spatial directions, and then obtain a lead field matrix;

[0010] Utilize the inverse problem processing mechanism of ESI technology and the lead field matrix to map the preprocessed EEG signals of the subject's brain scalp to the cortex of the head model as cortex mapping signals;

[0011] Based on the coordinates of the subject's scalp electrodes, calculate the central dipole from the source dipoles, and then obtain a circular sub-region; use the cortex mapping signals within the circular sub-region as the source signals to be processed;

[0012] Convert the source signals to be processed into 3D SODP maps, and then calculate the three-dimensional geometric features of the 3D SODP maps; use the CSP method to extract the feature vectors corresponding to each source signal to be processed;

[0013] Input the three-dimensional geometric features into the first classifier corresponding to the 3D SODP to obtain a first classification result; input the feature vectors into the second classifier corresponding to the CSP method to obtain a second classification result; perform weighted fusion on the three-dimensional geometric features and the feature vectors, input the fused data into the third classifier to obtain a third classification result; input the first classification result, the second classification result, and the third classification result into a majority voting mechanism to obtain a final classification result.

[0014] In an implementation manner of the present application, the preset number of sampling points of the EEG signal is n, and the number of acquisition channels is N;

[0015] Collect the EEG signals generated when the same subject performs the running imagination task, specifically including:

[0016] Collect the EEG signals of the i th channel; among them, the EEG signal of the i th channel is represented as , i ∈ [1, N];

[0017] Process the MRI signal into a two-dimensional slice image, specifically including:

[0018] Collect the obtained MRI signal; among them, the MRI signal is a three-dimensional MRI image with the format of A*B*C, where A represents the number of pixels of the image in the left-right direction, B represents the number of pixels of the image in the front-back direction, C represents the number of pixels of the image in the up-down direction, and the sagittal plane, coronal plane, and transverse plane of the three-dimensional MRI image respectively correspond to the three directions perpendicular to the X-axis, Y-axis, and Z-axis;

[0019] Obtain the two-dimensional slice image corresponding to the MRI signal by slicing along the X-axis, Y-axis, and Z-axis directions.

[0020] In an implementation manner of the present application, through the common average reference and frequency domain filtering techniques, obtain the preprocessed EEG signal, specifically including:

[0021] Through the formula corresponding to the common average reference:

[0022] , filter out the common mode components in the EEG signal;

[0023] Among them, represents the signal after common average reference of the i th channel, i ∈ [1, N], N represents the number of acquisition channels, represents the signal of the i th channel;

[0024] Furthermore, through a fourth-order Butterworth band-pass filter, perform band-pass filtering on the signal after common average reference in the frequency band of 8 - 30 Hz to obtain the preprocessed EEG signal.

[0025] In an implementation manner of the present application, the cortex of the head model is divided to obtain a source model including a number of grid points, which specifically includes: processing two-dimensional slice maps using FreeSurfer software and MNE-python library, and generating a three-shell volume conduction model with a conductivity ratio of 1:1 / 50:1 using the boundary element model as the head model; wherein, the head model includes three layers of structures, and the three layers of structures are, from outside to inside, the scalp, the skull, and the cortex; evenly dividing the cortical surface of the three layers of structures to obtain a source model including 20,484 grid points; wherein, each grid point corresponds to a source dipole.

[0026] In an implementation manner of the present application, using the inverse problem processing mechanism and lead field matrix of ESI technology, the preprocessed EEG signals of the brain scalp of the current subject are mapped to the cortex of the head model, which specifically includes: using the sLORETA method to map the preprocessed EEG signals of the current subject to the cortex of the head model; wherein, the regularization parameter is set to 1.

[0027] In an implementation manner of the present application, based on the coordinates of the scalp electrodes of the subject, the central dipole is calculated from the source dipoles, and then a circular sub-region is obtained, which specifically includes:

[0028] Obtaining the coordinates of the scalp electrodes; calculating the source dipole closest to the position of the scalp electrodes as the central dipole; obtaining a circular sub-region with the central dipole as the center based on a preset radius.

[0029] In an implementation manner of the present application, the geometric features at least include: the total sum of vector lengths, the total sum of vector angles, the total sum of distances to the origin, the total sum of distances to the major axis, and the total sum of distances to the minor axis;

[0030] Converting the source signal to be processed into a 3D SODP map, and then calculating the three-dimensional geometric features of the 3D SODP map, which specifically includes: obtaining the central dipoles corresponding to the electrodes in the left central region of the brain scalp, the electrodes in the right central region of the brain scalp, and the electrodes in the midline central region of the brain scalp, drawing a 3D SODP map of the source signals in the circular sub-regions corresponding to the central dipoles, and obtaining the three-dimensional geometric features of the source signals;

[0031] Through the vector length sum formula:

[0032] , calculating the total sum of vector lengths SVL;

[0033] Wherein, 、 and represent the three-dimensional coordinate values of the j th sample point of the 3D SODP, and T is the total number of sample points of the 3D SODP;

[0034] Using the total formula for the angle between vectors:

[0035] , calculate the total sum of the angles between vectors SA;

[0036] where, ( , , ) = ( , , ),

[0037] ( , , ) = ( , , );

[0038] Using the total formula for the distance to the origin:

[0039] , calculate the total sum of the distances to the origin SDTO;

[0040] Using the total formula for the distance to the major axis:

[0041] ,

[0042] calculate the total sum of the distances to the major axis SDTMA;

[0043] where, ( , , ) = MajorAxis · , , * MajorAxis, , MajorAxis = ( , , ), ( , , ) represents the point with the largest Z - coordinate in 3D SODP, ( , , ) represents the point with the smallest Z - coordinate in 3D SODP, and the symbol · represents the dot - product of vectors;

[0044] Using the total formula for the distance to the minor axis:

[0045] , calculate the total sum of the distances to the minor axis ;

[0046] Among them, ( , , ) = MinorAxis · , , * MinorAxis, , where a, b, and c are the a, b, and c in the plane equation ax + by + cz = 0 of the fitting plane involved in the 3D SODP diagram.

[0047] In one implementation manner of the present application, using the CSP method, the feature vectors corresponding to each source signal to be processed are extracted, specifically including:

[0048] Obtain the CSP projection matrix; select a preset number of row vectors from the CSP projection matrix to form a spatial filter, and then multiply the source signal to be processed by the spatial filter to obtain the feature vectors; among them, the number of feature vectors is the same as the preset number.

[0049] In one implementation manner of the present application, before inputting the three-dimensional geometric features into the first classifier corresponding to 3D SODP to obtain the first classification result; inputting the feature vectors into the second classifier corresponding to the CSP method to obtain the second classification result; performing weighted fusion on the three-dimensional geometric features and the feature vectors, and inputting the fused data into the third classifier to obtain the third classification result, the method further includes:

[0050] Input the three-dimensional geometric features marked with the imagination task into several classifiers, determine the classifier with the highest classification accuracy as the first classifier, and record the accuracy acc1 of the first classifier; input the feature vectors marked with the imagination task into several classifiers, determine the classifier with the highest classification accuracy as the second classifier, and record the accuracy acc2 of the second classifier;

[0051] Through the formula: , calculate the weight weight1 of the three-dimensional geometric features during weighted fusion;

[0052] Through the formula: , calculate the weight weight2 of the feature vectors during weighted fusion;

[0053] Multiply the three-dimensional geometric features by the weight weight1, multiply the feature vectors by weight2, then serially splice the two to obtain the fused data, and input the fused data marked with the imagination task into several classifiers, and determine the classifier with the highest classification accuracy as the third classifier.

[0054] In a second aspect, the present application provides an electroencephalogram (EEG) signal multi-view collaborative integration classification system, which includes:

[0055] A processing module, configured to collect EEG signals generated by the same subject performing a running imagination task and MRI signals in a resting state; process the MRI signals into two-dimensional slice images; and obtain preprocessed EEG signals through common average reference and frequency domain filtering techniques.

[0056] An ESI module, configured to use the forward problem processing mechanism of ESI technology and the two-dimensional slice images to obtain a head model; wherein the head model includes a three-layer structure, and the three-layer structure from the outside to the inside is the scalp, skull, and cortex in sequence; divide the cortex of the head model to obtain a source model including a plurality of grid points; wherein each grid point corresponds to a source dipole; calculate the potentials generated by each dipole in three spatial directions at the scalp electrodes, and then obtain a lead field matrix; use the inverse problem processing mechanism of ESI technology and the lead field matrix to map the preprocessed EEG signals of the subject's brain scalp to the cortex of the head model as cortical mapping signals.

[0057] A data extraction module, configured to calculate a central dipole from the source dipoles based on the coordinates of the subject's scalp electrodes, and then obtain a circular sub-region; use the cortical mapping signals within the circular sub-region as source signals to be processed; convert the source signals to be processed into 3D SODP maps, and then calculate the three-dimensional geometric features of the 3D SODP maps; use the CSP method to extract the feature vectors corresponding to each source signal to be processed.

[0058] A classification module, configured to input the three-dimensional geometric features into a first classifier corresponding to 3D SODP to obtain a first classification result; input the feature vectors into a second classifier corresponding to the CSP method to obtain a second classification result; perform weighted fusion on the three-dimensional geometric features and the feature vectors, input the fused data into a third classifier to obtain a third classification result; and input the first classification result, the second classification result, and the third classification result into a majority voting mechanism to obtain a final classification result.

[0059] Those skilled in the art can understand that the present application has at least the following beneficial effects:

[0060] This application extracts the features of the source signal through multi-perspective features (3D SODP + CSP + weighted fusion), and realizes the maximum utilization of features through collaborative integrated classification. While ensuring the high accuracy and strong robustness of the classification results, the online processing time is reduced; based on the coordinates of the scalp electrodes of the subject, the central dipole is calculated from the source dipole, and then a circular sub-region is obtained; since the solutions generated by the inverse problem solving algorithm of ESI are restricted to the circular sub-region, the data volume and redundant information of the source signal are reduced, avoiding the problems of the increase in the online processing time of the brain signal processing system and the reduction of the decoding accuracy of the source signal. In addition, this application adopts a weighted fusion method, which fully fuses the dynamic change features (three-dimensional geometric features) of 3D SODP and the spatial feature information (feature vectors) of CSP, and comprehensively considers the dynamic change characteristics and spatial characteristics of the source signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the present invention, the drawings required for description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0062] Figure 1 It is a flowchart of a multi-perspective collaborative integrated classification method for electroencephalogram signals provided by an embodiment of this application.

[0063] Figure 2 It is a sagittal plane schematic diagram of a two-dimensional slice corresponding to a three-dimensional image involved in a multi-perspective collaborative integrated classification method for electroencephalogram signals provided by an embodiment of this application.

[0064] Figure 3 It is a coronal plane schematic diagram of a two-dimensional slice corresponding to a three-dimensional image involved in a multi-perspective collaborative integrated classification method for electroencephalogram signals provided by an embodiment of this application.

[0065] Figure 4 It is a cross-sectional plane schematic diagram of a two-dimensional slice corresponding to a three-dimensional image involved in a multi-perspective collaborative integrated classification method for electroencephalogram signals provided by an embodiment of this application.

[0066] Figure 5 It is a schematic diagram of the total vector length feature provided by an embodiment of this application.

[0067] Figure 6 It is a schematic diagram of the total vector angle feature provided by an embodiment of this application.

[0068] Figure 7 It is a schematic diagram of the total distance from the origin feature provided by an embodiment of this application.

[0069] Figure 8It is a schematic diagram of the sum of distances to the long axis provided by an embodiment of the present application.

[0070] Figure 9 It is a schematic diagram of the sum of distances to the short axis provided by an embodiment of the present application.

[0071] Figure 10 It is a schematic diagram of the elliptical area feature provided by an embodiment of the present application.

[0072] Figure 11 It is a schematic diagram of the standard deviation feature in two directions provided by an embodiment of the present application.

[0073] Figure 12 It is a schematic diagram of the sum of distances to the 45-degree line provided by an embodiment of the present application.

[0074] Figure 13 It is a schematic diagram of the sum of distances to the 135-degree line provided by an embodiment of the present application.

[0075] Figure 14 It is a schematic diagram of the measure of central tendency feature provided by an embodiment of the present application.

[0076] Figure 15 It is a schematic diagram of the 3DSODP diagram involved in the multi-view collaborative integration classification method for electroencephalogram signals provided by an embodiment of the present application.

[0077] Figure 16 It is a schematic diagram of the projection of the fitting plane of the 3DSODP diagram involved in the multi-view collaborative integration classification method for electroencephalogram signals provided by an embodiment of the present application

[0078] Figure 17 It is a schematic diagram of the method for feature weighted fusion of three-dimensional geometric features and feature vectors provided by an embodiment of the present application.

[0079] Figure 18 It is a schematic diagram of the multi-view collaborative integration classification method for left and right hand electroencephalogram signals provided by an embodiment of the present application.

[0080] Figure 19 It is a schematic diagram of the internal structure of the multi-view collaborative integration classification system for electroencephalogram signals provided by an embodiment of the present application. Detailed implementation manners

[0081] Those skilled in the art should understand that the embodiments described below are only the preferred embodiments of the present disclosure, and do not mean that the present disclosure can only be implemented through these preferred embodiments. These preferred embodiments are only used to explain the technical principles of the present disclosure, rather than to limit the protection scope of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts should still fall within the protection scope of the present disclosure.

[0082] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

[0083] The technical solutions proposed in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0084] The embodiment provides a multi-view collaborative integration classification method for electroencephalogram signals. As Figure 1 shown, the method provided by the embodiment of the present application mainly includes the following steps:

[0085] Step 110: Collect the EEG signals generated by the same subject performing a running imagination task and the head MRI signals in the resting state; process the MRI signals into two-dimensional slice images; and obtain the preprocessed EEG signals through common average reference and frequency domain filtering techniques.

[0086] It should be noted that the running imagination task performed can be imagining left hand movement or right hand movement, the preset number of sampling points of the EEG signal is n, and the number of acquisition channels is N.

[0087] In some embodiments, collecting the EEG signals generated by the same subject performing a running imagination task can specifically be:

[0088] Collect the EEG signals of the i th channel; wherein, the EEG signals of the i th channel are represented as , i ∈[1, N].

[0089] Processing the MRI signals into two-dimensional slice images can specifically be:

[0090] Collect the MRI signal (each subject undergoes head magnetic resonance imaging on an MRI scanner in a medical institution to obtain three-dimensional MRI images. During the collection process, the subject needs to keep their eyes closed as much as possible, lie quietly, but not fall asleep); among them, the MRI signal is a three-dimensional MRI image with the format of A*B*C, where A represents the number of pixels of the image in the left-right direction, B represents the number of pixels of the image in the front-back direction, and C represents the number of pixels of the image in the up-down direction. The sagittal plane, coronal plane, and transverse plane of the three-dimensional MRI image respectively correspond to three directions perpendicular to the X-axis, Y-axis, and Z-axis; by slicing along the X-axis, Y-axis, and Z-axis directions, two-dimensional slice images corresponding to the MRI signal are obtained (including: sagittal plane, coronal plane, transverse plane; specifically, as Figure 2 shown in the schematic diagram of the sagittal plane composed of the X-axis and Y-axis, as Figure 3 shown in the schematic diagram of the coronal plane composed of the X-axis and Z-axis, as Figure 4 shown in the schematic diagram of the transverse plane composed of the Y-axis and Z-axis).

[0091] Among them, through the common average reference and frequency domain filtering techniques, the preprocessed EEG signal is obtained. Specifically, it can be:

[0092] Through the formula corresponding to the common average reference:

[0093] , the common mode components in the EEG signal are filtered out;

[0094] Among them, represents the signal after common average reference of the i th channel, i ∈[1,N], where N represents the number of acquisition channels, represents the signal of the i th channel;

[0095] Furthermore, through a fourth-order Butterworth band-pass filter, band-pass filtering processing of the signal after common average reference in the 8 - 30 Hz frequency band is performed to obtain the preprocessed EEG signal.

[0096] It should be noted that the Event Related Desynchronization (ERD) and Event Related Synchronization (ERS) phenomena related to motor imagery are mainly concentrated in the 8 - 30 Hz frequency band. Therefore, in order to remove irrelevant information in other frequency bands, this application uses a fourth-order Butterworth band-pass filter to perform band-pass filtering processing of the signal after common average reference in the 8 - 30 Hz frequency band.

[0097] Step 120: Use the forward problem processing mechanism of the ESI technology and the two-dimensional slice images to obtain a head model.

[0098] It should be noted that the head model includes three layers of structures, and the three layers of structures are, from the outside to the inside, the scalp, the skull, and the cerebral cortex in sequence.

[0099] In this application, FreeSurfer software and MNE-python library are used to process two-dimensional slice images, and a three-shell volume conduction model (head model) with a conductivity ratio of 1:1 / 50:1 is generated by using the boundary element model BEM. The three layers of structures of this model are, from the outside to the inside, the scalp, the skull, and the cerebral cortex in sequence.

[0100] Step 130: Divide the cerebral cortex of the head model to obtain a source model containing a number of grid points; calculate the potential generated by each dipole at the scalp electrodes in three directions in space, and then obtain the lead field matrix.

[0101] It should be noted that each grid point corresponds to a source dipole.

[0102] Among them, dividing the cerebral cortex of the head model to obtain a source model containing a number of grid points can specifically be:

[0103] In this application, the cerebral cortex surface of the head model is evenly divided to obtain a source model containing 20,484 grid points, and each grid point corresponds to a source dipole. Finally, the potential generated by each dipole at the scalp electrodes in three directions in space is calculated, thereby obtaining the lead field matrix , where M represents the number of EEG electrodes and D represents the number of dipoles.

[0104] Step 140: Use the inverse problem processing mechanism of the ESI technology and the lead field matrix to map the preprocessed EEG signal of the cerebral scalp of the current subject to the cerebral cortex of the head model as the cortical mapping signal.

[0105] This step can specifically be:

[0106] Use the sLORETA method to map the preprocessed EEG signal of the current subject to the cerebral cortex of the head model of the current subject; among them, the regularization parameter is set to 1.

[0107] Step 150: Calculate the central dipole from the source dipoles based on the coordinates of the scalp electrodes of the subject, and then obtain a circular sub-region; use the cortical mapping signal within the circular sub-region as the source signal to be processed.

[0108] It should be noted that the traditional region division method is to simply subdivide on the basis of existing brain partitions to create smaller sub-regions. However, this method does not consider the position of scalp electrodes and the aliasing relationship between scalp electrode signals and numerous source dipole signals, resulting in insufficient accuracy of the created regions. To solve these problems, this step adopts a region division method that combines electrode position and dipole relationship here. This method takes the dipole closest to the scalp electrode as the center and expands outward to form a circular region, thereby establishing a more accurate circular sub-region. In addition, it should be added that there can be several circular sub-regions, and those skilled in the art can determine the specific quantity and the radius lengths of each circular sub-region according to actual needs. As an example, two circular sub-regions can be set, and the radius lengths can be 5 millimeters and 10 millimeters respectively.

[0109] Among them, based on the coordinates of the scalp electrodes of the subject, the central dipole is calculated from the source dipoles, and then the circular sub-region is obtained. Specifically, it can be:

[0110] Obtain the coordinates of the scalp electrodes; calculate the source dipole closest to the position of the scalp electrode as the central dipole; based on a preset radius, with the central dipole as the center, obtain the circular sub-region.

[0111] It should be noted that the coordinates of the scalp electrodes have been obtained in the initial stage, and the scheme for calculating the source dipole closest to the position of the scalp electrode is an existing scheme for calculating the distance between two coordinates.

[0112] Step 160: Convert the source signal to be processed into a 3D SODP map, and then calculate and obtain the three-dimensional geometric features of the 3D SODP map; use the CSP method to extract the feature vectors corresponding to each source signal to be processed.

[0113] It should be noted that feature extraction is a key step in successfully decoding the source signal. Therefore, this application comprehensively extracts features from multiple perspectives (3D SODP + CSP + weighted fusion). After determining the circular sub-region in this application, first obtain the source signal within this region as the source signal to be processed, and then extract features from three different angles: First, use 3D SODP to capture the dynamic change characteristics (three-dimensional geometric features) of the source signal; Second, adopt the common spatial pattern to extract the spatial features (feature vectors) of the source signal; Third, it is obtained by fusing the three-dimensional geometric features and the feature vectors. The specific fusion process is shown in step 170.

[0114] Among them, the geometric features at least include: the total vector length (the feature display form is as Figure 5 shown), the total vector angle (the feature display form is as Figure 6 shown), the total distance to the origin (the feature display form is as Figure 7The sum of the distances to the long axis (the characteristic display form is as shown in Figure 8 ), the sum of the distances to the short axis (the characteristic display form is as shown in Figure 9 ), the elliptical area (the characteristic display form is as shown in Figure 10 ), the standard deviations in two directions (the characteristic display form is as shown in Figure 11 ), the sum of the distances to the 45-degree line (the characteristic display form is as shown in Figure 12 ), the sum of the distances to the 135-degree line (the characteristic display form is as shown in Figure 13 ), and the measure of central tendency (the characteristic display form is as shown in Figure 14 ).

[0115] The three-dimensional features of the source signal are captured by 3D SODP, and then the geometric features of the three-dimensional space corresponding to the source signal are calculated. Specifically, it can be:

[0116] Obtain the central dipoles corresponding to the electrodes in the left central region of the cerebral scalp, the electrodes in the right central region of the cerebral scalp, and the electrodes in the midline central region of the cerebral scalp. Draw a 3D SODP map of the source signal for the circular sub-region corresponding to the central dipole to obtain the three-dimensional geometric features of the source signal.

[0117] More specifically,

[0118] Through the vector length sum formula:

[0119] , calculate the vector length sum SVL;

[0120] Among them, , and represent the three-dimensional coordinate values of the j th sample point of 3D SODP, and T is the total number of 3D SODP sample points.

[0121] Through the vector angle sum formula:

[0122] , calculate the vector angle sum SA;

[0123] Among them, ([[]] , , ) = ([[]] , , ),

[0124] ( , , ) = ([[]] , , ).

[0125] Through the formula for the sum of distances to the origin:

[0126] , calculate the sum of distances to the origin SDTO.

[0127] Through the formula for the sum of distances to the major axis:

[0128] .

[0129] Calculate the sum of distances to the major axis SDTMA;

[0130] where, ( , , ) = MajorAxis · , , * MajorAxis, , MajorAxis = ( , , ), ( , , ) represents the point with the maximum Z - coordinate in the 3D SODP, ( , , ) represents the point with the minimum Z - coordinate in the 3D SODP, and the symbol · represents the dot - product of vectors.

[0131] Through the formula for the sum of distances to the minor axis:

[0132] , calculate the sum of distances to the minor axis ;

[0133] where, ( , , ) = MinorAxis · , , * MinorAxis, , a, b, c are the a, b, c in the plane equation ax + by + cz = 0 of the fitting plane involved in the 3D SODP diagram; where, the 3D SODP diagram, as Figure 15 shown (unit: microvolt); the projection schematic diagram of the fitting plane corresponding to the 3D SODP diagram, as Figure 16 shown (unit: microvolt).

[0134] Through the ellipse area formula:

[0135] , calculate the ellipse area EA;

[0136] Among them, , , , , , , , represents the j-th pixel point of the fitting plane corresponding to the 3D SODP diagram, and π is the pi.

[0137] Through the standard deviation formulas in two directions:

[0138] , calculate the standard deviation TDSD in two directions;

[0139] Among them, , , represents the vector composed of the horizontal coordinate axes of all sample points, represents the vector composed of the vertical coordinate axes of all sample points, and Var represents variance.

[0140] Through the sum of distances to the 45-degree line formula:

[0141] , calculate the sum of distances SDT45 from the data points in the 3D SODP to the 45-degree line.

[0142] Through the sum of distances to the 135-degree line formula:

[0143] , calculate the sum of distances SDT135 from the data points in the 3D SODP to the 135-degree line.

[0144] Through the central tendency measure formula:

[0145] , calculate the central tendency measure;

[0146] Among them, , , and r represents the preset radius.

[0147] In the steps, use the CSP method to extract the eigenvectors corresponding to each source signal to be processed, specifically:

[0148] Obtain the CSP projection matrix;

[0149] Select a preset number of row vectors from the CSP projection matrix to form a spatial filter, and then perform matrix multiplication on the source signal to be processed and the spatial filter to obtain eigenvectors; where the number of eigenvectors is the same as the preset number.

[0150] Step 170: Input the three-dimensional geometric features into the first classifier corresponding to 3D SODP to obtain a first classification result; input the eigenvectors into the second classifier corresponding to the CSP method to obtain a second classification result; perform weighted fusion on the three-dimensional geometric features and the eigenvectors, and input the fused data into the third classifier to obtain a third classification result; input the first classification result, the second classification result, and the third classification result into a majority voting mechanism to obtain a final classification result.

[0151] It should be noted that the classifiers include Logistic Regression (LR), K-Nearest Neighbors (KNN), Decision Tree (DT), Support Vector Machine (SVM), Naïve Bayes (NB), and Multilayer Perceptron (MLP). In addition, this application takes into account that 3D SODP is superior in capturing the dynamic change characteristics of the source signal, while CSP can effectively extract the spatial characteristics of the source signal to be processed, and both have their own advantages. In order to further enhance the effectiveness of the features, this application performs weighted fusion on the 3D SODP features and the CSP-based spatial features to obtain a new feature perspective.

[0152] Before inputting the three-dimensional geometric features into the first classifier corresponding to 3D SODP to obtain a first classification result; inputting the eigenvectors into the second classifier corresponding to the CSP method to obtain a second classification result; performing weighted fusion on the three-dimensional geometric features and the eigenvectors, and inputting the fused data into the third classifier to obtain a third classification result, the method further includes the selection of classifiers, and the selection process can be:

[0153] Input the three-dimensional geometric features with labeled imagination tasks into several classifiers, determine the classifier with the highest classification accuracy as the first classifier, and record the accuracy acc1 of the first classifier;

[0154] Input the eigenvectors with labeled imagination tasks into several classifiers, determine the classifier with the highest classification accuracy as the second classifier, and record the accuracy acc2 of the second classifier;

[0155] Through the formula: , calculate the weight weight1 of the three-dimensional geometric features during weighted fusion;

[0156] Through the formula: calculate the weight weight2 of the feature vector when performing weighted fusion;

[0157] The weighted fusion scheme is as Figure 17 shown. Multiply the three-dimensional geometric feature by the weight weight1, multiply the feature vector by weight2, then serially splice the two to obtain the fused data. Input the fused data with the labeled imagination task into several classifiers, and determine the classifier with the highest classification accuracy as the third classifier.

[0158] Those skilled in the art can understand that there is a better fitting relationship between different feature vectors and specific classifiers. In this application, feature extraction is performed on the source signal from three different perspectives. Among them, the fused data can effectively integrate the advantages of 3D SODP features and spatial features, thereby enhancing the information content of the features. However, in the process of feature fusion, due to the possible high correlation between sub-features, this kind of fusion sometimes leads to redundancy, which in turn masks some subtle differences. Therefore, the fused data cannot completely replace the 3D SODP features and spatial features. The features from the above three perspectives each have their own advantages, and making full use of these features will help improve the classification effect of the signal. Based on this, this application first selects the most suitable sub-classifier for each feature, then combines the multi-perspective features with their most suitable classifiers, and determines the final classification result through a voting mechanism.

[0159] Based on the above description, a multi-perspective collaborative integration classification method for left and right hand EEG signals provided by an embodiment of this application is as Figure 18 shown. Specifically:

[0160] Signal acquisition: Signal acquisition is divided into EEG signal acquisition and MRI signal acquisition, and each acquisition needs to be from the same subject.

[0161] Among them, the EEG signal acquisition is as follows: Each subject needs to use an EEG signal acquisition device to collect EEG signals. During the process, the subject needs to perform different motor imagination tasks, such as imagining moving the left hand or the right hand. The MRI signal acquisition is as follows: Each subject undergoes a head magnetic resonance imaging on an MRI scanner in a medical institution to obtain three-dimensional MRI images, and then processes them into two-dimensional slice images. During the acquisition process, the subject needs to keep their eyes closed as much as possible, lie quietly, but not fall asleep.

[0162] EEG Signal Preprocessing: Due to its inherent randomness, EEG is vulnerable to external interference, which affects the accuracy of the analysis results. Therefore, in order to obtain more reliable analysis results, it is particularly important to preprocess EEG in this application by using common average reference and frequency domain filtering (using a fourth-order Butterworth band-pass filter to perform band-pass filtering on the signal after common average reference in the 8 - 30 Hz frequency band).

[0163] ESI Technology Forward Problem Processing Two-Dimensional Section Map:

[0164] Use FreeSurfer software and MNE-python library to process structural MRI data (two-dimensional section map), and use the boundary element model BEM to generate a three-shell volume conduction model (head model) with a conductivity ratio of 1:1 / 50:1. Uniformly divide the cortical surface to obtain a source model containing 20484 grid points. Calculate the potential generated by each dipole at the scalp electrodes in three spatial directions, thereby obtaining the lead field matrix (the calculation process can be implemented by existing technologies, and this application does not limit it here).

[0165] ESI Technology Inverse Problem Processing EEG Signal:

[0166] Since the number of sources is much larger than the number of EEG electrodes, and there are countless in-vivo source distributions that can generate the same in-vitro electric potential, this results in the solution of inverse inferring the source signal from scalp EEG signals being highly underdetermined and non-unique. This application uses the sLORETA method to perform source reconstruction on EEG signals; among them, the regularization parameter is set to 1.

[0167] Furthermore, in order to reduce the computational amount and computational complexity. This application uses a partitioning method that combines electrode positions and dipole relationships. The steps of the partitioning method used can be specifically as follows:

[0168] Obtain the coordinates of scalp electrodes;

[0169] Calculate the source dipole closest to the scalp electrode position as the central dipole;

[0170] Expand outward with the central dipole as the center to form circular sub-regions with different radii.

[0171] Based on the circular sub-regions, perform multi-view feature extraction:

[0172] On the one hand, 3D SODP is used to capture the dynamic change characteristics (three-dimensional geometric features) of the source signal (the signal within the circular sub-region), and the three-dimensional geometric features are input into the first classifier to obtain the first classification result (left hand or right hand); on the other hand, the common spatial pattern is adopted to extract the feature vectors of the source signal, and the feature vectors are input into the second classifier to obtain the second classification result (left hand or right hand). To further enhance the effectiveness of the features, by integrating the dynamic change characteristics and spatial characteristics, the present application performs weighted fusion on the features obtained by these two methods, and inputs the fused features into the third classifier to obtain the third classification result (left hand or right hand); thus, features from a third perspective are obtained.

[0173] The three results are input into a majority voting mechanism to obtain the final result (left hand or right hand).

[0174] In addition, the present application Figure 19 provides a multi-perspective collaborative integration classification system for electroencephalogram signals according to an embodiment of the present application. As Figure 19 shown, the system provided by the embodiment of the present application mainly includes:

[0175] A processing module 210, configured to collect EEG signals generated by the same subject performing a running imagination task and MRI signals in a resting state; process the MRI signals into two-dimensional slice images; and obtain preprocessed EEG signals through common average reference and frequency domain filtering techniques.

[0176] An ESI module 220, configured to obtain a head model by using the forward problem processing mechanism of ESI technology and the two-dimensional slice images; wherein, the head model includes a three-layer structure, and the three-layer structure from the outside to the inside is the scalp, skull, and cortex in sequence; divide the cortex of the head model to obtain a source model including a plurality of grid points; wherein, each grid point corresponds to a source dipole; calculate the potentials generated by each dipole at the scalp electrodes in three spatial directions, and further obtain a lead field matrix; use the inverse problem processing mechanism of ESI technology and the lead field matrix to map the preprocessed EEG signals of the brain scalp of the current subject to the cortex of the head model as cortex mapping signals.

[0177] A data extraction module 230, configured to calculate a central dipole from the source dipoles based on the coordinates of the scalp electrodes of the subject, and further obtain a circular sub-region; use the cortex mapping signals within the circular sub-region as source signals to be processed; convert the source signals to be processed into 3D SODP maps, and further calculate the three-dimensional geometric features of the 3D SODP maps; use the CSP method to extract the feature vectors corresponding to each source signal to be processed.

[0178] A classification module 240 is configured to input three-dimensional geometric features into a first classifier corresponding to 3D SODP to obtain a first classification result; input feature vectors into a second classifier corresponding to the CSP method to obtain a second classification result; perform weighted fusion on the three-dimensional geometric features and the feature vectors, input the fused data into a third classifier to obtain a third classification result; and input the first classification result, the second classification result, and the third classification result into a majority voting mechanism to obtain a final classification result.

[0179] So far, the technical solutions of the present disclosure have been described in combination with multiple embodiments in the foregoing text. However, those skilled in the art can easily understand that the protection scope of the present disclosure is not limited to these specific embodiments. Without departing from the technical principles of the present disclosure, those skilled in the art can split and combine the technical solutions in the above-mentioned various embodiments, and can also make equivalent changes or replacements to the relevant technical features. Any changes, equivalent replacements, improvements, etc. made within the technical concept and / or technical principle of the present disclosure will fall within the protection scope of the present disclosure.

Claims

1. A multi-view collaborative integrated classification method for EEG signals, characterized in that: The method comprises: Collect EEG signals generated by the same subject performing a running imagination task and head MRI signals in a resting state; process the MRI signals into two-dimensional slice images; obtain the pre-processed EEG signals through common average reference and frequency domain filtering technology; The head model is obtained by using the forward problem processing mechanism and two-dimensional slice diagram of ESI technology; wherein the head model includes a three-layer structure, and the three-layer structure is scalp, skull and cortex from outside to inside; The cortex of the head model is divided to obtain a source model containing a number of grid points, wherein each grid point corresponds to a source dipole; the potential generated by each source dipole at the scalp electrode in three directions of space is calculated, and then the lead field matrix is ​​obtained; Using the inverse problem processing mechanism and lead field matrix of the ESI technology, the pre-processed EEG signal of the current subject's cerebral scalp is mapped to the cortex of the head model as a cortical mapping signal; Based on the coordinates of the subject's scalp electrodes, the central source dipole is calculated from the source dipoles to obtain a circular sub-region; the cortical mapping signal in the circular sub-region is used as the source signal to be processed; The source signal to be processed is converted into a 3D SODP graph, and then the three-dimensional geometric features of the 3D SODP graph are calculated; the feature vector corresponding to each source signal to be processed is extracted using the CSP method; The three-dimensional geometric features are input into the first classifier corresponding to 3D SODP to obtain the first classification result; the feature vector is input into the second classifier corresponding to the CSP method to obtain the second classification result; the three-dimensional geometric features and feature vectors are weighted fused, and the fused data is input into the third classifier to obtain the third classification result; the first classification result, the second classification result and the third classification result are input into the majority voting mechanism to obtain the final classification result.

2. The EEG signal multi-view collaborative integrated classification method according to claim 1, characterized in that: The preset number of sampling points for EEG signals is n, and the number of acquisition channels is N; Collect EEG signals generated by the same subject performing a running imagination task, including: Collection i The EEG signal of the channel; i The EEG signal of each channel is expressed as , i ∈[1,N]; Processing MRI signals into two-dimensional slice images includes: Acquire MRI signals; wherein the MRI signals are three-dimensional MRI images in the format of A*B*C, where A represents pixels of the image in the left-right direction, B represents pixels of the image in the front-back direction, and C represents pixels of the image in the up-down direction. The sagittal plane, coronal plane, and transverse plane of the three-dimensional MRI image correspond to three directions perpendicular to the X-axis, the Y-axis, and the Z-axis, respectively; By slicing along the X-axis, Y-axis and Z-axis directions, a two-dimensional slice image corresponding to the MRI signal is obtained.

3. The EEG signal multi-view collaborative integrated classification method according to claim 1, characterized in that: The pre-processed EEG signal is obtained by using common mean reference and frequency domain filtering technology, including: The formula corresponding to the common average reference is: , filter out the common mode components in the EEG signal; in, Indicates i The signal after the average reference of the channels is i ∈[1,N], N represents the number of acquisition channels, Indicates i The signal of each channel; Then, the common-average referenced signal was band-pass filtered in the 8-30 Hz frequency band through a fourth-order Butterworth band-pass filter to obtain the preprocessed EEG signal.

4. The method for multi-view collaborative integration classification of EEG signals according to claim 1, characterized in that: The cortex of the head model is divided to obtain a source model containing several grid points, including: FreeSurfer software and MNE-python library were used to process the two-dimensional slice images, and the boundary element model was used to generate a three-shell volume conduction model with a conductivity ratio of 1:1 / 50:1 as the head model; the head model includes a three-layer structure, and the three-layer structure is scalp, skull and cortex from the outside to the inside; The cortical surface of the three-layer structure is evenly divided to obtain a source model containing 20484 grid points, where each grid point corresponds to a source dipole.

5. The method for multi-view collaborative integration classification of EEG signals according to claim 1, characterized in that: Using the inverse problem processing mechanism and lead field matrix of ESI technology, the pre-processed EEG signal of the current subject's cerebral scalp is mapped to the cortex of the head model, including: The sLORETA method is used to map the preprocessed EEG signal of the current subject to the cortex of the head model; the regularization parameter is set to 1.

6. The EEG signal multi-view collaborative integrated classification method according to claim 1, characterized in that: Based on the coordinates of the subject's scalp electrodes, the central source dipole is calculated from the source dipoles to obtain the circular sub-area, including: Obtain the coordinates of the scalp electrodes; The source dipole closest to the scalp electrode position was calculated as the central source dipole; Based on the preset radius, a circular sub-area is obtained with the central source dipole as the center.

7. The method for multi-view collaborative integration classification of EEG signals according to claim 1, characterized in that: The geometric features include at least: the sum of vector lengths, the sum of vector angles, the sum of distances to the origin, the sum of distances to the major axis, and the sum of distances to the minor axis; The source signal to be processed is converted into a 3D SODP graph, and then the three-dimensional geometric features of the 3D SODP graph are calculated, including: Obtain the central source dipoles corresponding to the electrodes located in the left central area of ​​the cerebral scalp, the electrodes located in the right central area of ​​the cerebral scalp, and the electrodes located in the midline central area of ​​the cerebral scalp, draw a 3D SODP diagram for the source signal of the circular sub-area corresponding to the central source dipole, and obtain the three-dimensional geometric characteristics of the source signal; By the vector length sum formula: , calculate the sum of vector lengths SVL; in, , and Indicates 3D SODP j The three-dimensional coordinate values ​​of the sample points, T is the total number of 3D SODP sample points; The total formula for the vector angle is: , calculate the sum of the vector angles SA; in,( , , )=( , , ), ( , , )=( , , )； By the formula of the sum of distances to the origin: , calculate the total distance to the origin SDTO; The formula for the sum of the distances to the major axis is: ； Calculate the sum of distances to the major axis SDTMA; in, , , , ( , , ) represents the point with the largest Z coordinate in 3D SODP, ( , , ) represents the point with the smallest Z coordinate in 3D SODP, and the symbol · represents vector dot product; The sum of the distances to the minor axis is calculated by: , calculate the sum of the distances to the minor axis ; in, , , a, b, c are a, b, c in the plane equation ax+by+cz=0 of the fitting plane involved in the 3D SODP diagram.

8. The method for multi-view collaborative integration classification of EEG signals according to claim 1, characterized in that: Using the CSP method, the feature vectors corresponding to each source signal to be processed are extracted, including: Get the CSP projection matrix; A preset number of row vectors are selected from the CSP projection matrix to form a spatial filter, and then the source signal to be processed is matrix multiplied with the spatial filter to obtain a eigenvector; wherein the number of eigenvectors is the same as the preset number.

9. The method for multi-view collaborative integration classification of EEG signals according to claim 1, characterized in that: After inputting the three-dimensional geometric features into a first classifier corresponding to the 3D SODP, a first classification result is obtained; Input the feature vector into the second classifier corresponding to the CSP method to obtain a second classification result; Before performing weighted fusion of the three-dimensional geometric features and the feature vectors and inputting the fused data into a third classifier to obtain a third classification result, the method further includes: Input the three-dimensional geometric features of the labeled imagination task into several classifiers, determine the classifier with the highest classification accuracy as the first classifier, and record the accuracy acc1 of the first classifier; Input the feature vector of the labeled imagination task into several classifiers, determine the classifier with the highest classification accuracy as the second classifier, and record the accuracy acc2 of the second classifier; By formula: , calculate the weight weight1 of the three-dimensional geometric features during weighted fusion; By formula: , calculate the weight weight2 of the feature vector during weighted fusion; Multiply the 3D geometric features by weight weight1, multiply the feature vector by weight2, and then serially concatenate the two to obtain fused data. Input the fused data with labeled imagination tasks into several classifiers, and determine the classifier with the highest classification accuracy as the third classifier.

10. A multi-view collaborative integrated classification system for EEG signals, characterized in that: The system comprises: A processing module is used to collect EEG signals generated by the same subject performing the running imagination task and MRI signals in a resting state; process the MRI signals into two-dimensional slice images; and obtain the pre-processed EEG signals through common average reference and frequency domain filtering technology; The ESI module is used to obtain a head model by using the forward problem processing mechanism and two-dimensional slice diagram of the ESI technology; wherein the head model includes a three-layer structure, and the three-layer structure is scalp, skull and cortex from outside to inside; the cortex of the head model is divided to obtain a source model including a number of grid points; wherein each grid point corresponds to a source dipole; the potential generated by each source dipole at the scalp electrode in three directions of space is calculated, and then a lead field matrix is ​​obtained; the inverse problem processing mechanism and lead field matrix of the ESI technology are used to map the preprocessed EEG signal of the scalp of the current subject to the cortex of the head model as a cortical mapping signal; The data extraction module is used to calculate the central source dipole from the source dipole based on the coordinates of the subject's scalp electrodes, thereby obtaining a circular sub-region; using the cortical mapping signal in the circular sub-region as the source signal to be processed; converting the source signal to be processed into a 3D SODP map, thereby calculating the three-dimensional geometric features of the 3D SODP map; and using the CSP method to extract the feature vector corresponding to each source signal to be processed; The classification module is used to input the three-dimensional geometric features into the first classifier corresponding to the 3D SODP to obtain the first classification result; input the feature vector into the second classifier corresponding to the CSP method to obtain the second classification result; perform weighted fusion of the three-dimensional geometric features and the feature vector, input the fused data into the third classifier to obtain the third classification result; input the first classification result, the second classification result and the third classification result into the majority voting mechanism to obtain the final classification result.

Images