A system and method for classifying states of consciousness based on multimodal brain signals

Abstract

The present invention discloses a consciousness state classification system and method for multimodal brain signals, which belongs to the field of biomedical signal processing. The system includes: a multimodal head data acquisition unit for collecting multi-channel EEG signals and binocular eye movement data, and a multimodal data preprocessing unit for preprocessing multimodal raw data of the human head and obtaining multi-channel EEG feature information and eye movement basic feature information sets; the preprocessing process includes signal preprocessing and feature vector construction steps; multi-channel EEG feature information extraction includes target frequency band EEG data filtering and multi-band EEG data extraction steps, and the obtained original multi-channel EEG signals are preprocessed to obtain multi-channel EEG feature signals that are convenient for subsequent Transformer model encoding. This method solves the problems of information loss in traditional convolutional neural networks during global feature extraction and low classification accuracy of single-input Transformer models.

Description

Technical Field

[0001] The present invention belongs to the field of biomedical signal processing, and specifically relates to a consciousness state classification system and method for multimodal brain signals. Background Art

[0002] With the rapid development of deep learning technology in the field of artificial intelligence, its excellent data parsing and processing performance is gradually being recognized by people. Its advantage is that it not only has good parsing results for single-modal datasets, but also Transformer, due to its unique data encoding and decoding methods, can perform good semantic representation and analysis of multi-modal datasets, and is attracting the attention of more and more researchers.

[0003] Currently, most research on consciousness recognition focuses on single-modal areas such as EEG signals and medical consciousness measurement scales, resulting in inconsistent results in classifying states of consciousness. The pupillary reflex is a physiological response that changes pupil size through contraction of the iris muscle. It involves multiple brain tissues related to human consciousness, such as the anterior tegmental area of ​​the midbrain, the bilateral EW nuclei, and the oculomotor nerve. Changes in pupillary light reflexes and the spatiotemporal characteristics of eye movements are two important signs reflecting the state of consciousness. However, most judgments of consciousness state are made by professional physicians manually observing EEG spectra and the patient's eye features to draw conclusions, which is inefficient. Therefore, multimodal automatic consciousness recognition that integrates EEG signals and eye movement signals has become a key area of ​​development in the medical field.

[0004] Most traditional AI-based consciousness state detection methods and deep learning approaches rely on single models. For example, the traditional CNN model, whose convolutional kernel excels at extracting local features, has limitations in representing global features. While the single-input Transformer model, based on a self-recurrent neural network, can effectively capture long-range feature dependencies, it processes only a single modal signal, which can negatively impact experimental accuracy and various metrics. These traditional and single-model approaches can negatively impact experimental accuracy and various metrics. Summary of the Invention

[0005] This paper aims to solve the problem of low accuracy in consciousness state classification caused by information missing in global feature extraction of traditional convolutional neural networks and the fact that single-input Transformer models can only process single-modal signals.

[0006] To achieve the above objectives, this application adopts the following technical solutions:

[0007] A multimodal brain signal consciousness state classification system, comprising:

[0008] Multimodal head data acquisition unit, multimodal data preprocessing unit, multimodal information encoding unit and multimodal information decoding unit;

[0009] The multimodal head data acquisition unit is used to acquire multimodal human brain data information of multi-channel EEG signals and binocular eye movement signals;

[0010] The multimodal data preprocessing unit is used to extract features from the multi-channel EEG signals and binocular eye movement signals respectively to obtain multi-channel multi-band EEG consciousness-related data and binocular eye movement-related feature information;

[0011] The multimodal information encoding unit is used to receive multi-channel multi-band EEG consciousness-related data and binocular eye movement-related feature information, parse and encode the EEG and eye movement modal information respectively, and obtain multimodal consciousness information encoding by data feature fusion;

[0012] The multimodal information decoding unit is used to receive the multimodal consciousness information encoding, perform decoding and type discrimination, and output the consciousness state classification result.

[0013] Preferably, the EEG preliminary features extracted by the multimodal data preprocessing unit are EEG characteristic signals of multi-channel and multi-consciousness related frequency bands;

[0014] The preliminary features of the binocular eye movement signal extracted by the multimodal data preprocessing unit are velocity and acceleration information features on the vertical and horizontal components of the eye movement.

[0015] A classification method for a multimodal brain signal consciousness state classification system comprises the following steps:

[0016] S11, acquiring multimodal human brain data information, wherein the multimodal human brain data information includes 2D data of multi-channel EEG signals of the head and binocular eye movement signals;

[0017] S12, performing feature extraction on the multimodal human brain data information to obtain multi-band multi-channel EEG feature information and binocular eye movement related feature information;

[0018] S13, respectively analyzing and encoding the multi-band multi-channel EEG feature information and binocular eye movement-related feature information, and obtaining a multimodal consciousness information code by data feature fusion, thereby obtaining a multimodal consciousness information code;

[0019] S14, the multimodal consciousness information is encoded and decoded and the type is determined through a decoder network, and the consciousness state classification result is output.

[0020] Preferably, in step S12, the extracted EEG preliminary features are EEG characteristic signals of multi-channel and multi-consciousness related frequency bands, and the extracted binocular eye movement signal preliminary features are velocity and acceleration information features on the vertical and horizontal components of the eye movement.

[0021] Preferably, in step S12, the multi-channel EEG feature information extraction includes filtering processing of target frequency band EEG data and extraction of multi-band EEG data.

[0022] Preferably, the specific operation of filtering the target frequency band EEG data is:

[0023] The multimodal human brain data information is filtered in turn through a low-pass filter, a band-pass filter and a notch filter. The low-pass filter is used to remove baseline interference, the band-pass filter is used to remove frequency bands not needed for consciousness evaluation, and the notch filter is used to remove baseline interference to obtain a multi-channel EEG signal for consciousness evaluation in a specific target frequency band.

[0024] Preferably, the specific operation of extracting multi-band EEG data is:

[0025] The EEG data of the specific target frequency band is decomposed into a plurality of bandpass filters of different frequency bands through a continuous wavelet transform operation to obtain EEG signals of different sub-interval frequency bands in the specific target frequency band.

[0026] Preferably, the wavelet transform is based on the Mollet wavelet basis function.

[0027] Preferably, in S12, the multi-band multi-channel EEG feature information, binocular eye movement related feature information and consciousness state labels are combined in series with time as the row dimension, and the row dimension where time is located is divided into three types of data: model training set, model verification set and model result test set according to a preset ratio.

[0028] Preferably, in S14, the decoder network is a recurrent neural network or a long short-term memory network. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flowchart of the consciousness state classification system based on multimodal brain signals according to an embodiment of the present application;

[0030] Figure 2 This is a flowchart of the Transformer model training in an embodiment of the present application;

[0031] Figure 3 This is a flowchart of the data pre-processing unit in an embodiment of the present application;

[0032] Figure 4 This is a diagram of the overall Transformer network architecture of an embodiment of the present application;

[0033] Figure 5 This is a diagram showing the actual working of the multimodal EEG signal consciousness classification system according to an embodiment of the present application;

[0034] Figure 6 This is a multi-band EEG information graph obtained by wavelet transforming a single-channel EEG signal in an embodiment of the present application;

[0035] Figure 7 A monocular eye movement trajectory diagram of an embodiment of the present application;

[0036] Figure 8 The xy-axis motion trajectory of the monocular eye movement in the embodiment of the present application;

[0037] Figure 9 This is the monocular eye movement xy axis feature information extracted in the embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of this application clearer, this application will be further described in detail below with reference to the accompanying drawings.

[0039] It should be noted that the illustrations provided in this embodiment are only used to schematically illustrate the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0040] The following combination Figure 1-9 To describe the consciousness state classification system of multimodal brain signals in an embodiment of the present application.

[0041] Example 1

[0042] A multimodal brain signal consciousness state classification system, the system comprising:

[0043] Multimodal header data acquisition unit, multimodal data preprocessing unit, multimodal information encoding unit and multimodal information decoding unit,

[0044] The multimodal head data acquisition unit is used to acquire multimodal human brain data information such as 4-channel EEG signals of the head and 2D data of binocular eye movement signals;

[0045] The multimodal data preprocessing unit is used to extract features from multi-channel EEG signals and binocular eye movement signals. The extracted EEG preliminary features are EEG feature signals in multi-channel and multi-consciousness related frequency bands. For example, the extracted binocular eye movement signal preliminary features are velocity and acceleration information features in the vertical and horizontal components of the eye movement.

[0046] The multimodal information encoding unit inputs the feature information preprocessed by the multimodal data preprocessing unit into the multimodal information encoding unit, and obtains the multimodal consciousness information code by data feature fusion;

[0047] The multimodal information decoding unit will perform decoding and type discrimination based on the multimodal consciousness information encoding and output the consciousness state classification result.

[0048] Next, the operation of the above system is described. Figure 1 shown

[0049] The operation of the system includes the following steps:

[0050] S11, based on the multimodal head data acquisition unit, obtains multimodal human brain data information. At the same time, it is also necessary to obtain the corresponding consciousness state classification results during the training phase, and use them as the human consciousness state label as the label result; the multimodal human brain data information includes multimodal human brain data information such as the head 4-channel EEG signal and binocular eye movement signal 2D data. It should be noted that the human consciousness state label results can be divided into six evaluation levels from 1 to 6, respectively representing the six stages of a person from complete coma to high concentration of attention. The data of these six stages are all the results of cross-induction and annotation by two medical experts after doctors with relevant experience observe the patient's state and the standard consciousness evaluation scale.

[0051] S12, the multimodal data preprocessing unit performs feature extraction based on the received multimodal human brain data information. The extracted EEG preliminary features are EEG feature signals in multi-channel and multi-consciousness related frequency bands. The extracted binocular eye movement signal preliminary features are velocity and acceleration information features in the vertical and horizontal components of the eye movement.

[0052] It should be noted that during this step, if the algorithm is in the training phase, the multimodal data preprocessing unit will divide the EEG characteristic signals and preliminary eye movement characteristic signals of the multi-channel and multi-consciousness-related frequency bands into three types of data: a model training set, a model validation set, and a model result test set in a 6:2:2 ratio in the time dimension. The model training set is responsible for the initial training of the algorithm, and the model validation set is used for preliminary testing to improve the algorithm structure. The model result data set is then input into the final improved model for testing. If the algorithm's accuracy in classifying human consciousness reaches the threshold standard of 97.5%, the algorithm is proven to be reliable.

[0053] S13, using a multimodal information encoding unit to input the pre-processed multi-channel multi-band EEG consciousness-related data and binocular eye movement-related feature information into the multimodal information encoding unit, respectively parse and encode the EEG and eye movement modal information, and obtain multimodal consciousness information encoding through data feature fusion;

[0054] S14: The multimodal information decoding unit decodes and classifies the multimodal consciousness information through a decoder network, and outputs the consciousness state classification result. It should be noted that the Transformer decoder network in this step can use a recurrent neural network or a long short-term memory network as a decoding method. The recurrent neural network, due to its internal network nodes being cyclical, can effectively analyze short-term data and is suitable for the task of accurately classifying short-term consciousness states. The long short-term memory network, due to its memory gate and forget gate structure, can encode and analyze long-term data and is suitable for the task of classifying long-term consciousness states.

[0055] In one embodiment, the training process of the Transformer model is as follows Figure 2 As shown, including:

[0056] S21, use the training set to train the network, specifically,

[0057] First, the training set containing EEG feature signals, eye movement preliminary feature signals, and human consciousness state label results is input into the network for training.

[0058] Adopting a predetermined algorithm model (such as back propagation algorithm, error probability update algorithm for the neural network parameters in the Transformer model) and updating it based on the probability-based attention mechanism algorithm,

[0059] When the inter-class classification accuracy of the algorithm's last predetermined batch (e.g., 10 iterations) is less than or equal to 0.5%, the neural network training converges and parameter updating stops;

[0060] S22, input the validation set data into the trained network model to check the classification effect of the model on the validation set;

[0061] If the network classification accuracy of the validation set exceeds the threshold standard (such as 97.5%), it indicates that the network architecture is reasonable and effective, and the network parameters can meet the requirements of the algorithm application;

[0062] If the validation set accuracy is similar to the test set accuracy, but the validation set accuracy is lower than the threshold standard (such as 97.5%), it indicates that the algorithm needs to be adjusted. The classification algorithm accuracy can be improved by increasing the depth of the neural network and changing the update rate of the attention network.

[0063] If the classification accuracy of the validation set is lower than that of the test set, it proves that the model is overfitting. The model hyperparameters should be adjusted, such as reducing the network update rate and ending network parameter training early, to prevent overfitting.

[0064] S23, input the test set data into the trained network model to check the classification effect of the model on the test machine; if the classification accuracy of the test set network reaches the threshold standard (such as 97.5%), it means that the algorithm is actually usable, and the network parameters and related parameters can be finally determined in the multimodal information encoding unit and the multimodal information decoding unit. The network can be directly executed as an application end.

[0065] In one embodiment, the Transformer model multimodal data preprocessing unit is as follows Figure 3 As shown, including:

[0066] S121, filter the 4-channel EEG signals to obtain EEG signals, and filter the obtained multi-channel EEG signals through a low-pass filter, a band-pass filter, and a notch filter respectively; in this step, the baseline interference is removed by the low-pass filter, the frequency band not needed for consciousness evaluation is removed by the band-pass filter, and finally the baseline interference is removed by the notch filter, and a multi-channel EEG signal with a specific frequency suitable for consciousness evaluation is obtained after the power frequency interference is removed. The multi-channel EEG signal is convenient for subsequent EEG signal processing.

[0067] It should be noted that in this step, the four-channel EEG signals are filtered through a smoothing filter with a time width of 0.03 milliseconds to eliminate external noise interference, and then the EEG target frequency band information suitable for consciousness evaluation is selected through a bandpass filter with a cutoff frequency of 0.5-100 Hz.

[0068] S122, the 4-channel EEG signals are transformed using wavelet transform and characteristic frequency band selection is performed; in this step, the multi-channel specific frequency band EEG signals used for consciousness evaluation are divided into multiple different frequency band sub-intervals through wavelet transform filters with different step lengths and scaling parameters, so as to extract more detailed EEG frequency data and also perform differentiated processing on brain waves of different frequency bands. It should be noted that the specific interval frequency band EEG signals of each channel in the 4 channels are amplified by wavelet transform basis functions with magnifications of 2 times, 4 times, 8 times, and 16 times to obtain 4-channel 4-characteristic frequency band EEG signals suitable for consciousness evaluation, such as Figure 6 As shown in the figure, the top curve is the single-channel original EEG signal, and the bottom curve is the EEG signal of the four frequency domain characteristic bands obtained after wavelet transform, which is used to further mine the EEG data.

[0069] Preferably, the formula of the wavelet basis function ψ is as follows, where ω0 represents the center frequency of the wavelet transform, and t represents the time component of the EEG signal. It is composed of a complex trigonometric function multiplied by an exponential decay function, which can decompose the effective information frequency band where the eye movement is located. The Fourier expression φ of the wavelet transform using the Mollet wavelet basis function is as follows, where w is the wavelet transform frequency parameter, b is the wavelet transform translation parameter, and a is the wavelet transform scaling parameter. By selecting parameters a and b, the wavelet transform can perform preliminary frequency feature extraction on the EEG signal on a time scale.

[0070]

[0071]

[0072] Further, the binocular eye movement feature information extraction includes binocular pupil size, binocular two-dimensional position feature, binocular two-dimensional velocity feature, and binocular two-dimensional acceleration feature. Binocular two-dimensional acceleration features The specific formula is as follows:

[0073]

[0074] in, For a small time component, is the vertical displacement distance of the eyeball in the tiny time component, is the horizontal displacement distance of the eyeball in the tiny time component, is the differential of a small time component, is the change in the vertical velocity of the eyeball in a small time component, It is the change in the horizontal velocity of the eyeball in a small time component.

[0075] S123, pre-processing and feature extraction of binocular eye movement information, extracting binocular pupil size, binocular two-dimensional position features, binocular two-dimensional velocity features, binocular two-dimensional acceleration features, and splicing these feature vectors with the original binocular two-dimensional position data by row to obtain processed multi-channel EEG signal vectors in different bands and binocular eye movement related feature vectors. It should be noted that the binocular two-dimensional velocity features in the binocular eye movement feature information are obtained by performing quadratic spline interpolation of the discrete derivatives of the binocular two-dimensional positions in the horizontal and vertical directions; the binocular two-dimensional acceleration features are obtained by performing quadratic spline interpolation of the discrete derivatives of the binocular two-dimensional velocity features in the horizontal and vertical directions, that is, the quadratic spline interpolation result after the second-order derivative of the binocular two-dimensional positions in the horizontal and vertical directions. The standardized eye movement recording results of the monocular eye movement 2D motion trajectory recorded by a single eye movement camera are as follows. Figure 7As shown, the circles represent the positions of the pupil center in the standardized eye movement records at different times, and the straight lines between the circles represent the movement trajectory of the monocular pupil center.

[0076] It should be noted that the horizontal and vertical positions of the original monocular pupil center in the standardized eye movement record before processing in step S123 are as follows: Figure 8 As shown in , it can record the movement position of the eyeball over time. Monocular two-dimensional velocity features, monocular two-dimensional acceleration features, such as Figure 9 As shown, the two curves above are the horizontal and vertical movement speeds in the standardized eye movement records of the monocular pupil center, and the two curves below are the horizontal and vertical movement accelerations in the standardized eye movement records of the monocular pupil center, which further explore the eye movement feature information.

[0077] S124, after processing, the EEG and binocular eye movement features of the 4-channel 4-feature frequency bands are fused and segmented. Since all data are sampled by row vector time interpolation, the row vector sizes of all feature data are unified. Using the splicing algorithm, all EEG and eye movement feature data are spliced ​​along the row dimension to obtain the pre-processed multimodal consciousness-related head data matrix. It should be noted that in this step, if the network model is in the training stage, the human consciousness state label result needs to be spliced ​​into the last column of the pre-processed multimodal consciousness-related head data as the training result set. The obtained pre-processed multimodal consciousness-related head data matrix is ​​shuffled according to the row dimension of time to increase randomness during training and improve the accuracy of network training effects. The pre-processed multimodal consciousness-related head data matrix is ​​divided into 60% data training set, 20% data validation set, and 20% data test set according to the ratio of 6:2:2 in the row dimension of time. Model training, model optimization, and model feasibility analysis are performed respectively.

[0078] In one embodiment, the Transformer model multimodal network processing module is as follows: Figure 4 As shown in Figure 2, the network model is divided into two parts: multimodal encoding unit and multimodal decoding unit.

[0079] In this multimodal encoding unit, the preprocessed 4-channel 4-band EEG consciousness-related data and the preprocessed binocular eye movement-related feature information are respectively input into a specific embedding layer network. Next, multi-channel EEG information feature encoding and multi-channel eye movement information feature encoding are used to perform preliminary position feature encoding on the embedded EEG consciousness-related data and the embedded eye movement-related data, respectively.

[0080] Next, the two modal data after preliminary position encoding are respectively input into the Transformer EEG semantic information encoding layer and the eye movement semantic information encoding layer to encode the EEG semantic information and eye movement semantic information. Next, the two encoded EEG data and eye movement data are input into the multimodal data fusion layer. Through the attention-related algorithm, the EEG and eye movement modal data are fused to obtain multimodal consciousness-related head signal features. It should be noted that the EEG semantic information encoding layer and the eye movement semantic information encoding layer have the same structure. The basic feature encoding data is first input into the multi-head attention mechanism module, and then into the residual accumulation and regularization layer. The basic feature encoding data is spliced ​​into the residual accumulation and regularization layer data, which plays the role of residual network information propagation. Next, the obtained feature data is respectively input into the feedforward neural network, residual accumulation and regularization layer, and the feature data is also spliced ​​on this data to obtain the semantic information encoding of the relevant modal data.

[0081] The multimodal data fusion layer first applies attention convolution kernels to the semantic information encoding of the input EEG data and eye movement data, normalizing the feature dimensions to the same number of feature channels and feature dimensions. Next, in the multimodal attention intermediate fusion layer, an attention algorithm is used to fuse the normalized EEG signal features with the eye movement signal features to obtain multimodal consciousness-related head signal features. Compared to traditional CNN networks, which can only analyze regional modular features, this method, based on the Transformer algorithm, can summarize contextual semantic information, thereby improving classification accuracy.

[0082] The multimodal decoding unit (see Figure 4 ), including:

[0083] The Transformer decoder network module takes as input the fused feature information encoded from the previous output stage of the algorithm's time series and the multimodal consciousness-related head signal features from the current stage of the algorithm. The encoded data is decoded to produce the state of consciousness output. The fused feature information encoded from the previous output stage is first masked through a head mask network to match the semantic information contained in the output with the semantic information to be decoded in the current output. Next, residual accumulation and regularization layers are used to prevent the algorithm from experiencing issues such as vanishing gradients. The resulting data and the multimodal consciousness-related head signal features from the current stage are input into a multi-head attention mechanism network for attention decoding and allocation. The attention decoding operators are then forward-propagated through a feedforward neural network for semantic decoding analysis. Finally, the semantic decoding information obtained by the feedforward network is analyzed through a linear segmentation and classification layer network to produce the final state of consciousness classification result.

[0084] In a specific application, a multimodal brain signal consciousness state classification system also includes a mechanical hard disk or SSD hard disk storage unit, random access memory data storage, and a computing instruction execution system based on the ARM or X86 architecture. The Transformer encoding network can effectively utilize head EEG signals and eye movement signals to enhance multimodal data, thereby extracting highly robust and versatile multimodal consciousness information encoding;

[0085] The specific formula used by the algorithm applied in the multimodal attention intermediate fusion layer of the multimodal EEG signal and binocular eye movement signal in the Transformer encoding network is:

[0086]

[0087] Among them, k is the group number label of multimodal EEG data, B is the multi-channel EEG encoding signal sequence, L is the binocular eye movement data encoding signal sequence, d is the similarity matrix between B and L data, and P is the network weight vector.

[0088] The feedforward path expression used by the forward propagation layer in the Transformer encoder network can estimate the activation parameters of the first layer of the network but not the second layer of the network.

[0089] Feature screening is performed in the first layer of the network, and the network results without activation function are output in the second layer of the network. The specific formula is as follows:

[0090] max(0,RELU(XW1+q1))W2+q2

[0091] Wherein, RELU represents the network activation function, X represents the network input parameter, W1 represents the first layer network matrix parameter, W2 represents the second layer network matrix parameter, q1 is the first layer network weight, and q2 is the second layer network weight. The algorithm applied in the multimodal attention intermediate fusion layer learns the cross-modal fusion information between EEG data and binocular eye movement data, and fuses the information of different modalities through the modal information fusion algorithm to obtain the feature processing result of cross-modal information. The binocular eye movement data and multi-channel EEG signals are split and spliced, and the binocular eye movement data and multi-channel EEG signals are learned through the above-mentioned internal attention algorithm to generate multimodal consciousness information encoding of the binocular eye movement data and multi-channel EEG signals. The multimodal consciousness information encoding is decoded and predicted by a multimodal information decoding unit embedded with a Transformer decoder network to obtain the consciousness state classification result. The Transformer decoder network can use a recurrent neural network or a long short-term memory network, and the network uses mean square error or variance sum error as the loss function. The Transformer model is trained using the training set. During the training phase, the training set is fed into the model, and the model parameters are updated via the backpropagation algorithm. The model output is analyzed using the validation set, and the Transformer network architecture and hyperparameters are adjusted. This entire model training and optimization process is repeated repeatedly, ensuring that the network achieves high state-of-consciousness classification results on both the training and validation sets. Finally, the test set is fed into the model for testing. If the model meets the required prediction accuracy, the relevant model parameters are determined for multimodal head signal state-of-consciousness classification. During training, the state-of-consciousness classification values ​​predicted by the network on the training set are compared with the true state-of-consciousness values ​​in the training set. The mean squared error between the two is used as the model backpropagation parameter to adjust the model parameters and improve the model classification accuracy. The model structure is adjusted using the validation set. After each training set training, the test set is fed into the model to obtain the predicted state-of-consciousness classification results for the test set. The predicted state-of-consciousness classification results for the test set are compared with the true state-of-consciousness values ​​in the test set, and the model structure is adjusted based on the difference and mean squared error between the two.

[0092] Example 2

[0093] like Figure 5 As shown, the present invention provides a consciousness state classification system based on multimodal brain signals, the system comprising:

[0094] Multimodal head data acquisition device, multimodal data preprocessing unit, Transformer encoder network, Transformer decoder network, and consciousness assessment result comprehensive display terminal.

[0095] The multimodal head data acquisition device is used to acquire in real time the four EEG raw signals and binocular two-dimensional eye movement signals related to human consciousness: the left middle temporal position of T3, which is 10% of the distance from the left ear preconcave to the line connecting the two ear preconcave; the left posterior temporal position of T5, which is 30% of the distance from the left ear preconcave to the line connecting the two ear preconcave; the right middle temporal position of T4, which is 10% of the distance from the right ear preconcave to the right side of T3; and the right posterior temporal position of T6, which is 30% of the distance from the intersection of the two ear preconcave to the right ear preconcave.

[0096] Put the multimodal header data into the multimodal data preprocessing unit for processing.

[0097] First, the multi-channel multimodal consciousness-related head data is extracted, and the real-time raw EEG data and binocular two-dimensional eye movement signals are extracted.

[0098] The multi-band EEG data is segmented using wavelet transform to obtain EEG feature data of 4 channels and 4 feature frequency bands.

[0099] The speed and acceleration related features of binocular eye movement data are extracted, and finally the extracted data is standardized and feature serialized. The 4-channel 4-feature frequency band EEG signal data, binocular eye movement raw data, binocular eye movement speed and acceleration data in the horizontal and vertical directions are transmitted to the comprehensive display terminal of consciousness assessment results.

[0100] The parsed multimodal feature sequence is input into the Transformer encoder network, and then the EEG signal encoding unit and eye signal encoding unit in the Transformer encoding network are converted into multimodal consciousness information encoding through the algorithm applied by the multimodal attention intermediate fusion layer.

[0101] The multimodal consciousness information is encoded and input into the Transformer decoder network to obtain the final consciousness state assessment result, which is then output to the comprehensive display terminal of the consciousness state classification result.

[0102] The comprehensive display terminal of consciousness assessment results can display not only the classification results of the current patient's consciousness state, but also the current 4-channel EEG-related feature data and binocular eye movement-related feature data, realizing comprehensive monitoring of information related to the human consciousness state. This implementation method effectively solves the problem of low accuracy in existing consciousness state classification algorithms and the inability to use EEG data and eye movement data at the same time.

[0103] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A multimodal brain signal consciousness state classification system, characterized by: include: Multimodal head data acquisition unit, multimodal data preprocessing unit, multimodal information encoding unit and multimodal information decoding unit; The multimodal head data acquisition unit is used to acquire multimodal human brain data information of multi-channel EEG signals and binocular eye movement signals; The multimodal data preprocessing unit is used to extract features from the multi-channel EEG signals and binocular eye movement signals respectively to obtain multi-channel multi-band EEG consciousness-related data and binocular eye movement-related feature information; The multimodal information encoding unit is used to receive multi-channel multi-band EEG consciousness-related data and binocular eye movement-related feature information, parse and encode the EEG and eye movement modal information respectively, and obtain multimodal consciousness information encoding by data feature fusion; The multimodal information decoding unit includes a Transformer decoder network module, which is used to receive the multimodal consciousness information encoding, perform decoding and type discrimination, and output the consciousness state classification result; The preliminary EEG features extracted by the multimodal data preprocessing unit are EEG feature signals in multi-channel and multi-consciousness related frequency bands; The preliminary features of the binocular eye movement signal extracted by the multimodal data preprocessing unit are velocity and acceleration information features on the vertical and horizontal components of the eye movement; The network model is divided into two parts: multimodal encoding unit and multimodal decoding unit. In the multimodal encoding unit, the preprocessed 4-channel 4-band EEG consciousness-related data and the preprocessed binocular eye movement-related feature information are respectively input into a specific embedding layer network, and multi-channel EEG information feature encoding and multi-channel eye movement information feature encoding are used to perform preliminary position feature encoding on the embedded EEG consciousness-related data and the embedded eye movement-related data respectively; the two modal data after preliminary position encoding are respectively input into the Transformer EEG semantic information encoding layer and the eye movement semantic information encoding layer to encode the EEG semantic information and eye movement semantic information. Next, the two encoded EEG data and eye movement data are input into the multimodal data fusion layer, and the EEG and eye movement two modal data are fused through the attention-related algorithm.

2. A classification method for a multimodal brain signal consciousness state classification system, characterized in that: The consciousness state classification system based on the multimodal brain signal according to claim 1 comprises the following steps: S11, acquiring multimodal human brain data information, wherein the multimodal human brain data information includes 2D data of multi-channel EEG signals of the head and binocular eye movement signals; S12, performing feature extraction on the multimodal human brain data information to obtain multi-band multi-channel EEG feature information and binocular eye movement related feature information; S13, respectively analyzing and encoding the multi-band multi-channel EEG feature information and binocular eye movement-related feature information, and obtaining a multimodal consciousness information code by data feature fusion, thereby obtaining a multimodal consciousness information code; S14, the multimodal consciousness information is encoded and decoded and the type is determined through the Transformer decoder network, and the consciousness state classification result is output.

3. The classification method of the multimodal brain signal consciousness state classification system according to claim 2, characterized in that: In step S12, the extracted EEG preliminary features are EEG characteristic signals of multi-channel and multi-consciousness related frequency bands, and the extracted binocular eye movement signal preliminary features are velocity and acceleration information features on the vertical and horizontal components of the eye movement.

4. The classification method of the multimodal brain signal consciousness state classification system according to claim 2, characterized in that: In step S12: The multi-channel EEG feature information extraction includes filtering processing of target frequency band EEG data and extraction of multi-band EEG data.

5. The classification method of the multimodal brain signal consciousness state classification system according to claim 4, characterized in that: The specific operation of the target frequency band EEG data filtering process is: The multimodal human brain data information is filtered in turn through a low-pass filter, a band-pass filter and a notch filter. The low-pass filter is used to remove baseline interference, the band-pass filter is used to remove frequency bands not needed for consciousness evaluation, and the notch filter is used to remove baseline interference to obtain a multi-channel EEG signal for consciousness evaluation in a specific target frequency band.

6. The classification method of the multimodal brain signal consciousness state classification system according to claim 5, characterized in that: The specific operation of extracting multi-band EEG data is as follows: The EEG data of the specific target frequency band is decomposed into a plurality of bandpass filters of different frequency bands through a continuous wavelet transform operation to obtain EEG signals of different sub-interval frequency bands in the specific target frequency band.

7. The classification method of the multimodal brain signal consciousness state classification system according to claim 6, characterized in that: The wavelet transform is based on the Mollet wavelet basis function.

8. The classification method of the consciousness state classification system of multimodal brain signals according to claim 2, It is characterized in that In S12, the multi-band and multi-channel EEG feature information, binocular eye movement-related feature information, and consciousness state labels are combined in series with time as the row dimension, and the row dimension where time is located is divided into three types of data according to a preset ratio: model training set, model verification set, and model result test set.

9. The classification method of the consciousness state classification system of multimodal brain signals according to claim 2, characterized in that: In S14, the Transformer decoder network is a recurrent neural network or a long short-term memory network.

Images