Method for detecting and classifying electroencephalogram events of electroencephalogram signals by using target detection model DINO

Abstract

The invention discloses a method for detecting and classifying electroencephalogram events of electroencephalogram signals by using a target detection model DINO, and belongs to the technical field of deep learning. The method mainly comprises three stages of data set acquisition, data preprocessing, feature extraction and detection classification evaluation. According to the method, a complete electroencephalogram data file can be input, a tedious post-processing process is not needed, many defects of post-processing are overcome, an end-to-end model is constructed, original data are directly input, and an event detection result is output; meanwhile, many limitations existing in manual detection are overcome, a universal automatic electroencephalogram event detection framework is established by using a deep learning method, a doctor can be assisted to play a role in tasks such as epileptic seizure detection, sleep staging and artifact detection in various clinical application scenes, the electroencephalogram event labeling efficiency is improved, meanwhile, the burden of the clinician is reduced, and the workload of the clinician is reduced. And the pressure of medical resources is relieved.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology and relates to an event detection and classification method for electroencephalogram (EEG) signals using a target detection model DINO, which is used to improve the efficiency of doctors' diagnosis and treatment of brain diseases such as epilepsy. Background Art

[0002] Current deep learning methods generally model EEG event detection as a classification problem. First, the complete EEG data is divided into several slices through a sliding window, and a label is added to each slice (whether it contains an event). The slices are input into the model to obtain slice-level classification results, and finally the slice-level classification results are converted into event detection results through complex post-processing. Some sequence-to-sequence models classify the input sequence point by point, and then convert the point-by-point classification results into event detection results through post-processing. However, the current research methods need to consider the impact of slice length and slice step size on model performance, and rely on cumbersome post-processing processes. Post-processing is usually not obtained by the model through data learning, but requires experts to design; and neither of them can take into account the detection and classification of EEG events.

[0003] In response to the mentioned situation that current research relies on post-processing and cannot take into account the detection and classification of EEG events, the present invention constructs an EEG event detection model based on the target detection model DINO in the field of computer vision, which can simultaneously detect and classify events in EEG signals. In order to extract the essential characteristics of EEG signals, the present invention mainly considers the time-frequency domain characteristics of EEG signals, retaining useful information while removing the noise and redundancy of pure time domain signals; and we introduce the DINO model, which is itself a model for target detection, which can be end-to-end, and its two prediction heads can output categories and positions at the same time.

[0004] The technology related to the present invention includes two aspects: the first is the short-time Fourier transform STFT used for data processing; the second is the main structure of the model Transformer.

[0005] Considering that the original time-domain EEG signal has a high dimension and a complex change pattern, it is not conducive to the model's learning of EEG features and patterns. For the sliced ​​EEG signal, the short-time Fourier transform is used to convert it from the time domain to the time-frequency domain. Compared with the frequency domain information obtained by the traditional Fourier transform, the time-frequency domain information contains the time domain information of the signal, which is very suitable for non-stationary signals such as EEG. The short-time Fourier transform is to slide a window on the signal, perform a Fourier transform on the signal inside the window, and obtain the frequency domain information within the window. The mathematical definition of the short-time Fourier transform is shown in the following formula:

[0006]

[0007] Where x(t) represents the signal to be analyzed, w(t-τ) represents the window function, and f represents the angular frequency. In this way, the short-time Fourier transform can provide both time domain information and frequency domain information.

[0008] The Transformer model was proposed by Vaswani et al. in the paper "Attention is all you need". Transformer was first used as a model in the field of natural language processing (NLP). When its powerful processing ability for sequence data was discovered, it was applied to research in various fields such as computer vision (CV) and EEG signal processing. Transformer is an end-to-end, sequence-to-sequence deep learning model architecture. Its main purpose and innovation lies in the introduction of a multi-head self-attention mechanism to process sequence data. Transformer mainly consists of two parts: an encoder and a decoder. The encoder part of Transformer mainly consists of a multi-head self-attention module and a feed forward neural network (FFN). It receives position-encoded sequence data and extracts the features and semantics of the input data. Position encoding of input data is an indispensable step in Transformer. Its function is to identify the relative position of the data in the input sequence. The representation of the specific position is shown in the following formula:

[0009] PE position(pos,2i) = sin(pos / Temperature 2i / d ) (2)

[0010]

[0011] Where pos represents the position in the sequence dimension, d represents the model dimension of the Transformer, i represents the i-th element of the position vector; Temperature determines the smoothness of the attention map. The attention is calculated by the following formula:

[0012]

[0013] The key to the calculation of attention lies in the query vector Q, the key value vector K and the value vector V, d k Represents the feature dimension of Transformer. For self-attention, these three vectors are respectively transformed from the input sequence through different linear mappings W Q , W K and W VFor the cross attention that appears in the decoder part of Transformer, the Q vector and K, V vectors come from the linear mapping of two different inputs. Transformer also uses a multi-head attention mechanism, as shown in the following formula:

[0014] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0015] head i =Attention(QW i Q ,KW i K ,VW i V ) (5)

[0016] That is, attention will pass through multiple heads in parallel at the same time, and the attention of each head will be calculated independently. After completion, these results are connected together and passed through a linear output mapping layer, where W O , W Q , W K , W V Respectively represent the weights of the corresponding linear output mapping matrix. The decoder part of the Transformer is composed of a multi-head self-attention module, a multi-head cross-attention module and a feedforward neural network. The data input to the decoder will first pass through the multi-head self-attention module, and then do multi-head cross-attention with the output generated by the encoder, and the generated output will pass through the feedforward neural network. Finally, the output of the decoder will pass through a linear output mapping layer and Softmax calculation to obtain the final output. Transformer also has an important application in the DINO model used in the present invention. The Swin-L model, a variant of the Transformer model proposed by Liu et al. in "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows", is used in the backbone network of the model; the deformable (Deformable) Transformer model proposed by Zhu et al. in the document "Deformable DETR: Deformable Transformers for End-to-End Object Detection" is used in the neck part of the DINO model. Summary of the invention

[0017] In response to the mentioned situation that current research relies on post-processing and cannot take into account the detection and classification of EEG events, the present invention constructs an EEG event detection model based on the target detection model DINO in the field of computer vision, which can simultaneously detect and classify events in EEG signals. In order to extract the essential characteristics of EEG signals, the present invention mainly considers the time-frequency domain characteristics of EEG signals, retaining useful information while removing the noise and redundancy of pure time domain signals; and we introduce the DINO model, which is itself a model for target detection, which can be end-to-end, and its two prediction heads can output categories and positions at the same time.

[0018] The technical solution of the present invention:

[0019] A method for EEG event detection and classification using the target detection model DINO is proposed. The goal of EEG event detection and classification is to locate events in single-channel EEG data and to give the detected events the correct classification. The specific steps are as follows:

[0020] (1) Dataset acquisition stage

[0021] (2) EEG signal preprocessing stage

[0022] (3) Feature extraction and detection classification evaluation stage

[0023] Beneficial effects of the present invention: The present invention can input complete EEG data files without the need for tedious post-processing, overcomes many deficiencies of post-processing, builds an end-to-end model, directly inputs raw data and outputs event detection results; at the same time, it overcomes many limitations of manual detection, and uses deep learning methods to establish a general automatic EEG event detection framework, which can assist doctors in a variety of clinical application scenarios such as epileptic seizure detection, sleep staging, artifact detection and other tasks, improves the efficiency of EEG event labeling while reducing the burden on clinicians and alleviating the pressure on medical resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a schematic diagram of the data preprocessing process.

[0025] Figure 2 This is the structure of the DINO model.

[0026] Figure 3 It is a schematic diagram of the comparison denoising module. DETAILED DESCRIPTION

[0027] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0028] Data processing flow chart is as follows Figure 1 As shown, it is described in detail in the examples below.

[0029] The model structure diagram is as follows Figure 2 As shown in Figure 1, the model mainly consists of a backbone network, a Transformer, and a prediction head for obtaining the classification and location of events. Assume that the preprocessed single-channel signal input data is Where 1 represents single-channel data, F represents the frequency domain part after short-time Fourier transform, and T represents the time domain part. The model receives single-channel signal input data constructed by short-time Fourier transform as input. We first extract the multi-scale features of the input data through the backbone network and encode the corresponding position of each layer of features. The multi-scale features are then input into the deformable Transformer model. The output of the decoder of the deformable Transformer model is sent to two prediction heads. The prediction heads are two independent networks that distribute the location and category of the event.

[0030] Figure 3 It is a schematic diagram of the contrast denoising module in DINO.

[0031] Example

[0032] A method for detecting and classifying EEG events of EEG signals using a target detection model DINO, the steps are as follows:

[0033] The EEG event detection and classification method of EEG signals is mainly divided into four stages: data set acquisition, data preprocessing, feature extraction, and detection and classification evaluation;

[0034] (1) Dataset acquisition

[0035] The EEG data is obtained to construct a data set, which at least includes the EEG signal, signal sampling frequency, EEG signal event location and category labeling.

[0036] (2) Data preprocessing

[0037] The preprocessing stage is divided into five stages: data screening (removing incorrectly labeled EEG data), resampling, detection and reconstruction of bad conductors, TCP (Temporal Central Parasagittal) montage, short-time Fourier transform, and time-frequency matrix normalization. The data preprocessing process is as follows Figure 1 As shown in the figure, “*” indicates the process that can be omitted depending on the input data and task objectives.

[0038] (2.1) Data screening

[0039] Exclude EEG signal event category samples with very small data volume and EEG samples with incorrect annotations.

[0040] (2.2) Resampling

[0041] If the EEG signals in the data set have multiple sampling frequencies, in order to achieve a uniform temporal resolution, the selected data must first be resampled to a uniform sampling rate of 200 Hz.

[0042] (2.3) Inspection and reconstruction of damaged conductors

[0043] Bad conductors will produce a large number of artifacts. Depending on the data set and task objectives, it may be necessary to reduce the impact of artifacts on the detection results. The focus of bad conductor detection is the detection of electrode-related artifacts. Therefore, the present invention uses the tools provided by the relevant library to calculate the standard deviation of the EEG signal, detect completely flat EEG signals and EEG signals that differ too much from the standard deviation. These EEG signals generally come from loose electrodes and poor electrode contact, so they are excluded. After excluding bad conductors, the data needs to be reconstructed, and the spherical spline interpolation function is used to reconstruct the EEG channel that has just been deleted.

[0044] (2.4) TCP Montage

[0045] This paper performs TCP montage processing on EEG signals. The specific implementation method is to subtract the EEG signals of two corresponding channels to obtain a new single-channel EEG signal.

[0046] (2.5) Short-time Fourier transform and time-frequency matrix normalization

[0047] Considering that the original time-domain EEG signal has low information density, it is not conducive to the model's learning of EEG features and patterns. For the sliced ​​EEG signal, short-time Fourier transform is used to convert it from the time domain to the time-frequency domain. Compared with the frequency domain information obtained by the traditional Fourier transform, the time-frequency domain information contains the time domain information of the signal, which is very suitable for non-stationary signals such as EEG. The short-time Fourier transform is to slide a window on the signal, perform a Fourier transform on the signal inside the window, and obtain the frequency domain information within the window. In this way, the short-time Fourier transform can provide both time domain information and frequency domain information. The mathematical definition of the short-time Fourier transform is shown in the following formula:

[0048]

[0049] Where x(t) represents the signal to be analyzed, w(t-τ) represents the window function, and f represents the angular frequency.

[0050] In order to avoid large differences in data between EEG records, the present invention normalizes the time-frequency matrix generated by short-time Fourier transform. Specifically, for each time window of the time-frequency matrix, the proportion of different frequencies within the time window is calculated. The detailed processing method is shown in the formula:

[0051]

[0052] Where M is the time-frequency matrix of a time window, M(t) represents the amplitude value of the time-frequency matrix at time point t, and sum(M) represents the sum of the amplitude values ​​of all time points in the time window. The normalized data is used as the input of the model.

[0053] After preprocessing, the sampling frequency of the EEG record is 200Hz, and each original multi-channel EEG data will produce 22 independent single-channel EEG data. For short-time Fourier transform, the present invention uses a Hanning window with a length of 1 second, an overlap length of 0.5 seconds, and a zero-filling length of 200.

[0054] (3) Feature extraction

[0055] We propose to use the object detection model DINO based on computer vision to detect and classify EEG events. It is an improved version of the DETR (DEtection TRansformers) series of models for end-to-end object detection. We believe that the target object can be defined as a time period in a time series, and by making some modifications to the model, it can be used as the EEG time series version of the DINO model. Given a single-channel EEG data Where L represents the length of the EEG signal, and the task goal is to obtain the start and end time of the EEG event, recorded as and the categories to which these events correspond[c 1 ,c 2 ,…,c N ]. The goal of this task is similar to the target detection task in computer vision tasks, that is, to identify the target and the corresponding category from a given picture.

[0056] In order to further enhance the model's ability to detect targets and discard empty targets, the EEG data containing targets and the data without EEG events are downsampled 1:1 when inputting data; since the model can input EEG sequence data of any length, and in order to reduce the impact of data of different lengths on model performance, when inputting data, the data of each batch is padded with 0 values ​​to the same length as the longest data in the batch, and the input data length of each batch is manually controlled to be roughly similar. In specific implementation:

[0057] Assume that the dimension of the preprocessed EEG signal data is Among them, 1 represents single-channel data, F represents the frequency domain part after short-time Fourier transform, and T represents the time domain part; the data is input into the DINO model for training, and the model with the highest F1 score on the validation set is selected as the final model, and tested on the test set. The DINO model is as follows:

[0058] (3.1) Backbone network module

[0059] The backbone network is used to extract the representation of the input signal. The present invention currently uses the Transformer-based Swin-L model to output the extracted features in multiple feature dimensions. Research results show that the backbone network pre-trained on the image dataset is still effective when facing other tasks. Therefore, at the beginning of training, the weights of the Swin-L model pre-trained for classification on the ImageNet-22k image dataset are initialized.

[0060] (3.2) Position encoding module

[0061] Since the features output by the backbone network module contain multiple scales, the position encoding is divided into two parts. The first part is the position encoding within each layer of features, and the second part is the scale-level position encoding, which is used to positionally encode the tokens within each layer of features and identify which layer the feature map is in.

[0062] PE=PE postion +PE scale-level (6)

[0063] Among them, PE represents the output of the position encoding module, PE postion Represents the position encoding within each layer of features, PE scale-level Positional encoding to represent scale levels;

[0064] For the position coding within each layer of features, the present invention currently uses absolute position coding based on sine and cosine trigonometric functions. And because the model only needs to locate events in the time dimension, the present invention only performs one-dimensional position coding in the time dimension, as shown in the following formula:

[0065] PE position(pos,2i) = sin(pos / Temperature 2i / d ) (7)

[0066]

[0067] Where pos represents the position in the time dimension, d represents the model dimension of the deformable Transformer, and i represents the i-th element of the position vector; Temperature determines the smoothness of the attention map; and unlike the absolute position encoding of the original version (Temperature = 10000), we follow previous studies and use the same Temperature setting as DINO, setting it to a smaller value (Temperature = 20) to avoid the Transformer's attention map being too smooth.

[0068] For scale-level positional encoding, use learnable positional encoding Among them, e l Represents the learnable positional encoding embedding vector of layer l, N represents the number of layers in the multi-layer scale, and d represents the model dimension of the deformable Transformer to identify which layer the feature map belongs to.

[0069] (3.3) Deformable Transformer Module

[0070] Before inputting the deformable Transformer module, the multi-scale feature maps output by the backbone network are first positionally encoded, and then their number of channels is unified into dimension d using a linear projection layer to obtain a multi-scale input feature map in is a multi-scale feature with L layers, H l and W l Represents the size of the feature map of layer l, i.e., height and width.

[0071] Multi-scale deformable attention and multi-scale deformable Transformer are proposed in Deformable DETR. Figure 2 As shown in the figure, the structure of the deformable Transformer module is roughly similar to the original Transformer model, consisting of an encoder module composed of several encoding layers, a decoder module composed of several decoding layers, plus an additional query selection (Query Selection) module and a contrastive denoising (Contrastive De Noising, CDN) module.

[0072] 3.3.1) Encoder module

[0073] The input received by the deformable Transformer module is first fed into the encoder module. The encoder module consists of several encoder layers, each of which consists of a multi-scale deformable attention module, a dropout layer and a layer normalization layer, and a 2-layer feedforward neural network.

[0074] First, after receiving the multi-scale feature input, the encoder module needs to generate the reference point coordinates required by the internal multi-scale deformable attention module so that it can be input into the multi-scale deformable attention module together with the multi-scale features. The specific process of generating the reference point coordinates is as follows:

[0075] 3.3.1.1) For multi-scale input feature maps The size of each layer is H l ×Wl The feature map generates reference point coordinates normalized to between 0 and 1 Where 2 represents the normalized 2D horizontal and vertical coordinates of the feature point with a value between 0 and 1.

[0076] 3.3.1.2) Concatenate the multi-scale reference point coordinates together and expand the dimensions to form the final reference point coordinates: Where B represents the batch size, Q = H1×W1+H2×W2+…+H L ×W L represents the multi-scale feature map connected after flattening, L represents the multi-scale features with L layers, and 2 is the normalized 2D horizontal and vertical coordinates of the reference point. That is, each feature point of the original input corresponds to L reference points, because the original input becomes an L-layer multi-scale feature.

[0077] Next, the generated reference point coordinates and multi-scale features are sent to several encoder layers together. For an encoder layer: the received input first enters the multi-scale deformable attention module for self-attention operation, then undergoes dropout and layer normalization operations, and finally enters the feedforward neural network. The final output of an encoder layer is The output reference point of the encoder layer passes through the next encoder layer through the loop. After passing through several encoder layers, the output of the encoder is

[0078] For the Multi-Scale Deformable Attention module in the encoder, the details are as follows:

[0079] Multi-scale deformable attention is proposed by the deformable Transformer to improve the computational efficiency of the original multi-head attention module while integrating features of multiple scales. Therefore, it is also a type of multi-head attention. First, for the multi-scale input feature map For ease of calculation, it is flattened to Where B represents the batch size, Q is the length of the multi-scale feature map connected after flattening, and d is the model dimension of the deformable Transformer.

[0080] Secondly, the multi-scale deformable attention module maps the sampling offsets to the matrix Attention weights mapping matrix The value mapping matrix W′ and the output mapping matrix W are initialized. These four matrices are essentially linear projection layers. And the output mapping matrix The weight part of is initialized using Xavier, and the bias part is initialized to 0. Where d represents the model dimension, d v Indicates the feature dimension size calculated based on the number of multi-head attention, the number of multi-scale feature layers, and the number of sampling points. Attention weight mapping matrix The weight part and bias part of are initialized to 0, where M is the number of multi-head attention, L is the number of layers of multi-scale features, and K is the number of sampling points. For the sampling point offset mapping matrix Where d represents the model dimension, M represents the number of multi-head attention, L represents the number of layers of multi-scale features, K represents the number of sampling points, and 2 represents the 2D coordinate of each sampling point. Its weight part is initialized to 0. In the multi-head multi-scale deformable attention module, the number of multi-head attention heads is always fixed to 8, and these 8 heads can be responsible for just one initial offset direction; based on this sample point offset mapping matrix, the offset part is always initialized to {Δp 1lqk =(-k,-k),Δp 2lqk =(-k,0),Δp 3lqk =(-k,k),Δp 4lqk =(0,-k),Δp 5lq =(0,k),Δp 6lqk =(k,-k),Δp 7lqk =(k,0),Δp 8lqk =(k,k)}, where k∈{1,2,…,K} is the number of sampling points in each direction. Different sampling points in the same direction are responsible for different offset lengths from 1 to K. For example, in the case of K=4 sampling points, Δp 1lqk =(-k,-k) and the offset coordinates of the four sampling points are {(-1,-1),(-2,-2),(-3,-3),(-4,-4)} respectively.

[0081] The operation of the multi-scale deformable attention module is defined as follows:

[0082]

[0083] in is a multi-scale input feature map. In order to facilitate calculation, it is flattened and connected as Q is the query length of the deformable attention (the query length is the length of the multi-scale feature map connected after flattening). For ease of understanding, we use To express; Multi-scale features Add position encoding as a query for the multi-scale deformable attention module; represents the query (i.e. z q ) is generated by the corresponding reference point 2D coordinate Ref between 0 and 1 mentioned in the encoder section above q ; The entire attention weight matrix is ​​used In order to make the formula clear, A is used. mlqk represents a point in the attention weight matrix A, that is, the attention weight of the kth sampling point of the qth feature point of the lth layer of the mth attention head. The same is true for other places in the formula; By attention weight mapping matrix Then it is obtained through the softmax normalization layer, which can ensure Represents the denormalization operation, that is, the reference point coordinates with values ​​between 0 and 1 Restored to coordinates based on the true size of the feature, l represents the lth layer; the sampling point offset matrix is ​​the same as the attention weight A, Represents the entire sampling point offset matrix, which is also z q Offset the mapping matrix by sampling points In the encoder, the sampling point offset matrix learns the real coordinates, which are compared with the denormalized reference point coordinates. Add together to get the offset reference point coordinates, that is, the sampling point coordinates sampled in the value matrix. The value matrix passes through the linear mapping layer through the input feature x We get W in the corresponding formula. m ' m (m represents the mth attention head, W′ represents the entire linear mapping layer matrix); in the decoder, the sampling point offset matrix learns the offset real coordinates of the prior anchor box based on the previous position (the prior anchor box will be introduced in the query selection section below). This is the iterative bounding box refinement strategy adopted by DINO from Deformable DETR. The reference point coordinates input to the decoder are After sampling in the value matrix, it is weighted by the attention weight matrix A and finally output through the linear mapping layer W (corresponding to W in the formula m ), and obtain the output of the final multi-scale deformable attention module These outputs then go through the dropout layers and so on as described above.

[0084] 3.3.2) Query Selection Module

[0085] The output of the encoder module is input into the query selection module. DINO proposes a mixed query selection strategy (MixedQuery Selection), which uses the output of the encoder module Generate initial anchor boxes (initial anchor boxes / proposal boxes), and select the first few anchor boxes to generate reference points for the input decoder module. These reference points have two functions: one is to generate the position encoding of the query as the input decoder module The second is to serve as a reference point for the deformable attention in the decoder module.

[0086] Specifically:

[0087] 3.3.2.1) Query selection First, the position and size of all anchor boxes (proposal boxes) need to be initialized: the specific method for generating the initial anchor box position is consistent with the principle of generating reference point coordinates in the encoder module, and the normalized coordinates of the center point of each layer of feature map grid are obtained as the initial anchor box position; for the initial size of each anchor box, the length and width of each layer are initialized to 0.05*2 by default l , where l is the lth layer;

[0088] 3.3.2.2) Connect the initial anchor box position coordinates and size together to get The first two digits in 4 are the 2D coordinates of the proposal, and the last two digits are the size of the proposal; finally, an unsigmoid function is used for denormalization.

[0089] 3.3.2.3) For the output of the received encoder module First it passes through a linear layer and a normalization layer, and then passes through two separate linear layers to generate a classification confidence score for each feature point. and an anchor box offset (4 represents 2D coordinates and width and height), C represents the number of categories, and the anchor box offset is combined with the initial anchor box position coordinates of each feature point generated in step 3.3.3.2) Add together to get the final prior anchor box. Then, for the classification confidence score Sort by the confidence score of the first category from high to low, and select the top-NQ corresponding prior anchor boxes as the reference point coordinates of the input decoder module after sigmoid normalization. Here NQ is the number of queries input to the decoder module. The content part of the query input to the decoder module is randomly initialized and learned in the decoder module.

[0090] 3.3.3) Decoder

[0091] The decoder also consists of several decoder layers. A decoder layer consists of a multi-head self-attention module, a multi-scale deformable attention module, and a 2-layer feedforward neural network. Each attention module is followed by a dropout layer and a normalization layer. The multi-scale deformable attention module in the decoder layer is the same as that in the encoder layer.

[0092] The query input to the decoder module consists of two parts, the position encoding and content encoding The position code is the reference point coordinates generated in 3.3.3). Generate, the generation method is as follows.

[0093] For a decoder layer: first use the reference point coordinates initialized in step 3.3.3) to generate the position encoding of the sine and cosine trigonometric functions for the 2D horizontal and vertical coordinates and the width and height of each reference point, and then connect the four separately generated position encodings together and pass them through a 2-layer multi-layer perceptron, as shown in the following formula.

[0094]

[0095] Among them, PE y PE x PE w PE h Corresponding to the position codes generated for the 2D vertical and horizontal coordinates and width and height of each reference point mentioned above; The coordinates of the reference point Generates the positional encoding of the query as input to the decoder module.

[0096] For a certain decoder layer, the query input to the decoder module is first passed through the multi-head self-attention module, and the position encoding of the multi-head self-attention module is used Then, the multi-scale deformable cross attention module and the output of the encoder module are used to perform cross multi-scale deformable attention. At this time, the query position encoding is also The reference point uses the 2D horizontal and vertical coordinates and width and height of the prior anchor box. Finally, the feedforward neural network is used. In the decoder module, due to the iterative bounding box fine-tuning strategy, each decoder layer does not directly predict the new detection box position, but predicts its relative offset relative to the previous position, and generates a new reference point coordinate by adding. And generate a new position code Input to the next decoder layer.

[0097] 3.3.4) Comparison denoising module

[0098] The denoising module is an independent module, which is added to the DINO module as a plug-in. It was first proposed in DN-DETR and improved in DINO to become a contrastive denoising module. This module can greatly accelerate the convergence of the model. The idea of ​​the contrastive denoising module is to add noise to the true value to generate two different denoising queries: positive query and negative query, and add these two denoising queries to the query input into the decoder module; for positive samples, the model needs to reconstruct the real detection box, and for negative samples, the model needs to judge it as a background class. The generation of positive and negative samples is controlled by two given hyperparameters λ1 and λ2.

[0099] like Figure 3 As shown in the figure, when the added noise size is within λ1, a positive sample is generated, and the model needs to reconstruct the real detection box; conversely, when the added noise size is between λ1 and λ2, a negative sample is generated, and the model needs to classify it as the background class. In the DINO model, the number of added denoising queries is fixed to ND. For an input containing nt real detection boxes, All ND queries generated will be concatenated with the NQ queries input to the decoder module. At this time, the two parts of the query input to the decoder are encoded as and content encoding as

[0100] 3.4) Prediction Head

[0101] The prediction head consists of two 3-layer multi-layer perceptrons (MLPs), which respectively accept the output of NQ queries from the decoder module of the model and generate the location and category of the event for each query.

[0102] (4) Classification and evaluation

[0103] Two tasks are designed:

[0104] Task A: EEG event detection task, the model locates the start and end time of EEG events;

[0105] Task B: EEG event detection and classification task, the model locates the start and end time of EEG events and classifies the events at the same time.

[0106] When the model is inferring, the NQ queries are first sorted by confidence, the top-NI predictions are selected, and then the final prediction events are screened out using the threshold method. The threshold used in the threshold method is the threshold at which the model obtains the highest F1 score on the validation set.

[0107] The present invention uses event-based evaluation indicators to evaluate model performance by calculating the overlap between predicted events and real events. TP, FP, and FN are defined. The calculation method and process are as follows:

[0108] First, the intersection of the start and end coordinates of the real event and the predicted event is calculated to account for the proportion of the position union of the two events, and the intersection ratio is calculated. Then, the intersection ratio between each predicted event and each real event is calculated. If the intersection ratio is greater than a certain threshold, it is considered a successful match. Then, the Hungarian algorithm is used to perform bipartite graph matching to match each real event to the prediction. According to the different intersection ratio thresholds, three commonly used indicators can be obtained: (1) IoU>0. That is, any overlap, as long as the real value event and the predicted event have any length of overlap. This is the most relaxed evaluation indicator. (2) IoU>0.50. That is, if the intersection ratio of a prediction and a real event exceeds 0.5, it is considered a successful match. (3) IoU>0.75. Similar to IoU>0.50, but a higher intersection ratio requires a high degree of overlap between the predicted event and the real event, which is a more stringent evaluation indicator.

[0109] For Task B, when the predicted event matches the real event and is correctly classified, it is counted as a true positive (TP). When the predicted event does not match any real event or is misclassified, a false positive (FP) occurs. When the real event does not match any predicted event or matches a misclassified event, it is counted as a false negative (FN). For Task A, there is no need to consider the classification result, only whether the predicted event matches the real event.

[0110] Calculate TP, FP and FN for each category respectively, then calculate precision and recall, and finally get the F1 score as the evaluation indicator. The calculation method is as follows:

[0111]

[0112] Task B is different from Task A. It uses a weighted F1 score, namely Weighted-F1 Score. The calculation method is as follows:

[0113]

[0114] Where i represents the category, w (i) represents the proportion of samples of category i to all samples in the test set, P (i) , R (i) , F (i) represents the precision, recall and F1 score of the i-th class.

[0115] The present invention adopts the focal loss function as the classification loss, and the L1 loss and the GIoU loss are used together as the regression loss.

Claims

1. A method for detecting and classifying EEG events of EEG signals using a target detection model DINO, characterized in that: Here are the steps: (1) Dataset acquisition Obtaining EEG data to construct a data set, the data set at least includes EEG signals, signal sampling frequency, EEG signal event locations and category labels; (2) Data preprocessing The preprocessing stage is divided into five stages: data screening, resampling, bad conductor detection and reconstruction, TCP montage, short-time Fourier transform, and time-frequency matrix normalization; (3) Feature extraction Assume that the dimension of the preprocessed EEG signal data is Among them, 1 represents single-channel data, F represents the frequency domain part after short-time Fourier transform, and T represents the time domain part. The data is input into the DINO model for training, and the model with the highest F1 score on the validation set is selected as the final model, and tested on the test set. (4) Classification and evaluation Two tasks are designed: Task A: EEG event detection task, the model locates the start and end time of EEG events; Task B: EEG event detection and classification task, the model locates the start and end time of EEG events and classifies the events; When the model is inferring, the NQ queries are first sorted by confidence, the top-NI predictions are selected, and then the final prediction events are screened out using the threshold method; the threshold used in the threshold method is the threshold at which the model obtains the highest F1 score on the validation set; Using event-based evaluation indicators, the model performance is evaluated by calculating the overlap between predicted events and real events. TP, FP, and FN are defined. The calculation method and process are as follows: First, the ratio of the intersection of the start and end coordinates of the real event and the predicted event to the position union of the two is calculated to obtain the intersection ratio. Then, the intersection ratio between each predicted event and each real event is calculated. If the intersection ratio is greater than a certain threshold, it is considered a successful match. Then, the Hungarian algorithm is used to perform bipartite graph matching to match each real event with the prediction. According to the different intersection ratio thresholds, three commonly used indicators can be obtained: (1) IoU>0; that is, any overlap, as long as the real value event and the predicted event have any length of overlap; this is the most relaxed evaluation indicator; (2) IoU>0.50; that is, if the intersection ratio of a prediction and a real event exceeds 0.5, it is considered a successful match; (3) IoU>0.75; similar to IoU>0.50, but a higher intersection ratio requires a high degree of overlap between the predicted event and the real event, which is a more stringent evaluation indicator. For Task B, when the predicted event matches the real event and is correctly classified, it is counted as a true positive TP; when the predicted event does not match any real event or is misclassified, a false positive FP occurs; when the real event does not match any predicted event or matches a misclassified event, it is counted as a false negative FN; for Task A, there is no need to consider the classification result, only whether the predicted event matches the real event; Calculate TP, FP and FN for each category respectively, then calculate precision and recall, and finally get the F1 score as the evaluation indicator; the calculation method is as follows: Task B is different from Task A. It uses a weighted F1 score, namely Weighted-F1 Score. The calculation method is as follows: Where i represents the category, w (i) represents the proportion of samples of category i to all samples in the test set, P (i) , R (i) , F (i) represents the precision, recall and F1 score of the i-th class; The present invention adopts the focal loss function as the classification loss, and the L1 loss and GIoU loss are used together as the regression loss.

2. According to claim 1, a method for detecting and classifying EEG events using a target detection model DINO is characterized in that: The DINO model is as follows: (3.1) Backbone network module The backbone network is used to extract the representation of the input signal and output the extracted features in multiple feature dimensions using the Transformer-based Swin-L model. Therefore, at the beginning of training, the weights of the Swin-L model pre-trained for classification on the dataset are initialized. (3.2) Position encoding module Since the features output by the backbone network module contain multiple scales, the position encoding is divided into two parts. The first part is the position encoding within each layer of features, and the second part is the scale-level position encoding, which is used to positionally encode the tokens within each layer of features and identify which layer the feature map is in. PE=PE postion +PE scale-leve (6) Among them, PE represents the output of the position encoding module, PE postion Represents the position encoding within each layer of features, PE scale-leve Positional encoding to represent scale levels; For the position encoding within each layer of features, absolute position encoding based on sine and cosine trigonometric functions is used; and only one-dimensional position encoding is performed in the time dimension, as shown in the following formula: PE position(pos,2i) =sin(pos / Temperature 2i / d ) (7) Where pos represents the position in the time dimension, d represents the model dimension of the deformable Transformer, i represents the i-th element of the position vector; Temperature determines the smoothness of the attention map; For scale-level positional encoding, use learnable positional encoding Among them, e l represents the learnable positional encoding embedding vector of layer l, N represents the number of layers in the multi-layer scale, and d represents the model dimension of the deformable Transformer to identify which layer the feature map belongs to; (3.3) Deformable Transformer Module Before inputting the deformable Transformer module, the multi-scale feature maps output by the backbone network are first positionally encoded, and then their number of channels is unified into dimension d using a linear projection layer to obtain a multi-scale input feature map in is a multi-scale feature with L layers, H l and W l Represents the size of the feature map of layer l, i.e., height and width; The deformable Transformer module consists of an encoder module consisting of several encoding layers, a decoder module consisting of several decoding layers, plus an additional query selection module and a contrast denoising module; (3.4) Prediction head The prediction head consists of two 3-layer multilayer perceptrons, which accept the output of NQ queries from the decoder module of the model and generate the location and category of the event for each query.

3. According to the method for detecting and classifying EEG events of EEG signals using the target detection model DINO according to claim 1, it is characterized in that the deformable Transformer module is specifically as follows: 3.3.1) Encoder module The input received by the deformable Transformer module is first fed into the encoder module; the encoder module consists of several encoder layers, each of which consists of a multi-scale deformable attention module, a dropout layer and layer normalization layer, and a 2-layer feedforward neural network; First, after receiving the multi-scale feature input, the encoder module needs to generate the reference point coordinates required by the internal multi-scale deformable attention module so that it can be input into the multi-scale deformable attention module together with the multi-scale features; the specific process of generating the reference point coordinates is as follows: 3.3.1.1) For multi-scale input feature maps The size of each layer is H l ×W l The feature map generates reference point coordinates normalized to between 0 and 1 Where 2 represents the normalized 2D horizontal and vertical coordinates of the feature points between 0 and 1; 3.3.1.2) Connect the multi-scale reference point coordinates together and expand the dimensions to form the final reference point coordinates: Where B represents the batch size, Q = H1×W1+H2×W2+…+H L ×W L represents the multi-scale feature map connected after flattening, L represents the multi-scale features with a total of L layers, and 2 is the normalized 2D horizontal and vertical coordinates of the reference point; that is, each feature point of the original input corresponds to L reference points, because the original input becomes an L-layer multi-scale feature; Next, the generated reference point coordinates and multi-scale features are sent to several encoder layers together; for an encoder layer: the received input first enters the multi-scale deformable attention module for self-attention operation, then undergoes dropout and layer normalization operations, and finally enters the feedforward neural network; the final output of an encoder layer is The output reference point of the encoder layer passes through the next encoder layer through the loop. After passing through several encoder layers, the output of the encoder is For the multi-scale deformable attention module in the encoder, the details are as follows: First, for the multi-scale input feature map For ease of calculation, it is flattened to Where B represents the batch size, Q is the length of the multi-scale feature map connected after flattening, and d is the model dimension of the deformable Transformer; Secondly, the multi-scale deformable attention module offsets the mapping matrix of the sampling points Attention weight mapping matrix The value mapping matrix W′ and the output mapping matrix W are initialized; these four matrices are essentially linear projection layers; the value mapping matrix And the output mapping matrix The weight part of is initialized using Shevel, and the bias part is initialized to 0; Where d represents the model dimension, d v Indicates the feature dimension size calculated based on the number of multi-head attention, the number of multi-scale feature layers, and the number of sampling points; attention weight mapping matrix The weight part and bias part of are initialized to 0, where M is the number of multi-head attention, L is the number of layers of multi-scale features, and K is the number of sampling points; for the sampling point offset mapping matrix Where d represents the model dimension, M represents the number of multi-head attention heads, L represents the number of layers of multi-scale features, K represents the number of sampling points, and 2 represents the 2D coordinate of each sampling point; its weight part is initialized to 0. In the multi-head multi-scale deformable attention module, the number of multi-head attention heads is always fixed to 8, and these 8 heads can be responsible for just one initial offset direction; based on this sample point offset mapping matrix, the offset part is always initialized to {Δp 1lqk =(-k,-k),Δp 2lq =(-k,0),Δp 3lqk =(-k,k),Δp 4lqk =(0,-k),Δp 5lqk =(0,k),Δp 6lqk =(k,-k),Δp 7lqk =(k,0),Δp 8lqk =(k,k)}, where k∈{1,2,…,K} is the number of sampling points in each direction. Different sampling points in the same direction are responsible for different offset lengths ranging from 1 to K; The operation of the multi-scale deformable attention module is defined as follows: in is a multi-scale input feature map, which is flattened and concatenated in the actual process. Q is the query length of the deformable attention and is used in the formula To express; Multi-scale features Add position encoding as a query for the multi-scale deformable attention module; Represents the query z q Each element q is generated by the corresponding reference point 2D coordinate Ref between 0 and 1 mentioned in the encoder section above q ; The entire attention weight matrix is ​​used In order to make the formula clear, A is used. mlqk represents a point in the attention weight matrix A, that is, the attention weight of the kth sampling point of the qth feature point of the lth layer of the mth attention head. The same is true for other places in the formula; By attention weight mapping matrix Then it is obtained through the softmax normalization layer, which can ensure Represents the denormalization operation, that is, the reference point coordinates with values ​​between 0 and 1 Restored to coordinates based on the true size of the feature, l represents the lth layer; the sampling point offset matrix is ​​the same as the attention weight A, Represents the entire sampling point offset matrix, which is also z q Offset the mapping matrix by sampling points In the encoder, the sampling point offset matrix learns the real coordinates, which are compared with the denormalized reference point coordinates. Add together to get the offset reference point coordinates, that is, the sampling point coordinates sampled in the devalued matrix. The value matrix passes through the linear mapping layer through the input feature x We get W′ in the corresponding formula. m , m represents the mth attention head, W′ represents the entire linear mapping layer matrix; in the decoder, the sampling point offset matrix learns the offset real coordinates of the prior anchor frame based on the last position, and the reference point coordinates input to the decoder are After sampling in the value matrix, it is weighted by the attention weight matrix A and finally output through the linear mapping layer W, corresponding to W in the formula m , and get the final output of the multi-scale deformable attention module These outputs then go through the dropout layer as described above, etc. 3.3.2) Query selection module The output of the encoder module is input to the query selection module; the query selection module uses the output of the encoder module Generate an initial anchor box and select the first few anchor boxes to generate reference points for the input decoder module. These reference points have two functions: one is to generate the position encoding of the query as the input decoder module Second, it serves as a reference point for the deformable attention in the decoder module; 3.3.2.1) Query selection First, the position and size of all anchor boxes need to be initialized: the specific method for generating the initial anchor box position is consistent with the principle of generating reference point coordinates in the encoder module, and the normalized coordinates of the center point of each layer of feature map grid are obtained as the initial anchor box position; for the initial size of each anchor box, the length and width of each layer are initialized to 0.05*2 by default l , where l is the lth layer; 3.3.2.2) Connect the initial anchor box position coordinates and size together to get The first two bits in 4 are the 2D coordinates of the proposal, and the last two bits are the size of the proposal; finally, an unsigmoid function is used for denormalization; 3.3.2.3) For the output of the received encoder module First it passes through a linear layer and a normalization layer, and then passes through two separate linear layers to generate a classification confidence score for each feature point. and an anchor box offset 4 represents the 2D coordinates and width and height, C represents the number of categories, and the anchor box offset is combined with the initial anchor box position coordinates of each feature point generated in step 3.3.3.2) Add together to get the final prior anchor box; Then, for the classification confidence score Sort by the confidence score of the first category from high to low, and select the top-NQ corresponding prior anchor boxes as the reference point coordinates of the input decoder module after sigmoid normalization. Here NQ is the number of queries input to the decoder module; the content part of the query input to the decoder module is randomly initialized and learned in the decoder module; 3.3.3) Decoder The decoder also consists of several decoder layers; a decoder layer consists of a multi-head self-attention module, a multi-scale deformable attention module, and a 2-layer feedforward neural network. Each attention module is followed by a dropout layer and a layer normalization. The multi-scale deformable attention module in the decoder layer is the same as that in the encoder layer; The query input to the decoder module consists of two parts, the position encoding and content encoding The position code is the reference point coordinates generated in 3.3.3). Generate, the generation method is as follows; For a decoder layer: First, use the reference point coordinates initialized in step 3.3.3) to generate the position encoding of the sine and cosine trigonometric functions for the 2D horizontal and vertical coordinates and the width and height of each reference point, and then connect the four separately generated position encodings together and pass them through a 2-layer multi-layer perceptron, as shown in the following formula; Among them, PE y PE x PE w PE h Corresponding to the position codes generated for the 2D vertical and horizontal coordinates and width and height of each reference point mentioned above; The coordinates of the reference point The generated positional encoding of the query that is input to the decoder module; For a certain decoder layer, the query input to the decoder module is first passed through the multi-head self-attention module, and the position encoding of the multi-head self-attention module is used Then, the multi-scale deformable cross attention module and the output of the encoder module are used to perform cross multi-scale deformable attention. At this time, the query position encoding is also The reference point uses the 2D horizontal and vertical coordinates and width and height of the prior anchor box. Finally, through the feedforward neural network; in the decoder module, due to the iterative bounding box fine-tuning strategy, each decoder layer does not directly predict the new detection box position, but predicts its relative offset relative to the previous position, and generates a new reference point coordinate by adding And generate a new position code Input to the next decoder layer; 3.3.4) Comparison denoising module The idea of ​​the contrast denoising module is to add noise to the true value to generate two different denoising queries: positive sample query and negative sample query, and add these two denoising queries to the query input into the decoder module; for positive samples, the model needs to reconstruct the real detection box, and for negative samples, the model needs to judge them as background classes; the generation of positive and negative samples is controlled by two given hyperparameters λ1 and λ2; When the added noise size is within λ1, a positive sample is generated, and the model needs to reconstruct the real detection frame; conversely, when the added noise size is between λ1 and λ2, a negative sample is generated, and the model needs to classify it as the background class; in the DINO model, the number of added denoising queries is fixed to ND, so for an input containing nt real detection frames, All ND queries generated will be concatenated with the NQ queries input into the decoder module. At this time, the two parts of the query input into the decoder are encoded as and content encoding as

Images