A method and device for recognizing electroencephalogram signals based on convolutional neural network

Abstract

The present application provides a method and device for recognizing EEG signals based on a convolutional neural network. An input signal is obtained by preprocessing and data standardization of the EEG signal, and the input signal is input into a conscious movement classification recognition model to obtain a conscious movement recognition result after classification and recognition. The design and construction of the conscious movement classification recognition model can consider the signal components of multiple signal channels in the input signal, identify the conscious movement channel components that reflect the classification of the signal components, and use the channel data of each signal component to merge into an overall one-dimensional data to identify the overall conscious movement component, and then combine the conscious movement channel components and the conscious movement overall component to realize the conscious movement classification recognition of the EEG signal. In this way, the characteristics of each signal component and the overall synergistic characteristics of multiple channels can be considered to improve the accuracy of recognition; and multi-channel recognition can be carried out simultaneously, while ensuring the recognition accuracy, the recognition efficiency can be taken into account.

Description

Technical Field

[0001] The present application relates to the technical field of electroencephalogram (EEG) signal recognition, and in particular to an EEG signal recognition method and device based on a convolutional neural network. Background Art

[0002] With the continuous development and intersection of biomedicine and artificial intelligence, BCI (Brain Computer Interface) technology has been continuously applied in research and clinical treatment (such as rehabilitation therapy). For example, for severe neuromuscular diseases such as spinal cord injury and stroke (also known as stroke), BCI technology can help patients "recover" some motor functions to a certain extent, thereby improving the quality of life of patients and their family members.

[0003] The recognition of EEG signals is the top priority in the process of BCI technology moving towards practical application, especially the recognition of conscious movement (also known as motor imagery EEG recognition), which is the core of brain-computer control. Accurate recognition of conscious movement is the key to reliable brain-computer control. However, the existing EEG signal recognition is more applied in EEG signal monitoring (such as sleep monitoring, abnormality monitoring, etc.), and further development is needed for conscious movement recognition. Improving the accuracy of conscious movement recognition is the key. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method and device for EEG signal recognition based on convolutional neural networks to improve the accuracy of conscious movement recognition.

[0005] In order to achieve the above purpose, the embodiments of the present application are implemented in the following manner:

[0006] In a first aspect, an embodiment of the present application provides an EEG signal recognition method based on a convolutional neural network, comprising: obtaining an EEG signal of a user; performing preprocessing and data normalization on the EEG signal to obtain an input signal; inputting the input signal into a preset conscious movement classification and recognition model to obtain a conscious movement recognition result after the conscious movement classification and recognition model classifies and recognizes the input signal.

[0007] In combination with the first aspect, in a first possible implementation of the first aspect, the conscious movement classification and recognition model includes an input layer, n convolution channels, a fully connected layer, an overall output layer and a model output layer, and each convolution channel is sequentially provided with a convolution layer Con i1 , Pooling layer i1 , Convolutional layer i2 , Pooling layer i2 , Convolutional layer i3 , Pooling layer i3 , channel connection layer FCi And channel output layer Out i , i=1,2,…,n, the process of the conscious movement classification and recognition model classifying and identifying the input signal includes: the input layer is used to receive the input signal, wherein the input signal includes signal components of m signal channels, and any signal channel of the m signal channels has a unique corresponding convolution channel in the n convolution channels, m≤n; n convolution channels are respectively connected to the input layer, each convolution channel is used to receive the signal component of the input layer transmitted to the convolution channel, and the signal components of the convolution channel are convoluted and pooled by each convolution layer and pooling layer of the convolution channel, and the channel connection layer of the convolution channel is used to flatten to obtain the channel data corresponding to the signal components of the convolution channel, and then through The channel output layer classifies and identifies the channel data to obtain the conscious movement channel component, wherein the input layer transmits the corresponding signal component to the corresponding convolution channel based on the correspondence between the signal channel and the convolution channel; the fully connected layer is connected to the channel connection layers of the n convolution channels respectively, and is used to receive the channel data output by each convolution channel and splice to obtain one-dimensional data; the overall output layer is used to receive the one-dimensional data and classify and judge the one-dimensional data using the Softmax classification function to obtain the conscious movement overall component; the model output layer is connected to the channel output layer of each convolution channel respectively, and is connected to the overall output layer, and is used to determine the conscious movement recognition result based on each conscious movement channel component and the conscious movement overall component.

[0008] In combination with the first possible implementation method of the first aspect, in a second possible implementation method of the first aspect, the EEG signal is preprocessed and data normalized to obtain an input signal, including: filtering the EEG signal and performing bad conduction correction on the filtered EEG signal; using an ICA algorithm to remove artifact interference from the EEG signal after the bad conduction correction; and performing data normalization on the EEG signal after the artifact interference is removed to obtain the input signal.

[0009] In combination with the second possible implementation method of the first aspect, in a third possible implementation method of the first aspect, data normalization processing is performed on the EEG signal after artifact interference is removed to obtain the input signal, including: using a sliding window to intercept the EEG signal after artifact interference is removed, with a sliding window size of τ and a step size of s, to obtain several groups of signal segments with adjacent time intervals of s; performing correlation analysis on each signal segment to obtain the input signal corresponding to each signal segment, so as to obtain several input signals corresponding to the EEG signal.

[0010] In combination with the third possible implementation method of the first aspect, in a fourth possible implementation method of the first aspect, each signal segment contains m′ signal components, m′≥m, and a correlation analysis is performed on each signal segment to obtain an input signal corresponding to each signal segment, including: for each signal segment: calculating the correlation index between every two signal components in the signal segment; determining the independence index of this signal component based on the correlation index between each signal component in the signal segment and other signal components; based on the independence index of each signal component in the signal segment, determining the m signal components with the highest independence index in the signal segment as the input signal containing m signal components corresponding to the signal segment.

[0011] In combination with the fourth possible implementation manner of the first aspect, in the fifth possible implementation manner of the first aspect, after obtaining the input signal corresponding to each signal segment, the method also includes: calculating the confidence channel component corresponding to each signal component based on the independence index of the m signal components in the input signal, and calculating the confidence overall component based on the independence index of the m signal components in the input signal and the independence index of the m′ signal components in the corresponding signal segment; correspondingly, the model output layer determines the conscious movement recognition result based on each conscious movement channel component and the conscious movement overall component, specifically including: calculating the probability of the input signal being classified into each category based on the confidence channel component and the conscious movement channel component corresponding to each signal component, as well as the confidence overall component and the conscious movement overall component, and determining the category with the highest probability as the conscious movement recognition result based on the probability of each category.

[0012] In combination with the fifth possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, based on the confidence channel component and the conscious motion channel component corresponding to each signal component, as well as the confidence overall component and the conscious motion overall component, calculating the probability that the input signal is classified into each category includes: using the following formula to calculate the probability that the input signal is classified into each category:

[0013]

[0014] Among them, P j represents the probability that the input signal is classified into the jth category, j = 1, 2, ..., k, k is the total number of categories, Dw represents the overall confidence component of the input signal, ToP j represents the probability of the jth category in the overall component of conscious movement, Represents the lth confidence channel component of the input signal, ChP lj Represents the probability of the jth category in the lth conscious motion channel component.

[0015] In the second aspect, an embodiment of the present application provides an EEG signal recognition device based on a convolutional neural network, comprising: an EEG signal acquisition unit, used to acquire the EEG signal of a user; an EEG signal processing unit, used to preprocess and normalize the EEG signal to obtain an input signal; and a conscious movement recognition unit, used to input the input signal into a preset conscious movement classification and recognition model to obtain a conscious movement recognition result after the conscious movement classification and recognition model classifies and recognizes the input signal.

[0016] In a third aspect, an embodiment of the present application provides a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the EEG signal recognition method based on a convolutional neural network described in any one of the first aspect or the possible implementation methods of the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, and when the program instructions are loaded and executed by the processor, the method for recognizing electroencephalogram signals based on a convolutional neural network described in any one of the first aspect or the possible implementation methods of the first aspect is implemented.

[0018] Beneficial effects: The input signal is obtained by preprocessing and data standardization of the user's EEG signal, and then the input signal is input into the preset conscious movement classification and recognition model to obtain the conscious movement recognition result after classification and recognition. 1. The design and construction of the conscious movement classification and recognition model can consider the signal components of multiple signal channels in the input signal (the EEG signal usually collects signals from multiple channels), use the corresponding channel to identify each signal component, obtain the conscious movement channel component reflecting the classification of the signal component, and use the channel data of each signal component to merge into a one-dimensional data as a whole, identify the overall component of conscious movement, and then combine the conscious movement channel component and the conscious movement overall component to realize the conscious movement classification and recognition of the EEG signal. In this way, not only can the characteristics of each signal component be comprehensively considered, but also the characteristics of the overall coordination of multiple channels can be considered to improve the accuracy of recognition. The structural design of the model ensures that multi-channel recognition can be carried out simultaneously, while ensuring the recognition accuracy, the recognition efficiency can be taken into account. 2. The preprocessing and data standardization of EEG signals can obtain relatively pure EEG signals and accelerate the convergence speed. 3. Use sliding window to intercept the EEG signal to obtain the signal segment after standardization (including m' signal components), calculate the correlation between every two signal components for the same signal segment, further calculate the independence of each signal component, and select the m signal components with the highest independence as the input signal, so as to further remove the signal components of the redundant signal channel, ensure the classification accuracy while ensuring the stability of the number of signal components in the input signal, which is conducive to improving the reliability of the classification results, and can also reduce the data processing amount of the model and improve the recognition efficiency. 4. Use the independence index of the m signal components in the input signal to calculate the confidence channel component of each signal component, as well as the independence index of the m signal components in the input signal and the independence index of the m' signal components in the corresponding signal segment, and calculate the confidence overall component as the weight parameter for calculating the probability of the input signal being classified in each category. The independence of each signal component and the overall confidence of the signal components in the input signal can be considered, which is conducive to improving the accuracy of classification and recognition, especially for the classification and recognition of conscious movement of EEG signals, which is difficult to classify, and can consider the features and corresponding weights at different levels, which is more conducive to improving the classification effect.

[0019] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic diagram of a conscious movement classification and recognition model provided in an embodiment of the present application.

[0022] Figure 2 A flowchart of a method for recognizing electroencephalogram signals based on a convolutional neural network is provided in an embodiment of the present application.

[0023] Figure 3 A structural block diagram of an EEG signal recognition device based on a convolutional neural network provided in an embodiment of the present application.

[0024] Icons: 10- EEG signal recognition device based on convolutional neural network; 11- EEG signal acquisition unit; 12- EEG signal processing unit; 13- conscious movement recognition unit. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0026] In order to realize the classification and recognition of conscious movement of EEG signals, this embodiment designs a conscious movement classification and recognition model based on CNN (Convolutional Neural Network) as the core of conscious movement classification and recognition. In order to facilitate the understanding of this solution, the conscious movement classification and recognition model is first introduced here.

[0027] See also Figure 1 , Figure 1 A schematic diagram of a conscious movement classification and recognition model provided in an embodiment of the present application.

[0028] In this embodiment, the conscious movement classification and recognition model may include an input layer, n convolution channels, a fully connected layer, an overall output layer and a model output layer. The input layer is connected to the n convolution channels respectively, and the n convolution channels are connected to the fully connected layer respectively, and the n convolution channels are also connected to the model output layer respectively. The fully connected layer is connected to the overall output layer, and the overall output layer is connected to the model output layer.

[0029] Exemplarily, an input layer is used to receive an input signal, where the input signal includes signal components of m signal channels, and any one of the m signal channels has a uniquely corresponding convolutional channel among n convolutional channels, where 1 < m ≤ n. For example, the number of convolutional channels n = 32, and the number of signal components m of the input signal = 16. This is only an example and should not be regarded as a limitation.

[0030] The input layer can transmit the corresponding signal components in the input signal to the corresponding convolutional channels based on the correspondence between the signal channels and the convolutional channels, so that the convolutional channels can process the transmitted signal components.

[0031] Exemplarily, the n convolutional channels can be designed as follows: Each convolutional channel is provided with a convolutional layer Con i1 connected in sequence to a pooling layer Pool i1 connected in sequence to a convolutional layer Con i2 connected in sequence to a pooling layer Pool i2 connected in sequence to a convolutional layer Con i3 connected in sequence to a pooling layer Pool i3 connected in sequence to a channel connection layer FC i and a channel output layer Out i , and the label of the channel i = 1, 2, …, n. The convolutional layer Con i1 is connected to the input layer, the channel connection layer FC i is also connected to the fully connected layer, and the channel output layer Out i is also connected to the model output layer. The design of the convolutional channels, convolutional layers, and pooling layers here is only exemplary. Some can have one or more additional convolutional layers and pooling layers designed, or one less convolutional layer and pooling layer designed. This is not limited here. Additionally, for different convolutional channels, the selection of convolutional kernels in their convolutional layers can be different (to adapt to different signal components), or they can be the same convolutional kernels. This is not limited here and is subject to actual needs.

[0032] For each of the n convolutional channels:

[0033] Each convolutional channel is respectively connected to the input layer, used to receive the signal components transmitted by the input layer to this convolutional channel, and use the respective convolutional layers and pooling layers of this convolutional channel to perform convolutional processing and pooling processing on the signal components of this convolutional channel, use the channel connection layer of this convolutional channel to flatten, obtain the channel data corresponding to the signal components of this convolutional channel, and then perform classification and recognition on the channel data through the channel output layer to obtain the conscious motor channel components.

[0034] Exemplarily, the fully connected layer is respectively connected to the channel connection layers of the n convolutional channels, used to receive the channel data output by each convolutional channel and splice them to obtain one-dimensional data.

[0035] Exemplarily, the overall output layer is connected to the fully connected layer to receive one-dimensional data, and the Softmax classification function is used to classify and judge the one-dimensional data to obtain the overall component of conscious movement.

[0036] Exemplarily, the model output layer is connected to the channel output layer of each convolution channel, and is connected to the overall output layer, so as to determine the conscious motion recognition result based on each conscious motion channel component and the conscious motion overall component.

[0037] The above is an introduction to the structural design of the conscious movement classification and recognition model. The following is an introduction to the training and operation of the conscious movement classification and recognition model.

[0038] In this embodiment, the original data set may include EEG data sets of multiple subjects, and each experimental data of each EEG data set (an EEG signal with a label, such as an EEG signal of imagining controlling the movement of the "left hand") corresponds to a label (this label is the conscious movement category, for example, including four categories such as "left hand", "right hand", "foot", and "tongue").

[0039] In order to ensure the uniformity of input data, each test data can be filtered (for example, high-pass filtering to control the passage of signals above 0Hz, and low-pass filtering to control the passage of signals below 100Hz), and the filtered EEG signal can be corrected for bad conduction (for example, removing the signal of this channel, or using the mean value of multiple electrodes around to calculate the signal of this channel), and then the ICA algorithm can be used to remove artifacts from the EEG signal after bad conduction correction. In this way, a relatively pure EEG signal can be obtained, and the convergence speed can also be accelerated.

[0040] After that, the EEG signal after removing the artifact interference can be processed for data standardization: for example, the EEG signal after removing the artifact interference can be intercepted using a sliding window, the sliding window size is τ (for example, 4 seconds), the step size is s (for example, 0.1 seconds, for EEG signal bands of different frequencies, adaptively adjustable sliding window size and step size can also be designed to adapt to EEG signal bands of different frequencies, which is not limited here), and several groups of signal segments with adjacent time intervals of s are obtained (for example, each signal segment contains 800 sampling points). For signal segments derived from the same EEG signal, these signal segments correspond to the same label. In this way, multiple training data can be obtained through one test data, effectively expanding the training set and improving the fine-grained recognition effect of the model.

[0041] After obtaining the signal segments, correlation analysis can be performed on each signal segment to obtain the input signal corresponding to each signal segment, thereby obtaining several input signals corresponding to the EEG signal.

[0042] Exemplarily, it is assumed that each signal segment contains m′ signal components, m′≥m, for example, m′=22, m=16. Then, for each signal segment:

[0043] First, the correlation index between every two signal components in the signal segment can be calculated. For example, the correlation index between two signal components can be calculated using the following formula:

[0044]

[0045] Among them, a i ,a j are two signal components in this signal segment, i, j = 1, 2, ..., m', Sn is the number of sampling points of each signal component in this signal segment, is the signal component a i With signal component a j The correlation index between is the signal component a i The potential value of the kth sampling point in, is the signal component a i The potential mean of the sampling points is is the signal component a i The standard deviation of the sampling point potential, and is the signal component a j The potential value of the kth sampling point in, is the signal component a j The potential mean of the sampling points is is the signal component a j The standard deviation of the sampling point potential, and

[0046] Thus, the correlation index between every two signal components in the signal segment can be calculated. Then, based on the correlation index between each signal component and other signal components in the signal segment, the independence index of this signal component can be determined. For example, the independence index of each signal component can be calculated using the following formula:

[0047]

[0048] in, is the signal component a in this signal segment i independence index.

[0049] Thus, the independence index of each signal component in the present signal segment can be calculated. Then, based on the independence index of each signal component in the present signal segment, the m signal components with the highest independence index in the present signal segment can be determined as the input signal containing m signal components corresponding to the present signal segment. For example, the signal components are sorted from large to small using the independence index of each signal component in the present signal segment, and the first m signal components are taken to form the input signal (the input signal formed here retains the signal channel number of the signal component in the signal segment, which can be any signal channel number from 1 to m′).

[0050] After obtaining the input signal, the input signal can be input into the input layer of the conscious movement classification and recognition model, and the input layer can match the channel number (any value from 1 to m′) of each signal component in the received input signal with the convolution channel number in the conscious movement classification and recognition model, and pass the corresponding signal component in the input signal to the corresponding convolution channel, so that the convolution channel can process the passed-in signal component.

[0051] Since each signal component has 800 sampling point potential values, we take a convolution channel as an example to introduce: i1 64 convolution kernels of size 33×1 are used. Each convolution kernel performs a one-dimensional convolution operation on the input data with a step size of 8, and 64 groups of EEG features of size [96, 1] are obtained. Pooling layer Pool i1 The pooling window is 3×1, the step size is 3×1, and the EEG features of [96, 1] are downsampled to obtain 64 groups of EEG features of size [32, 1]. i2 64 convolution kernels of size 5×1 are used. Each convolution kernel performs a one-dimensional convolution operation on the EEG features of [32, 1] with a step size of 2, and 64 groups of EEG features of size [14, 1] are obtained. The pooling layer Pool i2 The pooling window is 2×1, the step size is 2×1, and the EEG features of [14, 1] are downsampled to obtain 64 groups of EEG features of size [7, 1]. i3 64 convolution kernels of size 3×1 are used. Each convolution kernel performs a one-dimensional convolution operation on the EEG features of [7, 1] with a step size of 1, and 64 groups of EEG features of size [5, 1] ​​are obtained. Pooling layer Pool i3 The pooling window is 3×1, the step size is 1×1, and the EEG features of [5, 1] ​​are downsampled to obtain 64 groups of EEG features of size [3, 1]. iThe 64 groups of EEG features of size [3, 1] are flattened, and the 64 groups of [3, 1] EEG features are connected in sequence to obtain a channel data of [192, 1]. The channel data is then classified and identified through the channel output layer to obtain the conscious movement channel components (the probability of belonging to "left hand", "right hand", "foot", and "tongue"), a total of m.

[0052] The fully connected layer can fill-in the channel data output by the channel connection layer of each convolution channel (fill the set data template according to the number of the convolution channel where the channel data is located, and fill the unused channels with 0) to obtain a one-dimensional data of [192n, 1] (or, for a relatively stable EEG acquisition mode, that is, the acquisition channel used is constant, a one-dimensional data template of [192m′, 1] can be determined, and the one-dimensional data of [192m′, 1] can be filled), and then the one-dimensional data is classified and identified to obtain the overall component of conscious movement (the probability of belonging to "left hand", "right hand", "foot" and "tongue").

[0053] The model output layer can receive the conscious motion channel component output by each convolution channel and the conscious motion overall component output by the overall output layer, and determine the conscious motion recognition result based on each conscious motion channel component and the conscious motion overall component.

[0054] Exemplarily, the model output layer can calculate the confidence channel component corresponding to each signal component based on the independence index of the m signal components in the input signal. For example, the confidence channel component corresponding to each signal component is calculated using the following formula:

[0055]

[0056] in, is the signal component b in the input signal l The confidence channel component, l = 1, 2, ..., m, the signal component b in the input signal l is the m′ signal component a in the signal segment i One of the first m signal components after sorting, α is the channel confidence weight, is the signal component b l independence index.

[0057] And, the model output layer can calculate the overall component of confidence based on the independence index of the m signal components in the input signal and the independence index of the m′ signal components in the corresponding signal segment.

[0058]

[0059] Wherein, Dw is the overall confidence component of the input signal, β is the overall confidence weight, and α+β = 1. The allocation of α and β can be a preset fixed value, or can be designed to be continuously updated during the operation of the model, which is not limited here.

[0060] Based on this, the model output layer can calculate the probability of the input signal being classified into each category based on the confidence channel component and the conscious-motor channel component corresponding to each signal component, as well as the confidence overall component and the conscious-motor overall component.

[0061] Specifically, the model output layer can use the following formula to calculate the probability of the input signal being classified into each category:

[0062]

[0063] Among them, P j represents the probability that the input signal is classified into the jth category, j = 1, 2, ..., k, k is the total number of categories, Dw represents the overall confidence component of the input signal, ToP j represents the probability of the jth category in the overall component of conscious movement, Represents the lth confidence channel component of the input signal, ChP lj Represents the probability of the jth category in the lth conscious motion channel component.

[0064] Afterwards, the model output layer can determine the category with the highest probability as the conscious movement recognition result based on the probability of each category.

[0065] In this embodiment, all neurons before the model output layer, the overall output layer, and the channel output layer of the conscious movement classification and recognition model use ReLU (Rectified Linear Unit) as the activation function, and the weight values ​​of the network connections use random values ​​in a normal distribution with a mean of 0 and a standard deviation of 0.1 as the initial values, and the bias is uniformly initialized to 0.1 for training. The overall output layer and the channel output layer use the softmax classification function (output the probability of each category), and the loss function can use cross entropy.

[0066] For example, it is assumed that the training set is (a different symbol system is used here to represent each quantity, which is not unified and distinguished from some symbols in the previous text. This is only to facilitate the understanding of this part and simplify the deduction of formulas. It should not be regarded as a limitation of this application. The meaning expressed in this part shall prevail):

[0067] (x (1) ,y (1) ), (x (2) ,y (2) ),…,(x (r),y (r) ),…,(x (m) ,y (m) ), (6)

[0068] Among them, x (r) ∈R n , represents the EEG signal, and y (r) Represents the EEG signal x (r) The corresponding label, and the activation function is selected as σ(x) = Re LU.

[0069] Then, the forward calculation results of each layer of each convolution channel in the conscious movement classification and recognition model are expressed as:

[0070]

[0071] in, is the lth convolution channel 1 -1 layer to l 1 The weight matrix of the layer, where l 1 =2,3,…,L, It is the first 1 -1 layer activation function, It is the first 1 The bias vector of the layer, It is the first 1 The input vector of the layer, σ is the sigmoid function, which is the threshold function of the neural network and maps the variables to [0, 1].

[0072] Then, the error of the channel output layer of the convolution channel is:

[0073]

[0074] H(p,q)=-∑ x p(x)ln(q(x)), (9)

[0075] in, is the error of the channel output layer, l 1 is the layer number of the channel output layer of the convolution channel, is the partial derivative of H(p, q), H(p, q) is the cross entropy loss function, p and q are the probability distributions of the calculated and original categories respectively.

[0076] Using the chain rule, formula (8) can be expressed as:

[0077]

[0078] in, is the loss function at the lth 1The error vector of the layer, ⊙ represents the Hadamard product, that is, the product of the corresponding elements of the two matrices.

[0079] The reverse calculation results of each layer of each convolution channel in the conscious movement classification and recognition model are expressed as:

[0080]

[0081] Among them, l 2 =L-1, L-2, ..., 2, resize so that the output of the convolution layer in the convolution channel is still an integer, rot means to convert the matrix Rotate 180° and recalculate.

[0082] The forward calculation result of the overall output layer is expressed as:

[0083]

[0084] a L+1 =σ(z L+1 ), (13)

[0085] z L+2 =a L+1 ·w L+2 +b L+2 , (14)

[0086] a L+2 =σ(z L+2 ), (15)

[0087] Among them, the L+1th layer is the fully connected layer, the L+2th layer is the overall output layer, and z L+1 is the input vector of the L+1th layer, is the activation function of the Lth layer in the ith convolution channel, is the weight matrix from the Lth layer to the fully connected layer (the L+1th layer) in the i-th convolution channel, is the bias vector corresponding to the L+1th layer and the i-th convolution channel; a L+1 is the activation function of the L+1th layer, w L+2 is the weight matrix from the L+1th fully connected layer to the L+2th layer, b L+2 is the bias vector of the L+2th layer.

[0088] Then, the errors of the fully connected layer and the overall output layer are:

[0089]

[0090]

[0091] Among them, δ (L+1) ,δ (L+2)are the errors of the fully connected layer and the overall output layer, L+1 is the number of layers where the fully connected layer is located, and L+2 is the number of layers where the overall output layer is located. is the partial derivative of H(p, q), H(p, q) is the cross entropy loss function, p and q are the probability distributions of the calculated and original categories respectively.

[0092] The reverse calculation results of the fully connected layer and the overall output layer in the conscious movement classification and recognition model are expressed as:

[0093]

[0094] δ (L+2) = resize(δ) L+1 *rot180°(w) L+1 ⊙σ′(z L+2 ), (18)

[0095] Among them, resize makes the output of the fully connected layer and the overall output layer still an integer, and rot means to transform the matrix (w) Li or matrix(w) L+1 Rotate 180° and recalculate. The superscript Li represents the Lth layer of the i-th convolution channel.

[0096] After that, the values ​​of weight ω and bias b can be updated by gradient descent. For the case of l ≤ L (that is, the case within each convolution channel), we have:

[0097]

[0098] in, represents the weight from the kth neuron in layer l-1 to the jth neuron in layer l, represents the bias of the jth neuron in the lth layer, represents the output value of the activation function of the kth neuron in the l-1th layer, α is the learning rate and α∈(0,1].

[0099] For the case of l=L+2 (i.e. the case of the overall output layer), we have:

[0100]

[0101] For the case of l=L+1 (i.e. the case of the fully connected layer), we have:

[0102]

[0103] Among them, n represents the total number of convolution channels, and i represents the number of the convolution channel.

[0104] In order to realize the training of the consciousness and movement classification and recognition model, the original data set can be allocated into training set, verification set and test set according to the ratio of 7:2:1. The consciousness and movement classification and recognition model is trained with the training set. After the training meets the standard (the number of training times meets the conditions or the training classification index meets the conditions), the consciousness and movement classification and recognition model that meets the training standard is verified with the verification set. According to the verification result, the data in the verification set is put into the training set for secondary training. After the training meets the standard again, it is tested with the test set. Then, the performance of the consciousness and movement classification and recognition model is statistically analyzed and evaluated by experimental data (the consciousness and movement classification and recognition model can also be improved in a targeted manner), and finally the trained consciousness and movement classification and recognition model is obtained.

[0105] After the conscious movement classification and recognition model is trained, it can be used to recognize conscious movement in EEG signals.

[0106] In this embodiment, the trained conscious movement classification recognition model can be loaded into an electronic device to run the EEG signal recognition method based on a convolutional neural network. The electronic device here can be an intelligent terminal (such as a personal computer, a laptop, or even a smart phone, etc.), or a server (such as a cloud server, a network server, a server cluster, etc.), or a dedicated monitoring and control terminal (such as a control device in a rehabilitation medical device for paralyzed patients), which is not limited here.

[0107] See also Figure 2 , Figure 2 A flowchart of a method for recognizing electroencephalogram signals based on a convolutional neural network is provided in an embodiment of the present application.

[0108] In this embodiment, in order to realize the recognition of the user's electroencephalogram signal, the electronic device can execute step S10.

[0109] Step S10: Acquire the user's EEG signal.

[0110] First, the electronic device can obtain the user's EEG signal. The EEG signal here can be an EEG signal collected in real time by an EEG monitoring device, or it can be an EEG signal transmitted after the EEG monitoring device collects it. In order to improve the recognition accuracy as much as possible, the EEG signal here is collected by the same collection method as the original data set as much as possible. For example, the original data set uses an EEG collection device G (with several electrodes, the electrode placement position is relatively fixed, and 22 channels of EEG signals can be sampled). When collecting the user's EEG signal, if conditions permit, try to use the same device as the EEG device G to collect the EEG signal, or adopt a similar scheme, such as an EEG collection device H, which has more electrodes. When collecting the user's EEG signal, some of the electrodes can be arranged at the corresponding collection position to obtain the user's EEG signal (including 22 channels of EEG signals). The user's EEG signal collected in this embodiment includes several signal channels (no less than m').

[0111] After acquiring the user's EEG signal, the electronic device may execute step S20.

[0112] Step S20: preprocessing and data normalization processing are performed on the EEG signal to obtain an input signal.

[0113] In this embodiment, the electronic device can filter the EEG signal (high-pass filtering 0Hz, low-pass filtering 100Hz), and perform bad conduction correction on the filtered EEG signal (for example, the potential mean value can be calculated using the average value of 3 to 6 surrounding electrodes as the signal component corresponding to the bad conduction). The EEG signal after bad conduction correction contains signal components of m′ channels, m′≥m. Then the electronic device can use the ICA (Independent Component Analysis) algorithm to remove artifacts from the EEG signal after bad conduction correction in order to obtain a pure EEG signal; then perform data standardization on the EEG signal after artifacts are removed to obtain the input signal.

[0114] Exemplarily, the electronic device can use a sliding window to intercept the EEG signal after removing artifact interference, with a sliding window size of τ and a step size of s, to obtain a plurality of groups of signal segments with adjacent time intervals of s. Afterwards, the electronic device can perform a correlation analysis on each signal segment to obtain an input signal corresponding to each signal segment (including m′ signal components), so as to obtain a plurality of input signals corresponding to the EEG signal.

[0115] Specifically, for each signal segment:

[0116] The electronic device may calculate the correlation index between every two signal components in the current signal segment. For example, the correlation index between every two signal components in the current signal segment may be calculated using formula (1) to obtain the correlation index between every two signal components in the current signal segment.

[0117] Then, the independence index of each signal component in the signal segment can be determined based on the correlation index between each signal component and other signal components in the signal segment. For example, the independence index of each signal component can be calculated using formula (2), thereby calculating the independence index of each signal component in the signal segment.

[0118] After obtaining the independence index of each signal component in the signal segment, the electronic device can determine the m signal components with the highest independence index in the signal segment based on the independence index of each signal component in the signal segment as the input signal containing m signal components corresponding to the signal segment. For example, the signal components are sorted from large to small using the independence index of each signal component in the signal segment, and the first m signal components are taken to form the input signal (the input signal formed here retains the signal channel number of the signal component in the signal segment, which can be any signal channel number from 1 to m′).

[0119] After obtaining the input signal, the electronic device may execute step S30.

[0120] Step S30: inputting the input signal into a preset conscious movement classification and recognition model to obtain a conscious movement recognition result after the conscious movement classification and recognition model classifies and recognizes the input signal.

[0121] In this embodiment, the electronic device can input the input signal into a trained conscious movement classification and recognition model, and the conscious movement classification and recognition model can perform classification and recognition based on the input signal to obtain a conscious movement recognition result.

[0122] Since the specific classification and recognition method of the conscious movement classification and recognition model for the input signal has been introduced in detail in the previous article when introducing the conscious movement classification and recognition model, it will not be repeated here. Finally, the conscious movement recognition result can be obtained after the conscious movement classification and recognition model classifies and recognizes the input signal.

[0123] By preprocessing and data standardization of the user's EEG signal, the input signal is obtained, and then the input signal is input into the preset conscious movement classification and recognition model to obtain the conscious movement recognition result after classification and recognition. It has the following advantages:

[0124] 1. The design and construction of the conscious movement classification recognition model can consider the signal components of multiple signal channels in the input signal (EEG signals usually collect signals from multiple channels), use the corresponding channel to identify each signal component, and obtain the conscious movement channel component that reflects the classification of the signal component, and use the channel data of each signal component to merge into a one-dimensional data as a whole, identify the overall conscious movement component, and then combine the conscious movement channel component and the conscious movement overall component to realize the conscious movement classification recognition of the EEG signal. In this way, not only can the characteristics of each signal component be comprehensively considered, but also the characteristics of the overall coordination of multiple channels can be considered to improve the accuracy of recognition. The structural design of the model ensures that multi-channel recognition can be carried out simultaneously, while ensuring the recognition accuracy, the recognition efficiency can be taken into account.

[0125] 2. Preprocessing and data standardization of EEG signals can obtain relatively pure EEG signals and accelerate the convergence speed.

[0126] 3. Use a sliding window to intercept the EEG signal to obtain a standardized signal segment (containing m′ signal components). For the same signal segment, calculate the correlation between every two signal components, further calculate the independence of each signal component, and select the m signal components with the highest independence as the input signal. This can further remove the signal components of redundant signal channels, while ensuring the classification accuracy and the stability of the number of signal components in the input signal, which is conducive to improving the reliability of the classification results, reducing the data processing amount of the model, and improving the recognition efficiency.

[0127] 4. Utilize the independence index of the m signal components in the input signal to calculate the confidence channel component of each signal component, as well as the independence index of the m signal components in the input signal and the independence index of the m′ signal components in the corresponding signal segment, and calculate the confidence overall component as the weight parameter for calculating the probability of the input signal being classified into each category. The independence of each signal component and the overall confidence of the signal components in the input signal can be considered, which is beneficial to improving the accuracy of classification and recognition, especially for the classification and recognition of conscious movement of EEG signals, which is difficult to classify. It can consider the features at different levels and the corresponding weights, which is more conducive to improving the classification effect.

[0128] See also Figure 3 Based on the same inventive concept, the present application also provides an electroencephalogram signal recognition device 10 based on a convolutional neural network, comprising:

[0129] The EEG signal acquisition unit 11 is used to acquire the EEG signal of the user.

[0130] The EEG signal processing unit 12 is used to perform preprocessing and data standardization on the EEG signal to obtain an input signal.

[0131] The conscious movement recognition unit 13 is used to input the input signal into a preset conscious movement classification and recognition model to obtain a conscious movement recognition result after the conscious movement classification and recognition model classifies and recognizes the input signal.

[0132] In this embodiment, the conscious movement classification and recognition model includes an input layer, n convolution channels, a fully connected layer, an overall output layer and a model output layer, and each convolution channel is sequentially provided with a convolution layer Con i1 , Pooling layer i1 , Convolutional layer i2 , Pooling layer i2 , Convolutional layer i3 , Pooling layer i3 , channel connection layer FC i And channel output layer Out i , i=1,2,…,n, the process of the conscious movement classification and recognition model classifying and identifying the input signal includes: the input layer is used to receive the input signal, wherein the input signal includes signal components of m signal channels, and any signal channel of the m signal channels has a unique corresponding convolution channel in the n convolution channels, m≤n; n convolution channels are respectively connected to the input layer, each convolution channel is used to receive the signal component of the input layer transmitted to the convolution channel, and the signal components of the convolution channel are convoluted and pooled by each convolution layer and pooling layer of the convolution channel, and the channel connection layer of the convolution channel is used to flatten to obtain the channel data corresponding to the signal components of the convolution channel, and then through The channel output layer classifies and identifies the channel data to obtain the conscious movement channel component, wherein the input layer transmits the corresponding signal component to the corresponding convolution channel based on the correspondence between the signal channel and the convolution channel; the fully connected layer is connected to the channel connection layers of the n convolution channels respectively, and is used to receive the channel data output by each convolution channel and splice to obtain one-dimensional data; the overall output layer is used to receive the one-dimensional data and classify and judge the one-dimensional data using the Softmax classification function to obtain the conscious movement overall component; the model output layer is connected to the channel output layer of each convolution channel respectively, and is connected to the overall output layer, and is used to determine the conscious movement recognition result based on each conscious movement channel component and the conscious movement overall component.

[0133] In this embodiment, the EEG signal processing unit 12 is specifically used for: filtering the EEG signal and performing bad conduction correction on the filtered EEG signal; using the ICA algorithm to remove artifact interference from the EEG signal after bad conduction correction; performing data normalization on the EEG signal after artifact interference removal to obtain the input signal.

[0134] In this embodiment, the EEG signal processing unit 12 is specifically used to: use a sliding window to intercept and process the EEG signal after artifact interference is removed, the sliding window size is τ, the step size is s, and obtain a number of groups of signal segments with adjacent time intervals of s; perform correlation analysis on each signal segment to obtain the input signal corresponding to each signal segment, so as to obtain a number of input signals corresponding to the EEG signal.

[0135] In this embodiment, each signal segment contains m′ signal components, m′≥m, and the EEG signal processing unit 12 is specifically used to: for each signal segment: calculate the correlation index between every two signal components in the signal segment; based on the correlation index between each signal component in the signal segment and other signal components, determine the independence index of this signal component; based on the independence index of each signal component in the signal segment, determine the m signal components with the highest independence index in the signal segment as the input signal containing m signal components corresponding to the signal segment.

[0136] An embodiment of the present application provides a storage medium, which includes a stored program, wherein when the program is run, the device where the storage medium is located is controlled to execute an electroencephalogram signal recognition method based on a convolutional neural network.

[0137] In summary, the embodiment of the present application provides a method and device for recognizing EEG signals based on a convolutional neural network, by preprocessing and data standardization of the user's EEG signal to obtain an input signal, and then inputting the input signal into a preset conscious movement classification recognition model to obtain a conscious movement recognition result after classification recognition. The design and construction of the conscious movement classification recognition model can consider the signal components of multiple signal channels in the input signal (the EEG signal usually collects signals from multiple channels), and use the corresponding channel to identify each signal component to obtain the conscious movement channel component reflecting the classification of the signal component, and use the channel data of each signal component to merge into an overall one-dimensional data to identify the overall component of conscious movement, and then combine the conscious movement channel component and the conscious movement overall component to realize the conscious movement classification recognition of the EEG signal. In this way, not only can the characteristics of each signal component be considered comprehensively, but also the characteristics of multi-channel overall coordination can be considered to improve the accuracy of recognition. The structural design of the model ensures that multi-channel recognition can be carried out simultaneously, while ensuring the recognition accuracy, the recognition efficiency can be taken into account. The preprocessing and data standardization of the EEG signal can obtain a relatively pure EEG signal and can also accelerate the convergence speed. The EEG signal is intercepted by sliding window to obtain the signal segment after standardization (including m′ signal components). The correlation between each two signal components of the same signal segment is calculated, and the independence of each signal component is further calculated. The m signal components with the highest independence are selected as the input signal. This can further remove the signal components of the redundant signal channel, ensure the stability of the number of signal components in the input signal while ensuring the classification accuracy, which is conducive to improving the reliability of the classification result, and can also reduce the data processing amount of the model and improve the recognition efficiency. The confidence channel component of each signal component is calculated by using the independence index of the m signal components in the input signal and the independence index of the m′ signal components in the corresponding signal segment, and the confidence overall component is calculated as the weight parameter for calculating the probability of the input signal being classified in each category. The independence of each signal component and the overall confidence of the signal components in the input signal can be considered, which is conducive to improving the accuracy of classification and recognition. Especially for the classification and recognition of conscious movement of EEG signals, which is difficult to classify, it can consider the features and corresponding weights at different levels, which is more conducive to improving the classification effect.

[0138] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0139] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0140] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for recognizing electroencephalogram signals based on convolutional neural networks, characterized in that: include: Obtain the user's EEG signal; Preprocessing and data normalization processing are performed on the EEG signal to obtain an input signal; Inputting the input signal into a preset conscious movement classification and recognition model to obtain a conscious movement recognition result after the conscious movement classification and recognition model classifies and recognizes the input signal; The conscious movement classification and recognition model comprises an input layer, n convolution channels, a fully connected layer, an overall output layer and a model output layer. Each convolution channel is sequentially provided with a convolution layer Con i1 , Pooling layer i1 , Convolutional layer i2 , Pooling layer i2 , Convolutional layer i3 , Pooling layer i3 , channel connection layer FC i And channel output layer Out i , i=1,2,...,n, the process of the conscious movement classification and recognition model classifying and recognizing the input signal includes: The input layer is used to receive an input signal, wherein the input signal includes signal components of m signal channels, and any signal channel of the m signal channels has a unique corresponding convolution channel in the n convolution channels, m≤n; n convolution channels are connected to the input layer respectively, each convolution channel is used to receive the signal component transmitted to the convolution channel by the input layer, and use the convolution layers and pooling layers of the convolution channel to perform convolution and pooling processing on the signal component of the convolution channel, and use the channel connection layer of the convolution channel to flatten it to obtain the channel data corresponding to the signal component of the convolution channel, and then classify and identify the channel data through the channel output layer to obtain the conscious movement channel component, wherein the input layer transmits the corresponding signal component to the corresponding convolution channel based on the corresponding relationship between the signal channel and the convolution channel; The fully connected layer is connected to the channel connection layers of the n convolution channels respectively, and is used to receive the channel data output by each convolution channel and concatenate them to obtain one-dimensional data; The overall output layer is used to receive one-dimensional data and classify and judge the one-dimensional data using the Softmax classification function to obtain the overall component of consciousness movement; The model output layer is connected to the channel output layer of each convolution channel, and is connected to the overall output layer, and is used to determine the conscious motion recognition result based on each conscious motion channel component and the conscious motion overall component; The EEG signal is preprocessed and data normalized to obtain an input signal, including: Filtering the EEG signal and performing bad conduction correction on the filtered EEG signal; removing artifact interference from the EEG signal after bad conduction correction using an ICA algorithm; performing data normalization on the EEG signal after artifact interference removal to obtain the input signal; The EEG signal after removing artifact interference is subjected to data standardization processing to obtain the input signal, including: The EEG signal after artifact interference is removed is intercepted by using a sliding window, the sliding window size is τ, the step length is s, and several groups of signal segments with adjacent time intervals of s are obtained; correlation analysis is performed on each signal segment to obtain the input signal corresponding to each signal segment, so as to obtain several input signals corresponding to the EEG signal; Each signal segment contains m′ signal components, m′≥m. Correlation analysis is performed on each signal segment to obtain the input signal corresponding to each signal segment, including: For each signal segment: calculate the correlation index between every two signal components in this signal segment; determine the independence index of this signal component based on the correlation index between each signal component and other signal components in this signal segment; based on the independence index of each signal component in this signal segment, determine the m signal components with the highest independence index in this signal segment as the input signal containing m signal components corresponding to this signal segment.

2. The method for recognizing electroencephalogram signals based on a convolutional neural network according to claim 1, characterized in that: After obtaining the input signal corresponding to each signal segment, the method further includes: Based on the independence indexes of the m signal components in the input signal, a confidence channel component corresponding to each signal component is calculated, and based on the independence indexes of the m signal components in the input signal and the independence indexes of the m′ signal components in the corresponding signal segment, a confidence overall component is calculated; Correspondingly, the model output layer determines the conscious motion recognition result based on each conscious motion channel component and the conscious motion overall component, specifically including: Based on the confidence channel component and conscious motion channel component corresponding to each signal component, as well as the confidence overall component and conscious motion overall component, the probability of the input signal being classified into each category is calculated, and based on the probability of each category, the category with the highest probability is determined as the conscious motion recognition result.

3. The method for recognizing electroencephalogram signals based on convolutional neural networks according to claim 2, characterized in that: Based on the confidence channel component and the consciousness movement channel component corresponding to each signal component, as well as the confidence overall component and the consciousness movement overall component, the probability of the input signal being classified into each category is calculated, including: The probability of the input signal being classified into each category is calculated using the following formula: Among them, P j represents the probability that the input signal is classified into the jth category, j = 1, 2, ..., k, k is the total number of categories, Dw represents the overall confidence component of the input signal, ToP j represents the probability of the jth category in the overall component of conscious movement, Represents the lth confidence channel component of the input signal, ChP lj Represents the probability of the th category in the lth conscious motion channel component.

4. An electroencephalogram signal recognition device based on convolutional neural network, characterized in that: include: An electroencephalogram signal acquisition unit, used to acquire an electroencephalogram signal of a user; An EEG signal processing unit, used for performing preprocessing and data standardization on the EEG signal to obtain an input signal; The conscious movement recognition unit is used to input the input signal into a preset conscious movement classification recognition model to obtain the conscious movement recognition result after the conscious movement classification recognition model classifies and recognizes the input signal; the conscious movement classification recognition model includes an input layer, n convolution channels, a fully connected layer, an overall output layer and a model output layer, and each convolution channel is sequentially provided with a convolution layer Con i1 , Pooling layer i1 , Convolutional layer i2 , Pooling layer i2 , Convolutional layer i3 , Pooling layer i3 , channel connection layer FC i And channel output layer Out i , i=1, 2, ..., n, the process of the conscious movement classification and recognition model classifying and identifying the input signal includes: the input layer is used to receive the input signal, wherein the input signal contains signal components of m signal channels, and any signal channel of the m signal channels has a unique corresponding convolution channel in the n convolution channels, m≤n; n convolution channels are respectively connected to the input layer, each convolution channel is used to receive the signal components of the input layer transmitted to the convolution channel, and the signal components of the convolution channel are convoluted and pooled by using the convolution layers and pooling layers of the convolution channel, and the channel connection layer of the convolution channel is used to flatten to obtain the channel data corresponding to the signal components of the convolution channel, and then pass The channel data is classified and identified through the channel output layer to obtain the channel component of consciousness movement, wherein the input layer transmits the corresponding signal component to the corresponding convolution channel based on the correspondence between the signal channel and the convolution channel; the fully connected layer is respectively connected to the channel connection layers of the n convolution channels, and is used to receive the channel data output by each convolution channel, and splice to obtain one-dimensional data; the overall output layer is used to receive the one-dimensional data, and classify and judge the one-dimensional data using the Softmax classification function to obtain the overall component of consciousness movement; the model output layer is respectively connected to the channel output layer of each convolution channel, and is connected to the overall output layer, and is used to determine the consciousness movement recognition result based on each consciousness movement channel component and the consciousness movement overall component; The EEG signal processing unit is specifically used to: filter the EEG signal and perform bad conduction correction on the filtered EEG signal; use the ICA algorithm to remove artifact interference from the EEG signal after bad conduction correction; perform data standardization on the EEG signal after artifact interference is removed to obtain the input signal; The EEG signal processing unit is specifically used to: intercept and process the EEG signal after removing artifact interference using a sliding window, where the sliding window size is τ and the step length is s, to obtain a plurality of groups of signal segments with adjacent time intervals of s; perform correlation analysis on each signal segment to obtain an input signal corresponding to each signal segment, so as to obtain a plurality of input signals corresponding to the EEG signal; Each signal segment contains m′ signal components, m ′ ≥m, the EEG signal processing unit is specifically used to: for each signal segment: calculate the correlation index between every two signal components in the signal segment; based on the correlation index between each signal component in the signal segment and other signal components, determine the independence index of this signal component; based on the independence index of each signal component in the signal segment, determine the m signal components with the highest independence index in the signal segment as the input signal containing m signal components corresponding to the signal segment.

5. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the EEG signal recognition method based on convolutional neural network according to any one of claims 1 to 3.

6. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, and when the program instructions are loaded and executed by the processor, the EEG signal recognition method based on a convolutional neural network described in any one of claims 1 to 3 is implemented.

Images