A target recognition and feedback system based on deep learning

Abstract

The present invention relates to a target recognition and feedback system based on deep learning, which belongs to the field of visual search human-computer interaction. It discloses a method for classifying and identifying EEG signals using a deep learning network based on an external attention mechanism and an EEG automatic correction method, which has the effect of improving target recognition efficiency. The system includes an offline training module: used to generate task-related EEG by visually stimulating the user, and collect EEG data as training samples; an online training module: used to collect the user's EEG data in real time when the user searches for a target, and to feed back the result of the target recognition module to the user in real time; an EEG signal automatic correction module: automatically learns a correction function according to the EEG differences between different users, thereby correcting the EEG, and inputting the corrected EEG signal into a target recognition module; a target recognition module: used to analyze the EEG signal and identify the result of the user's target search.

Description

Technical Field

[0001] The present invention belongs to the field of visual search human-computer interaction and relates to a target recognition and feedback system based on deep learning. Background Art

[0002] With the continuous development of society, brain science has also entered a period of rapid development. The world's scientific research powers have successively sounded the clarion call to explore the mysteries of the brain. Therefore, the China Brain Project, which aims to explore the mysteries of the brain, develop human-computer collaboration, and develop artificial intelligence technology-oriented brain-like research, has also been officially launched. EEG is the change in the potential of brain neurons recorded from the surface of the scalp. It is the sum of the postsynaptic potentials of a large number of neurons that occur synchronously. It is an electric field potential that manifests itself as a change in the potential difference between different points on the head surface. It mainly comes from excitatory and inhibitory postsynaptic potentials. Although EEG signals have been used to describe the neurophysiological changes that accompany psychological processes for decades, it was not until the late 1990s that it was used to study cognitive-related tasks.

[0003] Traditional machine learning algorithms (such as SVM and KNN) have the following disadvantages: (1) Feature extraction and feature classification are separated, which increases the workload; (2) EEG is subjected to complex preprocessing and then a large amount of prior knowledge is used to extract classification features; (3) Since EEG signals involve multiple dimensions of information such as time, space, and frequency domains, manual feature extraction may ignore discriminative information.

[0004] Invention patent application CN110610168A discloses a method based on self-attention, channel attention and LSTM, which uses convolutional neural network (CNN) and channel-wise attention mechanism to extract spatial information from the original EEG signal, and then uses recurrent neural network (RNN) and self-attention mechanism to extract temporal information from the encoded sample, and finally obtains the classification features of the EEG signal. However, the channel attention mechanism and self-attention mechanism used in this method are both based on the attention mechanism of the data itself, without considering the commonality of the EEG of users under the same task; the attention mechanism and RNN network calculation complexity used in this method are very high, thus consuming more computing resources; this method uses the entire EEG time feature, without considering the dynamic change process in time, and without considering the individual differences and commonality between data.

[0005] Invention patent application CN103077205A discloses an EEG-based image retrieval system, which uses the advantages of the human visual system in target image detection to design a new type of human-machine collaborative image retrieval system. This method does not use a feedback module to help users improve their status and improve the accuracy of user retrieval when the user's image retrieval status is not good, and it uses traditional machine learning methods in the classification module, which requires complicated work such as manual extraction. Summary of the invention

[0006] In view of this, the purpose of the present invention is to solve the problems that there is no system that combines recognition with training in the existing recognition system, the EEG recognition method has a complex calculation process, the attention mechanism in the neural network does not consider the relationship between data and the high computational complexity, and to provide a deep learning target recognition and feedback system. The system detects the user's state of target recognition based on the neural response during visual stimulation, and gives feedback on the result of target recognition. This form of feedback can improve the user's target recognition accuracy and speed. At the same time, a target recognition method with low computational complexity and capable of learning the correlation between data is proposed.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A target recognition and feedback system based on deep learning, including an offline training module, an online training module, an EEG signal correction module and a target recognition module;

[0009] The target recognition module uses a deep learning network based on an external attention mechanism to identify the target. The deep learning network includes an attention block, a spatiotemporal convolution block and a classification block. Each spatiotemporal convolution block is composed of one or more hidden layers. The attention block uses an external attention mechanism to enhance and learn the characteristics of the EEG signal.

[0010] The EEG signal correction module is used to automatically correct the EEG differences between different users. According to the differences between the EEGs of different users, the EEG correction function learns correction parameters, thereby automatically correcting the differences between the EEGs, and inputting the corrected EEG signals into the target recognition module;

[0011] The offline training module is used to stimulate the user visually, so that the user searches for the target and generates EEG related to the visual search task, and collects the user's EEG data as training samples, which are input into the EEG signal correction module for automatic correction;

[0012] The online training module is used to collect the user's EEG data in real time when the user is searching for a target, transmit it to the EEG signal correction module and the target recognition module in turn, and feed back the recognition result of the target recognition module to the user in real time.

[0013] Furthermore, when the user identifies the target correctly, a positive feedback interface is provided to the user; when the user identifies the target incorrectly, a negative feedback interface is provided and the correct target position is marked, and the user adjusts his or her status according to the feedback result.

[0014] Furthermore, both the offline training module and the online training module provide visual stimulation to the user through the visual stimulation submodule, and the visual target stimulation process is as follows:

[0015] When the process starts, a fixed “X” appears in the middle of a screen with a black background, reminding the user to start training;

[0016] In the following process, the stimulus image and the black background image with a "cross" in the middle appear alternately, and the presentation time of the stimulus image is adjusted appropriately according to the user's adaptation;

[0017] When the recognition target appears on the left side of the "cross", the user needs to press the left button of the mouse. When the recognition target appears on the right side of the "cross", the user needs to press the right button of the mouse.

[0018] During user training, the international 10-20 system is used to record the user's EEG.

[0019] Furthermore, the correction steps of the EEG signal correction module are as follows:

[0020] S11: First, find the activation point i related to a specific cognitive task through the max function:

[0021] i=max(x 2 )

[0022] S12: Obtaining the specific cognitive state distribution EEG required by deep learning networks

[0023] x={c i-a ,…,c i-a+499}

[0024] S13: Map specific cognitive EEG signals. The mapping range is learned based on EEG. The formula is:

[0025] o=A(x)+β

[0026] in, s is the stimulation start point, a is a constant set according to a specific cognitive task, A is a learnable parameter, β is the range correction, and o is the EEG signal after correction.

[0027] Furthermore, the deep learning network of the external attention mechanism adopted by the target recognition module includes an attention block, a spatiotemporal convolution block and a classification block. Each convolution block is composed of one or more hidden layers. The specific process of network construction is as follows:

[0028] In the attention block, two steps are performed: a convolution and a feature attention layer. First, when performing a 2D convolution, the output contains a feature map of EEG signals of different frequencies. Then, when performing the attention step, the attention mechanism weights the learned EEG signals to obtain the salient signals of attention and reallocate resources.

[0029] In the spatiotemporal convolution block, two convolution steps are performed: first, Depthwise convolution is used to learn a spatial filter; then, Separable convolution is used to learn the temporal features of EEG, and the separable convolution kernel size is (1,16); before using the nonlinear activation function ReLU, batch normalization technology is used on the feature map to prevent gradient disappearance and gradient explosion problems; then, dropout technology is used to solve the problem of overfitting of the model during training; finally, a pooling layer is used to reduce the dimension of the features;

[0030] Classify the features through the classification block: First, the previously learned features are expanded into a one-dimensional vector through the Flatten layer, and the probabilities of the two neurons are calculated. The neuron with the largest probability is the category obtained by the deep learning network.

[0031] Furthermore, the external attention mechanism is used to reallocate attention to features in the EEG and enhance features, including the following steps:

[0032] S21: Transform the original features into multiple feature layers:

[0033]

[0034] Where i represents the i-th feature layer generated, A represents the learnable parameter matrix, and I represents the input feature;

[0035] S22: Redistribute channel weights of features:

[0036]

[0037] Where e is a natural number, λ i,j It represents the feature of the ith feature layer after calculating the channel weight, α c,j represents the jth feature of the cth channel;

[0038] S23: Calculate the weights of the features at the time points and assign new significance to the brain regions, and learn the temporal characteristics of EEG-related cognitive tasks:

[0039]

[0040] where β i,jIt is represented as the feature of the jth feature of the i-th feature layer after the weight is redistributed;

[0041] S24: Through the features after weight distribution, the relationship and commonality between the learning data through the linear layer:

[0042]

[0043] where χ i It represents the feature of the jth feature of the i-th layer after the global weight update. represents the global learnable parameters;

[0044] S25: The learned multi-layer features are fused as the feature output of the entire feature attention layer, and its expression is:

[0045]

[0046] Where o is the output feature, χ h represents the h-th learned feature.

[0047] The beneficial effects of the present invention are: the present invention can effectively help users improve their target recognition capabilities, the present invention effectively uses EEG correction to eliminate differences between data, and further improves the accuracy of target recognition by the neural network; the target recognition module of this system proposes and uses a lightweight neural network, and effectively uses a learnable external attention mechanism to help the system have good target recognition efficiency and capabilities.

[0048] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0050] Figure 1 This is the overall structure diagram of the target recognition and feedback system based on deep learning;

[0051] Figure 2 This is the workflow diagram of the visual stimulation submodule;

[0052] Figure 3 is a convolutional neural network diagram;

[0053] Figure 4 Target recognition accuracy graph for new users. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0055] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0056] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0057] The present invention provides a target recognition feedback system based on deep learning, such as Figure 1 As shown in the figure, it includes offline training module, online training module, EEG correction module and target recognition module. In both online and offline training modules, specific EEG is generated by visual stimulation, and the EEG is corrected by the EEG correction module. The target recognition module is responsible for training the deep learning model and identifying EEG. In the online training module, the user's recognition results will be fed back, so that the user can adjust his or her status to achieve the training effect. Signal transmission adopts the UDP protocol to improve real-time performance.

[0058] In order to better illustrate the technical effect of the present invention, this embodiment adds technical content for the purpose of result verification on the basis of the original design of the present invention.

[0059] The target recognition and feedback system based on deep learning provided by the present invention includes

[0060] 1) Target recognition module: This paper proposes a new neural network framework that uses an external attention mechanism based on two external, learnable, shared memory units. It can be implemented using only two specific linear layers and two normalization layers. It effectively replaces the self-attention mechanism in the existing popular architecture and reduces the complexity of the existing attention mechanism. At the same time, its external attention has linear complexity and can implicitly consider the correlation between all data samples.

[0061] 2) EEG signal correction module: Considering the individual differences in brain nerve signals between users, correction of EEG signals can improve the decoding performance of the neural network. In the EEG correction module of the present invention, a method for EEG signal correction is proposed, which can adaptively construct a learning parameter matrix to correct EEG according to the cognitive characteristics of a specific task;

[0062] 3) Offline training module, which is used to stimulate the user visually, so that the user searches for the target and generates EEG related to the visual search task, and collects the user's EEG data as training samples to train the deep learning model;

[0063] 4) An online training module is used to collect the user's EEG data when the new user is visually stimulated, and use the deep learning model to classify the EEG data to identify the results of the user's target search, and at the same time feed back the classification results to the user to improve the user's search efficiency.

[0064] In the present invention, Python language is used to develop a visual stimulation program, and the visual target stimulation process is as follows: Figure 2 As shown. At the beginning of the experiment, a fixed "cross" appears in the middle of a screen with a black background to remind the user to start training, which lasts for 0.5s. In the following process, the stimulus picture and the black background picture with a "cross" in the middle appear alternately. The presentation time of the stimulus picture can be adjusted appropriately according to the user's adaptation (from 500ms to 5s). During the entire experiment, the user should maintain a high level of attention and try to blink as little as possible. When the recognition target appears on the left side of the "cross", the user needs to press the left button of the mouse, and when the recognition target appears on the right side of the "cross", the user needs to press the right button of the mouse.

[0065] During the user training process, the international 10-20 system is used to record the user's EEG, with a sampling frequency of 1000 Hz and 62 channels.

[0066] Since the EEG acquisition process contains artifacts and EEG drift to a greater or lesser extent in both the offline training module and the online training module, it is necessary to correct the EEG.

[0067] Eliminate the differences between data, learn the EEG correction parameters, and correct the differences between EEG, so as to improve the accuracy of neural network recognition. The correction steps are:

[0068] S1: First, find the activation point i related to a specific cognitive task through the max function:

[0069] i=max(x 2 )

[0070] S2: Get the specific cognitive state distribution EEG required by the neural network:

[0071] x={c i-a ,…,c i-a+499}

[0072] S3: Map specific cognitive EEG signals. The mapping range is learned based on EEG. The formula is:

[0073] o=A(x)+β

[0074] in, s is the stimulation start point, a is a constant set according to a specific cognitive task, A is a learnable parameter, β is the range correction, and o is the EEG signal after correction.

[0075] In this invention, a neural network is used as a classifier for visual target search. At the same time, in this system, a neural network with an attention mechanism is designed as a classification model. The neural network graph architecture is as follows Figure 3 As shown in Figure 2. The network consists of three blocks: attention block, spatiotemporal convolution block and classification block. Each convolution block consists of one or more hidden layers. The specific design of the model is as follows:

[0076] In the attention block, two steps are performed: a convolution and a feature attention layer. First, when performing a 2D convolution, the output contains a feature map of EEG signals of different frequencies. Then, when performing the attention step, the attention mechanism weights the learned EEG signals, which can obtain the significant signals of attention and thus reallocate resources. At the same time, it can also reduce the noise in the EEG signals and improve the classification accuracy of the model.

[0077] In the spatiotemporal convolution block, two convolution steps are performed. First, a depthwise convolution is used to learn a spatial filter. The main benefit of depthwise convolution is that it reduces the number of fits of trainable parameters because these convolutions are not fully connected to all previous feature maps. Then, we use separable convolution with a kernel size of (1,16) to learn the temporal features of the EEG. In order to prevent the gradient vanishing and gradient exploding problems, batch normalization techniques are used on the feature maps before using the nonlinear activation function ReLU. In order to solve the problem of overfitting of the model during training, the dropout technique is used. Then, a pooling layer is used to reduce the dimensionality of the features.

[0078] Through the above feature extraction and enhancement, the classification block is used to classify the features. The first is a Flatten layer, which expands the previously learned features into a one-dimensional vector. Finally, the probability of the two neurons is calculated, and the neuron with the highest probability is the category obtained by the neural network.

[0079] The above attention mechanism redistributes attention and enhances features in EEG, which mainly includes five steps:

[0080] S1: The original features are transformed into multiple feature layers to learn EEG features from multiple dimensions. The expression formula is:

[0081]

[0082] Where i represents the i-th feature layer generated, A represents the learnable parameter matrix, and I represents the input feature.

[0083] S2: Redistribute the channel weights of the features to learn the brain area features related to cognitive tasks. The expression is:

[0084]

[0085] Where e is a natural number, λ i,j It represents the feature of the ith feature layer after calculating the channel weight, α c,j represents the j-th feature of the c-th channel.

[0086] S3: Calculate the weights of the features at the time points and assign new significance to the brain regions, and learn the temporal characteristics of EEG-related cognitive tasks. The expression is:

[0087]

[0088] where β i,j It is represented as the feature of the jth feature of the i-th feature layer after the weight is redistributed.

[0089] S4: Through the features after weight assignment, the relationship and commonality between the learning data are passed through the linear layer.

[0090]

[0091] where χ i It represents the feature of the jth feature of the i-th layer after the global weight update. Global learnable parameters.

[0092] S5: The learned multi-layer features are integrated as the feature output of the entire attention layer, and its expression is:

[0093]

[0094] Where o is the output feature, χ h represents the h-th learned feature.

[0095] Model training is performed using the EEG data deep model collected by the offline training module. The cross entropy loss function is used to optimize EAEEGNet. In order to improve the convergence speed of the model, the data is normalized before model training, and batch normalization technology is used in the model to improve the convergence speed of the model. In order to prevent the model from overfitting during training, the dropout technology is used, and its probability is set to 0.5. The learning rate used in this invention is 1×10 -3 The Adam optimizer is used to train the network.

[0096] During the online training process, new users should try to choose a time that suits their visual stimulation target presentation (it is recommended to use a presentation time of 1 second first). After the visual stimulation begins, the collected EEG will be sent to the EEG correction module. The EEG correction module corrects the EEG, and then the EEG data is sent to the neural network to identify the target. If the user is identified to have correctly found the target, positive feedback will be presented to the user; if the user cannot correctly find the target or does not respond at the end of the visual stimulation, negative feedback will be presented to the user and the location of the target will be indicated with a significant mark. This helps users adjust their state, thereby improving their search and recognition capabilities.

[0097] In order to verify the effectiveness of the present invention, five normal users were used to train their target recognition ability. The experimental results are shown in Figure 4 As shown in the figure, the experimental results show that their recognition accuracy reached 88.7% on average, achieving the expected effect.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A target recognition and feedback system based on deep learning, characterized in that: It includes offline training module, online training module, EEG signal correction module and target recognition module; The target recognition module uses a deep learning network based on an external attention mechanism to recognize the target, and the deep learning network includes an attention block, a spatiotemporal convolution block and a classification block, each of which is composed of one or more hidden layers; the attention block uses an external attention mechanism to enhance and learn the characteristics of the EEG signal; The EEG signal correction module is used to automatically correct the EEG differences between different users. According to the differences between the EEGs of different users, the EEG correction function learns correction parameters, thereby automatically correcting the differences between the EEGs, and inputting the corrected EEG signals into the target recognition module; The offline training module is used to stimulate the user visually, so that the user searches for the target and generates EEG related to the visual search task, and collects the user's EEG data as training samples, which are input into the EEG signal correction module for automatic correction; The online training module is used to collect the user's EEG data in real time when the user is searching for a target, transmit it to the EEG signal correction module and the target recognition module in turn, and feed back the recognition result of the target recognition module to the user in real time; The correction steps of the EEG signal correction module are as follows: S11: First, find the activation point i related to the cognitive task through the max function: i=max(x 2 ) S12: Obtaining the cognitive state distribution EEG required by deep learning networks x={c i-a ,…,c i-a+499 } S13: Map the cognitive EEG signals. The mapping range is learned based on the EEG. The formula is: o=A(x)+β Where x = {c s-200 ,c s-199 ,…,c s ,…,c 1000 }, s is the stimulation start point, a is a constant set according to the cognitive task, A is a learnable parameter, β is the range correction, and o is the EEG signal after correction.

2. The target recognition and feedback system based on deep learning according to claim 1, characterized in that: When the user identifies the target correctly, a positive feedback interface is provided to the user; when the user identifies the target incorrectly, a negative feedback interface is provided and the correct target position is marked, and the user adjusts his or her status based on the feedback results.

3. The target recognition and feedback system based on deep learning according to claim 1, characterized in that: Both the offline training module and the online training module provide visual stimulation to the user through the visual stimulation submodule. The visual target stimulation process is as follows: When the process starts, a fixed "X" appears in the middle of a black background screen to remind the user to start training; In the following process, the stimulus image and the black background image with a "cross" in the middle appear alternately, and the duration of the stimulus image presentation is adjusted appropriately according to the user's adaptation; When the recognition target appears on the left side of the "cross", the user needs to press the left button of the mouse. When the recognition target appears on the right side of the "cross", the user needs to press the right button of the mouse. During user training, the international 10-20 system is used to record the user's EEG.

4. The target recognition and feedback system based on deep learning according to claim 1, characterized in that: The deep learning network of the external attention mechanism adopted by the target recognition module includes an attention block, a spatiotemporal convolution block and a classification block. Each convolution block is composed of one or more hidden layers. The specific process of network construction is as follows: In the attention block, two steps are performed: a convolution and a feature attention layer. First, when performing a 2D convolution, the output contains a feature map of EEG signals of different frequencies. Then, when performing the attention step, the attention mechanism weights the learned EEG signals to obtain the salient signals of attention and reallocate resources. In the spatiotemporal convolution block, two convolution steps are performed: first, Depthwise convolution is used to learn a spatial filter; then, Separable convolution is used to learn the temporal features of EEG, and the separable convolution kernel size is (1,16); before using the nonlinear activation function ReLU, batch normalization technology is used on the feature map to prevent gradient disappearance and gradient explosion problems; then, dropout technology is used to solve the problem of overfitting of the model during training; finally, a pooling layer is used to reduce the dimension of the features; Classify the features through the classification block: First, the previously learned features are expanded into a one-dimensional vector through the Flatten layer, and the probabilities of the two neurons are calculated. The neuron with the largest probability is the category obtained by the deep learning network.

5. The target recognition and feedback system based on deep learning according to claim 1, characterized in that: The external attention mechanism is used to reallocate attention to features in EEG and enhance features, including the following steps: S21: Transform the original features into multiple feature layers: Where i represents the i-th feature layer generated, A represents the learnable parameter matrix, and I represents the input feature; S22: Redistribute channel weights of features: Where e is a natural number, λ i,j It represents the feature of the ith feature layer after calculating the channel weight, α c,j represents the jth feature of the cth channel; S23: Calculate the weights of the features at the time points and assign new significance to the brain regions, and learn the temporal characteristics of EEG-related cognitive tasks: where β i,j It is represented as the feature of the jth feature of the i-th feature layer after the weight is redistributed; S24: Through the features after weight distribution, the relationship and commonality between the learning data through the linear layer: where χ i It represents the feature of the jth feature of the i-th layer after the global weight update. represents the global learnable parameters; S25: The learned multi-layer features are fused as the feature output of the entire feature attention layer, and its expression is: Where o is the output feature, χ h represents the h-th learned feature.

Images