Method for selecting target based on picture decoding electroencephalogram signal

Abstract

A method for selecting a target based on a picture decoding electroencephalogram signal comprises the following steps: 1, carrying out full-image segmentation on a video frame to obtain a plurality of objects, judging the plurality of objects as target objects and non-target objects through an attention thermodynamic diagram, and coding all the objects as first embedded vectors; 2, inputting the electroencephalogram signal into an electroencephalogram encoder to obtain an electroencephalogram embedded vector; 3, the Euclidean distance between the electroencephalogram embedding vector and the first embedding vector is calculated, the similarity between the electroencephalogram and the target object serves as a positive sample, and the similarity between the electroencephalogram and a non-target object serves as a negative sample; training the electroencephalogram encoder by using the positive sample and the negative sample to obtain a trained electroencephalogram encoder; and 4, testing the trained electroencephalogram encoder to obtain a target object selected by the user in the picture. By constructing the direct mapping of the electroencephalogram signal and the image object, the problem of slow target selection of the electroencephalogram signal is solved, and a user can be helped to directly select an expected object through the electroencephalogram signal.

Description

Technical Field

[0001] The present invention relates to the field of cognitive engineering and machine vision technology, and in particular to a method for selecting a target based on picture-based decoding of electroencephalogram signals. Background Art

[0002] Brain Computer Interface (BCI) can provide technical support for direct communication between humans and computers. Among them, electroencephalography (EEG), as a non-invasive neurophysiological measurement technology, can record the electrical activity of the cerebral cortex and is the most commonly used signal source in BCI systems. The real-time nature, high temporal resolution and sensitivity to specific brain activities of EEG signals make it one of the ideal choices for decoding brain intentions and communicating with external devices. However, because EEG signals are collected from the scalp, they have disadvantages such as low signal-to-noise ratio and low spatial resolution. In the past, target selection through EEG often relied on specific visual evoked paradigms, such as event-related potentials (ERP) or steady-state visual potentials (SSVEP).

[0003] With the development of brain-computer interface technology, brain-computer interface has received widespread attention in the field of disability assistance because it does not require any limb movement. For example, users can control prosthetic limbs or robotic arms to grab the desired target through brain-computer interface. However, in past studies, the process of user selection of targets often requires specific visual paradigms, such as ERP or SSVEP. This type of visual stimulation paradigm can easily cause visual fatigue in users. Alternatively, through motor imagery (MI), the computer cursor can be controlled to move to the target position in the image captured by the camera. However, due to the limitations of the signal-to-noise ratio and information transmission rate of non-invasive EEG signals, this method is often inefficient and has a high cognitive load. Summary of the invention

[0004] The present invention provides a method for selecting a target based on picture-decoded electroencephalogram signals to solve the technical problems mentioned in the background technology.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] The present invention provides a method for selecting a target based on picture-decoded electroencephalogram signals, comprising:

[0007] S1. Use the image segmentation model to segment the video frame into multiple objects, and use the EEG data and eye movement data to build an attention heat map. Use the attention heat map to determine the multiple objects as target objects and non-target objects, and then use the CLIP picture encoder to encode all the pictures containing the objects into the first embedding vector.

[0008] S2, input the EEG signal collected synchronously during the user's gaze on the target into the EEG encoder, and finally obtain the EEG embedding vector;

[0009] S3, calculate the Euclidean distance between the EEG embedding vector and the first embedding vector in S1, take the similarity between the EEG embedding vector and the first embedding vector of the picture where the target object is located as a positive sample, and take the similarity between the EEG embedding vector and the first embedding vector of the picture where the non-target object is located as a negative sample; and use the positive samples and negative samples to compare and train the EEG encoder to obtain the trained EEG encoder;

[0010] S4. Test the trained EEG encoder to obtain the target object selected by the user in the picture.

[0011] Furthermore, the S1 specifically includes the following steps:

[0012] S11, performing full-image segmentation on the video frame through an image segmentation model to obtain masks of all target objects and non-target objects in the image scene, and extracting each object in the image through the masks of the target objects and non-target objects;

[0013] S12, performing sliding window acquisition on the EEG signal and the output signal of the eye tracker with a set step length and window length, eliminating the eye saccade component, obtaining fixation eye movement data, and counting the number of eye movement sampling points in each object area in the fixation eye movement data under the set window length to obtain an attention heat map;

[0014] S13, analyzing the gaze heat of each object in the attention heat map, taking the object with the highest gaze heat as the target object, and the objects in other areas are non-target objects;

[0015] S14. Encode all pictures containing all objects into first embedding vectors through a CLIP picture encoder to obtain multiple first embedding vectors.

[0016] Furthermore, the method of eliminating the eye saccade component in S12 to obtain the fixation eye movement data is as follows:

[0017] S121, every time a sampling point is obtained, the earliest sampling point is removed from the eye movement data buffer, and the latest sampling point is added to the eye movement data buffer;

[0018] S122, calculating the distance between each sampling point and the previous sampling point in the eye movement data buffer point by point through a speed threshold filtering algorithm; if the distance between a certain sampling point and the previous sampling point meets the standard of fixation eye movement, it is considered that fixation eye movement has occurred, and the sampling point is added to a fixation buffer for analyzing the attention heat map;

[0019] S123, loop S121 to S122, and then obtain gaze eye movement data.

[0020] Furthermore, the EEG encoder in S2 includes a temporal convolution layer, a spatial convolution layer, an average pooling layer, a self-attention layer and a fully connected layer which are connected in sequence.

[0021] Furthermore, the S2 specifically includes the following steps:

[0022] S21. The EEG encoder inputs the EEG signal into the temporal convolution layer, which contains w convolution kernels with dimensions (1, d) and step length (1, 1). The output results of the w convolution kernels are integrated to obtain the temporal feature. The dimension of the temporal feature is (w, ch, m t ), where ch represents the number of channels, m t Represents the intermediate value of the time dimension after convolution;

[0023] S22, input the temporal features into the spatial convolution layer; the spatial convolution layer contains w convolution kernels with dimensions (ch, 1) and stride (1, 1). After convolution, the spatiotemporal features are obtained, and the dimensions of the spatiotemporal features are (w, 1, m t );

[0024] S23, input the spatiotemporal features of the EEG signal into the average pooling layer; the average pooling layer smoothes the temporal features along the time direction with a size of (1, t) and a step size of (1, s) to obtain smoothed spatiotemporal features, and the dimension of the smoothed spatiotemporal features is (w, 1, m), where m represents the time series dimension;

[0025] S24, inputting the smoothed spatiotemporal features of the EEG signal into the self-attention layer to obtain the global correlation between the time points, and obtaining the global features;

[0026] S25, performing a weighted operation on the global features to obtain a one-dimensional EEG embedding vector, and then adjusting the length of the one-dimensional EEG embedding vector to be the same as the length of the first embedding vector in S1 through a fully connected layer.

[0027] Furthermore, the median value m of the time in S21 is t Calculated by the following formula:

[0028] m t =n-d+1

[0029] Among them, n represents the number of sampling points of the original EEG signal; d represents the dimension of the convolution kernel in the temporal convolution layer.

[0030] Furthermore, the self-attention layer in S24 is expressed by a formula as follows:

[0031]

[0032] Among them, k is the length of a vector, Q represents the time node currently being processed; K represents all nodes in the time series; V represents the actual feature information of each time point, and T represents the transpose of the matrix.

[0033] Furthermore, the implementation of S3 is as follows:

[0034] First, the Euclidean distances between the EEG embedding vector and all the first embedding vectors in S1 are calculated to evaluate the similarity of the EEG-image pair; the similarity between the EEG embedding vector and the target object is used as a positive sample, and the similarity between the EEG embedding vector and the non-target object is used as a negative sample; the loss function of the positive and negative samples is calculated; and S1 to S3 are looped until the loss function converges, and the weight with the smallest loss is selected as the weight of the EEG encoder to obtain the trained EEG encoder.

[0035] Furthermore, the loss function is expressed by a formula as follows:

[0036] Loss=max(d(P)-d(N)+margin,0)

[0037] Among them, Loss represents the loss function, d(p) represents the positive sample, d(N) represents the negative sample, and margin is a positive number.

[0038] Furthermore, the S4 specifically includes the following steps:

[0039] S41, using the trained EEG encoder to encode the EEG signal collected in real time to obtain a new EEG embedding vector;

[0040] S42, using a CLIP picture encoder to perform picture encoding on the pictures where all objects segmented from the picture are located, to obtain a plurality of first embedding vectors;

[0041] S42. Calculate similarities between the new EEG embedding vector and the multiple first embedding vectors in S42 respectively, so as to determine the target object selected by the user in the picture.

[0042] Beneficial effects of the present invention:

[0043] The present invention solves the problem of slow target selection by EEG signals by constructing a direct mapping between EEG signals and image objects, and can help users select desired objects directly through EEG signals. Through the present invention, the EEG signals of the user during the period of gazing at the target object can be decoded, and the desired object in the image recorded by the camera can be directly selected to control the robotic arm to grab it. The cognitive load when the user indicates the desired target is reduced, providing the user with a more natural operating experience; it can be used to establish an efficient brain-computer interface system. It can help patients with movement disorders to freely select objects and express their wishes. It has great practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the present invention;

[0045] Figure 2 It is one of the video frames captured in real time by the camera of the robotic arm in the embodiment of the present invention;

[0046] Figure 3 In the embodiment of the present invention, the image segmentation model is used to Figure 1 The segmented image;

[0047] Figure 4 is an attention heat map in an embodiment of the present invention;

[0048] Figure 5 is the target object obtained by the attention heat map in the embodiment of the present invention;

[0049] Figure 6 It is a non-target object obtained through the attention heat map in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many other different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0051] Reference Figure 1 , the embodiment of the present application provides a method for selecting a target based on picture-decoded EEG signals, comprising:

[0052] S1. The user watches the video frames collected by the robot arm camera in real time and looks at the desired target. At the same time, the image segmentation model is used to segment the video frames into multiple objects. The attention heat map is constructed using EEG data and eye movement data. The multiple objects are judged as target objects and non-target objects through the attention heat map. Then, the CLIP (Contrastive Language-Image Pre-training, pre-training large model) picture encoder is used to encode the pictures of all objects into the first embedding vector.

[0053] This embodiment uses Figure 2 This is one of the video frames. The picture contains a tissue and a white bottle. The picture segmented by the image segmentation model is shown in Figure 3 As shown;

[0054] S2, input the EEG signal collected synchronously during the user's gaze on the target into the EEG encoder, and finally obtain the EEG embedding vector;

[0055] S3, respectively calculate the Euclidean distance between the EEG embedding vector and all the first embedding vectors in S1, take the similarity between the EEG embedding vector and the target object as a positive sample, and take the similarity between the EEG embedding vector and the non-target object as a negative sample; and use the positive samples and negative samples to compare and train the EEG encoder to obtain the trained EEG encoder;

[0056] S4. Test the trained EEG encoder to obtain the target object selected by the user in the picture.

[0057] In some embodiments, the S1 specifically includes the following steps:

[0058] S11, performing full-image segmentation on the video frame through an image segmentation model to obtain masks of all target objects and non-target objects in the image scene, and extracting each object in the image through the masks of the target objects and non-target objects;

[0059] S12. Perform sliding window acquisition on the EEG signal and the output signal of the eye tracker with a step length of 200 ms and a window length of 3000 ms, remove the eye saccade component, obtain the fixation eye movement data, count the number of eye movement sampling points in each object area in the fixation eye movement data with a window length of 3000 ms, and obtain the attention heat map; see the attention heat map for details. Figure 4 As shown, Figure 4 The redder the middle is, the more sampling points there are;

[0060] S13, analyzing the gaze heat of each object in the attention heat map, taking the object with the highest gaze heat as the target object, and the objects in other areas are non-target objects;

[0061] In this embodiment, the target object obtained by the attention heat map is shown in FIG. Figure 5 As shown, the obtained non-target objects are shown in Figure 6 As shown;

[0062] S14. Encode all pictures containing all objects into first embedding vectors through a CLIP picture encoder to obtain multiple first embedding vectors.

[0063] In some embodiments, the method of eliminating eye saccade components in S12 to obtain fixation eye movement data is as follows:

[0064] S121, every time a sampling point is obtained, the earliest sampling point is removed from the eye movement data buffer, and the latest sampling point is added to the eye movement data buffer;

[0065] S122, calculating the distance between each sampling point and the previous sampling point in the eye movement data buffer point by point through a speed threshold filtering algorithm; if the distance between a certain sampling point and the previous sampling point meets the standard of fixation eye movement, it is considered that fixation eye movement has occurred, and the sampling point is added to a fixation buffer for analyzing the attention heat map;

[0066] S123, loop S121 to S122, and then obtain gaze eye movement data.

[0067] In some embodiments, the EEG encoder in S2 includes a temporal convolution layer, a spatial convolution layer, an average pooling layer, a self-attention layer and a fully connected layer connected in sequence.

[0068] In some embodiments, S2 specifically includes the following steps:

[0069] S21, the EEG encoder inputs the EEG signal into the temporal convolution layer to extract temporal features; the temporal convolution layer includes w convolution kernels of different time scales, such as (1, 20), (1, 50), etc. Specifically, the temporal convolution layer includes w convolution kernels of dimension (1, d) and step size (1, 1); preferably, w is 40 and d is 25;

[0070] Then, the output results of w convolution kernels are concatenated and integrated to obtain the temporal feature; the dimension of the temporal feature is (w, ch, m t ), where ch represents the number of channels, m t Represents the intermediate value of the time dimension after convolution;

[0071] S22, input the temporal features into the spatial convolution layer; the spatial convolution layer contains w convolution kernels with dimensions (ch, 1) and stride (1, 1). After convolution, the spatiotemporal features are obtained, and the dimensions of the spatiotemporal features are (w, 1, m t); The spatial convolution layer uses a (32, 1) convolution kernel along the EEG channel;

[0072] S23, input the spatiotemporal features of the EEG signal into the average pooling layer; the average pooling layer smoothes the temporal features along the time direction with a size of (1, t) and a step size of (1, s) to obtain smoothed spatiotemporal features, and the dimension of the smoothed spatiotemporal features is (w, 1, m), where m represents the time series dimension;

[0073] S24, inputting the smoothed spatiotemporal features of the EEG signal into the self-attention layer to obtain the global correlation between time points, thereby obtaining the global features; the self-attention layer learns different aspects of the features through a multi-head attention mechanism, and calculates the correlation between different time positions in the features to extract the global features;

[0074] S25. Perform a weighted operation on the global features to obtain an EEG embedding vector of a one-dimensional vector, and then adjust the length of the EEG embedding vector of the one-dimensional vector to be the same as the length of the first embedding vector in S1 through a fully connected layer.

[0075] In some embodiments, the median value m of the time in S21 is t Calculated by the following formula:

[0076] m t =n-d+1

[0077] Among them, n represents the number of sampling points of the original EEG sample; d represents the dimension of the convolution kernel in the temporal convolution layer.

[0078] In some embodiments, the self-attention layer in S24 is expressed by a formula as follows:

[0079]

[0080] Among them, k is the length of a token, query (Query, Q) represents the time node currently being processed, which is used to compare the relevance with other keys in the time series; key (Key, K) represents all nodes in the time series, and calculates the attention score together with the query; value (Value, V) represents the actual feature information at each time point, which will be weighted and summed according to the attention score calculated by the query and key. Q, K and V are all linear transformations of the input spatiotemporal features, and T represents the transpose of the matrix.

[0081] From the above formula, we can know that QK T To calculate the dot product of the spatiotemporal feature and its own transposition, that is, to find the correlation between each vector. Then multiply it with V, that is, use the correlation between each vector (that is, the correlation between each time point and other time points) as a weight multiplied by the vector itself for classification, thereby obtaining the global correlation between time points.

[0082] In some embodiments, the implementation of S3 is as follows:

[0083] First, the Euclidean distances between the EEG embedding vector and all the first embedding vectors in S1 are calculated to evaluate the similarity of the EEG-image pair; the similarity between the EEG embedding vector and the target object is used as a positive sample, and the similarity between the EEG embedding vector and the non-target object is used as a negative sample; the loss function of the positive and negative samples is calculated; and S1 to S3 are looped until the loss function converges, and the weight with the smallest loss is selected as the weight of the EEG encoder to obtain the trained EEG encoder.

[0084] In some embodiments, the loss function is expressed by a formula as follows:

[0085] Loss=max(d(P)-d(N)+margin,0)

[0086] Among them, Loss represents the loss function, d(p) represents the positive sample, and d(N) represents the negative sample. Margin is a positive number, so that even if the distance of the sample to the positive class is smaller than the distance to the negative class, there is still a loss, ensuring that the trained EEG encoder can make the distance from the EEG embedding vector to the first embedding vector of the target image smaller than the distance to the first embedding vector of the non-target image.

[0087] It can be seen from the formula of the loss function that, under the premise of keeping the parameters of the CLIP image encoder unchanged, as the gradient backpropagation reduces the loss function, the EEG encoder will make the Euclidean distance between the EEG embedding vector of the EEG signal and the first embedding vector of the target object smaller and smaller, while the Euclidean distance between the EEG signal and the first embedding vector of the non-target object larger and larger.

[0088] In some embodiments, the S4 specifically includes the following steps:

[0089] S41, using the trained EEG encoder to encode the EEG signal collected in real time to obtain a new EEG embedding vector;

[0090] S42, using a CLIP picture encoder to perform picture encoding on the pictures where all objects segmented from the picture are located, to obtain a plurality of first embedding vectors;

[0091] S42. Calculate similarities between the new EEG embedding vector and the multiple first embedding vectors in S42 respectively, so as to determine the target object selected by the user in the picture.

[0092] The present invention solves the problem of slow target selection by EEG signals by constructing a direct mapping between EEG signals and image objects, and can help users select desired objects directly through EEG signals. Through the present invention, the EEG signals of the user during the period of gazing at the target object can be decoded, and the desired object in the image recorded by the camera can be directly selected to control the robotic arm to grab it. The cognitive load when the user indicates the desired target is reduced, providing the user with a more natural operating experience; it can be used to establish an efficient brain-computer interface system. It can help patients with movement disorders to freely select objects and express their wishes. It has great practical significance.

[0093] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention. In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for selecting a target based on picture-decoded EEG signals, characterized in that: include: S1. Use the image segmentation model to segment the video frame into multiple objects, and use the EEG data and eye movement data to build an attention heat map. Use the attention heat map to determine the multiple objects as target objects and non-target objects, and then use the CLIP picture encoder to encode all the pictures containing the objects into the first embedding vector. S2, input the EEG signal collected synchronously during the user's gaze on the target into the EEG encoder, and finally obtain the EEG embedding vector; S3, calculate the Euclidean distance between the EEG embedding vector and the first embedding vector in S1, take the similarity between the EEG embedding vector and the target object as a positive sample, and take the similarity between the EEG embedding vector and the non-target object as a negative sample; The positive samples and negative samples are used to train the EEG encoder to obtain a trained EEG encoder; S4. Test the trained EEG encoder to obtain the target object selected by the user in the picture.

2. The method according to claim 1, characterized in that The S1 specifically includes the following steps: S11, performing full-image segmentation on the video frame through an image segmentation model to obtain masks of all target objects and non-target objects in the image scene, and extracting each object in the image through the masks of the target objects and non-target objects; S12, performing sliding window acquisition on the EEG signal and the output signal of the eye tracker with a set step length and window length, eliminating the eye saccade component, obtaining fixation eye movement data, and counting the number of eye movement sampling points in each object area in the fixation eye movement data under the set window length to obtain an attention heat map; S13, analyzing the gaze heat of each object in the attention heat map, taking the object with the highest gaze heat as the target object, and the objects in other areas are non-target objects; S14. Encode all pictures containing all objects into first embedding vectors through a CLIP picture encoder to obtain multiple first embedding vectors.

3. The method according to claim 1, characterized in that The method for eliminating the eye saccade component in S12 to obtain the fixation eye movement data is as follows: S121, every time a sampling point is obtained, the earliest sampling point is removed from the eye movement data buffer, and the latest sampling point is added to the eye movement data buffer; S122, calculating the distance between each sampling point and the previous sampling point in the eye movement data buffer point by point through a speed threshold filtering algorithm; If the distance between a sampling point and the previous sampling point meets the criterion of fixation eye movement, it is considered that fixation eye movement has occurred, and the sampling point is added to a fixation buffer for analyzing the attention heat map; S123, loop S121 to S122, and then obtain gaze eye movement data.

4. The method according to claim 1, characterized in that: The EEG encoder in S2 includes a temporal convolution layer, a spatial convolution layer, an average pooling layer, a self-attention layer and a fully connected layer which are connected in sequence.

5. The method according to claim 1, characterized in that The S2 specifically includes the following steps: S21. The EEG encoder inputs the EEG signal into the temporal convolution layer, which contains w convolution kernels with dimensions (1, d) and step length (1, 1). The output results of the w convolution kernels are integrated to obtain the temporal feature. The dimension of the temporal feature is (w, ch, m t ), where ch represents the number of channels, m t Represents the intermediate value of the time dimension after convolution; S22, input the temporal features into the spatial convolution layer; the spatial convolution layer contains w convolution kernels with dimensions (ch, 1) and stride (1, 1). After convolution, the spatiotemporal features are obtained, and the dimensions of the spatiotemporal features are (w, 1, m t ); S23, input the spatiotemporal features of the EEG signal into the average pooling layer; the average pooling layer smoothes the temporal features along the time direction with a size of (1, t) and a step size of (1, s) to obtain smoothed spatiotemporal features, and the dimension of the smoothed spatiotemporal features is (w, 1, m), where m represents the time series dimension; S24, inputting the smoothed spatiotemporal features of the EEG signal into the self-attention layer to obtain the global correlation between the time points, and obtaining the global features; S25, performing a weighted operation on the global features to obtain a one-dimensional EEG embedding vector, and then adjusting the length of the one-dimensional EEG embedding vector to be the same as the length of the first embedding vector in S1 through a fully connected layer.

6. The method according to claim 5, characterized in that The median value m of the time in S21 t Calculated by the following formula: m t =n-d+1 Among them, n represents the number of sampling points of the original EEG signal; d represents the dimension of the convolution kernel in the temporal convolution layer.

7. The method according to claim 5, characterized in that The self-attention layer in S24 is expressed by the formula as follows: Among them, k is the length of a vector; Q represents the time node currently being processed; K represents all nodes in the time series; V represents the actual feature information of each time point, and T represents the transpose of the matrix.

8. The method according to claim 1, characterized in that The implementation of S3 is as follows: First, the Euclidean distances between the EEG embedding vector and all the first embedding vectors in S1 are calculated to evaluate the similarity of the EEG-image pair; the similarity between the EEG embedding vector and the target object is used as a positive sample, and the similarity between the EEG embedding vector and the non-target object is used as a negative sample; the loss function of the positive and negative samples is calculated; and S1 to S3 are looped until the loss function converges, and the weight with the smallest loss is selected as the weight of the EEG encoder to obtain the trained EEG encoder.

9. The method according to claim 8, characterized in that The loss function is expressed by a formula as follows: Loss=max(d(P)-d(N)+margin,0) Among them, Loss represents the loss function, d(p) represents the positive sample, d(N) represents the negative sample, and margin is a positive number.

10. The method according to claim 1, characterized in that The S4 specifically includes the following steps: S41, using the trained EEG encoder to encode the EEG signal collected in real time to obtain a new EEG embedding vector; S42, using a CLIP picture encoder to perform picture encoding on the pictures where all objects segmented from the picture are located, to obtain a plurality of first embedding vectors; S42. Calculate similarities between the new EEG embedding vector and the multiple first embedding vectors in S42 respectively, so as to determine the target object selected by the user in the picture.

Images