Electroencephalogram 6D attitude decoding method based on visual guidance coupling learning

Abstract

The invention discloses an electroencephalogram 6D attitude decoding method based on visual guidance coupling learning. According to the method, a series of images of the same scene are shot from different angles; and then playing a series of images to the testee according to a sequence that shooting angles are adjacent, and simultaneously collecting electroencephalogram signals when the testee watches the series of images. Then the collected electroencephalogram signals and the series of images are input into an encoder for independent feature extraction and then input into a coupling module, and private features of all modes and coupling features between the modes are extracted; and then splicing the private features and the coupling features of each mode, inputting the spliced features into a full connection layer, and decoding a 6D attitude in a single mode. After model training is completed, electroencephalogram signals generated when a testee watches a series of images are independently input into the encoder and the coupling module, and 6D postures corresponding to the series of images are decoded through the electroencephalogram signals. According to the method, the robustness and accuracy of feature extraction are enhanced by combining visual information, and the limitation of the method is overcome.

Description

Technical Field

[0001] The present invention belongs to the technical field of electroencephalogram (EEG) processing, and relates to robot posture positioning based on EEG signals, and specifically to an EEG signal 6D posture decoding method based on vision-guided coupled learning. Background Art

[0002] Robot localization technology has been widely used in many fields, including virtual reality, drone delivery, and autonomous driving. To achieve robust robot localization, researchers have proposed a variety of methods, such as place recognition, simultaneous localization and mapping (SLAM), and posture localization. These methods have made significant progress in improving localization performance. Although visual SLAM technology has made some progress in the field of robot localization, it often relies on static environments and good lighting conditions, making it less robust than humans in complex dynamic environments with large lighting changes or constantly changing environmental elements.

[0003] In the process of spatial cognition, humans activate multiple neurons (such as spatial view cells, place cells, head direction cells, grid cells, and time cells) to form a representation of space-time portals, thereby supporting accurate navigation and positioning. The activity of these neurons will produce potential changes in the brain and can be recorded by electroencephalogram (EEG). Researchers use methods such as independent component analysis (ICA) to separate mixed EEG signals and analyze the relationship between different frequency bands and spatial cognition, providing a powerful tool for studying spatial cognition. Some researchers have drawn inspiration from brain mechanisms and proposed models such as RatSLAM and NeuroSLAM, which achieve robust positioning by simulating the activity of nerve cells. However, there are still many challenges to achieving human-like spatial recognition capabilities, especially in complex environments.

[0004] Existing technologies attempt to combine EEG signals with robotics, leveraging the brain's spatial cognition mechanism to improve positioning accuracy and reliability. However, EEG has low spatial resolution and poor signal-to-noise ratio, which limits its application in spatial positioning and cognition. Existing studies mostly rely on artificially designed stimuli to induce potentials, and are usually based on analysis of specific frequency bands or features, lacking flexibility and comprehensiveness. In addition, most current analysis methods lack the ability to perform comprehensive six-degree-of-freedom (6D) analysis of posture and position at the same time, and are unable to fully capture the complex motion information of spatial objects. Summary of the invention

[0005] The present invention proposes a 6D posture decoding method for EEG signals based on vision-guided coupled learning. By combining EEG signals with visual information, features related to spatial cognition are directly learned from EEG signals in a data-driven manner, and the robustness and accuracy of feature extraction are enhanced by combining visual information. The individual's spatial imagination posture is reconstructed through EEG signals, and 6D posture is decoded, including 3D position and 3D direction, which overcomes the limitations of existing EEG signal analysis.

[0006] A 6D posture decoding method of EEG signal based on visual guided coupled learning specifically comprises the following steps: Step 1: Take multiple images of the same scene from multiple angles and annotate the corresponding camera pose data. Play the images to the subject in the order of adjacent shooting angles to stimulate brain activity related to spatial cognition, and collect EEG signals when the subject observes the image stimulation. The EEG signals reflect the brain activity of the subject, including neural information related to spatial position and direction perception.

[0007] Step 2: Input the multiple images captured in step 1 into the visual encoder in the order of playback to extract visual features. Input the EEG signal of the subject collected in step 1 into the EEG encoder to extract the spatiotemporal features of the EEG signal. The two encoders independently process data of different modalities.

[0008] Preferably, the EEG encoder is EEGNet.

[0009] Preferably, the visual encoder extracts image features by introducing a self-attention module in the residual network, and then generates the output of the visual encoder through a 4096-dimensional fully connected layer.

[0010] Step 3: Input the visual features extracted in step 2 and the spatiotemporal features of the EEG signal into the coupling module. First, two independent fully connected networks are used to extract the private features I of the visual features and the spatiotemporal features of the EEG signal. p 、E p , and the coupled fully connected network extracts the coupled features of visual features and EEG signal spatiotemporal features I c 、E c Then the visual features and private features I p With coupling feature I c Splicing to obtain the fused visual features I cp , the private feature E of the EEG signal spatiotemporal features p With coupling feature E c Splicing to obtain the fused EEG signal spatiotemporal features E cp Finally, the fused visual features I cpInput two independent fully connected networks to predict the position and direction of visual features; the fused EEG signal spatiotemporal features E cp The input is then fed into two other independent fully connected networks to predict the position and direction of the EEG signal based on its spatiotemporal features.

[0011] Step 4: Compare the 6D posture data output by the coupling module in step 3 with the camera posture data marked in step 1, and train the network parameters of the encoder and coupling module in steps 2 and 3.

[0012] Step 5: Input the EEG signal when the subject is watching the scene image into the EEG encoder trained in step 4 to extract the spatiotemporal features of the EEG signal. Then use the fully connected layer in the coupling module to extract the private feature E of the spatiotemporal features of the EEG signal. p With coupling feature E c , after splicing, the predicted 6D pose data is output through the fully connected layer.

[0013] The present invention has the following beneficial effects: 1. By adopting a visually guided coupled learning method in the training phase, the model structure is designed as coupled feature channels and independent feature channels. Through the interaction of coupled channels, visual information is used to guide the learning of EEG modal features, thereby exploring the potential correlation between visual information and EEG signals, solving the problems of low signal-to-noise ratio and poor spatial resolution in traditional EEG signal processing, more effectively compensating for the inherent noise in EEG signals, and enhancing the robustness and accuracy of 6D posture decoding.

[0014] 2. The data-driven feature extraction method can adapt to the complex patterns of the input EEG signals, enhance the robustness of feature extraction, automatically capture information directly related to spatial cognition, and automatically learn spatial cognitive features related to 6D posture based on EEG signals, avoiding the limitations of manual design and providing higher flexibility and adaptability when facing different scenarios or individuals.

[0015] 3. Experimental results show that the present invention can simultaneously perform accurate six-degree-of-freedom estimation of position and attitude, and can achieve accurate 6D attitude decoding under different subjects and different indoor scenes, showing strong robustness, which significantly improves the stability and accuracy of the system in diverse environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Flowchart of the 6D posture decoding method of EEG signals based on vision-guided coupled learning.

[0017] Figure 2 To compare the prediction trajectory visualization results of EEGNet and this method for different sequences in Experiment 1.

[0018] Figure 3 To compare the prediction trajectory visualization results of EEGNet and this method for different sequences in Experiment 2.

[0019] Figure 4 To compare the prediction trajectory visualization results of different subjects on the sequence f1-desk1 in Experiment 3.

[0020] Figure 5 To compare the prediction trajectory visualization results of different subjects on the sequence f1-desk2 in Experiment 3.

[0021] Figure 6 This is the visualization result of EEGNet’s prediction trajectory for different sequences in Experiment 4.

[0022] Figure 7 This is the visualization result of the prediction trajectory of different sequences in comparison experiment 4. DETAILED DESCRIPTION

[0023] The present invention will be further explained below with reference to the accompanying drawings; A 6D posture decoding method of EEG signal based on visual guided coupled learning specifically comprises the following steps: Step 1: In this example, a scene image sequence from the TUM RGB-D dataset is played to the subject, and the EEG signal of the subject when observing the image stimulus is collected.

[0024] Step 2: Figure 1 As shown in the figure, the EEG signal of the subject collected in step 1 is input into EEGNet to extract the spatiotemporal features of the EEG signal. The image sequence played in step 1 is input into the visual encoder in the order of playback. By introducing the self-attention module in the residual network, the most critical features in the image for posture regression are extracted and focused, and then the output of the visual encoder is generated through a 4096-dimensional fully connected layer.

[0025] Step 3: Input the visual features extracted in step 2 and the spatiotemporal features of the EEG signal into the coupling module for feature fusion and output of the 6D posture.

[0026] The coupling module first extracts the private features I of visual features and EEG signal spatiotemporal features through two independent fully connected networks. p 、E p , and the coupled fully connected network extracts the coupled features of visual features and EEG signal spatiotemporal features I c 、E c The coupled features reflect the common information between the two modalities, while the private features reflect the information unique to each modality.

[0027] Then the visual features and private features I p With coupling feature I c Splicing to obtain the fused visual features I cp , the private feature E of the EEG signal spatiotemporal features p With coupling feature E c Splicing to obtain the fused EEG signal spatiotemporal features E cp Finally, the fused visual features I cp Input two independent fully connected networks to predict the position and direction of visual features; the fused EEG signal spatiotemporal features E cp The input is then fed into two other independent fully connected networks to predict the position and direction of the EEG signal based on its spatiotemporal features.

[0028] Step 4: Compare the 6D posture data output by the coupling module in step 3 with the camera posture data marked in step 1, and train the network parameters of the encoder and coupling module in steps 2 and 3.

[0029] The features of EEG and visual modalities guide each other through coupled training to ensure that visual features can help EEG signals extract high-quality features related to 6D posture transformation, thereby effectively overcoming the low signal-to-noise ratio problem of EEG signals and enhancing the robustness and accuracy of the decoding process.

[0030] Step 5: Input the EEG signal when the subject is watching the scene image into the EEG encoder trained in step 4 to extract the spatiotemporal features of the EEG signal. Then use the fully connected layer in the coupling module to extract the private feature E of the spatiotemporal features of the EEG signal. p With coupling feature E c , after splicing, the predicted 6D pose data is output through the fully connected layer.

[0031] The obtained 6D posture information is suitable for a variety of application scenarios, including robot positioning, spatial cognition in virtual reality, and even posture tracking and spatial navigation in dynamic and complex environments.

[0032] In order to verify the effectiveness of this method, several comparative experiments were carried out.

[0033] Comparative experiment 1 Two image sequences of the same scene but different shooting trajectories were selected from the TUM RGB-D dataset, denoted as freiburg1-desk1 and freiburg1-desk2. These two image sequences were played to the same subject in the order of adjacent shooting angles, and EEG signals were collected.

[0034] The 6D posture decoding is performed using EEGNet and the model trained by this method. The prediction results are as follows: Figure 2As shown in Figure 1, the black line is the actual shooting trajectory corresponding to the sequences f1-desk1 and f1-desk2, the red line is the predicted result, and the dotted line represents the corresponding error. The error comparison results are shown in Table 1:

[0035] Table 1

[0036] It can be seen that compared with the method of directly decoding 6D posture from EEG signals using only EEGNet, after combining the visual guidance of image features, the position error of this method is reduced from 0.65m to 0.053m, and the rotation error is reduced from 29.01° to 1.89°, indicating that this method can achieve high-quality 6D posture decoding.

[0037] Comparative experiment 2 Based on the comparative experiment 1, several images were randomly sampled from the sequence f1-desk1, randomly shuffled and recorded as the sequence random1~random5. These five image sequences were played to the same subject in turn, and EEG signals were collected.

[0038] The prediction results of different methods are as follows Figure 3 The error comparison results are shown in Table 2: Table 2

[0039] It can be seen that when the images are presented in a random order, the subjects will see discontinuous and disordered images, making it difficult for the brain to effectively integrate and analyze the original information. Compared with the image sequence played in the order of adjacent angles in the comparative experiment 1, the prediction results have dropped significantly, but are still better than the method of directly decoding 6D pose from EEG signals using only EEGNet.

[0040] Comparative experiment 3 In order to evaluate the performance of this method on different subjects, five college students aged between 20 and 22 were invited as subjects Subject 1 to Subject 5. Sequences f1-desk1 and f1-desk2 were played to all subjects in turn. The prediction results are shown in the figure below. Figure 4 , Figure 5 The error comparison results are shown in Table 3: Table 3

[0041] It can be seen that the 6D poses decoded from the EEG signals of different subjects all show high accuracy, indicating that this method has good stability and can complete the learning tasks of different subjects in related experiments.

[0042] Comparative experiment 4 In order to study the impact of different scenes on the performance of the method, different objects in the indoor office environment were used as the main subjects to shoot different image sequences, denoted as f1-desk, f2-desk, f3-office, f4-plants, f5-teddy, and f6-floor. These five sequences were played to the same subject in the order of adjacent shooting angles. The prediction results are shown in Figure 6 , Figure 7 The error comparison results are shown in Table 4:

[0043] Table 4

[0044] It can be seen that the prediction results of this method in different scenarios are significantly better than EEGNet, and the performance is stable, proving that the effectiveness of this method does not depend on a specific environment. Even in the plant scene f4-plants, facing the typical challenges posed by high-angle changes to posture estimation methods, this method can still give accurate decoding results.

Claims

1. A 6D posture decoding method for EEG signals based on visually guided coupled learning, characterized in that: The specific steps include: Step 1: Take multiple images of the same scene from multiple angles and annotate the corresponding camera pose data; play the images to the subjects and collect the EEG signals when the subjects observe the image stimulation; Step 2: Input the multiple images captured in step 1 into the visual encoder in the order of playback to extract visual features; input the EEG signal of the subject collected in step 1 into the EEG encoder to extract the spatiotemporal features of the EEG signal; and independently process data of different modalities through two encoders; Step 3: Input the multimodal feature data extracted in step 2 into the coupling module, extract the private features under the single mode through two independent fully connected layers, and extract the coupling features between the multimodal models through the coupled fully connected layer; then splice the private features of the single mode with the coupling features to predict the 6D posture data under the single mode; Step 4: Compare the 6D posture data output by the coupling module in step 3 with the camera posture data marked in step 1, and train the network parameters of the encoder and coupling module in steps 2 and 3; Step 5: Input the EEG signal when the subject is viewing the scene image into the EEG encoder trained in step 4 to extract the spatiotemporal features of the EEG signal; then use the fully connected layer in the coupling module to extract the private features and coupling features of the spatiotemporal features of the EEG signal, and output the predicted 6D posture data through the fully connected layer after splicing.

2. The method for 6D posture decoding of EEG signals based on visually guided coupled learning as claimed in claim 1, characterized in that: Multiple images of the same scene were shown to the subjects in an order adjacent to the shooting angle.

3. A method for 6D posture decoding of EEG signals based on visually guided coupled learning as claimed in claim 1 or 2, characterized in that: The EEG encoder is EEGNet.

4. A method for 6D posture decoding of EEG signals based on visually guided coupled learning as claimed in claim 1 or 2, characterized in that: The visual encoder extracts image features by introducing a self-attention module in the residual network, and then generates the output of the visual encoder through a fully connected layer.

5. A method for 6D posture decoding of EEG signals based on visually guided coupled learning as claimed in claim 1 or 2, characterized in that: The coupling module first extracts the private features I of visual features and EEG signal spatiotemporal features through two independent fully connected networks. p 、E p , and the coupled fully connected network extracts the coupled features of visual features and EEG signal spatiotemporal features I c 、E c ; Then the visual features and private features I p With coupling feature I c Splicing to obtain the fused visual features I cp , the private feature E of the EEG signal spatiotemporal features p With coupling feature E c Splicing to obtain the fused EEG signal spatiotemporal features E cp ; Finally, the fused visual features I cp Input two independent fully connected networks to predict the position and direction of visual features; the fused EEG signal spatiotemporal features E cp The input is then fed into two other independent fully connected networks to predict the position and direction of the EEG signal based on its spatiotemporal features.

Images