An electroencephalogram 6D posture decoding method based on visual guidance coupled learning

Abstract

The application discloses a visual guidance coupling learning-based electroencephalogram 6D posture decoding method. The method first shoots a series of images of the same scene from different angles. Then, the series of images are played to the subject in the order of adjacent shooting angles, and the electroencephalogram of the subject when watching the series of images is collected. Then, the collected electroencephalogram and the series of images are respectively input into the encoder for independent feature extraction, and then input into the coupling module to extract private features of each mode and coupling features between modes. Then, the private features and the coupling features of each mode are spliced and input into the full connection layer to decode the 6D posture under the single mode. After the model training is completed, the electroencephalogram of the subject when watching the series of images is separately input into the encoder and the coupling module, and the 6D posture of the corresponding series of images is decoded by using the electroencephalogram. The method combines the robustness and accuracy of feature extraction enhanced by visual information, and overcomes the limitations of the method.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of electroencephalogram processing, and relates to robot pose positioning based on electroencephalogram signals, in particular to an electroencephalogram signal 6D pose decoding method based on visual guidance coupled learning. BACKGROUND

[0002] Robot positioning technology has a wide range of applications in many fields, including virtual reality, unmanned delivery and autonomous driving. To achieve robust robot positioning, researchers have proposed various methods, such as place recognition, simultaneous localization and mapping (SLAM), and pose positioning. These methods have made significant progress in improving positioning performance. Although visual SLAM technology has made some progress in the field of robot positioning, it often relies on static environments and good lighting conditions, making its robustness in complex dynamic environments with large changes in lighting or changing environmental elements still not as good as that of humans.

[0003] Humans form spatiotemporal portal representations during spatial cognition by activating various neurons, such as spatial view cells, place cells, head direction cells, grid cells, and time cells, to support precise navigation and positioning. The activity of these neurons produces potential changes in the brain, which can be recorded by electroencephalogram (EEG). Researchers use methods such as independent component analysis (ICA) to separate mixed EEG signals and analyze the relationship between different frequency bands and spatial cognition, providing a powerful tool for studying spatial cognition. Some researchers have drawn inspiration from brain mechanisms and proposed models such as RatSLAM and NeuroSLAM to achieve robust positioning by simulating the activity of neural cells. However, to achieve spatial recognition capabilities similar to humans, especially in complex environments, there are still many challenges to overcome.

[0004] Existing technologies attempt to combine EEG signals with robot technology to improve the accuracy and reliability of positioning by leveraging the brain's spatial cognition mechanisms. However, the low spatial resolution and poor signal-to-noise ratio of EEG limit its application in spatial positioning and cognition. Existing research relies on artificially designed stimuli to induce potentials and is usually based on analysis of specific frequency bands or features, lacking flexibility and comprehensiveness. In addition, current analysis methods lack the ability to simultaneously analyze six degrees of freedom (6D) of pose and position, making it difficult to fully capture complex motion information of spatial objects. SUMMARY

[0005] The application provides a visual guidance coupling learning-based electroencephalogram 6D posture decoding method.

[0006] A visual guidance coupling learning-based electroencephalogram 6D posture decoding method, specifically comprising the following steps:

[0007] Step 1: A plurality of images are shot from multiple angles for the same scene, and corresponding camera pose data are labeled. The images are played to the subjects in the order of adjacent shooting angles, brain activities related to spatial cognition are stimulated, and EEG signals of the subjects when observing the image stimuli are collected. The EEG signals reflect the brain activities of the subjects, including neural information related to spatial position and direction perception.

[0008] Step 2: The plurality of images shot in step 1 are input into a visual encoder in the order of playing to extract visual features. The EEG signals of the subjects collected in step 1 are input into an electroencephalogram encoder to extract spatiotemporal features of the EEG signals. The two encoders independently process data of different modalities.

[0009] Preferably, the electroencephalogram encoder is EEGNet.

[0010] Preferably, the visual encoder extracts image features by introducing a self-attention module in a residual network, and then generates the output of the visual encoder through a 4096-dimensional fully connected layer.

[0011] Step 3: The visual features and the spatiotemporal features of the EEG signals extracted in step 2 are input into a coupling module. First, two independent fully connected networks are used to extract private features I p , E p of the visual features and the spatiotemporal features of the EEG signals respectively, and a coupled fully connected network is used to extract coupled features I c , E c of the visual features and the spatiotemporal features of the EEG signals. Then, the visual features and the private features I p and the coupled features I c are spliced to obtain fused visual features I cp , the private features E p of the spatiotemporal features of the EEG signals and the coupled features E c are spliced to obtain fused spatiotemporal features E cp of the EEG signals. Finally, the fused visual features I cpInput two independent fully connected networks to predict the position and direction under visual features; the fused EEG signal spatiotemporal features E cp Input another two independent fully connected networks to predict the position and direction under EEG signal spatiotemporal features.

[0012] Step 4, compare the 6D pose data output by the coupling module in step 3 with the camera pose data labeled in step 1, and train the network parameters of the encoder and the coupling module in steps 2 and 3.

[0013] Step 5, input the EEG signal of the subject when viewing the scene image into the EEG encoder trained in step 4 to extract the spatiotemporal features of the EEG signal. p and the coupling feature E c , and output the predicted 6D pose data through the fully connected layer after splicing.

[0014] The present application has the following beneficial effects:

[0015] 1. By using the visual-guided coupling learning method in the training stage, the model structure is designed as a coupling feature channel and an independent feature channel. Through the interaction of the coupling channel, the visual information is used to guide the learning of the EEG modality feature, thereby exploring the potential correlation between the visual information and the EEG signal, solving the problems of low signal-to-noise ratio and poor spatial resolution in traditional EEG signal processing, more effectively compensating for the inherent noise in the EEG signal, and enhancing the robustness and accuracy of 6D pose decoding.

[0016] 2. The data-driven feature extraction method can adapt to the complex mode of the input EEG signal, enhance the robustness of feature extraction, automatically capture information directly related to spatial cognition, automatically learn spatial cognition features related to 6D pose from EEG signals, avoid the limitations of manual design, and provide higher flexibility and adaptability when facing different scenes or individuals.

[0017] 3. The experimental results show that the present application can simultaneously perform precise six-degree-of-freedom estimation of position and attitude, and can realize accurate 6D pose decoding in different subjects and different indoor scenes, showing strong robustness and improving the stability and accuracy of the system in diversified environments. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 It is a flowchart of the EEG signal 6D pose decoding method based on visual-guided coupling learning.

[0019] Figure 2 It is the visualization result of the predicted trajectory of different sequences in the comparison experiment 1 between EEGNet and the present method.

[0020] Figure 3 Visualization results of the predicted trajectories of different sequences by the method of the present application in the comparative experiment 2.

[0021] Figure 4 Visualization results of the predicted trajectories of different subjects on sequence f1-desk1 in the comparative experiment 3.

[0022] Figure 5 Visualization results of the predicted trajectories of different subjects on sequence f1-desk2 in the comparative experiment 3.

[0023] Figure 6 Visualization results of the predicted trajectories of different sequences by EEGNet in the comparative experiment 4.

[0024] Figure 7 Visualization results of the predicted trajectories of different sequences by the method of the present application in the comparative experiment 4. DETAILED DESCRIPTION

[0025] The application will be further explained in conjunction with the accompanying drawings;

[0026] A 6D pose decoding method of electroencephalogram signal based on visual guidance coupled learning, specifically comprising the following steps:

[0027] Step 1, the present example plays a sequence of scene images from the TUM RGB-D dataset to the subject, while collecting the EEG signals of the subject when observing the image stimulus.

[0028] Step 2, as shown in Figure 1 , the EEG signals of the subject collected in step 1 are input into EEGNet to extract the spatiotemporal features of the EEG signals. The image sequence played in step 1 is input into the visual encoder according to the playing order, the self-attention module is introduced in the residual network to extract and focus on the features in the image that are most critical to pose regression, and then a 4096-dimensional fully connected layer is used to generate the output of the visual encoder.

[0029] Step 3, the visual features extracted in step 2 and the spatiotemporal features of the EEG signals are input into the coupling module for feature fusion, and the 6D pose is output.

[0030] The coupling module first extracts private features I p , E p of the visual features and the spatiotemporal features of the EEG signals through two independent fully connected networks, respectively, and extracts coupled features I c , E cThe coupling features reflect the common information between the two modalities, while the private features embody the information unique to each modality.

[0031] The visual features and the private features I p are then concatenated with the coupling features I c to obtain the fused visual features I cp The private features E p of the EEG signal spatiotemporal features are concatenated with the coupling features E c to obtain the fused EEG signal spatiotemporal features E cp Finally, the fused visual features I cp are input into two independent fully connected networks to predict the position and direction under the visual features; the fused EEG signal spatiotemporal features E cp are input into another two independent fully connected networks to predict the position and direction under the EEG signal spatiotemporal features.

[0032] Step 4: Compare the 6D pose data output by the coupling module in step 3 with the camera pose data labeled in step 1, and train the network parameters of the encoder and the coupling module in steps 2 and 3.

[0033] The features of the EEG and visual modalities are guided by the coupling training to ensure that the visual features can help the EEG signal extract high-quality features related to the 6D pose transformation, thereby effectively overcoming the low signal-to-noise ratio problem of the EEG signal and enhancing the robustness and accuracy of the decoding process.

[0034] Step 5: Input the EEG signal of the subject watching the scene image into the EEG encoder trained in step 4 to extract the spatiotemporal features of the EEG signal. Then use the fully connected layer in the coupling module to extract the private features E p of the EEG signal spatiotemporal features, which are concatenated with the coupling features E c to output the predicted 6D pose data through the fully connected layer.

[0035] The obtained 6D pose information is suitable for various application scenarios, including robot positioning, spatial cognition in virtual reality, and even pose tracking and spatial navigation in dynamic complex environments.

[0036] To verify the effectiveness of the method, a number of comparative experiments were conducted.

[0037] Comparative Experiment 1

[0038] Two image sequences with the same scene and different shooting trajectories were selected from the TUM RGB-D dataset, denoted as freiburg1-desk1 and freiburg1-desk2. These two image sequences were played to the same subject in order according to the adjacent shooting angles, and the EEG signals were collected.

[0039] The prediction results of 6D pose decoding using the model trained by EEGNet and the proposed method respectively are shown in FIG. 6, where the black line is the real trajectory corresponding to the sequence f1-desk1 and f1-desk2, the red line is the prediction result, and the dashed line represents the error. The error comparison results are shown in Table 1: Figure 2

[0040] Table 1

[0041]

[0042] It can be seen that, compared with the method of directly decoding 6D pose from EEG signal using EEGNet, the position error is reduced from 0.65 m to 0.053 m and the rotation error is reduced from 29.01° to 1.89° after the visual guidance combined with image features, which shows that the proposed method can realize high-quality 6D pose decoding.

[0043] Comparative Experiment 2

[0044] On the basis of Comparative Experiment 1, several images were randomly sampled from the sequence f1-desk1 and were recorded as sequences random1-random5 after being randomly shuffled. The five image sequences were played to the same subject in turn, and EEG signals were collected.

[0045] The prediction results of different methods are shown in FIG. 7, and the error comparison results are shown in Table 2: Figure 3 Table 2

[0046]

[0047]

[0048] It can be seen that when the images are presented in a random order, the subject will see discontinuous and disordered images, which makes it difficult for the brain to effectively integrate and analyze the original information. Compared with playing the image sequence in the order of adjacent angles in Comparative Experiment 1, the prediction result has a significant decline, but it is still better than the method of directly decoding 6D pose from EEG signal using EEGNet.

[0049] Comparative Experiment 3

[0050] In order to evaluate the performance of the proposed method in different subjects, five college students aged between 20 and 22 years old were invited as subjects Subject1-Subject5. The sequences f1-desk1 and f1-desk2 were played to all subjects in turn, and the prediction results are shown in FIG. 8, and the error comparison results are shown in Table 3: Figure 4 Figure 5

[0051] ​​​​Table 3

[0052]

[0053] It can be seen that the 6D poses decoded based on the EEG signals of different subjects all show high accuracy, indicating that the method has good stability and can complete the learning tasks of different subjects in related experiments.

[0054] Comparative experiment 4

[0055] In order to study the influence of different scenes on the performance of the method, different image sequences were shot with different objects in the indoor office environment as the main body, denoted as f1-desk, f2-desk, f3-office, f4-plants, f5-teddy, f6-floor, and the five sequences were played to the same subject in turn according to the shooting angle. The prediction results are shown in Figure 6 , Figure 7 The error comparison results are shown in Table 4:

[0056] Table 4

[0057]

[0058] It can be seen that the prediction results of the method in different scenes are significantly better than those of EEGNet, and the method performs stably, proving that the effectiveness of the method does not depend on a specific environment. Even in the plant scene f4-plants, which poses a typical challenge to pose estimation methods with high angle changes, the method can still give accurate decoding results.

Claims

1. A method for decoding 6D posture of EEG signals based on visually guided coupled learning, characterized in that: Specifically, the following steps are included: Step 1: Play multiple images of the same scene to the subject in an adjacent order of shooting angles, and label the corresponding camera pose data; play the images to the subject and collect the EEG signals when the subject observes the image stimuli. Step 2: Input the multiple images captured in Step 1 into the visual encoder in the playback order to extract visual features; input the subject's EEG signals collected in Step 1 into the EEG encoder to extract the spatiotemporal features of the EEG signals; process the data of different modalities independently through the two encoders. Step 3: Input the multimodal feature data extracted in Step 2 into the coupling module. Extract the private features of the single modality through two independent fully connected layers, and extract the coupling features between the multimodal features through the coupled fully connected layer. Then, concatenate the private features of the single modality with the coupling features to predict the 6D pose data of the single modality. The coupling module first extracts private features I from visual features and EEG signal spatiotemporal features through two independent fully connected networks, respectively. p E p And coupled features I, which extract visual features and spatiotemporal features of EEG signals using a coupled fully connected network. c E c Then, the private feature I of the visual features. p Coupling features with visual features I c By splicing, the merged visual features I are obtained. cp The private feature E of the spatiotemporal characteristics of EEG signal p Coupling characteristics E with the spatiotemporal characteristics of EEG signals c By splicing, the spatiotemporal characteristics E of the fused EEG signal are obtained. cp Finally, the fused visual features I cp Input two independent fully connected networks to predict position and orientation based on visual features; then input the spatiotemporal features of the fused EEG signal. cp Input two other independent fully connected networks to predict the position and orientation of EEG signals under spatiotemporal characteristics; Step 4: Compare the 6D pose data output by the coupling module in Step 3 with the camera pose data labeled in Step 1, and train the network parameters of the encoder and coupling module in Steps 2 and 3. Step 5: Input the EEG signal of the subject when viewing the scene image into the EEG encoder trained in Step 4 to extract the spatiotemporal features of the EEG signal; then use the fully connected layer in the coupling module to extract the private features and coupling features of the spatiotemporal features of the EEG signal, and output the predicted 6D pose data through the fully connected layer after splicing.

2. The 6D pose decoding method for EEG signals based on visually guided coupled learning as described in claim 1, characterized in that: The EEG encoder is EEGNet.

3. The 6D pose decoding method for EEG signals based on visually guided coupled learning as described in claim 1, characterized in that: The visual encoder extracts image features by introducing a self-attention module into the residual network, and then generates the output of the visual encoder through a fully connected layer.

Images