Image reconstruction method and system based on electroencephalogram and eye movement tracking

Abstract

The invention discloses an image reconstruction method and system based on an electroencephalogram and eye movement tracking. The method comprises the following steps: (1) collecting and preprocessing electroencephalogram signals and eye movement data; (2) scene-level semantic feature extraction based on a 3D-CTNet model; (3) target-level semantic feature extraction and visual attention weight distribution based on a YOLOv9 model; and (4) reconstructing the image of the multi-semantic driven diffusion model. The invention also provides an image reconstruction system based on electroencephalogram and eye movement tracking. The system sequentially comprises the following modules: (1) an electroencephalogram semantic feature extraction module; (2) a visual semantic feature extraction module; and (3) an image reconstruction module. According to the method, the problem of single semantics in traditional electroencephalogram image reconstruction is effectively solved, the semantic richness and structural similarity of the reconstructed image are improved, and the image reconstruction quality is improved.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction, brain-computer interface, and artificial intelligence generation, and specifically to an image reconstruction method and system based on electroencephalography and eye tracking. Background Technology

[0002] EEG-based image reconstruction plays a crucial role in bridging the gap between brain activity and perceptual experience, offering significant benefits across various fields. For individuals with communication disorders, such as locked-in syndrome, this technology can provide a new way to express their thoughts and visual experiences. It also has potential applications in artistic creation, allowing users to generate images directly from their brain activity, enabling new forms of artistic creation and interaction without traditional input methods. Furthermore, EEG-based image reconstruction can help understand cognitive processes associated with visual perception, contributing to advancements in neuroscience, mental health, and therapeutic interventions.

[0003] With the rapid development of EEG signal technology, brain-computer interfaces (BCIs) are now able to record visually evoked human brain activity and provide further analysis. A key research focus is on recording this complex brain signal data and reconstructing the stimuli used to elicit responses in participants during experiments, in order to understand and simulate human visual perception. One mainstream approach to simulating human visual perception is to extract effective features and reconstruct the seen content using computational methods such as deep neural networks, for example, images or videos used to elicit subjective responses in stimulus experiments. However, existing EEG-based BCIs typically only generate semantically simple images and struggle to reconstruct scenes containing complex semantic information. This necessitates the introduction of multimodal data to improve the quality of image reconstruction. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide an image reconstruction method and system based on electroencephalography and eye tracking.

[0005] This paper proposes a hybrid brain-computer interface based on EEG and eye-tracking for multi-semantic image reconstruction, which consists of three stages. In the first stage, we preprocess the EEG data and use the proposed 3D-CTNet to extract coarse-grained scene-level semantic features. In the second stage, we utilize the object detection model Yolov9 to identify object-level semantic features in the image to supplement the fine-grained semantics. Simultaneously, we use eye-tracking data to obtain the visual attention distribution of participants and then assign weights to the object-level semantic features in the scene. Finally, we fuse the semantic features to obtain a multi-semantic representation for image reconstruction and use a multi-semantic-driven diffusion model for image reconstruction. The inclusion of eye-tracking data compensates for the low spatial resolution of EEG data, making it possible to reconstruct more complex multi-semantic scenes using EEG data. Furthermore, the hybrid brain-computer interface based on EEG and eye-tracking utilizes visual attention mechanisms to improve image reconstruction quality, resulting in images that better conform to human visual perception mechanisms and significantly enhancing the overall image reconstruction quality.

[0006] This invention provides an image reconstruction method based on electroencephalography (EEG) and eye tracking, which mainly includes the following steps: (1) Acquisition and preprocessing of EEG signals and eye movement data; EEG signals and eye movement data of users are acquired through EEG acquisition equipment and eye movement tracking equipment. EEG signals are filtered and independent component analysis is used to remove noise and artifacts, and sliding window is used for data enhancement. (2) Extract scene-level semantic features; use the 3D-CTNet model to extract scene-level semantic features from the preprocessed EEG signals. The 3D-CTNet includes: a multi-core 3D convolutional network module for extracting spatial features of EEG signals; a Transformer module for temporal feature encoding to extract temporal features of EEG signals through multi-head attention mechanism; and a parallel structure of fusion of 3D convolutional network and Transformer to ensure maximum preservation and optimized extraction of spatiotemporal features. (3) Extract object-level semantic features; use the YOLOv9 object detection model to identify objects in the user's viewed image and extract object-level semantic features. The model adopts a multi-scale feature fusion strategy to enhance the ability to identify small objects; calculate visual attention weights based on eye-tracking data, considering the number of fixation points, fixation duration, and saccade path length. The calculation formula is as follows: (4) Image reconstruction using semantic fusion and multi-semantic driven diffusion model; scene-level semantic features and target-level semantic features are fused together and combined with visual attention weights to generate multi-semantic representations; using a pre-trained Stable Diffusion diffusion model, the multi-semantic representations are transformed into text prompts for image reconstruction, and the weights of different semantic features during the generation process are controlled through a cross-attention mechanism.

[0007] Specifically, step (1) includes: (11) The user wears an EEG cap and applies conductive gel to reduce electrode impedance and improve signal stability; (12) At the start of each run, a fixed crosshair appears on the black screen, accompanied by a brief sound prompt. The user looks at the crosshair on the screen and prepares to begin visualizing; (13) Two seconds later, an image appears on the screen and lasts for 4 seconds, prompting the participant to imagine the corresponding scene. Scene images appear on the screen, the user imagines the corresponding scene image, and the corresponding EEG signals are collected; (14) The user’s eye movement data is collected synchronously through the Tobii Pro Fusion eye tracker to obtain eye movement information such as the user’s fixation point, fixation time, and fixation trajectory; (15) The collected EEG signals are filtered. First, a 50Hz notch filter is used to eliminate power frequency interference, and then a 0.1Hz high-pass filter and a 50Hz low-pass filter are used to extract the EEG frequency bands that are highly related to the semantic features of the scene. (16) Independent Component Analysis (ICA) was applied to eliminate noise caused by eye, head, and muscle movements. ICA uses statistical methods to decompose the mixed signal into independent signal sources and combines expert experience to identify and remove significant noise components to improve the signal-to-noise ratio. (17) Use a sliding window to increase the sample size and improve training quality. A sliding window was designed to slide the data.

[0008] Preferably, a 3.5-second sliding window was designed to slide data for 2-6 seconds, with the window sliding at 0.1-second intervals, increasing the number of experimental samples by five times.

[0009] Step (2) specifically includes: (21) The preprocessed EEG data was converted into a 2D matrix, and the EEG data from different channels were stacked and combined to create 3D EEG data. Spatial features were extracted from the EEG data using multiple 3D convolutional kernels of different sizes to obtain EEG data I; (22) Use a Transformer with attention mechanism to directly process the preprocessed EEG data to extract temporal features and obtain EEG data II; (23) The two sets of EEG data are expanded into one dimension and merged through a connection layer. The merged EEG data is then classified using softmax to obtain scene labels for scene image reconstruction.

[0010] Step (3) specifically includes: (31) Object-level semantic features and their corresponding quantity information in the image are obtained using the YOLOv9 object detection model. When using the quantity corresponding to the object-level semantic features, we do not uniformly use the actual quantity, but control the expression strategy of different object-level semantic feature quantities: when a semantic feature appears less than 6 times, we use the actual quantity, while when it appears 6 times or more, we use "many" to indicate the degree; (32) Calculate the attention weights corresponding to visual semantic information and the fixation point. G More, longer viewing time T The longer the scanning path L The longer the length, the larger the proportion of the semantic object in the image, and the higher the user's attention to that semantic object. The formula is as follows: (1) in, n Indicates the number of semantics detected. α , β , γ These represent the fixation point, fixation time, and saccade length in the calculation of attention weights, respectively. c i Weighting of time.

[0011] (33) Add visual attention to the detected visual semantic information by using attention weights. c i This scales the cross-attention map assigned to each visual semantic, making the effect stronger or weaker. The formula is as follows: (2) in M t It is the original cross-attention map. M t ) i,j Indicating time steps t How can the pixel at position i in the image focus on the token in the text? j .

[0012] Step (4) specifically includes: (41) In the single-person image reconstruction method, the pre-trained diffusion model Stable Diffusion is used as the image generator. The scene-level semantic features obtained by the EEG semantic feature extractor and the object-level semantic features obtained by the visual semantic feature extractor and their corresponding attention weights are combined to form complete image semantic features and handed over to Stable Diffusion for image generation. (42) In the multi-user image reconstruction method, an independent classification combined with consistency judgment is used for EEG scene classification. If the classification results of several users are consistent, the category is directly used as the final scene category. If the classification results of several users are different, the two categories with the most classification results are used as the input features for image reconstruction to preserve the scene cognitive information of different users. The quantitative information and attention weight under visual semantic features are weighted averaged.

[0013] This invention also provides an image reconstruction system based on electroencephalography and eye tracking, comprising the following modules in sequence: (1) The EEG semantic feature extraction module is used to extract EEG signal features for scene classification. First, signal acquisition and preprocessing are performed, including filtering, artifact removal, and data augmentation to improve signal quality. Then, a pre-trained 3D-CTNet is used for feature extraction, followed by spatial feature extraction using a 3D CNN, and time-series modeling is performed using a Transformer encoder. Finally, scene category prediction is performed to generate EEG semantic features, which are then passed to the image reconstruction module. Furthermore, in multi-user mode, a multi-user EEG semantic fusion strategy is adopted to enhance the semantic understanding of the scene. (2) The visual semantic feature extraction module is used to extract object-level semantic features, corresponding quantitative information, and attention weights corresponding to eye-tracking data. By judging the user's gaze point and combining it with the YOLOv9 object detection model, object-level semantic features in the scene are extracted. Furthermore, by using eye-tracking data such as gaze time and gaze point distribution, we can calculate the attention distribution of each user. Through attention assignment in single-user mode and attention weighting assignment in multi-user mode, we can better restore object-level semantic information in the image, making the reconstructed image more consistent with human visual perception and improving the image reconstruction quality. (3) The image reconstruction module integrates EEG semantic features and visual semantic features to generate the final reconstructed image. In single-user mode, users can view the generated image and provide subjective ratings. In multi-user mode, collaborative optimization by multiple users is supported, adjusting the generated image by fusing feature data from different users. This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the multi-factor user authentication method based on eye tracking and gesture interaction in the metaverse environment of this invention.

[0014] The present invention also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of the present invention.

[0015] The image reconstruction method based on electroencephalography (EEG) and eye tracking of this invention utilizes the YOLOv9 object detection model to identify object-level semantic features in images, supplementing the fine-grained semantics of the images. Simultaneously, it employs a visual attention mechanism to improve image reconstruction quality, and performs image reconstruction through a multi-semantic-driven diffusion model.

[0016] Furthermore, the developed image reconstruction-based hybrid brain-computer interface system boasts excellent operational convenience and visualization capabilities, enabling intuitive expression and accurate reconstruction of user cognitive content. This system provides users with a novel way to express thoughts and visual experiences, representing an innovative exploration and practice of brain-computer interfaces in the field of image reconstruction.

[0017] The working principle of this invention is: This system fuses electroencephalogram (EEG) signals with eye-tracking data to drive a diffusion model called StableDiffusion to generate complex, multi-semantic images. The system's workflow consists of three main stages: 1. Semantic Feature Extraction: First, the system collects EEG signals generated when the user views an image. Using a novel, parallel 3D convolutional neural network and Transformer (3D-CTNet) model, coarse-grained, scene-level semantic features (e.g., "street") are extracted from the EEG signals. Simultaneously, the system uses an eye-tracking device to capture the user's visual attention data (fixation point, fixation duration, etc.). This is combined with an object detection model (YOLOv9) to identify specific objects in the image the user is viewing, supplementing with fine-grained, object-level semantic features (e.g., "cars") and quantity. 2. Multi-Semantic Fusion and Cue Generation: The system fuses the "scene semantics" from the EEG with the "object semantics" from object detection to form a richly detailed "multi-semantic cue" (e.g., "a photo of a 'street' with 'multiple cars'"). Regarding the number of objects, the system employs a specific strategy: when the number is less than 6, it uses an exact number; when the number is greater than or equal to 6, it uses the word "many," which is then further adjusted through an attention mechanism. 3. Attention-Controlled Image Generation: The fused "multi-semantic cue words" are input into the diffusion model, Stable Diffusion. Crucially, during the model's denoising and generation process, the system utilizes the eye-tracking visual attention weights obtained in the first stage (calculated based on the number of gazes, duration, and saccade path) to control and adjust the cross-attention map of the diffusion model. If the user gazes at an object (such as a "car") for a longer time, the attention weight corresponding to that object will be higher, thus appearing more prominently in the generated image, making the final reconstructed image more consistent with the user's actual visual perception.

[0018] The innovation of this invention is: (1) Pioneering hybrid BCI architecture: This invention is the first hybrid brain-computer interface system that combines EEG and eye-tracking data and applies them to image reconstruction tasks.

[0019] (2) Multi-granularity semantic fusion strategy: This invention innovatively uses EEG to extract coarse-grained scene semantics, while using eye tracking and object detection to supplement the fine-grained object-level semantics missing by EEG. This "coarse-fine combination" approach successfully solves the problem that when images are reconstructed solely by EEG, the content is monotonous and can only produce a single semantic.

[0020] (3) Novel 3D-CTNet Model: For scene semantic extraction from EEG, this invention designs a parallel 3DCNN and Transformer (3D-CTNet) deep learning model. Its innovations are: 1. The CNN and Transformer modules work in parallel, extracting spatial and temporal features from the original data respectively, preserving the semantic features in the original EEG data to the maximum extent, which is superior to the traditional serial structure. 2. The use of 3D convolutional kernels effectively extracts spatial features between different EEG channels. 3. The use of various convolutional kernels of different sizes to capture features of different granularities.

[0021] (4) Visual attention-based generation control: Instead of using eye-tracking data as supplementary information, it quantifies it into attention weights and uses these weights to actively adjust the cross-attention mechanism of the Stable Diffusion model. This allows the image generation process to be guided by the user's visual focus, achieving precise control over the degree of expression of the reconstructed content (especially high-frequency objects).

[0022] (5) Optimization strategy for the quantitative representation of the diffusion model: This invention found that the diffusion model does not understand the specific numbers in the prompt words well. Therefore, it innovatively proposes a combination strategy of "exact number (<6) + descriptive word (>=6, such as 'many')" and uses the visual attention weight (innovation point 4) to control the actual degree of representation of "many", which cleverly avoids the defects of the model.

[0023] The advantages of this invention are: (1) High accuracy of EEG semantic extraction: The 3D-CTNet model used to extract scene semantics achieved a classification accuracy of 75.13% and an F1 score of 0.734 on a six-class scene dataset. This performance is significantly better than existing mainstream methods (such as SVM, LSTM, EEGNet, etc.).

[0024] (2) Significantly improved image quality: Compared with reconstruction using only EEG data, the hybrid system (EEG + eye movement) of the present invention improves the SSIM value of the reconstructed image by 41.5%. The LPIPS value of the hybrid system is reduced by 23.1% (the lower the LPIPS, the more similar the images are perceptually and the better the quality).

[0025] (3) Achieving complex multi-semantic reconstruction: Traditional EEG methods can only reconstruct a single scene (such as "desert"). By fusing object-level semantics, this invention can successfully reconstruct complex scenes containing multiple objects (such as "desert with four camels"), greatly enriching the details and accuracy of the reconstructed content.

[0026] (4) The generated results are more consistent with human visual perception: By introducing a visual attention mechanism to control the generation process, the system is able to highlight and reconstruct the key areas that the user is focused on. Subjective interviews also confirmed that participants believed the reconstructed images were “clearer” and “more accurate”, and felt that the system “understood” their focus of attention, making the visual output more consistent subjectively.

[0027] (5) Wide range of applications: This technology can be used to help people with communication disorders express their visual experiences. It can also be applied to BCI-assisted visual design and AIGC fields to achieve “what you think is what you see” creation without the need for hands or words. Attached Figure Description

[0028] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of the process of the method of the present invention.

[0030] Figure 2 This invention is the 3D-CTNet scene image classification model.

[0031] Figure 3 The flowchart illustrates the process of extracting scene-level semantic features according to the present invention.

[0032] Figure 4 This is a schematic diagram of the image reconstruction method framework of the multi-semantic driven diffusion model of the present invention.

[0033] Figure 5 This is a schematic diagram of the basic logical structure of the system of the present invention. Detailed Implementation

[0034] The image reconstruction method and system based on electroencephalography (EEG) and eye tracking of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them, and should not be construed as limiting the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0035] Example 1

[0036] See Figure 1 The present invention provides a flowchart of an image reconstruction method based on electroencephalography and eye tracking, and the steps therein are described in detail: (1) Acquisition and preprocessing of EEG signals and eye movement data; Users wear EEG caps and have conductive gel applied to reduce electrode impedance and improve signal stability. The experiment then begins collecting EEG data. At the start of each experiment, a fixed crosshair appears on a black screen, accompanied by a brief audio prompt. Users look at the crosshair on the screen, preparing to imagine. Two seconds later, an image appears on the screen, lasting four seconds, prompting participants to imagine the corresponding scene. The scene image appears on the screen, and the user imagines the scene, resulting in the acquisition of corresponding EEG signals. Simultaneously, eye movement data is collected using a Tobii Pro Fusion eye tracker, obtaining eye movement information such as fixation point, fixation time, and fixation trajectory. The collected EEG signals are filtered: first, a 50Hz notch filter is used to eliminate power frequency interference; then, a 0.1Hz high-pass filter and a 50Hz low-pass filter are used to extract EEG frequency bands highly correlated with scene semantic features. Independent component analysis (ICA) is then applied to eliminate noise caused by eye, head, and muscle movements. ICA uses statistical methods to decompose the mixed signal into independent signal sources and combines expert experience to identify and remove significant noise components to improve the signal-to-noise ratio. Finally, a sliding window is used to increase the sample size and improve training quality. A 3.5-second sliding window was designed to slide through data from 2 to 6 seconds. The window slid at 0.1-second intervals, increasing the experimental sample size fivefold. Through these operations, preprocessed EEG and eye-tracking data were obtained.

[0037] (2) Extract scene-level semantic features; See Figure 2 Scene-level semantic feature extraction requires classification of preprocessed EEG data. This invention uses the proposed 3D-CTNet for classification. (See attached document.) Figure 3Specifically, the preprocessed EEG data is converted into a 2D matrix, and EEG data from different channels are stacked and combined to create 3D EEG data. Spatial features are extracted from the EEG data using multiple 3D convolutional kernels of different sizes, resulting in EEG data I. Temporal features are extracted by directly processing the preprocessed EEG data using a Transformer with an attention mechanism, resulting in EEG data II. Finally, the two sets of EEG data are expanded into one dimension and merged through a connection layer. The merged EEG data is then classified using softmax to obtain scene labels for scene image reconstruction.

[0038] (3) Extracting entity-level semantic features; Object-level semantic features and their corresponding quantity information in the image were obtained using the YOLOv9 object detection model. When using the quantity of object-level semantic features, we did not uniformly use the actual quantity, but controlled the expression strategy for different object-level semantic feature quantities: when a semantic feature appeared less than 6 times, we used the actual quantity, while when it appeared 6 times or more, we used "many" to indicate the degree; we calculated the attention weights and fixation points corresponding to the visual semantic information. G More, longer viewing time T The longer the scanning path L The longer the length, the larger the proportion of the semantic object in the image, and the higher the user's attention to that semantic object. The formula is as follows: (1) in, n Indicates the number of semantics detected. α , β , γ These represent the fixation point, fixation time, and saccade length in the calculation of attention weights, respectively. c i Weighting of time.

[0039] To add visual attention to the detected visual semantic information, attention weights are used. c i This scales the cross-attention map assigned to each visual semantic, making the effect stronger or weaker. The formula is as follows: (2) in M t It is the original cross-attention map. M t ) i,j Indicating time steps t How can the pixel at position i in the image focus on the token in the text? j .

[0040] (4) Image reconstruction methods based on semantic fusion and multi-semantic driven diffusion models; See Figure 4 In the single-person image reconstruction method, a pre-trained diffusion model, Stable Diffusion, is used as the image generator. It combines scene-level semantic features obtained from the EEG semantic feature extractor with object-level semantic features obtained from the visual semantic feature extractor, along with their corresponding attention weights, to form complete image semantic features, which are then fed into Stable Diffusion for image generation. In the multi-person image reconstruction method, an independent classification combined with consistency judgment is used for the EEG scene classification task. If the classification results of several users are consistent, that category is directly used as the final scene category. If the classification results of several users are different, the two categories with the most frequent classification results are used together as the input features for image reconstruction to preserve the scene cognitive information of different users. The quantitative information and attention weights under the visual semantic features are weighted and averaged.

[0041] Example 2

[0042] See Figure 5 This invention provides a schematic diagram of the basic logical structure of a system for reconstructing images based on electroencephalography (EEG) and eye-tracking. For ease of explanation, only the parts relevant to the embodiments of this invention are shown. The functional modules / units in this system can be hardware modules / units or software modules / units, and mainly include:

[0043] (1) EEG semantic feature extraction module, used to extract EEG signal features for scene classification; (2) Visual semantic feature extraction module, used to extract volumetric semantic features, corresponding quantitative information and attention weights corresponding to eye movement data; (3) Image reconstruction module, used to fuse EEG semantic features and visual semantic features to generate the final reconstructed image.

[0044] Example 3

[0045] A computer-readable storage medium is characterized in that it stores a program thereon, which, when executed by a processor, implements the image reconstruction method based on electroencephalography and eye tracking as described in Embodiment 1.

[0046] Example 4

[0047] A computing device includes a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in Embodiment 1.

[0048] In the embodiments of this invention, the modules can be integrated into one unit, deployed separately, or further divided into multiple sub-modules. The modules can be distributed within the system of the embodiments as described, or they can be modified to reside in one or more systems different from those of the embodiments of this invention.

[0049] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, embodiments of the present invention are not limited to any particular combination of hardware and software.

[0050] The above-disclosed embodiments are merely specific examples of the present invention, but the present invention is not limited thereto. Those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Obviously, all such modifications and variations should fall within the protection scope claimed by the present invention.

Claims

1. An image reconstruction method based on electroencephalography (EEG) and eye tracking, characterized in that, Includes the following steps: (1) Acquisition and preprocessing of EEG signals and eye movement data; EEG signals and eye-tracking data of users are collected using EEG acquisition equipment and eye-tracking equipment. The EEG signals are filtered and independent component analysis to remove noise and artifacts, and data augmentation is performed using a sliding window. (2) Extract scene-level semantic features; The 3D-CTNet model is used to extract scene-level semantic features from preprocessed EEG signals. 3D-CTNet includes: a multi-kernel 3D convolutional network module for extracting spatial features of EEG signals; a Transformer module for temporal feature encoding to extract temporal features of EEG signals through a multi-head attention mechanism; and a parallel structure that integrates the 3D convolutional network and the Transformer to ensure maximum preservation and optimized extraction of spatiotemporal features. (3) Extracting entity-level semantic features; The YOLOv9 object detection model is used to identify objects in images viewed by users and extract target-level semantic features. The model adopts a multi-scale feature fusion strategy to enhance the recognition ability of small targets. Visual attention weights are calculated based on eye-tracking data, taking into account the number of fixation points, fixation duration, and saccade path length. (4) Image reconstruction using semantic fusion and multi-semantic driven diffusion models; Scene-level semantic features are fused with target-level semantic features, and combined with visual attention weights to generate multi-semantic representations. A pre-trained Stable Diffusion model is used to transform the multi-semantic representations into text prompts for image reconstruction, and the weights of different semantic features during the generation process are controlled through a cross-attention mechanism.

2. The multi-semantic image reconstruction method based on the combination of electroencephalogram (EEG) signals and eye-tracking data according to claim 1, characterized in that, Step (1) specifically includes: (11) The user wears an EEG cap and applies conductive gel to reduce electrode impedance and improve signal stability; (12) At the start of each run, a fixed crosshair appears on the black screen, accompanied by a brief sound prompt. The user looks at the crosshair on the screen and prepares to begin visualizing; (13) Two seconds later, an image appears on the screen and lasts for 4 seconds, prompting the participant to imagine the corresponding scene. Scene images appear on the screen, the user imagines the corresponding scene image, and the corresponding EEG signals are collected; (14) The user’s eye movement data is collected synchronously through the Tobii Pro Fusion eye tracker to obtain eye movement information such as the user’s fixation point, fixation time, and fixation trajectory; (15) The collected EEG signals are filtered. First, a 50Hz notch filter is used to eliminate power frequency interference, and then a 0.1Hz high-pass filter and a 50Hz low-pass filter are used to extract the EEG frequency bands that are highly related to the semantic features of the scene. (16) Independent Component Analysis (ICA) was applied to eliminate noise caused by eye, head, and muscle movements. ICA uses statistical methods to decompose the mixed signal into independent signal sources and combines expert experience to identify and remove significant noise components to improve the signal-to-noise ratio. (17) Use a sliding window to increase the sample size and improve training quality. Design a sliding window to slide the data.

3. The method for extracting scene-level semantic features as described in claim 2, characterized in that, Step (2) specifically includes: (21) The preprocessed EEG data was converted into a 2D matrix, and the EEG data from different channels were stacked and combined to create 3D EEG data. Spatial features were extracted from the EEG data using multiple 3D convolutional kernels of different sizes to obtain EEG data I; (22) Use a Transformer with attention mechanism to directly process the preprocessed EEG data to extract temporal features and obtain EEG data II; (23) The two sets of EEG data are expanded into one dimension and merged through a connection layer. The merged EEG data is then classified using softmax to obtain scene labels for scene image reconstruction.

4. The method for extracting entity-level semantic features according to claim 1, characterized in that, Step (3) specifically includes: (31) Object-level semantic features and corresponding quantity information in the image are obtained using the YOLOv9 object detection model. When using the quantity corresponding to the object-level semantic features, the actual quantity is not used uniformly. The expression strategy for the quantity of different object-level semantic features is controlled: when a semantic feature appears less than 6 times, the actual quantity is used, while when it appears 6 times or more, "many" is used to indicate the degree. (32) Calculate the attention weights corresponding to visual semantic information and the fixation point. G More, longer viewing time T The longer the scanning path L The longer the length, the larger the proportion of the semantic object in the image, and the higher the user's attention to that semantic object. The formula is as follows: (1) in, n Indicates the number of semantics detected. α , β , γ These represent the fixation point, fixation time, and saccade length in the calculation of attention weights, respectively. c i Weighting of time. (33) Add visual attention to the detected visual semantic information by using attention weights. c i This scales the cross-attention map assigned to each visual semantic, making the effect stronger or weaker. The formula is as follows: (2) in M t It is the original cross-attention map. M t ) i,j Indicating time steps t How can the pixel at position i in the image focus on the token in the text? j .

5. The image reconstruction method based on semantic fusion and multi-semantic driven diffusion model according to claim 1, characterized in that, Step (4) specifically includes: (41) In the single-person image reconstruction method, the pre-trained diffusion model Stable Diffusion is used as the image generator. The scene-level semantic features obtained by the EEG semantic feature extractor and the object-level semantic features obtained by the visual semantic feature extractor and their corresponding attention weights are combined to form complete image semantic features and handed over to Stable Diffusion for image generation. (42) In the multi-user image reconstruction method, an independent classification combined with consistency judgment is used for EEG scene classification. If the classification results of several users are consistent, the category is directly used as the final scene category. If the classification results of several users are different, the two categories with the most classification results are used as the input features for image reconstruction to preserve the scene cognitive information of different users. The quantitative information and attention weight under visual semantic features are weighted averaged.

6. An image reconstruction hybrid brain-computer interface system, characterized in that... include: The EEG semantic feature extraction module is used to extract EEG signal features for scene classification. First, signal acquisition and preprocessing are performed, including filtering, artifact removal, and data augmentation to improve signal quality. Then, a pre-trained 3D-CTNet is used for feature extraction, followed by spatial feature extraction using a 3D CNN, and temporal series modeling is performed in conjunction with a Transformer encoder. Finally, scene category prediction is performed to generate EEG semantic features, which are then passed to the image reconstruction module. Furthermore, in multi-user mode, a multi-user EEG semantic fusion strategy is employed to enhance the semantic understanding of the scene. The visual semantic feature extraction module is used to extract object-level semantic features, corresponding quantitative information, and attention weights corresponding to eye-tracking data. By determining the user's gaze point and combining it with the YOLOv9 object detection model, object-level semantic features are extracted from the scene. Furthermore, by using eye-tracking data such as gaze duration and gaze point distribution, we can calculate the attention distribution for each user. Through attention assignment in single-user mode and weighted attention assignment in multi-user mode, we can better restore object-level semantic information in the image, making the reconstructed image more consistent with human visual perception and improving image reconstruction quality. The image reconstruction module integrates EEG semantic features and visual semantic features to generate the final reconstructed image. In single-user mode, users can view the generated image and provide subjective ratings. In multi-user mode, collaborative optimization is supported, adjusting the generated image by fusing feature data from different users.

7. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the multi-semantic image reconstruction method based on the combination of EEG signals and eye movement data as described in the claims.

8. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method described in the claims.

Images