High-fidelity 3D image reconstruction method based on brain EEG signals

Abstract

The present invention relates to a high-fidelity 3D reconstruction method based on brain EEG signals. First, the fine features of brain activity are obtained through EEG correlation features for 3D image signal reconstruction to achieve accurate encoding of the participant's visual perception or imagination content; then, EEG signals are mapped to 2D images, and the features obtained by encoding are used to fine-tune the pre-trained 2D image generation module and CLIP model to achieve a graphical expression of the semantics of the EEG signal, thereby obtaining a pre-trained 2D image generation module; finally, the pre-trained 2D image generation model is combined with the 3D representation technology, and the 3D scene is parameterized through a joint loss function to achieve an intuitive mapping from the EEG signal encoding features to the specific 3D image, thereby generating a 3D image with high fidelity and rich details. The present invention generates the corresponding 3D image through the EEG signal generated by the participant to the object, providing a new perspective and tool for research in the fields of cognitive science, psychology and neuroscience.

Description

Technical Field

[0001] The present invention relates to the field of electroencephalogram (EEG) data processing, and in particular to a high-fidelity 3D image reconstruction method based on brain EEG signals. Background Art

[0002] As one of the cutting-edge technologies in the current field of artificial intelligence, generative artificial intelligence technology has been widely used in various visual synthesis tasks and has promoted the explosive growth of research in the fields of 2D and 3D creation. The Stable Diffusion Model (SD) has promoted the development of text-to-image synthesis tasks in this field with its powerful generation capabilities and the CLIP (Contrastive Language-Image Pre-training) model's effective response to text conditions. The Imagen model achieves highly realistic text-to-image synthesis by effectively combining the two, which marks a major progress in image quality and language understanding; in addition, the introduction of models such as DALL-E 2 and Muse has further enriched the technical means and application scenarios of text-to-image synthesis. DALL-E 2 has demonstrated its outstanding ability in understanding and generating multimodal content through its innovative architecture and training strategy. The Muse model has improved the quality and diversity of image generation by combining generative adversarial networks and other generative models, providing new tools and inspiration for artistic creation and design. With the continuous advancement of 3D synthesis technology, a large number of studies have also begun to focus on generating 3D objects from text descriptions. Dreamfussion uses a pre-trained 2D diffusion model and a novel score distillation sampling method to generate high-fidelity 3D models from text descriptions without the need for 3D training data; GSGEN proposes an innovative optimization strategy based on 3D Gaussian Splatting (3DGS) technology to achieve 3D content generation with fine details and accurate geometric shapes; In order to improve generation efficiency, Instant3D builds 3D triplanes directly from text prompts, greatly speeding up generation while maintaining a high degree of alignment with text descriptions. These studies, based on large models, achieve the transformation of abstract text descriptions into abstract 3D visual information.

[0003] With the continuous development of brain-computer interface (BCI) technology, a large number of studies have begun to explore how to visualize abstract brain signals into understandable visual content. Brain2Image attempts to generate images seen by observers from EEG signals by combining long short-term memory networks and generative adversarial network technology; Ye et al. proposed a self-supervised cross-modal retrieval method to restore the precise visual stimulation of EEG fragments by maximizing the mutual information between EEG encoding and visual stimulation. This method not only improves the accuracy of EEG signal-to-image synthesis, but also reduces the dependence on labeled data through self-supervised learning; Lan et al. further explored methods to reconstruct visual stimulation images from EEG signals, generating high-resolution visual stimulation images through multi-level perceptual information decoding and potential diffusion models; the Dream Diffusion method proposed by Bai et al. in 2023 can generate high-quality images directly from EEG signals. This method uses a pre-trained text-to-image model and combines the CLIP image encoder to provide additional supervision for the alignment of EEG, text, and image embeddings.

[0004] However, these studies on image synthesis based on EEG signals mainly focus on the reconstruction of 2D images. For application fields such as immersive virtual reality, 3D game development, and film production, the synthesis of EEG signals into 3D images can not only promote the development of human-computer interaction technology, but also provide new perspectives and tools for research in cognitive science, psychology, and neuroscience. At present, research on directly converting EEG signals into 3D images is still in its infancy. The generation of 3D images not only needs to consider the depth information of the image, but also requires more complex models to capture and reconstruct the structure and shape in three-dimensional space. Therefore, how to accurately extract rich visual information from EEG signals and generate 3D images that are closely related to brain activity is still an open problem.

[0005] Therefore, this study intends to develop a new 3D image reconstruction method by combining the SD model with 3DGS technology. This method will not only provide an intuitive 3D visualization method for understanding brain activity, but will also promote the development of EEG signal processing and 3D image generation technology. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a high-fidelity 3D image reconstruction method based on brain EEG signals. The reconstruction method uses the key feature encoding of the EEG signal as a guide, combines the three-dimensional representation of the 2D image generation module and the 3D image generation module, and generates a 3D image matching the EEG signal; firstly, the EEG key feature encoding is performed, the collected EEG signal is preprocessed, and a mask signal is constructed to train the EEG encoder to obtain the EEG feature vector of the key feature; the EEG feature vector is input into the 2D image generation model to realize the high-fidelity conversion from brain activity to 2D image, Output a rough 2D image, calculate the similarity between the EEG feature vector and the image feature vector extracted by the CLIP image decoder, reversely fine-tune the EEG encoder so that the EEG feature vector is aligned with the image text information of CLIP in the feature space, and use the camera elevation and azimuth information as auxiliary information; input the rough 2D image into the 3D image generation module, perform geometric optimization through the joint loss function composed of the 2D image score function and the 3D image score function, and finally refine the appearance through compactness densification technology to obtain a high-fidelity and detail-rich 3D image, including:

[0007] Step 1: EEG signal acquisition, by showing the target image to the subject and collecting its EEG signal;

[0008] Step 2: Preprocess the EEG signal by filtering and transforming to obtain an embedding vector;

[0009] Step 3: Learn the contextual information of the EEG signal by training the encoder of the EEG feature extractor to predict the masked embedding vector, including:

[0010] Step 31: Input the preprocessed EEG signal into the EEG feature extractor, perform random masking with a preset masking rate, and the mask signal model will use the embedding vector to model the mask signal;

[0011] Step 32: The difference between the EEG signal reconstructed by the model and the original EEG signal is measured by using the mean square error as the loss function to obtain an EEG feature vector with a dimension of 1024;

[0012] Step 4: Input the EEG feature vector into the 2D image generation module for training and output a rough 2D image. The 2D image generation module is based on the SD model, including the variational autoencoder, the denoising U-Net and the EEG encoder trained in step 3. The training steps include:

[0013] Step 41: Use the public ImageNet EEG dataset as a training set and the self-collected dataset as a validation set. The dataset contains both EEG signals and images corresponding to the EEG signals.

[0014] Step 42: The EEG feature vector is used as conditional information and input into the cross-attention layer in the denoising U-Net inside the 2D image generation module. The cross-attention layer matches the EEG feature vector with each position in the feature map of the denoising U-Net to generate attention weights; then, the aggregated representation of the conditional information at different positions is calculated by the weights and added to the feature map of the denoising U-Net; in this way, the EEG feature vector is encoded into each layer of the denoising U-Net, guiding the 2D image generation module to generate a rough 2D image consistent with the conditional information;

[0015] Step 43: Perform end-to-end training on the SD model. During the training process, the difference between the generated image and the target image is calculated using a loss function.

[0016] Step 44: Fine-tune the EEG encoder to align the EEG feature vector with the feature space of the CLIP image decoder to achieve multimodal alignment of EEG signals, text, and images;

[0017] Step 5: Input the rough 2D image into the 3D image generation module for geometric optimization. Use the semantic information in the EEG feature vector, the joint loss function composed of the 2D image score estimation function and the 3D image score estimation function to jointly perform geometric optimization;

[0018] Step 51: First, the Gaussian position is initialized, and the pre-trained 3D diffusion model extracts semantic information from the text embedding, and generates a related 3D point cloud based on the extracted semantic information;

[0019] Step 52: Input the camera pose and EEG feature vector into the 2D image generation model pre-trained in step 4, and train it through the 2D image score estimation function to optimize the color and opacity in the 3D Gaussian ellipsoid;

[0020] Step 53: Calculate a 3D image score estimation function for the 3D diffusion model according to the optimization process of the 2D image generation model loss, and optimize the shape and position of the 3D Gaussian ellipsoid through the 3D image score estimation function;

[0021] Step 54: Minimize the joint loss function composed of the 2D image score estimation function and the 3D image score estimation function, optimize the parameters in the 3D Gaussian ellipsoid, achieve the purpose of geometric optimization, and generate a rough 3D image that matches the output of the 2D image generation model in appearance;

[0022] Step 6: Appearance refinement, which realizes the gradual refinement of the rough 3D image, not only using the 2D image generation model as a guide, but also increasing the number of Gaussian points through a densification strategy to improve the density of the model, including:

[0023] Step 61: the rough 3D image obtained in step 5 is further combined with the rough 2D image output by the 2D image generation module to calculate a 2D image score estimation function, thereby continuing to adjust the color, opacity and shape of the Gaussian ellipsoid;

[0024] Step 62: During the refinement process, the opacity and position are regularized by a compactness densification technique to avoid potential degradation problems, while those insignificant Gaussian points are periodically removed, and finally a refined 3D image is output.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. The EEG encoder part uses the temporal mask signal modeling technology (TMSM). This method does not rely on limited EEG image paired samples, but uses a large amount of EEG data to train the encoder by predicting missing tags based on context clues. This pre-training strategy not only enhances the model's adaptability to individual differences, but also enables the pre-trained encoder to deeply understand EEG data from different populations and various brain activities through signal reconstruction in the time domain.

[0027] 2. In order to solve the multi-view problem, the 2D image generation module introduces auxiliary information embedding. Different position information embedding is added according to the postures of different sampling perspectives, so that images can be generated according to different camera postures, providing more guidance information for subsequent 3D generation.

[0028] 3. Rendering method of 3D scene sampling 3D Gaussian. Many 3D generation methods face a major challenge: overfitting to multiple views, resulting in multi-faceted and collapsed geometric shapes of generated objects. However, the geometry of 3D Gaussian can be directly corrected using point cloud priors, and through compactness densification technology and regularization of 3D Gaussian, a 3D model with stable structure, reasonable geometry and rich details can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is the overall structure diagram of the model of the present invention;

[0030] Figure 2 A schematic diagram of the overall technical route of the present invention;

[0031] Figure 3 It is a schematic diagram of the structure of the mask mechanism of the present invention;

[0032] Figure 4It is a schematic diagram of the processing flow of the 2D image generation module of the present invention;

[0033] Figure 5 It is a structural schematic diagram of a 3D image generation module of the present invention;

[0034] Figure 6 It is a diagram of the experimental results of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0036] The EEG of the present invention refers to: EEG (Electroencephalogram) is an electrical signal generated by the activity of brain neurons. Neurons are connected to each other through synapses to form a complex neural network. When neurons are activated, bioelectric phenomena are generated, and these electrical signals can be captured by placing electrodes on the scalp or directly implanted in the brain.

[0037] The token of the present invention refers to: Token refers to a data representation method for processing EEG signals into a data representation method suitable for deep learning models. Token is a method of discretizing continuous EEG signals so that the signals can be processed by deep learning models in the same way as sequence data. This method allows the model to capture the dynamic changes of EEG signals over time and learn useful feature representations from them.

[0038] The SD model of the present invention refers to: Stable Diffusion Model is a latent diffusion model (Latent Diffusion Model) for text-to-image generation, which converts the input text prompt words into semantic vectors through a text encoder, which are then passed to the image generator as control input. The image generator consists of a U-Net network and a sampler algorithm, which is responsible for generating image information in a low-dimensional space, and then the VAE decoder is used to increase the dimension and generate the final image. The present invention replaces the text encoder with an EEG encoder, thereby guiding image generation according to EEG.

[0039] The CLIP model of the present invention refers to: The CLIP (Contrastive Language-Image Pre-training) model is a multimodal pre-trained neural network released by OpenAI in 2021. It is trained on a large number of image and text pairs through contrastive learning to learn the semantic relationship between images and text. The innovation of the CLIP model is that it can map images and text into a shared vector space, so that the model can understand the semantic relationship between images and text.

[0040] The 3D diffusion model of the present invention refers to: 3D point cloud diffusion model (Point-E) is a new 3D model generator developed by OpenAI, which can quickly generate 3D point clouds based on text prompts. The present invention converts text prompts into EEG prompts to obtain initial point cloud information that matches the EEG signal.

[0041] The core of this invention is to explore cutting-edge technology for reconstructing 3D images directly from brain EEG signals. Although there are more and more studies on image generation based on brain signals, most of them focus on generating 2D graphics from fMRI or EEG signals. The research on converting these brain signals into 3D images is not only more technically challenging, but also innovative in both theory and application.

[0042] In view of the shortcomings of the prior art, the present invention proposes a high-fidelity 3D reconstruction method based on brain EEG signals. The method uses EEG signal feature encoding as a guide, combines the 2D image generation model and the three-dimensional representation of the 3D image generation module, and generates a 3D image that matches the EEG signal and has high fidelity and rich details. First, the collected EEG initial data is preprocessed to obtain more accurate and clean data. In addition, considering the scarcity of EEG image data sets, the method adopts the temporal masked signal modeling technology TMSM (Temporal Masked Signal Modeling) to obtain an effective and stable EEG representation by pre-training a large amount of EEG data; the representation of EEG key features is combined with the image generation of the 2D image generation model through the cross-attention mechanism in the 2D image generation model, thereby achieving a high-fidelity conversion from brain activity to visual images; in order to improve the semantic accuracy of the generated image, the EEG feature vector is combined with CLIP (Contrastive Language-Image The image feature vector extracted by the image decoder in the pre-training model is used for similarity calculation, and the EEG encoder is reversely fine-tuned to make the EEG feature vector closer to the CLIP image text information in the feature space; and through random camera position sampling, its auxiliary perspective information is embedded according to the size of its elevation and azimuth angles to provide pose and multi-perspective images for subsequent 3D generation; the three-dimensional scene is represented as a differentiable parameterization, and the joint loss function composed of the 2D image score function established with the 2D image generation model and the 3D image score function established with the 3D point cloud diffusion model is used for geometric optimization. Finally, the appearance is refined through 2D image guidance, densification technology, and regularized pruning to obtain a high-fidelity and detail-rich 3D image. Figure 1 It is a structural schematic diagram of the model of the present invention. Figure 2 The following is a process flow chart of the method of the present invention. The technical solution is described below in conjunction with the accompanying drawings.

[0043] Step 1: EEG signal acquisition, including:

[0044] Step 11: The target image used is the ImageNet dataset, which has 40 categories, each category contains 50 images, and a total of 2000 images;

[0045] Step 12: Select 6 subjects and show each image to them for 0.5 seconds. After showing 50 images, there is a 10-second pause. While the subjects are watching the images, the 128-channel Brainvision EEG system is used to record the EEG signals of the subjects.

[0046] Step 13: Each subject viewed 2,000 images, each image corresponding to an EEG recording, so a total of 12,000 128-channel EEG signals were obtained;

[0047] Step 2: EEG signal preprocessing helps improve signal quality, reduce the complexity of model training, and ensure that the model can learn useful features from the data. The steps include:

[0048] Step 21: All EEG signals are filtered before training; the specific filtering range is 5 to 95 Hz to eliminate noise and unwanted frequency components.

[0049] Step 22: The filtered EEG signal is truncated to a preset uniform time length; here it is 500 time steps, in order to ensure that all EEG signals have a uniform processing standard in the time dimension.

[0050] Step 23: The EEG signal is a sequence of potential measurements that vary over time, so every 4 adjacent time steps are combined into a window, the EEG signal in each window is converted into a token, and each token is converted into a 1024-dimensional embedding vector through a projection layer; for subsequent mask signal modeling.

[0051] Step 3: Learn the contextual information of the EEG signal by training the encoder of the EEG feature extractor to predict the masked embedding vector, including:

[0052] Step 31: The preprocessed EEG signal is input into the EEG feature extractor. The masked signal model uses the embedding vector for masked signal modeling (MSM) and performs random masking with a preset masking rate (the masking rate is 0.75).

[0053] like Figure 3 As shown, through the mask signal model, the encoder of the EEG feature extractor is able to learn deep semantic representations of the EEG signal, enabling the model to handle the noise and variability in the EEG signal. These representations are crucial for subsequent image generation tasks.

[0054] Step 32: By using the mean square error as the loss function to measure the difference between the model reconstructed EEG signal and the original EEG signal, the encoder in the EEG feature extractor is optimized and an EEG feature vector with a dimension of 1024 is obtained;

[0055] Step 3 obtains a trained EEG feature extractor. In subsequent tasks, only the EEG encoder in the EEG feature extractor needs to be used. The figure shows how to train the EEG feature extractor.

[0056] Step 4: Input the EEG feature vector into the 2D image generation module for training. The 2D image generation module is based on the SD model, and its core components include the variational auto-encoder VAE (Variational Auto-Encoder), denoising U-Net, and the EEG encoder trained in step 3. During the fine-tuning process, the EEG feature vector and the auxiliary vector are weighted and fused into a new embedding vector. Then, the EEG features are integrated through the cross-attention mechanism in U-Net to guide the denoising process. The denoising U-net outputs the latent vector, and finally the 2D image is obtained through the VAE decoder. The CLIP model is used to achieve multi-modal alignment of EEG, text, and image. The processing flow diagram of the 2D image generation module is shown in the figure. Figure 4 shown.

[0057] However, since EEG cues are usually not accurate enough for different perspectives, it is proposed to embed auxiliary information for different camera postures based on random sampling of camera positions, so as to obtain multi-perspective images and provide camera posture and geometric details guidance for subsequent 3D image generation. The steps include:

[0058] Step 41: Use the public ImageNet EEG dataset as a training set, and the dataset collected by the present invention as a validation set; the dataset contains both EEG signals and images corresponding to the EEG signals; and use the EEG feature vector as conditional information to guide image generation.

[0059] Step 42: The EEG feature vector is used as conditional information and input into the cross-attention layer in the denoising U-Net inside the 2D image generation module. The cross-attention layer matches the EEG feature vector with each position in the feature map of the denoising U-Net to generate attention weights. The weights are then used to calculate the aggregated representation of the conditional information at different positions and added to the feature map of the denoising U-Net. In this way, the EEG feature vector is encoded into each layer of the U-Net, guiding the 2D image generation module to generate a 2D image consistent with the condition.

[0060] In U-Net, the specific cross-attention formula is as follows: Query (Q): the middle layer feature of the encoder; Key (K): obtained by projecting the output of the EEG encoder, used as conditional information to help the model focus on the image area related to the EEG signal; Value (V): obtained by projecting the output of the EEG encoder, providing specific content guidance required to generate the image:

[0061] Q=W(i) Q ·φ i (z t ) (1)

[0062] K=W(i) K ·τ θ (y) (2)

[0063] V=W(i) V ·τ θ (y) (3)

[0064] Among them, φ i (z t ) represents the intermediate features of the denoising U-Net at time step t; τ θ (y) represents the output of the encoder; W(i) K and W(i) V is a learnable projection matrix that maps the EEG feature vectors to a space of the same dimensionality as the U-Net features.

[0065] The mathematical expression for attention weight calculation is as follows:

[0066]

[0067] Among them, d k Represents the dimension of the Key vector, which is used to scale the dot product to avoid gradient disappearance.

[0068] Step 43: The SD model is trained end-to-end to optimize the model parameters. During the training process, the difference between the generated image and the target image is calculated using a loss function, which can be expressed as:

[0069]

[0070] Where x represents the target image that matches the EEG signal; ∈ represents the image from the standard normal distribution Random noise in ∈ θ represents the denoising model parameterized by θ; x t is the noise sample of image x at a given time step t; y represents the output feature vector of the EEG encoder. τ θ (y) represents the projected EEG feature vector, which is used as the conditional information for generating a 2D image. Finally, through step 4, an image matching the EEG can be output;

[0071] Step 44: Optimizing the EEG encoder through the CLIP model aims to achieve semantic alignment between the EEG feature vector and the image feature vector. Since the CLIP model has been pre-trained on a large-scale image and text dataset, its feature space can already reflect the semantic relationship between the image and text. Therefore, the EEG feature vector is aligned with the image feature space of the CLIP model to achieve multimodal alignment of EEG, text and image. The steps include:

[0072] Step 441: input the target image into the CLIP image decoder, and encode the target image to obtain an image feature vector;

[0073] Step 442: Map the dimension of the EEG feature vector of step 3 to the same semantic space as the image feature vector through the projection layer of the CLIP image decoder;

[0074] Step 443: Calculate the similarity loss between the EEG feature vector and the image feature vector. The loss function can be expressed as:

[0075]

[0076] Among them, E I (I) represents the image feature vector after the target image is encoded by the CLIP image encoder; τ θ represents the projection function parameterized as θ, which is used to map the EEG feature vector to a space with the same dimension as the image feature vector; h(τ θ (y)) represents the EEG feature vector after being processed by the projection function;

[0077] By minimizing the loss function and then optimizing and fine-tuning the EEG encoder, the EEG feature vector learned by the EEG encoder will be aligned with the image feature vector of the CLIP model as much as possible, thereby achieving semantic alignment of the EEG signal, text, and image, so that the EEG feature vector has the feature information of text and image, which can better guide image generation;

[0078] Step 45: According to the randomly sampled camera poses, different auxiliary information embeddings are added for different elevation angles, so as to obtain a 2D image with multiple perspectives and matching the EEG signal, such as Figure 4 As shown, the steps include: classifying the randomly sampled elevation angles, adding a "top view" for elevation angles greater than 60 degrees; adding a "front view", "side view" or "back view" for elevation angles less than or equal to 60 degrees according to the value of the azimuth angle using a weighted combination of text embeddings, so that the EEG feature vector contains generation guidance for different perspectives;

[0079] Through step 4, a pre-trained generative model is obtained that can generate corresponding 2D images according to EEG signals and random camera poses, providing more accurate guidance for subsequent 3D image generation;

[0080] Step 5: Geometric optimization, representing the 3D scene as a differentiable parameterization, and using the semantic information in the EEG feature vector, the 2D image score estimation function in the 3D image generation module, and the 3D image score estimation function in the 3D point cloud diffusion model Point-E to jointly perform geometric optimization;

[0081] In order to avoid the problems of multifacetedness and geometric collapse, the Gaussian Splatting method is used to represent the three-dimensional scene. Through the progressive optimization strategy, fine optimization from rough geometric shapes to detailed appearance is achieved. Finally, multi-view 3D images are obtained through rendering. The first step of geometric optimization is as follows:

[0082] Step 51: First, initialize the Gaussian position. The pre-trained 3D point cloud diffusion model can extract semantic information from the text embedding and generate the relevant 3D point cloud accordingly.

[0083] In step 4, the multimodal alignment of EEG, text and image has been achieved, so it is only necessary to fine-tune the 3D point cloud diffusion model (Point-E) so that it can pass the EEG feature vector obtained by the EEG encoder to the 3D point cloud corresponding to it;

[0084] Step 52: Input the camera pose and EEG feature vector into the 2D image generation model pre-trained in step 4, and estimate the function L through the 2D image score SDS Train to optimize the color, opacity, and 2D image score function L in the 3D Gaussian ellipsoid SDS The mathematical expression is as follows:

[0085]

[0086] Where xt, y and t represent the noise image, EEG embedding and time step respectively, ∈ is Gaussian noise, w(t) is the weight function, and θ represents the model parameters;

[0087] Step 53: Based on the optimization process of the 2D image generation model loss, calculate the 3D image score estimation function L for the 3D point cloud diffusion model 3D-SDS , the shape and position of the 3D Gaussian ellipsoid are optimized through the 3D image score estimation function, the 3D image score estimation function L 3D-SDS The mathematical expression is as follows:

[0088]

[0089] Where pt, y and t represent the noise point cloud, EEG embedding and time step respectively, ∈ is Gaussian noise, and w(t) is the weight function;

[0090] Step 54: Optimize the parameters in the 3D Gaussian ellipsoid through the joint loss function composed of the 2D image score estimation function and the 3D image score estimation function to achieve the purpose of geometric optimization; the mathematical expression of the joint loss function is as follows:

[0091]

[0092] Among them, w I (t) and w P (t) represents the weight function of the image and point cloud at time t, which is used to balance the contribution of different time steps t; ∈ I and ∈ P Represent the actual values ​​of image and 3D point cloud errors respectively; p t and x t Represent the Gaussian position and rendered image with noise respectively; ∈ φ and ∈ ψ Represent the score estimation functions of 2D images and 3D point clouds respectively. 3D is a weight parameter used to balance the 2D image diffusion loss and the 3D point cloud diffusion loss. ∈ φ (x t ; y, t) represents the model parameters, input x t and the error function of the label y at time t; ∈ ψ (p t ; y, t) represents the model parameters, input p t and the point cloud error function of label y at time t.

[0093] By minimizing the joint loss function, we can generate a rough 3D image that matches the output of the 2D image diffusion model in appearance while maintaining 3D geometric consistency.

[0094] like Figure 5 Each iteration is in a finite spherical scene, the camera position is randomly sampled in spherical coordinates, the elevation angle range is within (-10°, 90°), the azimuth angle range is within (0°, 360°), and the distance from the far origin is [1, 1.5];

[0095] Step 6: Appearance refinement. This step is as follows Figure 5 The appearance refinement part in the 3D image will achieve the gradual refinement of the rough 3D image. It not only uses the 2D image generation model as a guide, but also increases the number of Gaussian points through the densification strategy to improve the density of the model, which helps to create more detailed features and textures on the surface of the model. The steps of appearance refinement include:

[0096] Step 61: The rough 3D image obtained in step 5 is further compared with the 2D image output by the 2D image generation module to calculate the 2D image score estimation function L SDS , thereby continuing to adjust the color, opacity and shape of the Gaussian ellipsoid; the detail optimization process adds densification technology to further seek loss with the 2D image, thereby strengthening the optimization of details.

[0097] Step 62: During the refinement process, the opacity and position are regularized by compactness densification techniques to avoid potential degradation problems. At the same time, those insignificant Gaussian points (i.e., Gaussian points with opacity below a certain threshold) are periodically removed to maintain the compactness of the model and finally output a refined 3D image. In order to control the opacity and position changes of 3D Gaussian points, a regularization loss is introduced:

[0098] L regularize =λ mean ·mean∑ i ||p i ||+λ opacity ∑ i sg(||p i ||)o i (10)

[0099] Among them, p i is the position of the i-th Gaussian point, o i represents opacity, λ mean and λ opacity is a weight parameter, and sg(·) is a function used to adjust the regularization strength of opacity according to the position gradient of the Gaussian point.

[0100] Finally, the loss function for appearance refinement can be expressed as the weighted sum of the above losses:

[0101] L refine =L SDS +λ regularize ·L regularize (11)

[0102] Among them, λ regularize It is a weight parameter used to balance the contribution of different losses. By calculating the loss, it guides the model update;

[0103] Figure 6 This is the experimental result diagram of the present invention. A three-dimensional car image is provided to a subject. By collecting EEG brain wave signals and inputting them into the 3D image reconstruction device based on EEG signals proposed by the present invention, a series of 3D car images with different orientations are output. It can be seen from the output result images that the method of the present invention reconstructs the characteristics of the 3D image better.

[0104] It should be noted that the above specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall within the protection scope of the present invention. Those skilled in the art should understand that the present invention description and its drawings are illustrative and do not constitute a limitation of the claims. The protection scope of the present invention is defined by the claims and their equivalents.

Claims

1. A high-fidelity 3D image reconstruction method based on brain EEG signals, characterized in that: The reconstruction method uses the key feature encoding of the EEG signal as a guide, combines the three-dimensional representation of the 2D image generation module and the 3D image generation module, and generates a 3D image matching the EEG signal; firstly, the EEG key feature encoding is performed, the collected EEG signal is preprocessed, and a mask signal is constructed to train the EEG encoder to obtain the EEG feature vector of the key feature; the EEG feature vector is input into the 2D image generation model to realize the high-fidelity conversion from brain activity to the 2D image, and a rough 2D image is output. At the same time, the EEG feature vector is similarity calculated with the image feature vector extracted by the CLIP image decoder, and the EEG encoder is reversely fine-tuned so that the EEG feature vector is aligned with the image text information of the CLIP in the feature space, and the camera elevation angle and azimuth angle information are used as auxiliary information; the rough 2D image is input into the 3D image generation module, and the geometric optimization is performed together through the joint loss function composed of the 2D image score function and the 3D image score function, and finally the appearance is refined through the compactness densification technology, so as to obtain a high-fidelity and detail-rich 3D image, which specifically includes: Step 1: EEG signal acquisition, by showing the target image to the subject and collecting its EEG signal; Step 2: Preprocess the EEG signal by filtering and transforming to obtain an embedding vector; Step 3: Learn the contextual information of the EEG signal by training the encoder of the EEG feature extractor to predict the masked embedding vector, including: Step 31: Input the preprocessed EEG signal into the EEG feature extractor, perform random masking with a preset masking rate, and the mask signal model will use the embedding vector to model the mask signal; Step 32: The difference between the EEG signal reconstructed by the model and the original EEG signal is measured by using the mean square error as the loss function to obtain an EEG feature vector with a dimension of 1024; Step 4: Input the EEG feature vector into the 2D image generation module for training and output a rough 2D image. The 2D image generation module is based on the SD model, including the variational autoencoder, the denoising U-Net and the EEG encoder trained in step 3. The training steps include: Step 41: Use the public ImageNet EEG dataset as a training set and the self-collected dataset as a validation set. The dataset contains both EEG signals and images corresponding to the EEG signals. Step 42: The EEG feature vector is used as conditional information and input into the cross-attention layer in the denoising U-Net inside the 2D image generation module. The cross-attention layer matches the EEG feature vector with each position in the feature map of the denoising U-Net to generate attention weights; then, the aggregated representation of the conditional information at different positions is calculated by the weights and added to the feature map of the denoising U-Net; in this way, the EEG feature vector is encoded into each layer of the denoising U-Net, guiding the 2D image generation module to generate a rough 2D image consistent with the conditional information; Step 43: performing end-to-end training on the SD model, during which the difference between the generated image and the target image is calculated using a loss function; Step 44: Fine-tune the EEG encoder to align the EEG feature vector with the feature space of the CLIP image decoder to achieve multimodal alignment of EEG signals, text, and images; Step 5: Input the rough 2D image into the 3D image generation module for geometric optimization. Use the semantic information in the EEG feature vector, the joint loss function composed of the 2D image score estimation function and the 3D image score estimation function to jointly perform geometric optimization; Step 51: First, the Gaussian position is initialized, and the pre-trained 3D diffusion model extracts semantic information from the text embedding, and generates a related 3D point cloud based on the extracted semantic information; Step 52: Input the camera pose and EEG feature vector into the 2D image generation model pre-trained in step 4, and train it through the 2D image score estimation function to optimize the color and opacity in the 3D Gaussian ellipsoid; Step 53: Calculate a 3D image score estimation function for the 3D diffusion model according to the optimization process of the 2D image generation model loss, and optimize the shape and position of the 3D Gaussian ellipsoid through the 3D image score estimation function; Step 54: Minimize the joint loss function composed of the 2D image score estimation function and the 3D image score estimation function, optimize the parameters in the 3D Gaussian ellipsoid, achieve the purpose of geometric optimization, and generate a rough 3D image that matches the output of the 2D image generation model in appearance; Step 6: Appearance refinement, which realizes the stepwise refinement of the rough 3D image, not only using the 2D image generation model as a guide, but also increasing the number of Gaussian points through a densification strategy to improve the density of the model, including: Step 61: the rough 3D image obtained in step 5 is further combined with the rough 2D image output by the 2D image generation module to calculate a 2D image score estimation function, thereby continuing to adjust the color, opacity and shape of the Gaussian ellipsoid; Step 62: During the refinement process, the opacity and position are regularized by a compactness densification technique to avoid potential degradation problems, while those insignificant Gaussian points are periodically removed, and finally a refined 3D image is output.

Images