High-quality 3D Mesh reconstruction method based on brain EEG signal

Abstract

The invention provides a high-quality 3D Mesh reconstruction method based on brain EEG (electroencephalogram) signals. The electroencephalogram signals can be quickly and efficiently converted into a high-fidelity three-dimensional mesh model. Firstly, an EEG encoder extracts time domain and frequency domain features from an EEG signal to enhance the representation capability of the signal. The time-frequency embedding is then input to a two-stage image generation module guided by the EEG, generating a high-quality two-dimensional image. And finally, through a multi-view diffusion model and a sparse view reconstruction technology, the generated two-dimensional image is used as input, and reconstruction of the three-dimensional grid is rapidly completed. According to the scheme, the technical problem of converting the EEG signal into the three-dimensional object is effectively solved, an important breakthrough is made in the fields of electroencephalogram signal processing and three-dimensional reconstruction, and a new view angle and a new tool can be provided for research in the fields of acknowledging science, psychology and neuroscience.

Description

Technical Field

[0001] The present invention relates to the field of electroencephalogram (EEG) data processing, and in particular to a high-quality 3DMesh reconstruction method based on brain EEG signals. Background Art

[0002] As one of the cutting-edge technologies in the current field of artificial intelligence, generative AI technology has been widely used in various visual synthesis tasks and has promoted the explosive growth of research in the fields of 2D and 3D creation. The Stable Diffusion Model (SD) with its powerful generation capabilities and the IPAdapter (Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models) model pair provide an efficient and flexible method for the AIGC (artificial intelligence generated content) field, making it easier for users to use pre-trained models to generate images with a specific style, greatly simplifying the image generation process and jointly promoting the development of text-to-image synthesis tasks in this field. The Imagen model achieves highly realistic text-to-image synthesis by effectively combining the two, which marks a major breakthrough in image quality and language understanding; in addition, the introduction of models such as DALL-E 2 and Muse further enriches the technical means and application scenarios of text-to-image synthesis. DALL-E 2 demonstrates its outstanding ability in understanding and generating multimodal content through its innovative architecture and training strategy.

[0003] With the continuous development of brain-computer interface technology, a large number of studies have begun to explore how to visualize abstract brain signals into understandable visual content. Brain2Image attempts to generate images seen by observers from EEG signals by combining long short-term memory networks and generative adversarial network technology; Ye et al. proposed a self-supervised cross-modal retrieval method to restore the precise visual stimulation of EEG fragments by maximizing the mutual information between EEG encoding and visual stimulation. This method not only improves the accuracy of EEG signal-to-image synthesis, but also reduces the dependence on labeled data through self-supervised learning; Lan et al. further explored methods to reconstruct visual stimulation images from EEG signals, generating high-resolution visual stimulation images through multi-level perceptual information decoding and potential diffusion models; the DreamDiffusion method proposed by Bai et al. in 2023 can generate high-quality images directly from EEG signals. This method uses a pre-trained text-to-image model and combines the CLIP image encoder to provide additional supervision for the alignment of EEG, text, and image embeddings.

[0004] Large Reconstruction Models (LRM) are a feed-forward image-to-3D model generation technique for creating 3D models from 2D images. These models benefit from the availability of large-scale 3D datasets, which enables them to train highly generalized reconstruction models. LRM demonstrates that a transformer-based backbone network can effectively map image tokens to implicit 3D triplanes and is trained with multi-view supervision. Instant3D (Efficient 3D Mesh Generation from a Single Image with Sparse-viewLarge Reconstruction Models) further extends LRM to sparse view inputs, significantly improving reconstruction quality. By combining a multi-view diffusion model, Instant3D is able to achieve highly generalized and high-quality single image to 3D model generation. Inspired by Instant3D, LGM (Large multi-view gaussian model for high-resolution 3D content creation) and GRM (Single image to 3D textured mesh with convolutional reconstruction model) replaced the three-plane neural radiation field (NeRF) representation with a three-dimensional Gaussian distribution to enjoy its superior rendering efficiency and avoid the memory-intensive volume rendering process. However, Gaussian distribution has shortcomings in explicit geometry modeling and high-quality surface extraction. Given the success of neural mesh optimization methods, parallel work such as MVD2 (Efficient multiview 3D reconstruction for multiview diffusion) and CRM choose to optimize directly on the mesh representation to achieve efficient training and high-quality geometry and texture modeling. Summary of the invention

[0005] In view of this, the present invention provides a high-quality 3D Mesh reconstruction method based on brain EEG signals. The reconstruction method uses the key feature encoding of the EEG signal as a guide, combines the 2D image generation model and the 3D Mesh reconstruction module, reconstructs the 3D Mesh corresponding to the EEG signal, and finally obtains the final result through rendering. The research on converting these brain signals into 3D images is not only more challenging in technology, but also groundbreaking in theory and application.

[0006] The method provided by the present invention passes the EEG signal through a time domain encoder and a frequency encoder to obtain time domain features and frequency features respectively. The time domain encoder adopts the time mask signal modeling technology TMSM (Temporal Masked Signal Modeling), which can extract meaningful contextual knowledge from large-scale noisy EEG signals, while the frequency encoder adopts Fast Fourier Transform (FFT), first converting the EEG signal from a time domain signal to a frequency signal, and in order to better obtain the distribution of the signal in frequency, LSTM (Long Short-Term Memory) is used to extract frequency-related features. The time-frequency embedding is input into the semantic alignment network, and semantic information is added to the time-frequency embedding through IPAdapter as a guiding condition for the 2D image generation module. At the same time, the time-frequency encoder will be fine-tuned with the EEG classifier to ensure that the EEG category information can be accurately obtained, thereby guiding the generation of a 2D image matching the EEG signal. The 2D image generation module consists of two stages. The first stage uses the semantically aligned time-frequency embedding through IPAdapter to guide Stable Diffusion through the cross-attention mechanism to generate a rough image with certain semantic information. The second stage uses the rough image generated in the first stage as a basis and uses the category information obtained by the EEG signal through the EEG classifier as a condition to obtain a more accurate 2D image. The generated 2D image is used as input to the multi-view generation module in Zero123++ after fine-tuning to obtain 6 multi-view white background images to provide guidance information for subsequent 3D Mesh reconstruction. Finally, the 6 images generated by Zero123++ are used as input to the reconstruction model, and their camera poses are fixed. The image token is obtained through the Vit Encoder of the reconstruction model, and then the triplane is obtained through the Triplane Decoder. Finally, in order to improve the reconstruction quality, the triplane is supervised by the differentiable isosurface extraction module FlexiCubes using full-resolution images and additional geometric information to obtain a smoother mesh output, and finally the final 3D object is obtained by rendering.

[0007] Compared with the prior art, the present invention has the following beneficial effects:

[0008] 1. EEG encoders are divided into time domain encoders and frequency domain encoders, which extract their own features from the time domain and frequency domain of EEG signals respectively. The time domain encoder uses the temporal mask signal modeling technology (TMSM). This method does not rely on limited EEG image paired samples, but uses a large amount of EEG data to train the encoder by predicting missing tags based on context clues. This pre-training strategy not only enhances the adaptability of the model to individual differences, but also enables the time domain encoder to deeply understand the EEG data of different populations and various brain activities through signal reconstruction in the time domain. The frequency domain encoder uses the fast Fourier transform (FFT) to first convert the EEG signal from the time domain to the frequency domain. In order to better capture the frequency distribution of the signal, the long short-term memory (LSTM) model is used to extract frequency-related features. The LSTM model has memory units (unit states) that can retain long-term contextual knowledge, enabling it to learn useful features from serialized EEG signals.

[0009] 2. The 2D image generation module solves the inevitable inaccurate alignment in the reconstruction process, which leads to some information loss and semantic noise. The present invention proposes a two-stage process to generate images as visual stimuli for EEG recordings. Due to the limitations and noise of the EEG signal itself, the images initially reconstructed using the time-frequency embedded diffusion model are of low quality and only contain basic contour and color information. Therefore, in order to enhance these reconstructed images, the rough images generated in the first stage are used as input to the diffusion model, and the EEG classification labels are used as new conditions at the same time, and finally accurate 2D images with richer detailed features are reconstructed. Provide accurate guidance for subsequent three-dimensional reconstruction.

[0010] 3. Multi-view white background generation module To solve the problem that the images generated by the 2D image generation module usually contain complex background elements, which may interfere with the reconstruction process of 3D objects, resulting in artifacts and floating elements in the final output. Therefore, it is crucial to generate multi-view images with a uniform white background. Therefore, the multi-view diffusion model in InstantMesh is used as a multi-view generation module. This model is fine-tuned by Zero123++ to produce images with consistent viewpoints with a white background, thereby ensuring stability during sparse view reconstruction.

[0011] 4. 3D Mesh Reconstruction Module In order to efficiently obtain high-quality 3D mesh objects, the Sparse-ViewLarge Reconstruction Model is used. This model uses meshes as 3D representations, while previous methods based on large reconstruction models (LRM) generate three planes, which require volume rendering to synthesize images. However, during training, volume rendering is memory intensive, which limits the use of high-resolution images and normals for supervision. In order to improve the quality of reconstruction, InstantMesh integrates a differentiable isosurface extraction module (FlexiCubes) in the reconstruction process, allowing the use of full-resolution images and supplementary geometric information for supervision. This geometric supervision helps to generate smoother meshes. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 It is a schematic diagram of the overall model structure of the present invention;

[0014] Figure 2 The overall technical route structure schematic diagram of the present invention;

[0015] Figure 3 It is a schematic diagram of the structure of the EEG encoder in the present invention;

[0016] Figure 4 It is a graph of the experimental results of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0018] The EEG of the present invention refers to: EEG (Electroencephalogram) is an electrical signal generated by the activity of brain neurons. Neurons are connected to each other through synapses to form a complex neural network. When neurons are activated, bioelectric phenomena are generated, and these electrical signals can be captured by placing electrodes on the scalp or directly implanted in the brain.

[0019] The IPAdapter of the present invention refers to: IPAdapter (Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models) is to separate the image as a prompt feature. Compared with the previous method of simply extracting image features and text features and splicing them together, IP-Adapter distinguishes the Cross-Attention of text features from the Cross-Attention of image features through an adapter module with decoupled cross attention, and adds a new Cross-Attention module in the U-Net module to introduce image features. It is equivalent to separating the original method of splicing img and prompt into one vector in SD. Both img and prompt will form vectors separately and then be handed over to the U-Net layer, so that the features in img can be better preserved, thereby achieving more explicit inheritance and preservation of image features.

[0020] The SD generation model of the present invention refers to: Stable Diffusion Model Stable Diffusion is a latent diffusion model (Latent Diffusion Model) for text-to-image generation, which converts the input text prompt words into semantic vectors through a text encoder, which are then passed to the image generator as control input. The image generator consists of a U-Net network and a sampler algorithm, which is responsible for generating image information in a low-dimensional space, and then the VAE decoder is used to increase the dimension and generate the final image. The present invention replaces the text embedding with an EEG embedding that is semantically aligned with the IPAdapter.

[0021] The multi-view generation model of the present invention refers to: the module is fine-tuned by Zero123++ to obtain a white background image with consistent view angle, ensuring stability in the sparse view reconstruction process. The azimuth, elevation and camera distance of the image reconstruction image are randomly sampled from the predefined range of Zero123++. The postures of the 6 target images consist of alternating absolute elevation angles of 20° and -10°, combined with the azimuth relative to the query image, starting from 30° and increasing by 60° for each posture.

[0022] The 3D Mesh reconstruction model of the present invention refers to: In InstantMesh, the large-scale reconstruction model of the sparse view is represented by a mesh as a 3D representation. The image is synthesized by volume rendering based on the output of triplanes. In order to improve the reconstruction quality, the differentiable isosurface extraction module (FlexiCubes) is integrated into the reconstruction module, so that full-resolution images and additional geometric information can be used for supervision, and the mesh can be made smoother through these geometric supervisions.

[0023] The triplanes of the present invention refer to: triplanes (three planes) are an efficient structure for representing three-dimensional volume data, and are widely used in three-dimensional reconstruction and generation tasks related to neural radiation fields (NeRFs). This method reduces the computational and storage burden of directly processing three-dimensional voxel grids by projecting three-dimensional scene information onto three orthogonal two-dimensional planes. When a point in three-dimensional space needs to be sampled, the system will obtain the corresponding feature information from these three planes respectively, and then infer the properties of the point (such as color, density, etc.) by fusing these features.

[0024] FlexiCubes of the present invention refers to: FlexiCubes is a differentiable isosurface extraction module for three-dimensional scene reconstruction, which is mainly used in the fields of differentiable rendering, 3D reconstruction and scene understanding. In three-dimensional space, isosurface extraction is a technology that extracts an isosurface (or isosurface) of a certain value from volume data (such as a three-dimensional scalar field). FlexiCubes provides a mechanism to enhance the representation ability of three-dimensional objects through geometric refinement, which can capture more delicate details in space. It allows the isosurface to be flexibly adjusted according to the feedback of the neural network, making the generated three-dimensional scene more accurate and detailed.

[0025] The core of this invention is to explore the cutting-edge technology of reconstructing 3D images directly from brain EEG signals. Although the current research on image generation based on brain signals is gradually increasing, most of the research focuses on generating 2D graphics from fMRI or EEG signals. The research on converting these brain signals into 3D Mesh is not only more technically challenging, but also innovative in both theory and application.

[0026] The present invention proposes a reconstruction method for converting brain EEG signals into high-quality 3D Mesh. The method passes the EEG signal through a time domain encoder and a frequency encoder to obtain time domain features and frequency features respectively. The time domain encoder adopts the time mask signal modeling technology TMSM (Temporal Masked Signal Modeling), which can extract meaningful contextual knowledge from large-scale noisy EEG signals, while the frequency encoder adopts Fast Fourier Transform (FFT). First, the EEG signal is converted from a time domain signal to a frequency signal. In order to better obtain the distribution of the signal in frequency, LSTM (Long Short-Term Memory) is used to extract frequency-related features. The time-frequency embedding is input into the semantic alignment network, and semantic information is added to the time-frequency embedding through IPAdapter as a guiding condition for the 2D image generation module. At the same time, the time-frequency encoder will be fine-tuned with the EEG classifier to ensure that the EEG category information can be accurately obtained, thereby guiding the generation of a 2D image matching the EEG signal. The 2D image generation module consists of two stages. The first stage uses the semantically aligned time-frequency embedding through IPAdapter to guide Stable Diffusion through the cross-attention mechanism to generate a rough image with certain semantic information. The second stage uses the rough image generated in the first stage as a basis and uses the category information obtained by the EEG signal through the EEG classifier as a condition to obtain a more accurate 2D image. The generated 2D image is used as input to the multi-view generation module in Zero123++ after fine-tuning to obtain 6 multi-view white background images to provide guidance information for subsequent 3D Mesh reconstruction. Finally, the 6 images generated by Zero123++ are used as input to the reconstruction model, and their camera poses are fixed. The image token is obtained through the VitEncoder of the reconstruction model, and then the triplane is obtained through the Triplane Decoder. Finally, in order to improve the reconstruction quality, the triplane is supervised by the differentiable isosurface extraction module FlexiCubes using full-resolution images and additional geometric information to obtain a smoother mesh output, and finally the final 3D object is obtained by rendering. Figure 1 It is a schematic diagram of the overall model structure of the present invention. Figure 2 The overall technical route structure diagram of the present invention is as follows: The technical solution is described in detail with reference to the accompanying drawings.

[0027] Specifically, the high-quality 3D Mesh reconstruction method based on brain EEG signals of the present invention comprises the following steps:

[0028] Step 1: EEG signal acquisition, including:

[0029] Step 11: The target image used is the ImageNet dataset, which has 40 categories, each category contains 50 images, and a total of 2000 images;

[0030] Step 12: Select 6 subjects and show each image to them for 0.5 seconds. After showing 50 images, there is a 10-second pause. While the subjects are watching the images, the 128-channel Brainvision EEG system is used to record the EEG signals of the subjects.

[0031] Step 13: Each subject viewed 2,000 images, each image corresponding to an EEG recording, so a total of 12,000 128-channel EEG signals were obtained.

[0032] Step 2: EEG signal preprocessing helps improve signal quality, reduce the complexity of model training, and ensure that the model can learn useful features from the data. The steps include:

[0033] Step 21: All EEG signals are filtered before training; the specific filtering range is 5 to 95 Hz to eliminate noise and unwanted frequency components;

[0034] Step 22: The filtered EEG signal is truncated to a preset uniform time length; here it is 500 time steps, in order to ensure that all EEG signals have a uniform processing standard in the time dimension;

[0035] Step 23: The EEG signal is a sequence of potential measurements that vary over time, so every 4 adjacent time steps are combined into a window, the EEG signal in each window is converted into a token, and each token is converted into a 1024-dimensional embedding vector through a projection layer; for subsequent mask signal modeling.

[0036] Step 3: Extract the time domain features in the EEG signal by training the time domain encoder in the EEG encoder. Figure 3 is a schematic diagram of the structure of the EEG encoder. Step 3 specifically includes:

[0037] Step 31: Input the preprocessed EEG signal into the EEG time domain encoder. The masked signal model will use the embedding vector for masked signal modeling (MSM) and perform random masking with a preset masking rate (the masking rate is 0.75).

[0038] Step 32: By using the mean square error as the loss function to measure the difference between the model-reconstructed EEG signal and the original EEG signal, the EEG time domain encoder is optimized and an EEG feature vector with a dimension of 1024 is obtained.

[0039] Step 4: Extract frequency domain features from the EEG signal by training the frequency domain encoder in the EEG encoder, including:

[0040] Step 41: In the frequency domain feature extraction process, the EEG signal is first converted from the time domain to the frequency domain through fast Fourier transform (FFT). This step enables the frequency domain information of the EEG signal to be effectively extracted as the basis for further processing;

[0041] Step 42: Considering the small size of the training data set, directly learning the frequency domain features of EEG through complex neural networks may lead to overfitting problems. Therefore, the present invention uses a long short-term memory network (LSTM) model as a frequency encoder to extract visual semantic frequency domain features in EEG signals for embedding. The LSTM model is trained by supervising the EEG signal with paired image labels and optimizing the cross entropy loss function to ensure that the extracted frequency domain features can accurately reflect the semantic information in the EEG signal.

[0042] Step 5: In order to construct a unified time-frequency feature, the present invention further jointly fine-tunes the time encoder and the frequency encoder. The specific steps include: first, the output features of the time encoder and the frequency encoder are connected to form a joint time-frequency embedding. Then, the cross-entropy loss function is used to optimize the connected output, and the image label is input into the model as a supervisory signal. Through this process, the model can make more accurate predictions of EEG signals. Finally, the predicted EEG signal label will be used in the subsequent image reconstruction stage to improve the quality and accuracy of the reconstructed image.

[0043] Step 6: Input the time-frequency joint embedding into the EEG classifier for training, so as to accurately classify the EEG signal and obtain its category prompt label, providing guidance information for the subsequent 2D image generation stage. The specific steps include:

[0044] Step 61: First, assign a one-hot code l to each masked segment through the tokenizer m , as the self-supervised label, where l m is defined as a codeword. The total number of codewords is recorded as n t A linear layer is used to transform each masked fragment into a t The probability representation of code words;

[0045] Step 62: By mClassify and get the probability representation p m , and calculate p m With l m The cross entropy error between them is used to obtain the classification target. The final classification loss L cls :

[0046] L cls =-l m ·log(pm) (1)

[0047] Through the above classification loss L cls Training is performed to obtain category labels that can accurately predict EEG and provide prompt word conditions for subsequent generation tasks.

[0048] Step 7: At the same time, since there is a lot of noise in the EEG signal, in order to enhance the semantic information of the EEG signal. The present invention inputs the time-frequency embedding obtained in step 6 into IPAdapter for semantic alignment. The picture viewed by the subject is input into IPAdapter together with the time-frequency embedding, and the time-frequency embedding and the picture are semantically aligned, so that the time-frequency embedding can obtain more accurate semantic information. In the original Stable Diffusion model, the text features extracted by the text encoder of CLIP are injected into the cross-attention layer to complete the semantic alignment of text prompts and generated images. On this basis, IPAdapter adds a new cross-attention layer, which can process EEG features, and then realize the parallel processing of EEG signals and image features. The semantically enhanced time-frequency embedding is finally obtained through step 7.

[0049] Step 8: Input the obtained time-frequency embedding into the 2D image generation module. The 2D image generation module of the present invention consists of two stages. In the first stage, the time-frequency embedding is input into the SD model as a condition, and the EEG features are integrated through the cross-attention mechanism to guide the denoising process, and finally the 2D image is obtained by VAE (Variational Auto-Encoder) decoding. Due to the complexity of the EEG signal itself and the presence of a large amount of noise, the first stage can only obtain a rough image with color and contour information; in the second stage, the rough image obtained in the first stage is used as input, and the prompt label obtained from the EEG classifier in step 6 is added during the denoising process, and the cross-attention mechanism is used to guide the conversion of the rough image into a high-fidelity 2D image. The specific steps include:

[0050] Step 81: In the first stage, the time-frequency embedding of EEG is used as conditional information and input into the cross-attention layer in the U-Net module inside the SD module. The cross-attention layer matches the time-frequency embedding with the U-Net and generates attention weights. The weights are then used to calculate the aggregated representation of the conditional information at different locations and added to the feature map of the denoising U-Net. In this way, the EEG feature vector is encoded into each layer of the U-Net, guiding the 2D image generation module to generate a 2D image consistent with the condition.

[0051] In U-Net, the specific cross-attention formula is as follows: Query (Q): the middle layer feature of the encoder; Key (K): obtained by projecting the output of the EEG encoder, used as conditional information to help the model focus on the image area related to the EEG signal; Value (V): obtained by projecting the output of the EEG encoder, providing specific content guidance required to generate the image:

[0052] Q=W(i) Q ·φ i (z t ) (2)

[0053] K=W(i) K ·τ θ (y) (3)

[0054] V=W(i) V ·τ θ (y) (4)

[0055] Among them, φ i (z t ) represents the intermediate features of the denoising U-Net at time step t; τ θ (y) represents the output of the encoder; W(i) K and W(i) V is a learnable projection matrix that maps the EEG feature vectors to a space of the same dimensionality as the U-Net features.

[0056] The mathematical expression for attention weight calculation is as follows:

[0057]

[0058] Among them, d k Represents the dimension of the key vector, which is used to scale the dot product to avoid gradient disappearance; through the cross attention mechanism, the SD model can obtain a rough graph of the first stage based on the time-frequency embedding.

[0059] Step 82: In the second stage, the rough image generated from the first stage is used as the input of the denoising process. The rough image contains preliminary structural information and color distribution, but the details are not complete and there is a certain degree of noise. The prompt label is extracted from the EEG classifier in step 6. The classification result generated by the classifier based on the EEG signal can provide prompt labels of the image viewed by the subject. These labels will serve as auxiliary information in the image generation process to guide the generated image to conform to the image generated by the subject's stimulation.

[0060] Step 821: Initialize the cross-attention mechanism to associate the cue labels obtained from the EEG classifier with the local features in the rough image. The cross-attention mechanism can capture the relationship between image pixels and cue labels and dynamically adjust the weight distribution of different regions during image refinement.

[0061] Step 822: First, the local features of the rough image are extracted through the encoder, and cross-attention mapping is performed between them and the prompt label to generate a weighted feature map. Then, the weighted feature map is input into the denoising network to gradually remove the noise in the image, while strengthening the specific areas related to the prompt label, so that the image generation process is more in line with the guidance of the prompt label. Finally, through multiple iterations of the denoising process, a high-fidelity 2D image with rich details is finally generated. Finally, through step 7, the present invention realizes the conversion of EEG signals to high-fidelity 2D images.

[0062] Step 9: Reconstructing a 3D object from a single image often requires obtaining multi-view information through a multi-view generation module to reconstruct the 3D object. Traditional multi-view image generation modules usually generate images containing complex background elements, which may interfere with the 3D object reconstruction process and cause artifacts and floating elements to appear in the final output. Therefore, the present invention obtains an image with a white background and consistent viewing angle by fine-tuning the Zero123++ model to ensure the stability of the subsequent 3D reconstruction process. The specific steps are as follows:

[0063] Step 91: The high-quality 2D image obtained in step 7 is used as input and passed through the encoder in the Zero123++ model to extract image features. And in order to generate consistent multi-view images, the model adopts a tiling strategy to tile the six views into a single image in a 3×2 layout. This layout allows the model to correctly model the joint distribution of multiple images.

[0064] Step 92: In Zero123++, the azimuth, elevation, and camera distance of the reconstructed image are randomly sampled from a predefined range. The poses of the six target images consist of alternating elevations of 20° and -10°, combined with the azimuth relative to the query image, starting from 30° and increasing by 60° for each pose, resulting in six images with different viewpoints.

[0065] Step 93: Perform the final background removal using SAM (Segment Anything Model) in Zero123++ to obtain a multi-view image with a white background. SAM uses the self-attention mechanism of Transformer to capture the global dependency between any two pixels in the image, which enables the model to better understand the image content and perform accurate segmentation. And by combining features of different scales, SAM can capture both local details and global context information. Through these techniques, multiple views of the white background are finally obtained, making it more suitable for subsequent applications.

[0066] Step 10: Input the white background multi-view obtained in step 9 as conditions into the reconstruction model to perform the final 3DMesh reconstruction. The specific steps are as follows:

[0067] Step 101: First, input the multi-view image obtained in step 9 into the reconstruction module. The ViT Encoder in the module will first divide the input image into multiple fixed-size blocks, then flatten each block into a vector, and embed it through a linear layer to map it into a high-dimensional space for extracting more features. In order to retain spatial information, a position code is added to each block, which contains the position information of the block in the original image. The embedded blocks are processed by multiple Transformer layers, each of which contains a self-attention mechanism, so that the model can capture the relationship between different parts of the image. After processing by the Transformer layer, the representation of each block is further encoded to form a feature representation of the image, which can be used for subsequent 3D reconstruction tasks.

[0068] Step 102: The output of ViT Encoder is a series of image features that need to be converted into coordinates in 3D space. The image features are then depth estimated through Triplane Decoder to convert the 2D image features into points in 3D space. The next step is to project these points onto three orthogonal planes (XY, YZ, ZX) and fuse the features extracted from the three planes to obtain comprehensive 3D information. Finally, the fused features are used to generate the Triplane, which is an implicit 3D representation that describes the entire 3D space through density and color information on three orthogonal planes (usually XY, YZ, and ZX planes).

[0069] Step 103: The process of extracting mesh from implicit 3D field Triplane using FlexiCubes is as follows: First, initialize a grid structure in 3D space, which is composed of multiple small cubes; then, for each small cube in the grid, calculate the Signed Distance Function (SDF) value of its internal point. The SDF value represents the distance from the point to the nearest surface, and the positive and negative signs represent the internal or external position of the point relative to the surface; using the SDF value, FlexiCubes can determine the isosurface inside each small cube, that is, the surface with zero SDF value. This is achieved by solving the zero level set of the SDF function; finally, connect the isosurfaces in all small cubes to form a complete 3D Mesh, and obtain the final result by rendering.

[0070] The present invention takes the key feature encoding of the EEG signal as a guide, combines the 2D image generation model and the 3D Mesh reconstruction module to reconstruct the 3D Mesh corresponding to the EEG signal, and finally obtains the final result through rendering. Figure 4 This is a diagram of the experimental results of the present invention.

[0071] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0072] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0073] It should be noted that the above specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall within the protection scope of the present invention. Those skilled in the art should understand that the present invention description and its drawings are illustrative and do not constitute a limitation of the claims. The protection scope of the present invention is defined by the claims and their equivalents.

Claims

1. A high-quality 3D Mesh reconstruction method based on brain EEG signals, characterized in that: The following steps are involved: Step 1: EEG signal acquisition; Step 2: EEG signal preprocessing helps improve signal quality, reduce the complexity of model training, and ensure that the model can learn useful features from the data; Step 3: Extract the time domain features in the EEG signal by training the time domain encoder in the EEG encoder, including: Step 31: Input the preprocessed EEG signal into the EEG time domain encoder, and the mask signal model will use the embedding vector to model the mask signal and perform random masking with a preset masking rate; Step 32: By using the mean square error as the loss function to measure the difference between the model-reconstructed EEG signal and the original EEG signal, the EEG time domain encoder is optimized and an EEG feature vector with a dimension of 1024 is obtained; Step 4: Extract frequency domain features from the EEG signal by training the frequency domain encoder in the EEG encoder; Step 5: Jointly fine-tune the time encoder and the frequency encoder, which includes: concatenating the output features of the time encoder and the frequency encoder to form a joint time-frequency embedding; then optimizing the concatenated output using the cross entropy loss function, and inputting the image label as a supervisory signal into the model; Step 6: Input the joint time-frequency embedding into the EEG classifier for training, so as to classify the EEG signal and obtain its category prompt label to provide guidance information for the subsequent 2D image generation stage. The specific steps include: Step 61: First, assign a one-hot code l to each masked segment through the tokenizer m , as the self-supervised label, where l m is defined as a codeword, and the total number of codewords is recorded as n t ; Use a linear layer to transform each masked fragment into a t The probability representation of code words; Step 62: By m Classify and get the probability representation p m , and calculate p m With l m The cross entropy error between them is used to obtain the classification target, and the final classification loss is L cls : L cls =-l m ·log(pm) (1) Through the above classification loss L cls Conduct training to obtain category labels that can accurately predict EEG and provide prompt word conditions for subsequent generation tasks; Step 7: Input the time-frequency embedding obtained in step 6 into IPAdapter for semantic alignment to obtain the semantically enhanced time-frequency embedding; Step 8: Input the obtained time-frequency embedding into the 2D image generation module, which includes the first and second stages. In the first stage, the time-frequency embedding is input as a condition into the SD model, and the EEG features are integrated through the cross-attention mechanism to guide the denoising process. Finally, the 2D image is obtained through VAE decoding. Due to the complexity of the EEG signal itself and the presence of a large amount of noise, the first stage obtains a rough image with color and contour information. In the second stage, the rough image obtained in the first stage is used as input, and the prompt label obtained from the EEG classifier in step 6 is added to the denoising process. The cross-attention mechanism is used to guide the conversion of the rough image into a high-fidelity 2D image. Step 9: Reconstructing a 3D object from a single image requires obtaining multi-view information through a multi-view generation module to reconstruct the 3D object. The image with a white background and consistent view is obtained by fine-tuning the Zero123++ model. The specific steps are as follows: Step 91: The high-quality 2D image obtained in step 7 is used as input and passed through the encoder in the Zero123++ model to extract image features; and in order to generate consistent multi-view images, the model adopts a tiling strategy to tile the six views into a single image in a 3×2 layout; Step 92: In Zero123++, the azimuth, elevation, and camera distance of the reconstructed image are randomly sampled from a predefined range; the poses of the six target images consist of alternating elevations of 20° and -10°, combined with the azimuth relative to the query image, starting from 30° and increasing by 60° for each pose, resulting in six images with different viewpoints; Step 93: Perform the final background removal using SAM in Zero123++ to obtain a multi-view image with a white background. SAM uses the Transformer's self-attention mechanism to capture the global dependency between any two pixels in the image, which enables the model to better understand the image content and perform accurate segmentation. By combining features of different scales, SAM can capture both local details and global context information. Through these techniques, multiple views of a white background are finally obtained. Step 10: Input the white background multi-view obtained in step 9 as conditions into the reconstruction model, perform the final 3D Mesh reconstruction, and obtain the final result through rendering.

2. The method according to claim 1, characterized in that In step 1, specifically include: Step 11: The target image used is the ImageNet dataset, which has 40 categories, each category contains 50 images, and a total of 2000 images; Step 12: Select 6 subjects and show each image to them for 0.5 seconds. After showing 50 images, there is a 10-second pause. While the subjects are watching the images, the 128-channel Brainvision EEG system is used to record the EEG signals of the subjects. Step 13: Each subject viewed 2,000 images, each image corresponding to an EEG recording, so a total of 12,000 128-channel EEG signals were obtained.

3. The method according to claim 2, characterized in that Step 2 includes: Step 21: All EEG signals are filtered before training; the specific filtering range is 5 to 95 Hz to eliminate noise and unwanted frequency components; Step 22: The filtered EEG signal is truncated to a preset uniform time length, thereby ensuring that all EEG signals have a uniform processing standard in the time dimension; Step 23: Every 4 adjacent time steps are combined into a window, the EEG signal in each window is converted into a token, and each token is converted into a 1024-dimensional embedding vector through a projection layer for subsequent mask signal modeling.

4. The method according to claim 3, characterized in that: Step 4 specifically includes: Step 41: Convert the EEG signal from the time domain to the frequency domain by fast Fourier transform; Step 42: Use a long short-term memory network LSTM model as a frequency encoder to extract visual semantic frequency domain feature embedding in the EEG signal; train the LSTM model by supervised training the EEG signal with paired image labels and optimizing the cross entropy loss function so that the extracted frequency domain features can accurately reflect the semantic information in the EEG signal.

5. The method according to claim 4, characterized in that Step 8 includes: Step 81: In the first stage, the time-frequency embedding of EEG is used as conditional information and input into the cross-attention layer in the U-Net module inside the SD module. The cross-attention layer matches the time-frequency embedding with the U-Net and generates attention weights. The weights are then used to calculate the aggregated representation of the conditional information at different locations and added to the feature map of the denoising U-Net. In this way, the EEG feature vector is encoded into each layer of the U-Net, guiding the 2D image generation module to generate a 2D image consistent with the condition. In U-Net, the specific cross-attention formula is as follows: Q is the middle layer feature of the encoder; K is obtained by projecting the output of the EEG encoder, which is used as conditional information to help the model focus on the image area related to the EEG signal; V is obtained by projecting the output of the EEG encoder, which provides specific content guidance required to generate the image: Q=W(i) Q ·φ i (z t ) (2) K=W(i) K ·τ θ (y) (3) V=W(i) V ·τ θ (y) (4) Among them, φ i (z t ) represents the intermediate features of the denoising U-Net at time step t; τ θ (y) represents the output of the encoder; W(i) K and W(i) V is a learned projection matrix that maps the EEG feature vector to a space with the same dimensions as the U-Net feature; The mathematical expression for attention weight calculation is as follows: Among them, d k Represents the dimension of the key vector, which is used to scale the dot product to avoid gradient disappearance. Through the cross-attention mechanism, the SD model can obtain a rough map of the first stage based on the time-frequency embedding. Step 82: In the second stage, the rough image generated in the first stage is used as the input of the denoising process; the cue labels are extracted from the EEG classifier in step 6; the classification result generated by the classifier based on the EEG signal can provide cue labels of the image viewed by the subject, and these labels will serve as auxiliary information in the image generation process to guide the generated image to conform to the image generated by the subject's stimulation; Step 821: Initialize a cross-attention mechanism to associate the cue labels obtained from the EEG classifier with the local features in the rough image; the cross-attention mechanism can capture the relationship between image pixels and cue labels and dynamically adjust the weight distribution of different regions during the image refinement process; Step 822: First, the local features of the rough image are extracted through the encoder, and cross-attention mapping is performed between them and the prompt label to generate a weighted feature map; then, the weighted feature map is input into the denoising network to gradually remove the noise in the image, while strengthening the specific areas related to the prompt label, so that the image generation process is more in line with the guidance of the prompt label; finally, through multiple iterations of the denoising process, a high-fidelity 2D image with rich details is finally generated; finally, the conversion of the EEG signal to a high-fidelity 2D image is realized through step 7.

6. The method according to claim 5, characterized in that Step 10 includes: Step 101: First, the multi-view image obtained in step 9 is input into the reconstruction module; the ViT Encoder in the module will first divide the input image into multiple fixed-size blocks, then flatten each block into a vector, and embed it through a linear layer to map it to a high-dimensional space for extracting more features; in order to retain spatial information, a position code is added to each block, which contains the position information of the block in the original image; the embedded blocks are processed by multiple Transformer layers, each layer contains a self-attention mechanism, so that the model can capture the relationship between different parts of the image; after being processed by the Transformer layer, the representation of each block is further encoded to form a feature representation of the image, which can be used for subsequent 3D reconstruction tasks; Step 102: The output of ViT Encoder is a series of image features, which need to be converted into coordinates in 3D space; then the image features are depth estimated by Triplane Decoder to convert the 2D image features into points in 3D space; the next step is to project these points onto three orthogonal planes and fuse the features extracted from the three planes to obtain comprehensive 3D information; finally, the fused features are used to generate Triplane, where Triplane is an implicit 3D representation that describes the entire 3D space through density and color information on three orthogonal planes; Step 103: The process of using FlexiCubes to extract mesh from the implicit 3D field Triplane is as follows: first, initialize a grid structure in 3D space, which is composed of multiple small cubes; then, for each small cube in the grid, calculate the SDF value of its internal points; the SDF value represents the distance from the point to the nearest surface, and the positive and negative signs represent the internal or external position of the point relative to the surface; using the SDF value, FlexiCubes can determine the isosurface inside each small cube, that is, the surface with an SDF value of zero; finally, connect the isosurfaces in all the small cubes to form a complete 3DMesh, and obtain the final result through rendering.

Images