A method and system for brain television reconstruction in conjunction with multi-modal alignment of speech

Abstract

This application provides a method and system for EEG-visual reconstruction using joint speech for multimodal alignment. By jointly training and aligning speech signals with EEG signals and image signals, the utilization rate of multimodal information is significantly improved and the quality of EEG signals is optimized. Compared with the existing single-modal reconstruction method based on EEG-image pair training, it can significantly improve the quality of image reconstruction.

Description

Technical Field

[0001] This application belongs to the field of interdisciplinary technology of artificial intelligence and neuroscience, specifically relating to a method and system for EEG visual reconstruction by combining speech for multimodal alignment, and particularly relating to a technology for optimizing EEG decoding and image generation by training with multimodal information of speech, image and EEG signals. Background Technology

[0002] Reconstructing visual images based on electroencephalogram (EEG) signals is an important research direction in neuroscience and artificial intelligence. Its goal is to reconstruct the visual stimuli perceived by subjects by decoding brain activity. Current techniques primarily use paired EEG-image data of a single visual modality (such as an image) to reconstruct visual images from EEG signals, such as using stable diffusion models or generative adversarial networks (GANs) for end-to-end training of EEG signals. However, such methods suffer from the following significant drawbacks:

[0003] 1. Modal singularity: It relies solely on visual stimuli and ignores the complexity of multimodal information such as speech and text in real-world scenarios, resulting in insufficient utilization of multimodal features in EEG signals. For example, although existing methods such as DreamDiffusion align EEG and image features through the CLIP model, they do not consider the synergistic effect of auxiliary modalities such as speech, which limits the efficiency of information utilization.

[0004] 2. Insufficient information alignment: The spatiotemporal characteristics of EEG signals differ significantly from the latent space of text or images in the pre-trained model, making it difficult to achieve effective alignment through direct fine-tuning;

[0005] 3. Limited robustness: Models trained on a single modality are susceptible to noise interference, resulting in unstable reconstructed image quality that is difficult to meet the needs of practical applications.

[0006] Therefore, a new technical solution is needed to overcome the above-mentioned technical problems, so as to perform visual reconstruction of EEG signals based on multimodal information and improve the quality of the reconstructed image. Summary of the Invention

[0007] A method and system for EEG visual reconstruction using multimodal alignment via joint speech is provided to address the aforementioned technical problems, thereby optimizing EEG signal representation through joint alignment of multimodal information and improving the quality and stability of the reconstructed image.

[0008] On one hand, embodiments of this application provide a method for EEG-visual reconstruction using combined speech for multimodal alignment, characterized by comprising the following steps:

[0009] S1. Select audio and video with a certain duration as external stimuli, and simultaneously collect EEG signals, video frame images, and speech data;

[0010] S2. Video frames and EEG signal segments are extracted at preset time intervals, manually screened and labeled to form EEG-image-audio pairs, and a time-series correlated multimodal dataset is obtained.

[0011] S3. Using a masked signal modeling pre-trained encoder, the spatiotemporal features of the EEG signal are extracted to generate an EEG embedding vector;

[0012] S4. Extract speech features using the HuBERT speech recognition model and reduce the dimensionality to CLIP semantic space using a GRU network to generate speech embedding vectors.

[0013] S5. Use CLIP to extract image features and generate image embedding vectors;

[0014] S6. In the CLIP semantic space, the cross-modal distance of the EEG embedding vector, CLIP image embedding vector, and speech embedding vector is jointly optimized by the cosine similarity loss function to align the EEG signal with the semantic representation of the image and speech modalities.

[0015] S7. Input the optimized EEG signal into the image generator for visual reconstruction to obtain the generated image;

[0016] S8. Calculate the similarity between the original image and the generated image in the CLIP semantic space to evaluate the visual reconstruction effect.

[0017] Specifically, step S2, which involves manual screening and category labeling to form EEG-image-audio pairs, includes:

[0018] Select video frames whose content belongs to the same category within n preset time intervals;

[0019] Audio segments and EEG signal segments corresponding to the n preset time intervals are extracted to form EEG-image-audio pairs, and the categories are labeled according to the description content of the video frames.

[0020] Specifically, step S6, which involves jointly optimizing the cross-modal distance of the EEG embedding vector, CLIP image embedding vector, and speech embedding vector using a cosine similarity loss function, includes:

[0021] Calculate the cosine similarity between the EEG embedding vector and the image embedding vector, and the cosine similarity between the EEG embedding vector and the speech embedding vector, respectively.

[0022] The overall loss function is calculated based on the average of the two values, and the parameters of the EEG feature encoder are updated in reverse.

[0023] Specifically, step S8 includes:

[0024] Select N pairs of original images and generated images, and calculate the semantic similarity between the original image and the generated image in each pair.

[0025] Calculate the average of N semantic similarities as the CLIP semantic similarity.

[0026] Step S8 further includes:

[0027] The degree of speech enhancement is calculated based on semantic similarity.

[0028] The degree of speech enhancement is calculated based on the CLIP semantic similarity of the generated image with speech enhancement and the CLIP semantic similarity of the generated image without speech enhancement.

[0029] Step S8 further includes:

[0030] The speech enhancement ratio is calculated based on semantic similarity.

[0031] The speech enhancement ratio is based on the number of speech-related video frames in the test set and the number of video frames in the test set whose CLIP semantic similarity is higher than that of the corresponding CLIP semantic similarity without speech enhancement.

[0032] In step S7, a finely tuned StableDiffusion is used as the image generator.

[0033] On the other hand, embodiments of this application provide a brainwave-visual reconstruction system for multimodal alignment using combined speech, characterized in that the system comprises:

[0034] The data acquisition module is used to select audio and video of a certain duration as external stimuli and simultaneously acquire EEG signals, video frame images, and speech data.

[0035] The dataset construction module is used to extract video frames and EEG signal segments at preset time intervals, perform manual screening and category labeling, and form EEG-image-audio pairs to obtain a time-series correlated multimodal dataset;

[0036] The embedding vector calculation module is used to extract the spatiotemporal features of EEG signals from a pre-trained encoder modeled with masking signals, and generate EEG embedding vectors.

[0037] Speech features are extracted using the Hubert speech recognition model and then reduced to the CLIP semantic space using a GRU network to generate speech embedding vectors.

[0038] Image features are extracted using CLIP to generate image embedding vectors;

[0039] The semantic representation alignment module is used to jointly optimize the cross-modal distance of EEG embedding vectors, CLIP image embedding vectors, and speech embedding vectors in the CLIP semantic space through a cosine similarity loss function, so as to align the semantic representations of EEG signals with those of image and speech modalities.

[0040] The visual reconstruction module inputs the optimized EEG signals into the image generator for visual reconstruction to obtain the generated image.

[0041] The reconstruction effect evaluation module is used to calculate the similarity between the original image and the generated image in the CLIP semantic space and evaluate the visual reconstruction effect.

[0042] The beneficial effects of the embodiments in this application compared with the prior art are:

[0043] This invention provides a method and system for EEG-visual reconstruction using combined speech signals for multimodal alignment. By jointly training and aligning speech signals with EEG signals and image signals, the utilization rate of multimodal information is significantly improved and the quality of EEG signals is optimized. Compared with existing single-modal reconstruction methods based on EEG-image pair training, this invention can significantly improve the quality of image reconstruction, providing new ideas and technical inspiration for multimodal EEG reconstruction research. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0045] Figure 1 This is a flowchart of a method for EEG-visual reconstruction based on combined speech for multimodal alignment, provided in an embodiment of this application.

[0046] Figure 2 This is a schematic diagram of an EEG-visual reconstruction system for multimodal alignment using combined speech, provided in an embodiment of this application. Detailed Implementation

[0047] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0048] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0049] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0050] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0051] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0052] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0053] Example 1

[0054] Figure 1 The flowchart shown is a method for EEG-visual reconstruction using combined speech for multimodal alignment, as provided in an embodiment of this application. Figure 1 As shown, the method specifically includes:

[0055] S1. Select audio and video with a certain duration as external stimuli, and simultaneously collect EEG signals, video frame images, and speech data;

[0056] Among them, when collecting signal data, synchronous signals of the same subject within the same time period are collected;

[0057] The audio and video signals that serve as external stimuli can be, for example, 5 minutes of popular science content; however, they can also be any other type of audio and video signal, and this invention does not impose any specific limitations on them.

[0058] S2. Video frames and EEG signal segments are extracted at preset time intervals, manually screened and labeled to form EEG-image-audio pairs, and a time-series correlated multimodal dataset is obtained.

[0059] The preset time interval is 0.5s;

[0060] The aforementioned manual screening and category labeling to form EEG-image-audio pairs specifically includes:

[0061] Select video frames whose content belongs to the same category within n preset time intervals;

[0062] For example, select five video frames whose content belongs to the same category within five preset time intervals;

[0063] This is because visual information reception time is usually short, such as 0.5 seconds, while speech has a delayed characteristic. It takes longer to describe the same scene using speech. Therefore, in order to align the semantic information of video frames and speech, and make the content described by speech consistent with the content described by video frames to increase the quality of reconstruction, we first acquire video frames whose content belongs to the same category within multiple preset time intervals, such as n=5, and then make these video frames share the same audio segment, thereby achieving alignment between audio and video frame content.

[0064] Audio segments and EEG signal segments corresponding to the n preset time intervals are extracted to form EEG-image-audio pairs, and the categories are labeled according to the description content of the video frames.

[0065] For example, when n=5, then 2.5 audio segments and EEG signal segments are extracted and combined with the corresponding video frames to form an EEG-image-audio pair;

[0066] The resulting time-series correlated multimodal dataset can be represented as follows: Where D is the multimodal dataset, Let i represent the EEG dataset, video frame set, and audio set, respectively, and let i be the number of EEG-image-audio pairs in the dataset.

[0067] S3. Using a masked signal modeling pre-trained encoder, the spatiotemporal features of the EEG signal are extracted to generate an EEG embedding vector;

[0068] S4. Extract speech features using the HuBERT speech recognition model and reduce the dimensionality to CLIP semantic space using a GRU network to generate speech embedding vectors.

[0069] S5. Use CLIP to extract image features and generate image embedding vectors;

[0070] S6. In the CLIP semantic space, the cross-modal distance of the EEG embedding vector, CLIP image embedding vector, and speech embedding vector is jointly optimized by the cosine similarity loss function to align the EEG signal with the semantic representation of the image and speech modalities.

[0071] Specifically, the cross-modal distance of EEG embedding vectors, CLIP image embedding vectors, and speech embedding vectors is jointly optimized using a cosine similarity loss function, including:

[0072] Calculate the cosine similarity between the EEG embedding vector and the image embedding vector, and the cosine similarity between the EEG embedding vector and the speech embedding vector, respectively.

[0073] The overall loss function is calculated based on the average of the two values, and the parameters of the EEG feature encoder are updated in reverse.

[0074] Specifically, the loss function is calculated using the following formula:

[0075]

[0076] Among them, E eeg E sp E img These are the EEG embedding vector, speech embedding vector, and image embedding vector calculated through steps S3-S5, respectively.

[0077] S7. Input the optimized EEG signal into the image generator for visual reconstruction to obtain the generated image;

[0078] Specifically, a fine-tuned StableDiffusion is used as the image generator, and the loss function is:

[0079]

[0080] Where ∈ θ It is the denoising function of UNet.

[0081] S8. Calculate the similarity between the original image and the generated image in the CLIP semantic space to evaluate the visual reconstruction effect.

[0082] Specifically, step S8 includes:

[0083] Select N pairs of original images and generated images, and calculate the semantic similarity between the original image and the generated image in each pair.

[0084] Calculate the average of N semantic similarities as the CLIP semantic similarity.

[0085] Specifically, the CLIP semantic similarity is represented as follows:

[0086]

[0087] Among them, E0 and E i These represent CLIP visual embeddings of the original image and the generated image, respectively;

[0088] The SCS score is limited to between 0 and 100. The closer the score is to 100, the more semantically consistent the generated image is with the original image.

[0089] Furthermore, to better compare the image reconstruction quality before and after adding speech, this invention proposes two indicators: "speech enhancement degree" and "speech enhancement ratio," specifically:

[0090] Calculate speech enhancement level based on semantic similarity;

[0091] The degree of speech enhancement is calculated based on the CLIP semantic similarity of the generated image with speech enhancement and the CLIP semantic similarity of the generated image without speech enhancement.

[0092] Specifically, the formula for calculating the degree of speech enhancement is as follows:

[0093]

[0094] in, and represents the CLIP semantic similarity of the generated image with speech enhancement and the CLIP semantic similarity of the generated image without speech enhancement, respectively.

[0095] The speech enhancement ratio is calculated based on semantic similarity; the speech enhancement ratio is based on the number of speech-related video frames in the test set, and the number of video frames in the test set whose CLIP semantic similarity is higher than the CLIP semantic similarity of the corresponding non-speech-enhanced video frames.

[0096] Specifically, the formula for calculating the speech enhancement ratio is as follows:

[0097]

[0098] in, This indicates the number of video frames in the test set that are related to speech. This indicates the number of video frames in the test set whose CLIP semantic similarity is higher than the CLIP semantic similarity of the corresponding video frames without speech enhancement.

[0099] The specific code implementation methods for feature extraction, model implementation, and training in the above steps can all be implemented using the deep learning framework PyTorch.

[0100] The method of this invention significantly improves the quality and stability of reconstructed images by introducing a speech processing module on the basis of the DreamDiffusion model and using CLIP semantic space to jointly align EEG, image and speech features.

[0101] Example 2

[0102] Figure 2 This application illustrates an EEG-visual reconstruction system for multimodal alignment using combined speech, as provided in an embodiment of this application. Figure 2 As shown, the EEG-visual reconstruction system 20, which combines speech for multimodal alignment, specifically includes:

[0103] Data acquisition module 21 is used to select audio and video with a certain duration as external stimuli and simultaneously acquire EEG signals, video frame images, and voice data;

[0104] The dataset construction module 22 is used to extract video frames and EEG signal segments at preset time intervals, perform manual screening and category labeling, form EEG-image-audio pairs, and obtain a time-series associated multimodal dataset;

[0105] Embedded vector calculation module 23 is used to extract the spatiotemporal features of EEG signals from a pre-trained encoder modeled with masking signals and generate EEG embedded vectors.

[0106] Speech features are extracted using the Hubert speech recognition model and then reduced to the CLIP semantic space using a GRU network to generate speech embedding vectors.

[0107] Image features are extracted using CLIP to generate image embedding vectors;

[0108] The semantic representation alignment module 24 is used to jointly optimize the cross-modal distance of the EEG embedding vector, CLIP image embedding vector, and speech embedding vector in the CLIP semantic space through the cosine similarity loss function, so as to align the semantic representation of the EEG signal with the image and speech modalities.

[0109] The visual reconstruction module 25 inputs the optimized EEG signal into the image generator to perform visual reconstruction and obtain the generated image.

[0110] The reconstruction effect evaluation module 26 is used to calculate the similarity between the original image and the generated image in the CLIP semantic space and evaluate the visual reconstruction effect.

[0111] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for EEG-visual reconstruction using combined speech for multimodal alignment, characterized in that, Includes the following steps: S1. Select audio and video with a certain duration as external stimuli, and simultaneously collect EEG signals, video frame images, and speech data; S2. Video frames and EEG signal segments are extracted at preset time intervals, manually screened and labeled to form EEG-image-audio pairs, resulting in a temporally correlated multimodal dataset; S3. Using a masked signal modeling pre-trained encoder, the spatiotemporal features of the EEG signal are extracted to generate an EEG embedding vector; S4. Extract speech features using the HuBERT speech recognition model and reduce the dimensionality to CLIP semantic space using a GRU network to generate speech embedding vectors. S5. Use CLIP to extract image features and generate image embedding vectors; S6. In the CLIP semantic space, the cross-modal distance of the EEG embedding vector, CLIP image embedding vector, and speech embedding vector is jointly optimized by the cosine similarity loss function to align the EEG signal with the semantic representation of the image and speech modalities. S7. Input the optimized EEG signal into the image generator for visual reconstruction to obtain the generated image; S8. Calculate the similarity between the original image and the generated image in the CLIP semantic space to evaluate the visual reconstruction effect; The manual screening and category labeling described in step S2 to form EEG-image-audio pairs specifically includes: Select video frames whose content belongs to the same category within n preset time intervals; Audio segments and EEG signal segments corresponding to the n preset time intervals are extracted to form EEG-image-audio pairs, and the categories are labeled according to the description content of the video frames; specifically, this includes making the video frames of the n preset time intervals share the same audio segment as the corresponding audio segment.

2. The method as described in claim 1, characterized in that, Step S6, which involves jointly optimizing the cross-modal distance of the EEG embedding vector, CLIP image embedding vector, and speech embedding vector using a cosine similarity loss function, specifically includes: Calculate the cosine similarity between the EEG embedding vector and the image embedding vector, and the cosine similarity between the EEG embedding vector and the speech embedding vector, respectively. The overall loss function is calculated based on the average of the two values, and the parameters of the EEG feature encoder are updated in reverse.

3. The method as described in claim 1, characterized in that, Step S8 specifically includes: Select N pairs of original images and generated images, and calculate the semantic similarity between the original image and the generated image in each pair. Calculate the average of N semantic similarities as the CLIP semantic similarity.

4. The method as described in claim 3, characterized in that, Step S8 further includes: The degree of speech enhancement is calculated based on semantic similarity.

5. The method as described in claim 4, characterized in that, The degree of speech enhancement is calculated based on the CLIP semantic similarity of the generated image with speech enhancement and the CLIP semantic similarity of the generated image without speech enhancement.

6. The method as described in claim 3, characterized in that, Step S8 further includes: The speech enhancement ratio is calculated based on semantic similarity.

7. The method as described in claim 6, characterized in that, The speech enhancement ratio is based on the number of speech-related video frames in the test set, and the number of video frames in the test set whose CLIP semantic similarity is higher than the CLIP semantic similarity of the corresponding non-speech-enhanced video frames.

8. The method as described in claim 1, characterized in that, In step S7, the finely tuned StableDiffusion is used as the image generator.

9. A brainwave-visual reconstruction system for multimodal alignment using combined speech, characterized in that, The system includes: The data acquisition module is used to select audio and video of a certain duration as external stimuli and simultaneously acquire EEG signals, video frame images, and speech data. The data acquisition and construction module is used to extract video frames and EEG signal segments at preset time intervals, perform manual screening and category labeling, form EEG-image-audio pairs, and obtain a time-series correlated multimodal dataset; The embedding vector calculation module is used to extract the spatiotemporal features of EEG signals from a pre-trained encoder modeled with masking signals, and generate EEG embedding vectors. Speech features are extracted using the Hubert speech recognition model and then reduced to the CLIP semantic space using a GRU network to generate speech embedding vectors. Image features are extracted using CLIP to generate image embedding vectors; The semantic representation alignment module is used to jointly optimize the cross-modal distance of EEG embedding vectors, CLIP image embedding vectors, and speech embedding vectors in the CLIP semantic space through a cosine similarity loss function, so as to align the semantic representations of EEG signals with those of image and speech modalities. The visual reconstruction module inputs the optimized EEG signals into the image generator for visual reconstruction to obtain the generated image. The reconstruction effect evaluation module is used to calculate the similarity between the original image and the generated image in the CLIP semantic space to evaluate the visual reconstruction effect. The manual screening and category labeling process described in the data acquisition and construction module to form EEG-image-audio pairs specifically includes: Select video frames whose content belongs to the same category within n preset time intervals; Audio segments and EEG signal segments corresponding to the n preset time intervals are extracted to form EEG-image-audio pairs, and the categories are labeled according to the description content of the video frames; specifically, this includes making the video frames of the n preset time intervals share the same audio segment as the corresponding audio segment.

Images