A high-quality video reconstruction method based on EEG signals

Abstract

The present invention relates to the field of image processing technology, and specifically discloses a high-quality video reconstruction method based on electroencephalogram (EEG) signals. The method comprises the following steps: preprocessing EEG data, downsampling video frames, and segmenting EEG data using a sliding window technique; selecting channels with high correlation with video reconstruction; extracting EEG temporal features; extracting EEG spatiotemporal features; extracting EEG time-frequency features; fusing EEG time-space-frequency features; extracting EEG categories and semantic information; aligning EEG-text-image features using a contrastive learning method; generating reference video frames using a stable diffusion model; and generating high-quality video using a video diffusion model. The method further deeply extracts EEG feature information, significantly enhancing the semantic content of the video. At the same time, the introduced video diffusion model gradually constructs a clear and natural picture through a step-by-step denoising process, significantly improving the consistency between video frames, thereby generating a high-quality video with rich details.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a high-quality video reconstruction method based on electroencephalogram (EEG) signals. Background Art

[0002] The human visual system is a highly intelligent information processing mechanism. The brain processes vast amounts of visual input daily, perceiving and understanding this information through complex neural processes. Advances in deep learning technology have enabled scientists to extract visual information from patterns of brain activity and reconstruct visual stimuli such as videos. This technology holds enormous potential and has a significant impact on the development of brain-computer interfaces (BCIs). The core goal of BCI technology is to establish a direct bridge between the human brain and external devices, enabling people to control devices such as computers and robotic arms through simple brain activity or slight muscle movements. This is particularly important for restoring communication and mobility for people with disabilities, and is also crucial for enhancing the efficiency and experience of interaction for people without disabilities.

[0003] Recent research on reconstructing videos from brain activity has largely relied on functional magnetic resonance imaging (fMRI) technology. However, fMRI technology measures changes in hemodynamics, a relatively slow process that prevents it from capturing rapid changes in brain activity and results in low temporal resolution, making it inherently disadvantageous for tasks requiring high temporal resolution, such as video reconstruction. Furthermore, fMRI data acquisition faces high equipment costs, complex operational procedures, and the need for highly specialized skills. These factors make the effective acquisition and analysis of fMRI data a complex and expensive undertaking, often requiring specialized technicians with in-depth domain knowledge and extensive practical experience, hindering the widespread use of this research. Furthermore, traditional research methods often extract video information directly from brain activity, resulting in insufficient semantic information capture, resulting in poor temporal consistency in the generated videos, choppy video, and artifacts.

[0004] To address these issues, the present invention uses a multimodal fusion video reconstruction method to reconstruct videos based on EEG data. This method preprocesses the EEG data; then, it fuses the EEG's temporal, spatial, and frequency domain features to comprehensively characterize brain activity, providing richer information for subsequent video reconstruction. Finally, it leverages contour and depth information to enhance temporal consistency in the video, generating high-quality videos. This method optimizes brain signal extraction, effectively captures the video's temporal and spatial information, and produces more informative videos. Summary of the Invention

[0005] To solve the problems existing in the prior art, the present invention provides a high-quality video reconstruction method based on EEG signals, which optimizes the extraction of brain signals, effectively captures the spatiotemporal information of the video, generates a video with richer information, and can achieve high-quality video reconstruction, solving the problems mentioned in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a high-quality video reconstruction method based on EEG signals, comprising the following steps:

[0007] S1. Preprocess the EEG data, downsample the video frames, and segment the EEG data using the sliding window technique;

[0008] S2. Select channels with high correlation with video reconstruction: Use the common spatial pattern (CSP) technique to spatially filter the EEG data to extract signal components related to specific tasks or stimuli, and combine it with the attention mechanism to select channels with high correlation with video reconstruction;

[0009] S3, using the masked coding method and the masked coding MAE model to extract EEG temporal features;

[0010] S4. Use the spatiotemporal attention mechanism to capture the correlation and importance between different time points and spatial positions and extract EEG spatiotemporal features;

[0011] S5. Use wavelet transform method to extract frequency domain information of EEG data, and use convolutional neural network to extract time domain features of EEG data to generate EEG time-frequency features;

[0012] S6. Using an adaptive fusion method, the EEG temporal features, spatiotemporal features, and time-frequency features are fused to generate EEG time-space-frequency features;

[0013] S7, using EEG classifier to extract video category information, using BLIP model to extract video semantic information, and using semantic fusion network SF-Net to fuse these two kinds of information and generate text features;

[0014] S8, using contrastive learning method to align EEG-text-image features to obtain optimized EEG features;

[0015] S9. Using the generated optimized EEG features as a condition, a stable diffusion model is used to generate a reference video frame; using the generated reference video frame as an auxiliary condition, a video diffusion model is used to generate a high-quality video.

[0016] Preferably, in step S1, it specifically includes: preprocessing the EEG data, the preprocessing steps include locating channel data, deleting useless electrodes, re-referencing, bandpass filtering, notch filtering and independent principal component analysis, so as to ensure the quality of EEG data; then downsampling the video to 3fps; and then using a fixed-length sliding window to segment the EEG data, and each window is processed as a time frame.

[0017] Preferably, in step S2, the EEG signal is spatially filtered using the common spatial pattern (CSP) technique, and the attention mechanism is applied to the relevant channels selected by the CSP method. The features of these channels are weighted by the learned weight vector to extract the signal components related to the specific task or stimulus. The objective function of the CSP spatial filter ω is as follows:

[0018]

[0019] in, represents the EEG data of the i-th channel, TS is all time points, AC is the number of all channels; C i is the spatial covariance matrix of the EEG data of the i-th channel; ω represents the spatial filter, and T represents the transpose of the matrix.

[0020] Preferably, in step S3, a masked coding method is used to extract the time domain features of the EEG data. The EEG data of the channel selected in step S2 is input into the masked coding MAE model, the EEG data is masked by 75%, and then information is extracted from the unmasked EEG data to reconstruct the EEG data. Finally, the predicted EEG data is compared with the original EEG data. The specific calculation formula of the loss function is as follows:

[0021]

[0022] Where N represents the number of samples, y i ' represents the predicted value generated by the MAE model, y i Represents a primitive value.

[0023] Preferably, in step S4, the EEG data of the channel selected in step S2 is input into a spatiotemporal feature extraction network;

[0024] First, the EEG data passes through a temporal convolution layer to capture the dynamic changes of the EEG data in the temporal dimension, thereby extracting temporal features. Then, a deep Transformer model is used to model the EEG data in the spatial dimension, thereby extracting the spatial information of the EEG data, including the associations and interactions between different brain regions. Finally, the extracted features are input into a spatial convolution model to further fuse the temporal and spatial information to obtain the spatiotemporal features of the EEG data.

[0025] Preferably, in step S5, the EEG data frequency domain features are extracted from the EEG data corresponding to the selected channels with high correlation with the video reconstruction task based on the wavelet transform method, and the EEG data time domain features are extracted from the EEG data corresponding to the selected channels with high correlation with the video reconstruction task based on the convolutional neural network. The extracted EEG data frequency domain features and EEG data time domain features are then feature fused to obtain EEG time-frequency features. The specific formula of the wavelet transform is as follows:

[0026]

[0027] in, Represents the data of all channels on the EEG at time t, a represents the scale factor, b represents the translation factor, γ * (t) represents the complex conjugate of the wavelet function, and dt represents the integral with respect to t.

[0028] Preferably, in step S6, the method specifically includes: first, using an adaptive feature fusion method to fuse the EEG spatiotemporal features and the EEG time-frequency features to obtain EEG spatiotemporal-time-frequency fusion features; then, using a convolutional neural network automatic learning method to obtain a weight coefficient matrix; then, based on the obtained weight coefficient matrix, pooling and fully connecting the EEG time features and the EEG spatiotemporal-time-frequency fusion features to obtain EEG time-space-frequency features; the specific formula is as follows:

[0029] F=[(m1,m2)·V',m3]

[0030] Among them, F represents the fusion feature of EEG time feature, EEG spatiotemporal feature and EEG time-frequency feature, m1 represents EEG time-frequency feature, m2 represents EEG spatiotemporal feature, m3 represents EEG time feature, and V′ represents the weight coefficient matrix.

[0031] Preferably, in step S7, it specifically includes: using an EEG classifier to obtain the category of the EEG corresponding to the video frame; using a guided language-image pre-training BLIP model to extract the semantic information of the video frame in the preprocessed original video; using a semantic fusion network SF-Net to fuse the video frame category information and the video frame semantic information to generate text features.

[0032] Preferably, in step S8, the downsampled video frame obtained in step S1 is input into the CLIP picture encoder to extract picture features; then the text features generated in step S7 are input into the CLIP text encoder to obtain text features; and together with the EEG time-space-frequency features generated in step S6, they are input into the CLIP model as features for comparative learning. Finally, EEG-picture and EEG-text alignment are performed by cosine similarity calculation. The specific formula is as follows:

[0033]

[0034] S clip =S ei +S et

[0035] Among them, l eeg is the projected EEG feature to make it more suitable for video generation, l image , l text are image features and text features respectively; S ei represents the cosine similarity of EEG-image, S et represents the cosine similarity of EEG-text, S clip It is the overall contrastive learning loss of the time-space-frequency features, image features and text features in CLIP. Finally, the EEG time-space-frequency features are further optimized to obtain the optimized EEG features.

[0036] Preferably, in step S9, it specifically includes: using the generated optimized EEG features as conditions, using the stable diffusion model to generate a reference video frame; using the Canny edge detection algorithm to extract the contour information of the reference video frame; extracting the depth information of the reference video frame based on the Monodepth depth estimation algorithm; using the preset spatiotemporal conditional fusion encoder STCF-net to integrate the contour information and depth information into the conditions to obtain the fusion conditions; generating a high-quality reconstructed video based on the video diffusion model and the fusion conditions.

[0037] The beneficial effects of the present invention are as follows: the multimodal fusion video reconstruction method based on brain activity of the present invention makes up for the problems of incomplete extraction of video dynamic information, low consistency between video frames, frame jumps and incoherence by adopting methods such as time window segmentation method, attention mechanism, masking coding method, spatiotemporal attention mechanism method, wavelet transform method, contrastive learning, etc., and realizes high-quality video reconstruction, thereby greatly improving the authenticity of the video, thereby making the application value higher. The present invention can extract visual information from human brain activity patterns, help develop brain-computer interfaces, provide new ways for disabled people to resume communication and control equipment, and enhance the interaction efficiency and experience of normal people. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of video reconstruction using EEG data according to an embodiment of the present invention;

[0039] Figure 2 This is a sample result diagram of an embodiment of the present invention;

[0040] Figure 3 Schematic diagram comparing sample results of an embodiment of the present invention and a traditional method. DETAILED DESCRIPTION

[0041] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] This implementation uses EEG data collected while watching a video stimulus as an example. The EEG data was self-collected. All participants were monitored using a Boruikang 64-electrode EEG acquisition device. During the acquisition process, participants maintained a comfortable and relaxed state, sitting approximately 1 meter from the screen. They were instructed to remain motionless while watching the video and minimize blinking and swallowing.

[0043] See also Figure 1 The present invention provides a technical solution: a high-quality video reconstruction method based on EEG signals, comprising the following steps:

[0044] Step S1: Preprocess the EEG data. First, downsample the original video to 3 fps. Then, use a sliding window of fixed length to segment the EEG data. Each window is processed as a time frame.

[0045] Step S2: Use CSP technology to perform spatial filtering on the EEG signal to extract signal components related to specific tasks or stimuli, and combine it with the attention mechanism to select channels with high correlation with video reconstruction.

[0046] Step S3: Adopting the masked coding method and using the MAE model to extract EEG temporal features.

[0047] Step S4: Use the spatiotemporal attention mechanism method to capture the correlation and importance between different time points and spatial positions and extract EEG spatiotemporal features.

[0048] Step S5: Using the wavelet transform method to extract the frequency domain information of the EEG data, and using the convolutional neural network to extract the time characteristics of the EEG data to generate EEG time-frequency features.

[0049] Step S6: Using an adaptive fusion method, the EEG time features, spatiotemporal features, and time-frequency features are fused to generate EEG time-space-frequency features.

[0050] Step S7: Use the EEG classifier to extract video category information, use the BLIP model to extract video semantic information, and use the ensemble learning method to fuse these two types of information to generate text features.

[0051] Step S8: Using the contrastive learning method, the CLIP model is used to align the EEG features of step S6 with the image features, and the EEG features of step S6 with the text features of step S7 to obtain optimized EEG features.

[0052] Step S9: Using the optimized EEG features generated in step S8 as conditions, a stable diffusion model is used to generate a reference video frame; using the generated reference video frame as an auxiliary condition, combined with contour and depth information, a video diffusion model is used to generate / reconstruct a high-quality video.

[0053] In step S1, the EEG data is preprocessed. The main steps of preprocessing include locating channel data, deleting useless electrodes, re-referencing, bandpass filtering, notch filtering, and independent principal component analysis. This process removes eye movement artifacts, electrocardiographic artifacts, and electromyographic artifacts during the acquisition process to the greatest extent possible, ensuring the quality of the EEG data.

[0054] For video downsampling, ffmpeg software was used to downsample the video from 30fps to 3fps.

[0055] For the division of the time window, the present invention adopts a time window division method with a length of 500 data points to make a one-to-one correspondence between EEG data and video frames.

[0056] In step S2, in order to effectively extract the key information most relevant to video reconstruction and improve the accuracy and quality of the reconstruction results, the present invention uses CSP technology to perform spatial filtering on the EEG signal. At the same time, the attention mechanism is applied to the relevant channels selected by the CSP method. The features of these channels are weighted by the learned weight vector to extract the signal components related to the specific task or stimulus. The objective function of the CSP spatial filter ω is as follows:

[0057]

[0058] in, represents the EEG data of the i-th channel, TS is all time points, AC is the number of all channels; C i is the spatial covariance matrix of the EEG data of the i-th channel; ω represents the spatial filter, and T represents the transpose of the matrix.

[0059] During training, the attention mechanism helps the model selectively focus on certain parts of the input signal, thereby extracting information relevant to the video reconstruction task. The attention mechanism dynamically adjusts the weights of channel features, enabling the model to effectively extract relevant information in different situations, optimizing tasks and improving performance.

[0060] In step S3, a masked coding method is used to extract the time domain features of the EEG data. The EEG data of the channel selected in step S2 is input into the MAE model, the EEG data is masked by 75%, and then information is extracted from the unmasked data to reconstruct the EEG data. Finally, the predicted data is compared with the original data. The specific calculation formula of the loss function is as follows:

[0061]

[0062] Where N represents the number of samples, y i ' represents the predicted value generated by the MAE model, y i Represents a primitive value.

[0063] In step S4, the spatiotemporal variation pattern of the EEG signal contains rich cognitive and behavioral information. Extracting these spatiotemporal features can improve the decoding ability of the brain to process and respond to video content. Therefore, in order to accurately reconstruct the details and dynamics in the video, the present invention introduces the spatiotemporal Transformer method to extract spatiotemporal features. Specifically, the EEG data of the channel selected in step S2 is first input into the spatiotemporal feature extraction network. Then, the data passes through the temporal convolution layer to capture the dynamic changes of the EEG data in the temporal dimension, thereby extracting its temporal features; then, through the deep Transformer model, the EEG data is modeled in the spatial dimension, thereby extracting the spatial information of the data, including the association and interaction between different brain regions; finally, the extracted features are input into the spatial convolution model to further fuse the temporal and spatial information to obtain the spatiotemporal features of the data and provide a richer and more comprehensive feature representation for subsequent tasks.

[0064] In the step S5, because the activity of EEG signals at different frequencies reflects the state and process of the brain in different cognitive tasks. Extracting time-frequency features can reveal these spectral changes, helping to better understand how the brain processes different levels of video content, thereby improving the accuracy and quality of video reconstruction. Therefore, the present invention introduces a wavelet transform and convolutional neural network fusion method to extract time-frequency features. Specifically, the wavelet transform method processes the EEG data of the channel selected in step S2. The wavelet transform first converts the EEG data into the wavelet domain. In this domain, by analyzing the wavelet basis functions of different scales and frequencies, the multi-scale decomposition of the EEG data is achieved, thereby capturing the characteristics of the EEG data in the frequency domain. Then, the efficient representation and compression properties of the wavelet transform are used to extract the important frequency domain features of the EEG data. The specific formula of the wavelet transform is as follows:

[0065]

[0066] in, represents the data of all channels on EEG at time t, a represents the scale factor, b represents the translation factor, γ * (t) represents the complex conjugate of the wavelet function, dt represents the integral of t, and the wavelet function used in the present invention is the Morlet wavelet. The Morlet wavelet is a commonly used complex wavelet function that has the advantage of simultaneously considering time domain and frequency domain characteristics. By applying the Morlet wavelet transform, the EEG data is converted into the wavelet domain, thereby realizing the time-frequency feature analysis of the EEG data. In the wavelet domain, the Morlet wavelet can provide local feature representations of the EEG data at different scales and frequencies to capture the time-frequency information of the EEG data.

[0067] After extracting frequency domain features using wavelet transform, the EEG data selected in step S2 is input into the convolutional neural network to extract time domain features, and finally the two are fused to obtain time-frequency features.

[0068] In step S6, the temporal, spatial, and frequency characteristics of the EEG signal provide different levels of understanding of the signal. Traditional feature fusion methods have some specific problems in feature selection and fusion effects. Redundant features may appear during the feature selection process, that is, different features are highly correlated, which increases model complexity but does not necessarily improve performance. At the same time, important features may be omitted. In terms of feature fusion, how to assign appropriate weights to different features is a challenge. Improper weight assignment may lead to insufficient feature differentiation ability after fusion. Therefore, the present invention introduces an adaptive feature fusion method to dynamically integrate these multi-dimensional features, extract information in an optimal manner, and ensure that valuable information obtained from different features is effectively utilized, thereby improving the accuracy and reliability of the final result. Therefore, an adaptive feature fusion method is used to fuse the time domain features obtained in step S3, the spatiotemporal features generated in step S4, and the time-frequency features in step S5 to obtain EEG time-space-frequency fusion features. First, the spatiotemporal features and time-frequency features are fused using the adaptive fusion method. Next, a convolutional neural network is used to automatically learn and obtain the weight coefficient matrix. Then, the two features are pooled and fully connected. Since the masked coding model masks and reconstructs the input data, it can capture subtle changes and important patterns in the signal. This high-fidelity feature representation can more comprehensively reflect EEG features. Therefore, the present invention assigns a higher weight to it and directly fuses the obtained features with the time domain features in step S3. The specific formula is as follows:

[0069] F=[(m1,m2)·V',m3]

[0070] Among them, F represents the fusion feature of the three, m1 represents the time-frequency feature obtained in step S5, m2 represents the spatiotemporal feature generated in step S4, m3 represents the time domain feature in step S3, and V' represents the weight coefficient matrix.

[0071] In step S7, an EEG classifier is first used to obtain the EEG category corresponding to the video frame. Specifically, EEG data and a recurrent neural network are combined to learn the brain's classification activities for visual objects. Next, the present invention introduces an attention mechanism to improve the model's attention to important features. On this basis, a regressor is trained through a convolutional neural network to transfer the learned classification capabilities to the machine, enabling it to automatically classify images and perform visual classification using human brain features integrated with the attention mechanism to obtain the category corresponding to the EEG data, such as "cat".

[0072] The BLIP model is used to obtain video semantic information. Specifically, the downsampled video frame from step S1 is fed into the BLIP model. The BLIP model's subtitle generator generates synthetic subtitles, converting the visual information into textual descriptions. A filter is then applied to the generated subtitles to remove any noise or irrelevant information, extracting the key information. Ultimately, the subtitles corresponding to the video frame are obtained, such as "A man in a white tracksuit runs again."

[0073] When reconstructing a video, category information can provide a basic classification of the video content (such as scenes, objects or activities), while semantic information can describe the specific meaning and context of these categories. By fusing these two types of information, the relationship between EEG signals and video content can be understood more comprehensively, ensuring that the reconstruction process not only takes into account the simple features of the signal, but also captures the deep semantics behind it. Therefore, the present invention proposes a semantic fusion network called SF-Net to fuse category information and semantic information. First, the category information and semantic information are used as different input data, and are effectively encoded and represented using unique hot encoding. Then, the convolutional neural network of the SF-Net category branch is used to extract category features, and the recurrent neural network branch is used to extract semantic features. Finally, at the top of the multimodal fusion network, a fusion layer is introduced to fuse features from different branches to integrate category information and semantic information to generate comprehensive text features.

[0074] In step S8, the downsampled video frame obtained in step S1 is input into the CLIP image encoder to extract image features. Then, the semantic features generated in step S7 are input into the CLIP text encoder to obtain text features. The text features are input into the CLIP model together with the time-space-frequency features generated in step S6 as features for comparative learning. Finally, EEG-image and EEG-text alignment are performed by cosine similarity calculation. The specific formula is as follows:

[0075]

[0076]

[0077] Among them, l eeg is the projected EEG feature to make it more suitable for video generation, l image , l text are the features extracted from pictures and texts respectively. ei represents the cosine similarity of EEG-image, S et represents the cosine similarity of EEG-text, S clip It is the overall contrastive learning loss of the three CLIPs, which ultimately further optimizes the EEG features.

[0078] In step S9, a pre-trained stable diffusion model is used to generate video frames, and stable diffusion gradually denoises the learning data distribution. First, the stable diffusion model is fine-tuned by using EEG and video frame datasets, and it is fine-tuned from a text-generated graph model that generates ordinary images to a model that generates images based on the EEG. Then, the EEG features optimized in step S8 are input as conditions into the stable diffusion model. The conditional signal is introduced through the cross-attention mechanism in the UNet network, and the cross-attention can merge the conditional information of the EEG data. The specific formula is as follows:

[0079]

[0080] Among them, Q stands for query. The query vector refers to the model's reference and dependence on previous frames and global information when generating the current frame. It is mainly used to obtain correlation with other vectors. K stands for key. The key vector refers to the characteristics of the conditional information embedded in the EEG data, which is used to match with Q to calculate the attention weight. V stands for value. The value vector is the part that is weighted according to the attention weight and used to generate the final output. It represents the EEG features weighted by the cross-attention mechanism. This information will be used to guide the image content in the generation process. d represents the column vector dimension of Q and K, and T represents the transpose of the matrix.

[0081] In step S9, the video diffusion model is used to generate high-quality video using the generated video frames as auxiliary conditions. First, the Canny edge detection algorithm is used to extract the outline of the video frame, followed by the Monodepth depth estimation algorithm to extract the depth information of the video frame. The present invention then proposes a spatiotemporal conditional fusion encoder, named STCF-net, which incorporates outline and depth information into the conditions, further capturing the spatiotemporal information of the video and improving video quality.

[0082] For STCF-net, specifically, in the spatial information extraction stage, two two-dimensional convolutional layers and a maximum pooling layer are first used to capture the input data features, and then contour information is introduced as auxiliary input, and channels are added to the convolution layer to process contour information, so as to better express the structure and content of the image; at the same time, by combining depth information with spatial information, a one-dimensional channel can be added to process depth information, so that the model can better understand the three-dimensional sense and depth information of the object; in the temporal feature extraction stage, compared with a single contour map, the contour sequence can provide more temporal information, so the temporal transformers module is continued to be used to extract the contour sequence and depth information, so that the model can fully utilize the advantages of multi-source information for video generation, and finally generate high-quality video through conditional fusion.

[0083] The present invention first proposes a time-space-frequency EEG feature fusion method to more comprehensively capture brain activity information, thereby more deeply understanding and analyzing the brain's activity patterns. Then, the present invention proposes SF-Net to fuse category and video semantic information, and more comprehensively fuse coarse-grained and fine-grained semantic information. Finally, in the generation stage, the present invention first uses a stable diffusion model to generate reference video frames. At the same time, in order to enhance inter-frame consistency, the proposed STCF-NET is used to fuse contour and depth information to generate high-quality video through a video diffusion model. Figure 2 From the above, the present invention reconstructs the video with accurate semantics, motion and scene dynamics. Figure 3 Compared with traditional methods, the method of the present invention is closer to the real video in terms of the motion, number of subjects, and color of the reconstructed video, and the reconstructed video is more realistic. This invention has made certain contributions to the development of brain-computer interface and computer vision.

[0084] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0085] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0086] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0087] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0088] The references to "first" and "second" in the embodiments merely distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or precedence of "first" and "second" can be interchanged where appropriate. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0089] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A high-quality video reconstruction method based on EEG signals, characterized in that: The steps include: S1. Preprocess the EEG data, downsample the video frames, and segment the EEG data using the sliding window technique; S2. Select channels with high correlation with video reconstruction: Use the common spatial pattern (CSP) technique to spatially filter the EEG data to extract signal components related to specific tasks or stimuli, and combine it with the attention mechanism to select channels with high correlation with video reconstruction; S3, using the masked coding method and the masked coding MAE model to extract EEG temporal features; S4. Use the spatiotemporal attention mechanism to capture the correlation and importance between different time points and spatial positions and extract EEG spatiotemporal features; S5. Use wavelet transform method to extract frequency domain information of EEG data, and use convolutional neural network to extract time domain features of EEG data to generate EEG time-frequency features; S6. Using an adaptive fusion method, the EEG temporal features, spatiotemporal features, and time-frequency features are fused to generate EEG time-space-frequency features; S7, using EEG classifier to extract video category information, using BLIP model to extract video semantic information, and using semantic fusion network SF-Net to fuse these two kinds of information and generate text features; S8, using contrastive learning method to align EEG-text-image features to obtain optimized EEG features; S9, generating a reference video frame using a stable diffusion model based on the generated optimized EEG features; The generated reference video frames are used as auxiliary conditions to generate high-quality videos using the video diffusion model.

2. The high-quality video reconstruction method based on EEG signals according to claim 1, characterized in that: In step S1, specifically, the following steps are performed: preprocessing the EEG data, including locating channel data, deleting useless electrodes, re-referencing, bandpass filtering, notch filtering, and independent principal component analysis to ensure the quality of the EEG data; then downsampling the video to 3fps; and then segmenting the EEG data using a fixed-length sliding window, with each window being processed as a time frame.

3. The high-quality video reconstruction method based on EEG signals according to claim 1, characterized in that: In step S2, the EEG signal is spatially filtered using the common spatial pattern (CSP) technique. At the same time, the attention mechanism is applied to the relevant channels selected by the CSP method. The features of these channels are weighted by the learned weight vector to extract the signal components related to the specific task or stimulus. The objective function of the CSP spatial filter ω is as follows: in, represents the EEG data of the i-th channel, TS is all time points, AC is the number of all channels; C i is the spatial covariance matrix of the EEG data of the i-th channel; ω represents the spatial filter, and T represents the transpose of the matrix.

4. The high-quality video reconstruction method based on EEG signals according to claim 1, characterized in that: In step S3, the masked coding method is used to extract the time domain features of the EEG data. The EEG data of the channel selected in step S2 is input into the masked coding MAE model, the EEG data is masked by 75%, and then information is extracted from the unmasked EEG data to reconstruct the EEG data. Finally, the predicted EEG data is compared with the original EEG data. The specific calculation formula of the loss function is as follows: Where N represents the number of samples, y i ' represents the predicted value generated by the MAE model, y i Represents a primitive value.

5. The high-quality video reconstruction method based on EEG signals according to claim 1, characterized in that: In step S4, the EEG data of the channel selected in step S2 is input into the spatiotemporal feature extraction network; First, the EEG data passes through a temporal convolution layer to capture the dynamic changes of the EEG data in the temporal dimension, thereby extracting temporal features. Then, a deep Transformer model is used to model the EEG data in the spatial dimension, thereby extracting the spatial information of the EEG data, including the associations and interactions between different brain regions. Finally, the extracted features are input into a spatial convolution model to further fuse the temporal and spatial information to obtain the spatiotemporal features of the EEG data.

6. The high-quality video reconstruction method based on EEG signals according to claim 1, characterized in that: In step S5, the EEG data frequency domain features are extracted from the EEG data corresponding to the selected channels with high relevance to the video reconstruction task based on the wavelet transform method, and the EEG data time domain features are extracted from the EEG data corresponding to the selected channels with high relevance to the video reconstruction task based on the convolutional neural network. The extracted EEG data frequency domain features and EEG data time domain features are then fused to obtain EEG time-frequency features. The specific formula of the wavelet transform is as follows: in, Represents the data of all channels on the EEG at time t, a represents the scale factor, b represents the translation factor, γ * (t) represents the complex conjugate of the wavelet function, and dt represents the integral with respect to t.

7. The high-quality video reconstruction method based on EEG signals according to claim 1, characterized in that: In step S6, the following steps are specifically performed: first, the EEG spatiotemporal features and the EEG time-frequency features are fused using an adaptive feature fusion method to obtain EEG spatiotemporal-time-frequency fusion features; then, a weight coefficient matrix is ​​obtained using a convolutional neural network automatic learning method; then, based on the obtained weight coefficient matrix, the EEG time features and the EEG spatiotemporal-time-frequency fusion features are pooled and fully connected to obtain EEG time-space-frequency features; the specific formula is as follows: F=[(m1,m2)·V',m3] Among them, F represents the fusion feature of EEG time feature, EEG spatiotemporal feature and EEG time-frequency feature, m1 represents EEG time-frequency feature, m2 represents EEG spatiotemporal feature, m3 represents EEG time feature, and V' represents the weight coefficient matrix.

8. The high-quality video reconstruction method based on EEG signals according to claim 1, characterized in that: In step S7, it specifically includes: using the EEG classifier to obtain the EEG category corresponding to the video frame; using the guided language-image pre-training BLIP model to extract the semantic information of the video frame in the preprocessed original video; using the semantic fusion network SF-Net to fuse the video frame category information and the video frame semantic information to generate text features.

9. The high-quality video reconstruction method based on EEG signals according to claim 1, characterized in that: In step S8, the downsampled video frame obtained in step S1 is input into the CLIP picture encoder to extract picture features; Then, the text features generated in step S7 are input into the text encoder of CLIP to obtain text features; together with the EEG time-space-frequency features generated in step S6, they are input into the CLIP model as features for comparative learning. Finally, the EEG-image and EEG-text alignment are performed through cosine similarity calculation. The specific formula is as follows: S clip =S ei +S et Among them, l eeg is the projected EEG feature to make it more suitable for video generation, l image , l text are image features and text features respectively; S ei represents the cosine similarity of EEG-image, S et represents the cosine similarity of EEG-text, S clip It is the overall contrastive learning loss of the time-space-frequency features, image features and text features in CLIP. Finally, the EEG time-space-frequency features are further optimized to obtain the optimized EEG features.

10. The high-quality video reconstruction method based on EEG signals according to claim 1, characterized in that: In step S9, it specifically includes: using the generated optimized EEG features as conditions, using the stable diffusion model to generate a reference video frame; using the Canny edge detection algorithm to extract the contour information of the reference video frame; extracting the depth information of the reference video frame based on the Monodepth depth estimation algorithm; using the preset spatiotemporal conditional fusion encoder STCF-net to integrate the contour information and depth information into the conditions to obtain the fusion conditions; generating a high-quality reconstructed video based on the video diffusion model and the fusion conditions.

Images