Scene imagination retrieval method, device and system, storage medium and program product

Abstract

The invention discloses a scene imagination retrieval method, device and system, a storage medium and a program product, and the method comprises the following steps: S1, collecting an electroencephalogram signal of a human body when the human body imagines a specific scene; s2, the electroencephalogram signals are preprocessed, and continuous electroencephalogram signal fragments in a short time window are obtained; s3, encoding and fusing the electroencephalogram signal segments to obtain high-dimensional feature vectors, and identifying and encoding retrieval targets in a retrieval target library to obtain a plurality of target feature vectors; s4, performing cross-modal projection on the high-dimensional feature vector and the multiple target feature vectors, and mapping the high-dimensional feature vector and the multiple target feature vectors to a unified semantic space; s5, calculating the similarity between the high-dimensional feature vector and the target feature vector; and S6, performing retrieval target classification based on the similarity, and outputting a retrieval result in the retrieval category with the highest similarity according to the matching confidence. The problems that in retrieval in the prior art, retrieval operation is inconvenient, and implicit retrieval is lacked are solved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of brain-computer interface, and particularly relates to a scene imagination retrieval device, system, storage medium and program product based on electroencephalogram signals and auxiliary physiological signals collected in real time when a human brain imagines a scene, and target retrieval in a target database. BACKGROUND

[0002] Traditional target retrieval relies on a user directly inputting retrieval elements such as text, voice or images into an input method program, and the input method program performs retrieval and feeds back retrieval results based on the direct input of the user. However, disabled people often cannot directly input retrieval elements, and thus cannot implement retrieval functions. In some other scenarios, a user pursues more natural and implicit expression of retrieval elements, and at this time, traditional retrieval cannot meet the needs of the user.

[0003] With the rapid development of brain-computer interface (BCI) technology, based on the brain-computer interface technology, human brain thinking activities can be analyzed to a certain extent by collecting brain electrical signals and other signals of the human body. Existing brain-computer interfaces and associated artificial intelligence content generation (AIGC) technologies focus on content output and device control, such as generating text on a display device based on brain electrical signals, controlling a mechanical arm based on brain electrical signals, and the like, and there are few related applications in the retrieval field.

[0004] Eric A Pohlmeyer of Columbia University in New York City, New York, USA and Jun Wang of Columbia University, in the paper "Closing the loop in cortically-coupled computer vision: a brain-computer interface for searching image databases", disclose a brain-computer interface for searching an image database, through which brain electrical signals of a user are collected, and based on the correlation of electroencephalogram (EEG) signals related to target detection, attention shift and self-monitoring processes, an interest score of the user is obtained, images of a selected interest category of the user are guessed, and images of interest to the user are pushed. However, under different conditions of different times, different body states and different moods of the user, the difference in interest points is very large. This retrieval method cannot decode the active needs of the user, and cannot perform targeted retrieval according to the needs of the user.

[0005] For the disabled, their daily life is monotonous and boring, and the demand for active retrieval such as multimedia retrieval is much greater than content generation and control of external devices. In the active retrieval scenario, the accuracy sensitivity is low, and even if the retrieved result is biased, as long as the candidate content pool is set, there is a high probability that there is an option that meets the demand in the candidate sequence. This makes brain-computer interface and artificial intelligence technology better play a value in the user active retrieval scenario, fill the technical gap in the retrieval field and effectively improve the quality of life of users.

[0006] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with regard to the present disclosure. SUMMARY

[0007] The embodiments of the present disclosure solve the problems in the foregoing retrieval technology. A scene imagination retrieval method and a scene imagination retrieval device are provided, which solve the problem that users cannot implicitly retrieve. The user can search for content such as video, audio, picture and text through "scene imagination" without explicit operation, which improves the naturalness and convenience of retrieval, has good interactive plasticity and application expansion. It can be widely applied to personal album retrieval, AR / VR, immersive experience, barrier-free interaction, information access and control in special scenes such as military or medical treatment. A scene imagination retrieval system, a non-transitory storage medium and a computer program product are also provided, which solve the problem of executing the scene imagination retrieval method.

[0008] The first aspect of the present disclosure provides a scene imagination retrieval method, comprising the following steps: S1: collecting brain electrical signals of a human body while the human body imagines a specific scene; S2: preprocessing the brain electrical signals to obtain continuous brain electrical signal segments in a short time window; S3: encoding and fusing the brain electrical signal segments to obtain a high-dimensional feature vector, and identifying and encoding retrieval targets in a retrieval target library to obtain a plurality of target feature vectors; S4: performing cross-modal projection on the high-dimensional feature vector and the plurality of target feature vectors, and mapping to a unified semantic space; S5: calculating the similarity of the high-dimensional feature vector and the target feature vectors; S6: classifying the retrieval targets based on the similarity, and outputting a retrieval result in the retrieval category with the highest similarity in the order of matching confidence.

[0009] For example, in at least one embodiment, step S1 further includes acquiring the human body's electroencephalogram (EEG) signals while simultaneously acquiring the human body's auxiliary physiological signals; step S2 further includes preprocessing the auxiliary physiological signals to obtain continuous auxiliary physiological signal segments within a short time window; step S3 further includes encoding the EEG signal segments and the auxiliary physiological signal segments, and performing cross-modal fusion to obtain the high-dimensional feature vector; the auxiliary physiological signals include one or more of the following: electrooculogram (EOG) signals, facial muscle electroencephalogram (EMG) signals, skin electroencephalogram (SEEG) signals, pulse signals, heart rate variability signals, and speech signals.

[0010] For example, in at least one embodiment, steps S3-S6 are implemented based on a pre-trained multimodal fusion classification model. The model pre-training includes the following steps: S01: Pre-setting a scene library including multiple scenarios; S02: Classifying the search targets in the search target library and generating search target labels corresponding to each scenario; S03: Collecting and recording the EEG signal while the user imagines a scenario; S04: Preprocessing the EEG signal to obtain continuous EEG signal segments within a short time window; S05: Embedding the corresponding search target label into the EEG signal segments to obtain training data; S06: Repeating steps S03-S05 multiple times for each scenario, recording the training data collected multiple times, and constructing a training dataset; S07: Setting the training parameters of the multimodal fusion classification model, executing training loops and validation loops in batches, optimizing model parameters using contrastive learning and triplet loss function in the training loop, evaluating model performance in the validation loop, and performing multiple batches of training.

[0011] For example, in at least one embodiment, the scene is one or more of a still image, a moving video, and audio; the retrieval target is one or more of an image, video, audio, and text.

[0012] For example, in at least one embodiment, step S6 includes obtaining the search results using the target feature vectors with the highest similarity as search elements through the Approximate Nearest Neighbor (ANN) indexing technique.

[0013] For example, in at least one embodiment, after step S6, step S7 is further included: collecting feedback information from the human body, continuously fine-tuning the algorithm model based on the feedback information, and optimizing the accuracy of the retrieval results.

[0014] A second aspect of this disclosure provides a scene imagination retrieval device, including a brain-computer interface module, a preprocessing module, and a computing module. The computing module includes an encoder, a projection layer, a contrast layer, and a decoder. The brain-computer interface module is configured to acquire electroencephalogram (EEG) signals of a person while they imagine a specific scene. The preprocessing module is configured to preprocess the EEG signals to obtain continuous EEG signal segments within a short time window. The encoder is configured to encode and fuse the EEG signal segments to obtain a high-dimensional feature vector, and to identify and encode a retrieval target in a retrieval target database to obtain multiple target feature vectors. The projection layer is configured to perform cross-modal projection of the high-dimensional feature vector and the multiple target feature vectors corresponding to the retrieval target database, mapping them to a unified semantic space. The contrast layer is configured to calculate the similarity between the high-dimensional feature vector and the target feature vectors. The decoder is configured to classify the retrieval target based on the similarity and output the retrieval results in order of matching confidence.

[0015] A third aspect of this disclosure provides a scene imagination retrieval system, comprising: a memory for non-temporarily storing computer-executable instructions; and a processor for running the computer-executable instructions, characterized in that the computer-executable instructions, when run by the processor, execute the scene imagination retrieval method described in any of the preceding claims.

[0016] The fourth aspect of this disclosure provides a non-transitory storage medium for non-transitory storage of computer-executable instructions, characterized in that, when the computer-executable instructions are executed by a computer, the scene imagination retrieval method described in any of the preceding claims is executed.

[0017] The fifth aspect of this disclosure provides a computer program product, including a computer program, characterized in that, when executed by a processor, the computer program implements the scene imagination retrieval method described in any of the preceding claims. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure, and are not intended to limit this disclosure.

[0019] Figure 1 This is a flowchart of a scene imagination retrieval method according to an embodiment of the present disclosure;

[0020] Figure 2 This is a flowchart of the pre-training of a multimodal fusion classification model according to an embodiment of the present disclosure;

[0021] Figure 3 This is a schematic diagram of a scene imagination retrieval device according to an embodiment of the present disclosure;

[0022] Figure 4 This is a schematic diagram of image retrieval according to an embodiment of the present disclosure. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0024] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes. In this disclosure, “multiple” means two or more.

[0025] According to embodiments of this disclosure, Figure 1 A scene imagination retrieval method is disclosed, including the following steps: Step S1: While the user imagines a specific scene, the user's electroencephalogram (EEG) signals are collected. There are no limitations on the brain-computer interface (BCI) device used to collect the EEG signals; it can be a non-invasive BCI device, an invasive BCI device, or even an EEG signal collected by a BCI configured by the user for other functions. For example, a user wearing a non-invasive BCI headpiece imagines a skier skiing under a sunset in their mind, and the non-invasive BCI collects the user's multi-channel EEG signals.

[0026] The scenario can take many forms; any scenario that can repeatedly reflect the user's needs can realize the technical solution disclosed herein. Preferably, the scenario is one or more of the following: static image, dynamic video, and audio.

[0027] Preferably, the EEG signal is a multi-channel EEG signal acquired by a non-invasive brain-computer interface device. Multi-channel EEG signals can complement and corroborate each other, which can improve the accuracy of input signals in machine learning.

[0028] Preferably, step S1 further includes collecting the human body's electroencephalogram (EEG) signals while simultaneously collecting auxiliary physiological signals. These auxiliary physiological signals can be any physiological signal that reflects factors such as attention and emotion. During different scene visualization processes, EEG signals exhibit changes with specific characteristics, which can serve as the basis for identifying and retrieving elements. In the same process, the changes in emotion and attention reflected by the auxiliary physiological signals also have specific characteristics. By fusing these with the EEG signals, multimodal physiological signals can enhance recognition robustness and effectively address the ambiguity and false recognition rate problems inherent in single-modality recognition.

[0029] Preferably, the auxiliary physiological signals include electrooculography (EOG) signals. Based on EOG signals, the micro-movements of the eyeballs can be tracked in real time, thereby reflecting the user's focus direction and dimension during the imagination process, such as whether the focus is on the overall scene or local details, thus generating specific characteristics for different scenes and effectively supplementing and verifying the characteristics of electroencephalogram (EEG) signals.

[0030] Preferably, the auxiliary physiological signals include facial electromyography (EMG) signals, which are used to monitor changes in facial micro-expressions or jaw muscles. These subtle changes in muscle activity can reflect the user's cognitive specificity when performing visual imagination, thereby effectively supplementing and verifying the characteristics of electroencephalogram (EEG) signals.

[0031] Preferably, the auxiliary physiological signals include electrodermal osmosis (EDA) signals. EDA signals can sensitively reflect the user's psychological arousal level, and thus represent the intensity of emotions evoked by the imagined content. The stronger the emotion, the more obvious the change in EDA response. Based on the user's emotional characteristics reflected by EDA signals, the characteristics of EEG signals can be effectively supplemented and verified.

[0032] Preferably, the auxiliary physiological signals include pulse (PPG) signals and / or heart rate variability (HRV) signals. Pulse (PPG) and heart rate variability (HRV) signals are based on changes in the human heart rate. Heart rate fluctuations are closely related to changes in emotion and attention. These two signals can reflect the user's emotional involvement and attentional state during the imagination process, thus effectively supplementing and verifying the characteristics of EEG signals.

[0033] Preferably, the auxiliary physiological signals include speech signals. It is well known that speech signals can be used as a standalone retrieval element. However, in cases where a user's speech is unclear and cannot be retrieved independently, speech signals can still serve as a supplement to EEG signal retrieval. In other embodiments, while the user is still able to speak, the user's speech signals can be used as the primary basis for training the large-scale model. As the large-scale model is trained and optimized, the weight of EEG signals in the retrieval process is gradually increased, gradually transitioning to relying primarily on EEG signals as the retrieval basis.

[0034] Step S2: Preprocess the EEG signals to obtain continuous EEG signal segments within a short time window. For multi-channel acquired EEG signals, filtering, artifact removal, and signal standardization are performed. Filtering uses techniques such as filters to remove noise from the original signal. EEG signals often contain artifacts from specific sources; artifact removal removes or compensates for these artifacts. Signal standardization uses normalization to obtain a standard signal format. Signal preprocessing improves signal quality and enhances the accuracy of signals input into large models. Slicing the signal into short time windows divides the long-term continuous EEG acquisition into continuous EEG signal segments within short time windows to ensure the continuity and resolvability of the time series. For example, processing multi-channel EEG signals used in non-invasive brain-computer interfaces compresses the data while retaining EEG signal segments with key signal features. Information in these segments can be used in subsequent processing to identify features such as "outdoors," "snow," and "sunset." The identified information can be multi-dimensional information that large models can recognize, or semantic information such as "outdoors," "snow," and "sunset," or image information such as "white" and "orange circles," or audio information including a specific melody.

[0035] Preferably, for multi-channel acquired EEG signals, independent component analysis (ICA) is used to remove artifacts and separate effective neural features. ICA eliminates noise while preserving almost no detail in other signals, and significantly improves denoising performance.

[0036] Preferably, step S2 further includes preprocessing the auxiliary physiological signals to obtain continuous auxiliary physiological signal segments within a short time window. The signal processing methods for auxiliary physiological signals are similar to those for electroencephalogram (EEG) signals; filtering, artifact removal, and signal standardization are performed on the multimodal physiological signals respectively. Signal segments are obtained by dividing the signals into short time windows.

[0037] Step S3: Encode and fuse EEG signal segments to obtain high-dimensional feature vectors. Identify and encode the search targets in the target database to obtain multiple target feature vectors. Based on, for example, STFT, CSP, or RNN / CNN fusion architectures, extract modal features of EEG signals through temporal features, perform temporal alignment of multiple modal features, and obtain high-dimensional feature vectors. Extract local and global features of the search targets, fuse features through an attention mechanism, and obtain the corresponding target feature vectors. The acquisition of multiple target feature vectors can be completed during the model pre-training stage to save computational resources, improve retrieval efficiency, and perform retrieval in a target database containing more search targets with less computational resources. For example, the high-dimensional feature vectors obtained by a large model can also identify features with information such as "outdoors," "snow," and "sunset." The search targets in the target database are videos. After the video as the search target is identified and encoded by the model, it also has features associated with information such as "outdoors," "snow," and "sunset."

[0038] Preferably, a multimodal fusion classification model is used, such as the Transformer algorithm model, the contrastive learning algorithm model, and the attention mechanism fusion network algorithm model, to implement steps S3-S6.

[0039] Preferably, steps S3-S6 are implemented using the Transformer algorithm model. The multimodal features are input to the encoder for encoding to obtain multimodal feature vectors. The input encoder adopts a Transformer encoder architecture, consisting of multiple identical encoder layers stacked together. Each layer contains two main modules: a multi-head self-attention mechanism and a feed-forward neural network. The multi-head attention mechanism calculates attention separately after segmenting the input and then concatenates the results. This is beneficial for capturing features in different subspaces. The output of each encoder layer passes through the feed-forward neural network, which transforms the representation at each position, facilitating further feature extraction.

[0040] Preferably, the search target is one or more of images, videos, audio, and text.

[0041] Preferably, step S3 further includes encoding the EEG signal segments and auxiliary physiological signal segments, and then performing cross-modal fusion to obtain a high-dimensional feature vector. The encoder module first performs temporal alignment on the EEG signal segments and auxiliary physiological signal segments to ensure the correspondence between EEG signal segments and auxiliary physiological signals acquired within the same short time window, and then performs multimodal fusion to obtain a high-dimensional feature vector.

[0042] Step S4: Perform cross-modal projection on the high-dimensional feature vector and the multiple target feature vectors corresponding to the retrieved target library, mapping them to a unified semantic space. In some embodiments, the high-dimensional feature vector and the multiple target feature vectors are low-dimensional dense vectors representing their respective objects. In the shared semantic space, the distance between the high-dimensional feature vector and the target feature vectors reflects the similarity between them. For example, the shared semantic space includes both high-dimensional feature vectors and target feature vectors containing information such as "outdoors," "snow," and "sunset," as well as other target feature vectors with extremely low correlation to information such as "outdoors," "snow," and "sunset."

[0043] Step S5: Calculate the similarity between the high-dimensional feature vector and the target feature vector. In some embodiments, cosine similarity, Euclidean distance, etc., are used to measure the similarity between samples. In a batch of data, for each high-dimensional feature vector, one positive target feature vector sample and multiple negative target feature vector samples are constructed. The sample with the highest similarity is labeled as the positive target feature vector sample. In other embodiments, a loss function is used to encourage similar high-dimensional feature vectors and target feature vectors to move closer together in the shared semantic space, while dissimilar high-dimensional feature vectors and target feature vectors are moved further apart in the shared semantic space. The similarity is ranked according to the distance in the shared semantic space. For example, in the shared semantic space, through contrastive learning, target feature vectors with high similarity to information such as "outdoors," "snow," and "sunset" are obtained.

[0044] Step S6: Classify the search targets based on similarity, and output the search results in the search category with the highest similarity, sorted by matching confidence. In the search target database, multiple search targets may be associated with the same target feature vector. The large model classifies these multiple associated search targets into a single category based on the target feature vector's matching, and outputs the results. Search targets with high matching confidence are ranked first in the output search results, while those with low matching confidence are ranked last. For example, based on target feature vectors with high similarity to information such as "outdoors," "snow," and "sunset," videos in the categories of "outdoors," "snow," and "sunset" are output. The output results for search targets are sorted according to the matching degree of these themes, allowing users to obtain search results with high matching degrees first.

[0045] Preferably, step S6 includes using the one or more target feature vectors with the highest similarity as search elements, and obtaining search results through Approximate Nearest Neighbor (ANN) indexing technology. While ANN indexing technology reduces search accuracy to some extent, it effectively improves search efficiency, reduces response time, and saves computing resources. In target searches, users often focus on response speed and do not have particularly stringent requirements for search accuracy. Therefore, using ANN indexing technology can effectively improve the user experience of the search process.

[0046] Step S7: Collect user feedback information and continuously fine-tune the algorithm model based on this feedback to optimize the accuracy of search results. Collect user feedback on search results, distinguish between positive and negative feedback samples, and input these samples into the algorithm model to update model parameters and improve search accuracy. For example, if a user actively selects a search target and spends the most time on that target, this target is marked as a positive feedback sample for that scenario. Search targets that the user ignored but are already in the list are marked as negative feedback samples. The model can update its parameters accordingly, resulting in higher accuracy in subsequent searches.

[0047] According to embodiments of this disclosure, Figure 2 A pre-training method for a multimodal fusion classification model is disclosed, including the following steps: Step S01: Pre-setting a scene library containing multiple scenarios. The pre-set scenarios can take many forms; any scenario that can repeatedly reflect the user's needs can implement the technical solution disclosed herein. For example, a scenario can be a static image imagined by the human brain, such as a landscape painting including grasslands, horses, and blue skies; it can be a dynamic scene imagined by the human brain, such as a football player performing a shot; or it can be an audio signal imagined by the human brain, such as a section of a popular song.

[0048] Step S02: Classify the search targets in the search target database and generate search target tags corresponding to each scenario. Search targets in the search target database can be one or more of images, videos, audio, and text. Search target tags can be pre-generated automatically by a large model based on massive amounts of unlabeled or micro-labeled data, automatically generated using a pre-trained image and text recognition model (e.g., the CLIP model), or generated using traditional manual labeling methods.

[0049] Step S03: While the user imagines a scenario, EEG signals are collected and recorded. When a user imagines different scenarios, physiological characteristics such as attention and emotion will differ, and the resulting EEG signals will reflect these differences. When imagining the same scenario, attention and emotion are relatively similar, resulting in multiple segments with identical features in the EEG signals. By training a large model, the model continuously learns these identical features and their contextual relationships, enabling relatively accurate prediction of the target semantic encoding corresponding to the user's imagined scenario. The specific method for collecting EEG signals is not limited; preferably, the same collection method used in the scene imagination retrieval stage is employed.

[0050] Step S04: Preprocess the EEG signals to obtain continuous EEG signal segments within a short time window. Slice the signal according to the short time window to divide the long-term continuously acquired EEG signals into continuous EEG signal segments within a short time window, so as to ensure the continuity and resolvability of the time series.

[0051] Step S05: Embed the corresponding retrieval target label into the EEG signal segment to obtain training data. In some embodiments, the corresponding retrieval target label is manually identified in the EEG signal segment for the model to learn. In other embodiments, positive samples of the retrieval target label corresponding to the EEG signal segment are identified, and the model is rewarded for matching the positive samples.

[0052] Step S06: Repeat steps S03-S05 multiple times for each scenario, recording the training data collected repeatedly to construct a training dataset. The signal features, including data labels, recorded from multiple acquisitions across multiple scenarios constitute the training and validation datasets for the large model. Using a dataset with manual annotations ensures that the dataset acquisition method used for large model pre-training is the same as that used during model application, which helps reduce the computational resources required for large model training and improves the accuracy of large model predictions.

[0053] S07: Set the training parameters for the multimodal fusion classification model, and execute training and validation loops in batches. In the training loop, contrastive learning and the triplet loss function are used to optimize the model parameters. In the validation loop, the model performance is evaluated. Multiple batches of training are performed until the multimodal fusion classification model reaches the pre-training target. Key training parameters include batch size, learning rate and its adjustment strategy, optimizer and its parameters, and stabilization techniques. Large-scale model training uses batch gradient descent to update model parameters. Early batches contain less data, while later batches gradually increase the amount of data. Through a training-validation-training-validation... loop, the accuracy of the large-scale model's predictions is continuously improved until the accuracy reaches the expected target.

[0054] There are various methods for pre-training large models. Figure 2Publicly available pre-training methods are a preferred approach, but not the only one. In training large existing models, especially in the classification of multimodal image and text data, pre-training can be performed using massive amounts of publicly available database information, even without labels or micro-labels. Other pre-training methods can also be employed to train and fine-tune multimodal fusion classification models to obtain usable large models.

[0055] According to embodiments of this disclosure, Figure 3 A scene imagination retrieval device is disclosed, including a brain-computer interface module 1, a preprocessing module 2, and a computing module 3. The computing module 3 has a built-in multimodal classification model, specifically including an encoder 31, a projection layer 32, a contrast layer 33, and a decoder 34.

[0056] The brain-computer interface module 1 is configured to collect the brain signals of a person while they imagine a specific scene. The preprocessing module 2 is configured to preprocess the brain signals to obtain continuous brain signal segments within a short time window. The encoder 31 is configured to encode and fuse the brain signal segments to obtain high-dimensional feature vectors, and to identify and encode the search targets in the search target library to obtain multiple target feature vectors. The projection layer 32 is configured to perform cross-modal projection of the high-dimensional feature vectors and the multiple target feature vectors corresponding to the search target library, mapping them to a unified semantic space. The comparison layer 33 is configured to calculate the similarity between the high-dimensional feature vectors and the target feature vectors. The decoder 34 is configured to classify the search targets based on similarity and output the search results in order of matching confidence.

[0057] According to embodiments of this disclosure, such as Figure 4 As shown, the user's head-mounted device integrates a brain-computer interface module 1, a cloud server integrates a preprocessing module 2 and a computing module 3, the computing module 3 has a built-in multimodal classification model, and the user's visible position includes an interactive device 4. Figure 4 In this embodiment, preprocessing module 2 and computation module 3 are integrated on a cloud server. In other embodiments, preprocessing module 2 and computation module 3 can also be integrated on the interactive device 4. During the model training phase, computation module 3 uses a visual encoder to extract local and global features of the target image, performs feature fusion through an attention mechanism, and combines geometric consistency constraints to improve the alignment ability of local structures, thereby obtaining multiple target feature vectors. These include some target feature vectors that correspond to the semantic features of "grassland," "horse," and "nature," while other target feature vectors have low correlation with the semantic features of "grassland," "horse," and "nature."

[0058] After the search begins, the user imagines a scene of horses grazing on a grassland in nature. Simultaneously, the brain-computer interface module 1 collects multi-channel EEG signals from the user. The preprocessing module 2 performs filtering, artifact removal, and signal standardization on the multi-channel EEG signals, and segments the signals into short time windows, dividing the long-term continuous EEG signals into continuous EEG signal segments within short time windows to ensure the continuity and resolvability of the time series. The encoder 31 of the built-in multimodal classification model in the computation module is a self-supervised deep encoder that maps continuous EEG signal segments into stable dynamic tokens (soft tokens), compresses the data, and retains key spatiotemporal features while compressing the data to obtain high-dimensional feature vectors. At this point, the obtained high-dimensional feature vectors have a high correlation with the semantic features of "grassland," "horse," and "nature." In the projection layer 32, the multimodal classification model performs cross-modal projection of the high-dimensional feature vectors and all target feature vectors obtained in the preprocessing stage, mapping them to a unified semantic space. In contrast layer 33, the multimodal classification model calculates the similarity between the high-dimensional feature vector and the target feature vector. Through contrastive learning, it obtains target feature vectors with high similarity to "grassland," "horse," and "nature." In decoder 34, the multimodal classification model obtains the image categories corresponding to "grassland," "horse," and "nature," and achieves efficient retrieval through Approximate Nearest Neighbor (ANN) indexing technology. During the matching process, retrieval targets with high matching confidence are ranked first in the output retrieval results, while retrieval targets with low matching confidence are ranked last. The retrieved images are then displayed on the program interface of interactive device 4, sorted by matching degree.

[0059] After the search is complete, the user can open search results including "grassland," "horse," and "nature" on the program interface of interactive device 4, and ignore search results including "cow" and "sheep." The multimodal classification model 3 obtains a record of the images opened by the user; opened images are marked as positive samples, and ignored images are marked as negative samples. The multimodal classification model optimizes its parameters based on the positive and negative samples. After multiple optimizations, images including "cow" and "sheep" no longer appear in the search results.

[0060] In another embodiment, the user recalls a melody in their mind and simultaneously hums it softly. Brain-computer interface module 1 acquires multi-channel EEG signals from the user, while a microphone acquires the user's humming sound signal. Preprocessing module 2 performs filtering, artifact removal, and signal standardization on the multi-channel acquired EEG and sound signals, and segments the signals into short time windows, dividing the long-term continuously acquired EEG and sound signals into continuous EEG signal segments within short time windows to ensure the continuity and resolvability of the time series. The encoder 31 of the multimodal classification model is a self-supervised deep encoder that maps continuous EEG signal segments into stable dynamic tokens (soft tokens), compresses the data, and retains key spatiotemporal features while compressing the data. Endpoint detection, denoising, and pitch tracking are performed on the sound signal segments to extract the melody backbone. After fusion, a high-dimensional feature vector is obtained. In projection layer 32, the multimodal classification model performs cross-modal projection of the high-dimensional feature vector and all target feature vectors acquired in the preprocessing stage, mapping them to a unified semantic space. In this embodiment, the target feature vectors correspond to different audio objects such as songs, instrumental music, and music videos. In the contrast layer 33, the multimodal classification model calculates the similarity between the high-dimensional feature vector and the target feature vector. Through contrastive learning, it obtains target feature vectors with high similarity to the recalled and hummed melodies. In the decoder 34, the multimodal classification model obtains the audio object category corresponding to the recalled and hummed melodies, achieving efficient retrieval through Approximate Nearest Neighbor (ANN) indexing technology. During the matching process, retrieval targets with high matching confidence are ranked first in the output retrieval results, while retrieval targets with low matching confidence are ranked last. The retrieved audio objects are then displayed on the program interface of the interactive device 4, sorted by matching confidence. In this embodiment, a threshold for the hummed sound signal can also be set. When the sound signal clarity is higher than the threshold, the sound signal has a higher weight in the retrieval process, and the sound signal is used as a positive sample to train the model for EEG signal recognition. When the sound signal clarity is lower than the threshold, the EEG signal has a higher weight in the retrieval process, and the retrieval is primarily based on the EEG signal.

[0061] According to an embodiment of this disclosure, a scene imagination retrieval system is also provided, comprising: a memory for non-temporarily storing computer-executable instructions; and a processor for running the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to perform the scene imagination retrieval method described in any of the above embodiments.

[0062] According to embodiments of this disclosure, a non-temporary storage medium is also provided for storing computer-executable instructions non-temporarily, wherein when the computer-executable instructions are executed by a computer, the scene imagination retrieval method described in any of the above embodiments is executed.

[0063] According to embodiments of this disclosure, a computer program product is also provided, including a computer program, characterized in that, when the computer program is executed by a processor, it implements the scene imagination retrieval method described in any of the above embodiments.

[0064] The following points also need to be explained:

[0065] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure, and other structures can be referred to the general design.

[0066] (2) For clarity, the thickness of devices, layers, or regions is enlarged or reduced in the drawings used to describe embodiments of the present disclosure, i.e., these drawings are not drawn to scale. It will be understood that when an element such as a layer, film, region, or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element or there may be intermediate elements.

[0067] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0068] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure shall be determined by the scope of the claims.

Claims

1. A scene imagination retrieval method, comprising the following steps: S1: Collect the brain signals of the human body while the human body imagines a specific scene; S2: Preprocess the EEG signal to obtain continuous EEG signal segments within a short time window; S3: Encode and fuse the EEG signal segments to obtain high-dimensional feature vectors; identify and encode the search targets in the search target database to obtain multiple target feature vectors; S4: Perform cross-modal projection on the high-dimensional feature vector and multiple target feature vectors to map them into a unified semantic space; S5: Calculate the similarity between the high-dimensional feature vector and the target feature vector; S6: Based on the similarity, classify the search target, and output the search results in the search category with the highest similarity according to the matching confidence.

2. The scene imagination retrieval method according to claim 1, characterized in that, Step S1 also includes collecting the human body's electroencephalogram (EEG) signals while simultaneously collecting the human body's auxiliary physiological signals. Step S2 further includes preprocessing the auxiliary physiological signal to obtain continuous auxiliary physiological signal segments within a short time window; Step S3 further includes encoding the EEG signal segment and the auxiliary physiological signal segment, and performing cross-modal fusion to obtain the high-dimensional feature vector; The auxiliary physiological signals include one or more of the following: electrooculogram (EOG) signals, facial muscle electromyography (EMG) signals, skin electromyography (SEM) signals, pulse signals, heart rate variability (HRV) signals, and speech signals.

3. The scene imagination retrieval method according to claim 1, characterized in that, Steps S3-S6 are implemented based on a pre-trained multimodal fusion classification model. Model pre-training includes the following steps: S01: Preset scene library including various scenarios; S02: Classify the search targets in the search target library and generate search target tags corresponding to each scenario; S03: While the user imagines one of the aforementioned scenarios, the EEG signals are collected and recorded; S04: Preprocess the EEG signal to obtain continuous EEG signal segments within a short time window; S05: Embed the corresponding retrieval target label into the EEG signal segment to obtain training data; S06: Repeat steps S03-S05 multiple times for each scenario, record the training data collected multiple times, and construct a training dataset. S07: Set the training parameters of the multimodal fusion classification model, execute training and validation cycles in batches, optimize the model parameters using contrastive learning and triplet loss function in the training cycle, evaluate the model performance in the validation cycle, and perform multiple batches of training.

4. The scene imagination retrieval method according to claim 1, characterized in that, The scene is one or more of the following: static image, dynamic video, and audio. The search target is one or more of the following: images, videos, audio, and text.

5. The scene imagination retrieval method according to claim 1, characterized in that, Step S6 includes using one or more of the target feature vectors with the highest similarity as retrieval elements and obtaining the retrieval results through Approximate Nearest Neighbor (ANN) indexing technology.

6. The scene imagination retrieval method according to claim 1, characterized in that, After step S6, step S7 is also included: collecting feedback information from the human body, continuously fine-tuning the algorithm model based on the feedback information, and optimizing the accuracy of the retrieval results.

7. A scene imagination retrieval device, comprising a brain-computer interface module, a preprocessing module, and a computing module, wherein the computing module includes an encoder, a projection layer, a contrast layer, and a decoder; wherein, The brain-computer interface module is configured to collect the brain signals of a person while the person imagines a specific scene. The preprocessing module is configured to preprocess the EEG signal to obtain continuous EEG signal segments within a short time window; The encoder is configured to encode and fuse the EEG signal segments to obtain high-dimensional feature vectors, and to identify and encode the search targets in the search target library to obtain multiple target feature vectors. The projection layer is configured to perform cross-modal projection of the high-dimensional feature vector and multiple target feature vectors corresponding to the search target library, mapping them to a unified semantic space. The comparison layer is configured to calculate the similarity between the high-dimensional feature vector and the target feature vector; The decoder is configured to classify the search target based on the similarity and output the search results in order of matching confidence.

8. A scene imagination retrieval system, comprising: Memory is used to store non-temporary executable instructions for a computer. And a processor for running the computer-executable instructions, characterized in that the computer-executable instructions are executed by the processor to perform the scene imagination retrieval method according to any one of claims 1-6.

9. A non-transitory storage medium for non-transitory storage of computer-executable instructions, characterized in that, When the computer-executable instructions are executed by a computer, the scene imagination retrieval method according to any one of claims 1-6 is performed.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the scene imagination retrieval method according to any one of claims 1-6.

Images