A brain stimulation reconstruction method, medium and device based on visual experience retrieval

Abstract

The present application discloses a brain stimulation reconstruction method, medium and device based on visual experience retrieval, which relates to the field of computational vision technology. The method ensures the retention of semantic information in the reconstruction process by extracting low-level visual cues and high-level visual cues of fMRI signals to be processed; retrieves visual memory images by simulating the memory retrieval process of the human brain, provides context information for degraded visual stimuli, and further improves the accuracy of image reconstruction and enriches semantic details; generates a plurality of reconstructed images by utilizing low-level visual cues and super visual cues (the fusion result of CLIP features of high-level visual cues and visual memory images), and selects the final reconstructed image therefrom, which can fully capture or retain the key semantic information in the visual stimulation and improve the accuracy of the image reconstruction result.

Description

Technical Field

[0001] The present application relates to the field of computational vision technology, and in particular to a method, medium and device for brain stimulation reconstruction based on visual experience retrieval. Background Art

[0002] Currently, there are two main methods for decoding visual stimuli from fMRI. The first method mainly uses linear regression to draw the relationship between fMRI signals and visual stimulus features, and then uses deep neural networks such as Gan networks to reconstruct stimulus images. The second method mainly uses pre-trained natural language models or generative models to map neural signals to pre-trained models to generate latent variables, which are then used to reconstruct stimuli through diffusion models.

[0003] However, both methods have certain limitations and cannot fully capture or retain the key semantic information in visual stimuli, which affects the accuracy of image reconstruction results. Summary of the invention

[0004] The purpose of this application is to provide a brain stimulation reconstruction method, medium and device based on visual experience retrieval, which can fully capture or retain the key semantic information in the visual stimulation and improve the accuracy of the image reconstruction results.

[0005] To achieve the above objectives, this application provides the following solutions.

[0006] In a first aspect, the present application provides a method for brain stimulation reconstruction based on visual experience retrieval, comprising the following steps.

[0007] Acquire fMRI signals to be processed.

[0008] A perceptual visual cue is constructed according to the fMRI signal to be processed, wherein the perceptual visual cue includes a low-level visual cue and a high-level visual cue, the low-level visual cue is a fuzzy image, and the high-level visual cue is an image semantic feature.

[0009] The high-level visual clues and CLIP features of the visual memory image are fused to obtain super visual clues, wherein the visual memory image is a target image retrieved from a visual memory database, the visual memory database includes a plurality of images, and the target image refers to an image whose similarity with the high-level visual clues is greater than a threshold.

[0010] The low-level visual cues and the super visual cues are taken as inputs, and a general diffusion model is used to generate a plurality of reconstructed images.

[0011] The reconstructed image having the highest similarity to the super visual clue is selected as the final reconstructed image.

[0012] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a brain stimulation reconstruction method based on visual experience retrieval as described in the first aspect above.

[0013] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the brain stimulation reconstruction method based on visual experience retrieval described in the first aspect above.

[0014] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0015] The present application provides a brain stimulation reconstruction method, medium and device based on visual experience retrieval. The method ensures the retention of semantic information in the reconstruction process by extracting low-level visual cues and high-level visual cues of the fMRI signal to be processed; retrieves visual memory images by simulating the memory retrieval process of the human brain, provides contextual information for degraded visual stimuli, and further improves the accuracy of image reconstruction and enriches semantic details; generates a number of reconstructed images by utilizing low-level visual cues and super visual cues (the fusion result of CLIP features of high-level visual cues and visual memory images), and selects the final reconstructed image therefrom, which can fully capture or retain the key semantic information in the visual stimulus and improve the accuracy of the image reconstruction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is an application environment diagram of a brain stimulation reconstruction method based on visual experience retrieval in Example 1 of the present application;

[0018] Figure 2 A flowchart of a method for brain stimulation reconstruction based on visual experience retrieval provided in Example 1 of the present application;

[0019] Figure 3 This is a technical concept diagram of a brain stimulation reconstruction method based on visual experience retrieval in Example 1 of the present application;

[0020] Figure 4 A schematic diagram of the structure of a computer device provided in Example 2 of the present application. DETAILED DESCRIPTION

[0021] The study found that in the first method (i.e., using linear regression techniques to depict the correspondence between fMRI signals and visual stimulus features), the direct use of deep neural network mapping makes it difficult to capture semantic information from the stimulus, resulting in reduced image readability. In the second method (using a pre-trained natural language model or a generative model, mapping neural signals to a pre-trained model to produce latent variables, which are then used to reconstruct the stimulus through a diffusion model), the use of this multi-step process may lead to the loss of key semantic details, thereby limiting the accuracy of the reconstruction.

[0022] Whether through direct mapping of deep neural networks or through a complex process of multi-step generation, both types of methods may fail to fully capture or preserve key semantic information in visual stimuli, thus affecting the accuracy of the final reconstructed image.

[0023] In this regard, this embodiment provides a brain stimulation reconstruction method based on visual experience retrieval and related devices to overcome the above-mentioned technical defects.

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0026] Example 1

[0027] The brain stimulation reconstruction method based on visual experience retrieval provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the fMRI signal to be processed to the server 104. After the server 104 receives the fMRI signal to be processed, it constructs a perceptual visual clue according to the fMRI signal to be processed, wherein the perceptual visual clue includes a low-level visual clue and a high-level visual clue, the low-level visual clue is a fuzzy image, and the high-level visual clue is an image semantic feature; the high-level visual clue and the CLIP feature of the visual memory image are fused to obtain a super visual clue, wherein the visual memory image is a target image retrieved from a visual memory database, and the visual memory database includes several images, and the target image refers to an image whose similarity with the high-level visual clue is greater than a threshold; the low-level visual clue and the super visual clue are used as input, and several reconstructed images are generated using a general diffusion model; the reconstructed image with the highest similarity with the super visual clue is selected as the final reconstructed image. The server 104 can feed back the final reconstructed image to the terminal 102. In addition, in some embodiments, the brain stimulation reconstruction method based on visual experience retrieval can also be implemented by the server 104 or the terminal 102 alone, such as the terminal 102 can directly use the brain stimulation reconstruction method based on visual experience retrieval to process the fMRI signal to be processed, or the server 104 can obtain the fMRI signal to be processed from the data storage system and use the brain stimulation reconstruction method based on visual experience retrieval to process it.

[0028] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0029] In an exemplary embodiment, Figure 2 As shown, a method for brain stimulation reconstruction based on visual experience retrieval is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 201 to 205. Among them:

[0030] Step 201, obtaining fMRI signals to be processed.

[0031] Step 202, constructing perceptual visual cues according to the fMRI signal to be processed, wherein the perceptual visual cues include low-level visual cues and high-level visual cues, the low-level visual cues are blurred images, and the high-level visual cues are image semantic features.

[0032] Step 203, fusing the CLIP features of the high-level visual cues and the visual memory image to obtain a super visual cues, wherein the visual memory image is a target image retrieved from a visual memory database, the visual memory database includes a plurality of images, and the target image refers to an image having a similarity with the high-level visual cues greater than a threshold.

[0033] Step 204 : Taking the low-level visual cues and the super visual cues as input, a general diffusion model is used to generate a plurality of reconstructed images.

[0034] Step 205 , selecting the reconstructed image with the highest similarity to the super visual cue as the final reconstructed image.

[0035] Specifically, the CLIP image embedding of each reconstructed image is extracted, and the reconstructed image with the highest similarity between the CLIP image embedding and the super visual cue is selected as the final reconstructed image.

[0036] Through the above steps 201 to 205, this embodiment can extract low-level visual cues and high-level visual cues of the fMRI signal to be processed, ensuring the retention of semantic information during the reconstruction process; by simulating the memory retrieval process of the human brain, visual memory images are retrieved to provide context information for degraded visual stimuli, further improving the accuracy of image reconstruction and enriching semantic details; by using low-level visual cues and super visual cues (the fusion result of high-level visual cues and visual memory images) to generate CLIP image embedding, and selecting the final reconstructed image, the key semantic information in the visual stimulus can be fully captured or retained, thereby improving the accuracy of the image reconstruction result.

[0037] In order to make those skilled in the art more aware of the specific implementation process of the brain stimulation reconstruction method based on visual experience retrieval provided by this embodiment, the following Figure 3 Provide specific explanation.

[0038] The present embodiment provides a method for brain stimulation reconstruction based on visual experience retrieval, which is divided into three steps: 1. Constructing damaged perceptual visual clues V; 2. Fusing perceptual visual clues V by retrieving memory E; 3. Reconstructing visual stimulation to obtain I.

[0039] The reconstruction process can be expressed by the following formula.

[0040] I=V⊙E.

[0041] Among them, V represents the lossy perceptual visual cues mapped from the fMRI signal, E represents the relevant information retrieved from the visual experience database (i.e., visual memory database) based on the fMRI signal, and I represents the reconstructed image obtained by fusing the two (i.e., CLIP image embedding). The multiplication sign ⊙ in the formula represents the fusion operation of perceptual visual cues and retrieved information. This process is guided by the attention scoring mechanism to ensure that the retrieved information can effectively improve the accuracy of image reconstruction. It specifically includes the following steps.

[0042] Step 1: Construct lossy perceptual visual cues V.

[0043] In this step, the fMRI signals are encoded into two forms to construct the perceptually degraded perceptual visual cues, which include low-level visual cues V low and high-level visual cues V high , the low-level visual clue is the blurred image, and the high-level visual clue is the image semantic feature, V⊂(V low , V high ).

[0044] 1.1 Low-level visual cues V low Reconstruction is performed through a variational autoencoder (VAE) based on a stable diffusion model to capture pixel similarities.

[0045] For low-level visual cues, the state-of-the-art variational autoencoder technology combined with a stable diffusion model is used to reconstruct low-level visual cues, and the low-level visual cues (i.e., blurred images) are used as initial image guesses based on pixel similarity.

[0046] The variational autoencoder consists of an encoder, a multilayer perceptron, and a decoder combined with a stable diffusion model.

[0047] An encoder is trained to map fMRI signals of brain voxels to the VAE embedding space in Stable Diffusion. This mapping is then decoded using a fixed decoder in Stable Diffusion to generate low-level visual cues V low .

[0048] During the training process, the training images are encoded using a VAE encoder to extract their latent variables. Then, a multi-layer perceptron (MLP) is used to map the fMRI signals of brain voxels into the stable diffusion VAE embedding space. The resulting mapping is upsampled and enhanced based on CNN, and the upsampling operation makes the latent space embedding more consistent with the stable diffusion VAE architecture.

[0049] The process is to map the fMRI signals of brain voxels into a 64×64×4 latent space embedding to facilitate feature alignment.

[0050] The mean square error (MSE) loss is used to measure the mapping difference between the fMRI signal of the brain voxel and the latent variable, which promotes the network to learn how to map the voxel signal (i.e., the fMRI signal of the brain voxel) to the appropriate latent variable space.

[0051] During inference, a pre-trained variational autoencoder is used to predict VAE latent variables from test fMRI data, and a fixed decoder is used to decode low-level visual cues from these variables to provide an “initial guess” for image reconstruction.

[0052] 1.2. Advanced Clues V high The residual multi-layer perceptron (MLP) is used to map the image to the CLIP embedding space to capture semantic information.

[0053] Advanced Visual Cues V high Aiming to capture deeper semantic information, an MLP structure is used to map the fMRI signals of voxels into the CLIP embedding space.

[0054] The MLP module consists of a residual multilayer perceptron backbone and a residual projector. In detail, voxels are mapped to a 257×768 feature space through the backbone and projector to facilitate alignment with the image embedding of the CLIP ViT / L-14 model. The backbone of the MLP learns high-level visual cues through diffusion prior training. The projector is connected to the backbone and learns memory retrieval through contrastive learning to supplement subsequent visual cues.

[0055] During training, the loss function combines the diffusion prior loss and the MLP projection mapping loss with a weight of α = 0.3. The loss function is defined as

[0056] .

[0057] Among them, L BiM ixCo| SoftCLIP represents the combined loss of BiMixCo and SoftCLIP, BiMixCo represents data enhancement loss, SoftCLIP represents soft contrastive learning loss, L prior represents the diffusion prior loss.

[0058] Step 2: Fusion of visual cues V by retrieving memory E.

[0059] First, a Visual Memory Database (VMD) was established to represent the memories stored in the brain. Then, an Attentive Visual Signal Fusion Network (AVSFN) was designed to process the perceived visual cues and visual memory. Specifically, the Visual Memory Database contains 154,116 images. During the memory recall process, the high-level visual cues V high To retrieve VMD, we need to find the most similar images.

[0060] That is, given the total number of images in the visual experience library (i.e., the visual memory database) is N = 154116, the goal is to find the top N images with the highest similarity to a given embedding. In this process, the cosine similarity between the given embedding and the embedding in the visual experience is calculated.

[0061] Cosine similarity is defined as follows.

[0062] .

[0063] Where p represents the query embedding, e i represents the embedding of the i-th image in the visual experience library, · represents the dot product, and ||x|| represents the norm of x.

[0064] The retrieval results construct a visual experience cluster of p, which consists of the embeddings of the first n images with the highest cosine similarity, forming the following set: E = {e1, e2, …, e n According to previous psychological research results, four images were selected as visual experience clusters to enhance visual cues.

[0065] In order to enhance the integration of visual cues and visual experience, this embodiment designs an Attentive Visual Signal Fusion Network (AVSFN). The network combines a unique attention scoring mechanism that enables the method to operate only based on visual experience and visual cues without interacting with the decoder. Since the method is independent of the decoder, it provides plug-and-play features and is easy to extend to other multimodal fusion tasks. The main goal of AVSFN is to integrate visual experience e1, e2, e3, e4 with high-level visual cues V high Integrate and thus enrich the contextual information of visual cues.

[0066] The focused visual signal fusion network includes a multi-head self-attention layer, a fully connected layer (using the ReLU activation function), a scoring layer, and a Softmax layer connected in sequence. These four layers constitute a processing flow of a multi-head self-attention mechanism, and each layer has a different function.

[0067] Multihead Attention: The input feature sequence is processed simultaneously by multiple attention heads to capture the relationship between different positions in the sequence. Each attention head calculates a weighted representation, and the outputs of multiple heads are merged and mapped back to the input dimension to generate attn_output, which has the same dimension as the input feature sequence.

[0068] Data transfer: After the input feature sequence passes through this layer, a weighted feature representation is obtained, which is the weighted output (attn_output), which is a weighted representation of the input sequence with the same dimension as the input.

[0069] Fully connected layer (Linear): maps the attn_output output of the self-attention layer from the input_dim dimension to the hidden_dim dimension, which helps the network learn more complex feature representations. The ReLU activation function introduces nonlinearity to improve the expressiveness of the model.

[0070] Data transfer: After attn_output undergoes nonlinear transformation and ReLU activation, a new feature representation (i.e., transformed feature) is generated for subsequent processing.

[0071] Scoring layer (score_fc): This layer maps the features of hidden_dim to a scalar output, representing the importance of each feature. Using the Sigmoid activation function, the score range is compressed to [0, 1] so that it represents the "weight" or importance of each feature.

[0072] Data transfer: The nonlinearly transformed features are mapped from hidden_dim to a single scalar, and the attention score of each feature is obtained through Sigmoid activation.

[0073] Softmax layer: It is used to normalize the score of each feature so that the sum of the scores of all features is 1. It ensures that the relative importance of each feature can be compared and provides a basis for subsequent weighted calculations.

[0074] Data transfer: The score vector normalized by Softmax is used to weight the features and finally generate a weighted feature representation.

[0075] The high-level visual clues and the visual memory images are used as input, and a pre-trained focused visual signal fusion network model is used to output super visual clues, specifically including:

[0076] Input feature sequence X={V high, e1, e2, e3, e4}∈R, the feature set is a B×257×768 matrix, which is passed through a multi-head attention mechanism to capture the relationships between these features, where B represents the batch size.

[0077] The attention score for each of the features is calculated using each attention head: The attention mechanism operates on three vectors: query (Q), key (K), and value (V). The query matrix, key matrix, and value matrix are calculated as Q = XW Q 、K=XW K and V=XW V , where W Q , W K and W V The size is R 768×dk The learnable weight matrix, d k represents the feature dimension of each head. Then the attention score of each feature is calculated using the following formula.

[0078] .

[0079] Then, the outputs of multiple attention heads are concatenated and linearly transformed to obtain the final attention output (i.e., the comprehensive attention score), expressed as:

[0080] .

[0081] In the method, each attention head is calculated as attention (Q i , K i , V i ), where W O ∈R h·dk×768 is a learnable weight matrix, and h represents the number of attention heads.

[0082] In addition, the focused visual signal fusion network of this embodiment introduces a fully connected layer activated by ReLU, and generates a standardized attention score through the Sigmoid function to obtain a standard attention score, which is expressed as follows.

[0083] .

[0084] in, represents the comprehensive attention score, ReLU() represents the ReLU function, H represents the feature sequence, b1 and b2 represent bias terms, S represents the standard attention score, σ is the Sigmoid function, W1 and W2 represent weight matrices, W1∈R 768×H , W2∈R H×1 (H is the hidden layer dimension), the attention score of each element in X is recorded as S=(S1, S2, S3, S4, S5)∈R, expressed as a B×257×1 matrix.

[0085] The advantage of this design is that it can assign appropriate weights to the attention scores, effectively fuse feature representations, maximize visual information, and avoid the training of the decoder.

[0086] The design accurately captures the importance of each feature component through nonlinear transformation and normalization, while mitigating the noise introduced by the attention mechanism.

[0087] By calculating the attention scores of visual experience and high-level visual cues, the weight of each feature component is determined, and weighted fusion is performed to obtain the super visual cues V super , the expression is as follows.

[0088] .

[0089] where ⊙ represents element-wise multiplication.

[0090] This fusion approach ensures that the contribution of each visual experience to the overall output is consistent with its importance, thereby improving the quality of the final generated visual signal.

[0091] Step 3: Reconstruct the visual stimulus I.

[0092] Use the general diffusion (VD) generative model to guide the use of super visual cues V super Low-level visual cues V low Perform denoising.

[0093] For each fMRI sample, use the super visual cue V super The CLIP image embedding is generated and then used to guide the denoising process of the general diffusion. This process starts from the low-level vision pipeline (V low ), involving a denoising step using the UniPCMultistep noise schedule.

[0094] In the reconstruction process, 16 images (i.e., reconstructed images) are generated and one of them is selected as the final output. Specifically, for each test fMRI sample, the super visual cue V super Guided by low-level visual cues V through VD generative model low Denoising, generating 16 reconstructed images. Then, CLIP embedding and V super The image with the highest cosine similarity is taken as the final output.

[0095] This embodiment improves the quality and accuracy of image reconstruction by simulating the human visual perception process and combining a pre-trained model with visual experience retrieval. Compared with the prior art, the advancement of this embodiment lies in the following.

[0096] Visual perception process simulation: This embodiment draws on the operating principle of the human visual system and imitates the brain to improve the degraded visual clue perception by retrieving stored visual experience. First, visual clues are obtained through the visual system, and then these clues are supplemented and enriched by retrieving relevant memories in the brain, and finally a complete visual perception is formed.

[0097] Involvement of pre-trained models: This embodiment combines VAE (variational autoencoder) and CLIP (contrastive language-image pre-training model). The model can extract low-level and high-level visual clues, ensuring the preservation of semantic information during the reconstruction process.

[0098] Retrieval enhancement mechanism: This embodiment designs a visual experience database for retrieving stored visual memories, simulating the memory retrieval process of the human brain, providing contextual information for degraded visual stimuli, and further improving the accuracy and semantic details of reconstruction.

[0099] Fusion mechanism: The attention score-based visual signal fusion network (AVSFN) combines the retrieved complementary information to ensure that the model can better capture details during the reconstruction process and improve the quality of the reconstructed image.

[0100] The present application also provides an application scenario, which applies the above-mentioned method of brain stimulation reconstruction based on visual experience retrieval. Specifically: The method of brain stimulation reconstruction based on visual experience retrieval provided in this embodiment can be applied in the scenario of neuroscience research. The scenario includes a brain activity data acquisition link, a visual information processing link, and a stimulation reconstruction and analysis link. The method of brain stimulation reconstruction based on visual experience retrieval provided in this embodiment is a key technology in the stimulation reconstruction and analysis link. This method can help researchers better understand how the brain processes and responds to visual information, and thus provide new ideas for the diagnosis and treatment of neurological diseases.

[0101] Example 2

[0102] This embodiment provides a computer device, which may be a server or a terminal. Its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in the brain stimulation reconstruction method based on visual experience retrieval. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a brain stimulation reconstruction method based on visual experience retrieval in Example 1 is implemented.

[0103] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0104] Example 3

[0105] This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program. When the processor executes the computer program, a brain stimulation reconstruction method based on visual experience retrieval in the above-mentioned embodiment 1 is implemented.

[0106] Example 4

[0107] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for brain stimulation reconstruction based on visual experience retrieval in the above-mentioned embodiment 1 is implemented.

[0108] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0109] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0110] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0111] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A brain stimulation reconstruction method based on visual experience retrieval, characterized in that: The brain stimulation reconstruction method based on visual experience retrieval comprises: Acquiring fMRI signals to be processed; Constructing perceptual visual cues according to the fMRI signals to be processed, wherein the perceptual visual cues include low-level visual cues and high-level visual cues, the low-level visual cues are fuzzy images, and the high-level visual cues are image semantic features; fusing the high-level visual clues and CLIP features of visual memory images to obtain super visual clues, wherein the visual memory image is a target image retrieved from a visual memory database, the visual memory database includes a plurality of images, and the target image refers to an image whose similarity to the high-level visual clues is greater than a threshold; Taking the low-level visual cues and the super visual cues as input, generating a plurality of reconstructed images using a general diffusion model; Selecting the reconstructed image with the highest similarity to the super visual clue as the final reconstructed image; The CLIP features of the advanced visual cues and visual memory images are integrated to obtain super visual cues, which specifically include: The high-level visual cues and the CLIP features of the visual memory image are taken as input, and a pre-trained focused visual signal fusion network model is used to output super visual cues, wherein the focused visual signal fusion network model is a focused visual signal fusion network based on a multi-head attention mechanism, and the focused visual signal fusion network includes a multi-head self-attention layer, a fully connected layer, a scoring layer and a Softmax layer connected in sequence; the multi-head attention layer is used to process the feature sequence through multiple attention heads to obtain a weighted feature representation, and the feature sequence is a sequence determined according to the high-level visual cues and the CLIP features of the visual memory image; the fully connected layer is used to perform a nonlinear transformation on the weighted feature representation to obtain a transformed feature; the scoring layer is used to calculate the attention score of the transformed feature; the Softmax layer is used to normalize the attention score.

2. The brain stimulation reconstruction method based on visual experience retrieval according to claim 1, characterized in that: The high-level visual clues and the CLIP features of the visual memory image are used as input, and the pre-trained focused visual signal fusion network model is used to output super visual clues, specifically including: Inputting the feature sequence; Calculate the attention score of each feature in the feature sequence using each attention head; Obtaining a comprehensive attention score according to the attention scores calculated by all the attention heads; Using the Sigmoid function to convert the comprehensive attention score into a standard attention score; Assigning a weight to each of the features according to the standard attention score; The super visual clue is obtained according to the weight and the feature.

3. The brain stimulation reconstruction method based on visual experience retrieval according to claim 1, characterized in that: The process of constructing the low-level visual clues specifically includes: The fMRI signal to be processed is taken as input, and the low-level visual clues are outputted using a pre-trained variational autoencoder, wherein the variational autoencoder includes an encoder, a multi-layer perceptron, and a decoder combined with a stable diffusion model.

4. The brain stimulation reconstruction method based on visual experience retrieval according to claim 1, characterized in that: The process of constructing the high-level visual clues specifically includes: The processed fMRI signals are mapped into the CLIP embedding space using a pre-trained residual multilayer perceptron to obtain high-level visual cues.

5. The brain stimulation reconstruction method based on visual experience retrieval according to claim 4, characterized in that: The loss function of the residual multilayer perceptron is expressed as: ； Among them, L BiM ixCo| SoftCLIP represents the combined loss of BiMixCo and SoftCLIP, BiMixCo represents data enhancement loss, SoftCLIP represents soft contrastive learning loss, α is the weight, L prior represents the diffusion prior loss.

6. The brain stimulation reconstruction method based on visual experience retrieval according to claim 2, characterized in that: The calculation formula of the standard attention score is: ； in, represents the comprehensive attention score, ReLU() represents the ReLU function, H represents the feature sequence, b1 and b2 represent bias terms, S represents the standard attention score, σ is the Sigmoid function, and W1 and W2 represent weight matrices.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the brain stimulation reconstruction method based on visual experience retrieval according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the brain stimulation reconstruction method based on visual experience retrieval described in any one of claims 1 to 6 is implemented.

Images