Gpt-based progressive migration training decoding method

Abstract

This invention discloses a progressive transfer training decoding method based on GPT, belonging to the field of visual neural decoding. Based on functional magnetic resonance imaging (fMRI) signal data recorded while subjects viewed a large number of visual perception images, this invention constructs a GPT-based category and text decoding model, which includes: an image encoder for embedding image feature vectors extracted using a pre-trained model into a latent feature space; a text encoder for embedding prompts and positional codes containing task information into the latent feature space; a visual information encoder for mapping voxel signals from the brain's visual regions to the latent feature space using a GRU; a whole-brain information encoder for guiding the selection of voxel signals from other brain regions based on the voxel signals from the brain's visual regions as context, and mapping global information to the latent feature space; and a language generator for fusing all the above latent features and selecting the token with the highest probability to generate text corresponding to the content. This invention improves decoding accuracy and enhances the model's generalization ability.

Description

Technical Field

[0001] This invention relates to the field of visual neural decoding, and more particularly to a progressive transfer training decoding method based on GPT. Background Technology

[0002] Visual neural decoding has received increasing attention in recent decades, with many researchers devoting considerable effort to exploring various mechanisms of neural decoding. These mechanisms can be categorized into three types based on decoding type: visual stimulus classification, visual stimulus recognition, and visual stimulus reconstruction. Research on visual stimulus classification began as early as 2001 with Haxby et al. They used the MVPA method based on fMRI (Functional Magnetic Resonance Imaging) signals to classify eight different categories presented to participants. Yargholi and Hossein-Zadeh used an enhanced Naive Bayes classifier to decode fMRI signals responding to handwritten digits into the correct categories.

[0003] These early neural decoding methods for classification typically used classifiers to learn the relationship between brain activity and a predefined set of labels, making it impossible to apply these models to new categories. The work of Horikawa and Kamitani broke through this limitation. They proposed decoding arbitrary object categories by learning a linear regression function that transforms fMRI patterns into convolutional neural network features of the viewed image. New image categories can be estimated by comparing the correlation coefficients between the predicted image features and the average visual features of various categories. However, the performance of this method is significantly dependent on a large amount of paired stimulus and response data. With the continuous updates and iterations of deep learning technology, emerging technologies such as VAEs and Transformers have been applied to the field of neural decoding. Du et al. proposed BraVL, a VAE-based model consisting of two parts: multimodal joint modeling and mutual information regularization, to learn the relationship between brain activity, visual stimuli, and Wikipedia descriptions of categories. Support vector machines are then used to classify the learned latent representations. Furthermore, Liu proposed the BrainCLIP model in 2023, which aims to train a mapping network to transform fMRI patterns into representations well aligned with the CLIP embedding space, decoding visual information by combining visual and textual supervision. Their research confirms the importance of multimodal information in visual decoding.

[0004] However, previous research still faces many challenges. Firstly, there is a lack of research on multi-task decoding; previous studies often focused on a single task, neglecting the connections between tasks. Secondly, with technological advancements, previous models have failed to achieve ideal performance and lack cross-modal generalization capabilities. Summary of the Invention

[0005] This invention provides a progressive transfer training decoding method based on GPT (an autoregressive language model based on the Transformer architecture) for multi-task decoding involving category and language decoding in brain decoding of biomedical images. This invention can solve the problem of the lack of cross-modal generalization ability in existing technologies and improve model performance through progressive transfer learning.

[0006] The technical solution adopted in this invention is as follows:

[0007] A progressive transfer learning decoding method based on GPT, comprising the following steps:

[0008] Step 1: Construct a GPT-based category and text decoding model based on the functional magnetic resonance signals collected from the subjects while they are viewing visually perceiving images. The model includes: a text encoder, an image encoder, a visual information encoder, a whole-brain information encoder, a feature fusion unit, and a language generator.

[0009] in,

[0010] The text encoder is used to embed the image description text containing cue words (i.e., image description text in natural language) and cue word position encoding of the visually perceived image viewed by the subject into the latent feature space to obtain the text latent feature vector;

[0011] An image encoder is used to embed multiple image feature vectors of the visually perceived image viewed by the subject into a latent feature space to obtain image latent feature vectors; wherein, each image feature vector is extracted through a pre-trained image feature extraction model;

[0012] The visual information encoder uses a recurrent neural network to map the activity signals of the brain's visual region in the acquired functional magnetic resonance imaging signals to the latent feature space, thereby obtaining the visual latent feature vector.

[0013] The whole-brain information encoder uses the activity signals of the visual region of the brain as context and maps the response signals of the whole brain other than the visual region of the brain to the latent feature space based on the multi-head attention mechanism to obtain non-visual latent feature vectors.

[0014] The feature fusion unit is used to fuse text latent feature vectors, image latent feature vectors, visual latent feature vectors, and non-visual latent feature vectors to generate a fused feature vector. That is, in this invention, the feature vector dimension of the latent feature space can be customized based on the actual scene, while the dimension of the feature vectors extracted by the text encoder, image encoder, visual information encoder, and whole brain information encoder remains consistent.

[0015] The language generator takes the fused feature vector output by the feature fusion ...

[0016] Step 2: Train the constructed GPT-based category and text decoding model based on the set training loss function. During training, the feature fusioner uses a progressive transfer learning training strategy to generate different fused feature vectors.

[0017] When the preset training convergence conditions are met, the category and text decoding models for visual neural decoding are obtained.

[0018] Furthermore, the text encoder specifically refers to:

[0019] The input image description text containing prompt words is segmented according to a pre-set vocabulary for encoding to obtain a word sequence of a specified length; that is, the image description text is segmented, and then the corresponding number in the vocabulary is queried for each segment, and the numbers are combined into a sequence to obtain a word sequence of length L (custom value);

[0020] Multiply the sequence by the preset word embedding matrix (WTE) to obtain an L*Num sequence of word embedding vectors, where Num represents the dimension of the word embedding vectors;

[0021] Multiply the word sequence by the position embedding matrix (WPE) of the cue words to calculate the position-encoded position embedding vector sequence, which also has a dimension of L*Num; where WPE is used to describe the order of the cue words in the sentence so that the model can understand the order information of the cue words;

[0022] The word embedding vector sequence and the position embedding vector sequence are linearly added to obtain the fused sentence embedding vector sequence, which is the text latent feature vector.

[0023] Furthermore, the image encoder is based on a learnable parameter vector (θ) (1) ,θ (2) ,...,θ (M) ), assign weights to each type of image feature vector, where M represents the number of types of image feature vectors, θ (m) Let θ represent the weights of the m-th image feature vector, where m = 1, 2, ..., M, and θ (m) The initial values ​​are all the same. When training the GPT-based category and text decoding model, after training each sample, the values ​​of θ are adjusted based on the loss function. (1) ,θ (2) ,...,θ (M) Update; based on weight θ (m)The image feature vectors of M types are weighted and fused to obtain the latent feature vector of the image output by the image encoder (such as described as image fusion feature I).

[0024] Furthermore, the image feature vectors include ten types: Semantic Representation Model Bert text features, Clip (Contrastive Language-Image Pre-training) image features, Clip text features, Dino (DETR with Improved deNoising AnchOr boxes) image features, Gemma text features, Gpt text features, Llama (Large Language Model Meta) text features, Swin (a visual model based on Transformer architecture) image features, Vgg (Visual Geometry Group Network) image features, and Vit (Vision Transformer) image features, which are the features extracted from the backbone networks of existing mainstream text and / or image processing models.

[0025] Furthermore, the visual information encoder is specifically as follows:

[0026] Based on the adopted brain region segmentation template, the voxel points distributed in multiple different visual regions are subjected to upsampling or downsampling operations using linear interpolation. The data dimension of the collected activity signals is unified to the latent feature space dimension to obtain the feature vector of each visual region. Then, based on the feature vectors of all visual regions, an N1*Num visual region feature sequence is formed, where N1 represents the number of visual regions in the brain region segmentation template.

[0027] The visual region feature sequence is input into a recurrent neural network (preferably a gated recurrent unit GRU). The recurrent neural network integrates the visual region feature sequence into a single feature vector, and the Num-dimensional hidden features in the final stage of the recurrent neural network are taken as the visual latent feature vector.

[0028] Furthermore, when the stride of the recurrent neural network is the same as the number of visual regions in the brain, the number of layers is 1, and the size of the input layer and the output layer are the same, both being Num.

[0029] Furthermore, the whole-brain information encoder specifically refers to:

[0030] Based on the adopted brain region segmentation template, the voxel points distributed in different non-visual regions are subjected to upsampling or downsampling operations using linear interpolation. The data dimension of the collected activity signals is unified to the latent feature space dimension to obtain the feature vectors of each non-visual region. Then, based on the feature vectors of all non-visual regions, an N2*Num non-visual region feature sequence is formed, where N2 represents the number of non-visual regions in the brain region segmentation template.

[0031] Initialize three fully connected linear layers q, k, and v to represent the query, key, and value, respectively, with weight matrices W for each layer. Q W K W V Each weight matrix has a dimension of Num*Num, and each linear fully connected layer includes the same number of weight matrices, all set to H. The N1*Num visual region feature sequences are input into the fully connected layers k and v, respectively, along with the H weight matrices W. K W V Multiplying these results in H N1*Num K and V matrices; then, the N2*Num non-visual region feature sequences are multiplied by the H weight matrices W. Q Multiplying them together yields H Q matrices of type N2*Num;

[0032] Multiply the H Q matrices by the transposes of the H K matrices, and then multiply them by the H V matrices to obtain H N2*Num Z matrices; concatenate the H Z matrices by rows and input them into a fully connected layer to adjust the dimensions to obtain an N2*Num Z' matrix; then average the matrix Z' by columns to obtain the Num-dimensional non-visual region feature vector.

[0033] Furthermore, the feature fusion unit is specifically as follows:

[0034] Add the visual latent feature vector to the non-visual region feature vector to obtain the feature vector A of the whole brain region;

[0035] Based on the weight α of the latent feature vector of the image (i.e., the image fusion feature I), it is fused with the feature vector A to obtain the feature vector B = α*I + (1-α)*A;

[0036] The feature vector B is then added to the vector corresponding to the prompt word part in the latent feature vector of the text to generate a fused feature vector, which is then used as the input to the GPT decoder.

[0037] The weight α gradually decreases from 1 to 0, and the rate of decrease is negatively correlated with the size of the training epochs. That is, the larger the training epochs, the slower the rate of decrease. If the number of training epochs is too small, the rate of decrease needs to be limited to prevent the model from learning insufficiently. If the number of training epochs is sufficient, the rate of decrease does not need to be limited.

[0038] Furthermore, the progressive transfer learning training strategy is divided into three stages based on the number of training rounds. For example, two training round thresholds can be set. The initial stage is the first stage. When the first training round threshold is reached, the second stage is considered to have been reached. When the second training round threshold is reached, the third stage is considered to have been reached. In the first stage, the weight α is set to feature 1, and feature B is I. In the second stage, the weight α decreases from 1 to 0 in an arithmetic progression, and feature B is α*I+(1-α)*A. In the third stage, the weight α is set to 0, and feature B is A.

[0039] The language generator takes the fused feature vector output by the feature fusion ...

[0040] Furthermore, the language generator specifically refers to:

[0041] (1) The fused feature vector is passed through all network layers of the decoder of the GPT model to output a text prediction vector of descriptive words that characterize the visual stimuli of the image.

[0042] Then, based on the pre-set vocabulary for generating words (i.e., the vocabulary of the GPT-based category and text decoding model, which can be customized by the user), each word in the vector is scored to obtain the predicted probability of each word. Then, the word with the highest probability is selected for output, which is the current predicted generated word.

[0043] (2) Add the generated words output in (1) to the current input sequence of the decoder of the GPT model, so that the decoder can make the next prediction;

[0044] Repeat steps (1) and (2) until the token at the end of the sentence is a preset terminator, and obtain the text prediction vector based on all generated words.

[0045] Furthermore, the predicted probability of each word is obtained by multiplying the output text prediction vector with the lm_head word embedding matrix of the lm_head component of the GPT language model to obtain the predicted probability list of the next word, i.e., the logits list, and selecting the maximum value as the generated token.

[0046] Furthermore, in step 2, the training loss function is set as follows:

[0047] L=β1L CE -β2Cosine Similarity(A′,B′)

[0048] Where L is the total loss of the GPT-based category and text decoding model, L CE The model's cross-entropy loss is β1, and the cosine similarity(A′,B′) is the feature cosine similarity loss. β1 and β2 are the preset weights for the two losses.

[0049] Cross-entropy loss Where N is the number of target tokens to be predicted, and g i It is an index of the vocabulary corresponding to manually annotated natural language words in the training samples, p i It is the predicted probability of generating the token;

[0050] Feature cosine similarity loss Here, A′ is the latent feature vector of the image output by the image encoder, and B′ is the feature vector of the whole brain region. The obtained value Cosine Similarity(A′,B′)∈[-1,1] is used to evaluate the similarity between the two features. A value of 1 indicates that the two features have the same orientation, and a value of 0 indicates that the two features have no similarity. Thus, in the optimization process, both fMRI and image features can be aligned, and the prediction accuracy of the model can be improved.

[0051] The technical solution provided by this invention brings at least the following beneficial effects:

[0052] This invention employs a training strategy based on progressive transfer learning, using a text encoder, image encoder, visual information encoder, whole-brain information encoder, and language generator to perform multi-task decoding of natural images, including category classification and text decoding. The resulting category accuracy is high, and the text descriptions closely match natural language. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the structure of the GPT-based category and text decoding model used in an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the confusion matrix results for category decoding in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the text decoding result according to an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0058] This invention provides a progressive transfer learning-based decoding method based on GPT. Referring to Figure 1, this method first constructs a GPT-based category and text decoding model based on functional magnetic resonance imaging (fMRI) signals collected from subjects while viewing visually perceived images. Second, it trains the constructed GPT-based category and text decoding model using a set training loss function. During training, the feature fusion unit employs a progressive transfer learning training strategy to generate different fused feature vectors. When a preset training convergence condition is met (such as reaching the maximum number of training iterations or the loss function value converging), a category and text decoding model for visual neural decoding is obtained. The model constructed in this method includes: a text encoder, an image encoder, a visual information encoder, a whole-brain information encoder, a feature fusion unit, and a language generator. The text encoder embeds the image description text containing prompts (i.e., image description text in natural language) and the prompt position encodings of the visually perceived images viewed by the subject into the latent feature space to obtain text latent feature vectors. The image encoder embeds multiple image feature vectors of the input visually perceived images viewed by the subject into the latent feature space to obtain image latent feature vectors. Each image feature vector is extracted using a pre-trained image feature extraction model. The visual information encoder uses a recurrent neural network to process the acquired functional magnetic field data. The activity signals of the brain's visual region in the resonance signal are mapped to the latent feature space to obtain visual latent feature vectors. The whole-brain information encoder, using the activity signals of the brain's visual region as context, maps response signals from all areas of the brain except the visual region to the latent feature space based on a multi-head attention mechanism to obtain non-visual latent feature vectors. A feature fusion unit is used to fuse text latent feature vectors, image latent feature vectors, visual latent feature vectors, and non-visual latent feature vectors to generate a fused feature vector. That is, in this invention, the dimension of the feature vectors in the latent feature space can be customized based on the actual scene, while the dimension of the feature vectors extracted by the text encoder, image encoder, visual information encoder, and whole-brain information encoder remains consistent. A language generator takes the fused feature vector output by the feature fusion unit as input, obtains the text prediction vector of the current input fused feature vector through the decoder of the GPT model, and then converts the text prediction vector into text describing the visual stimulus of the image based on a preset word embedding matrix.

[0059] In one embodiment, constructing a GPT-based category and text decoding model specifically includes the following steps:

[0060] Step 1. The text encoder transforms the text containing the Prompt and the natural language description of the image into vectors in the latent feature space. This vector sequence contains positional information.

[0061] Step 1.1: The input text containing "Prompt" is segmented according to the vocabulary to obtain a word sequence, that is, the corresponding number is queried in the vocabulary and the number is combined into a sequence to obtain a token sequence of length L.

[0062] Step 1.2: Multiply the token sequence with the pre-set word embedding matrix (WTE) to calculate the L*Num word embedding vector sequence, where Num represents the dimension of the word embedding vector;

[0063] Step 1.3: Multiply the word sequence obtained in Step 1.1 with the position embedding matrix (WPE) of the Prompt to calculate the position-encoded position embedding vector sequence;

[0064] Step 1.4: Linearly add the word embedding vector sequence and the position embedding vector sequence to obtain the fused sentence embedding vector sequence.

[0065] Step 2. The image encoder converts the stimulus images viewed by the subjects into image feature vectors that the model can input through ten different pre-trained architectures, and adds the ten features together through learnable parameters;

[0066] Step 2.1: Input the input image into ten different pre-trained models to obtain Bert text features, Clip image and text features, Dino image features, Gemma text features, Gpt text features, Llama text features, Swin image features, Vgg image features, and Vit image features, which include most of the mainstream pre-trained models.

[0067] Step 2.2: Through a learnable parameter vector (θ) (1) ,θ (2) ,...,θ (10) Each feature is assigned a weight, initially set to 1. The model learns to reduce the weight of useless features and increase the weight of effective features. These ten parameters are updated after each training iteration on a sample.

[0068] Step 2.3: Multiply each of the ten features by the ten weight parameters and add them together to obtain the final output feature I.

[0069] The length of the further image feature vector I is the same as the input vector length of GPT-2, which is 768.

[0070] Step 3. The visual information encoder converts the brain visual area activity signals obtained from magnetic resonance imaging during the viewing of visual perception images into latent feature space vectors:

[0071] Step 3.1: Perform upsampling or downsampling operations on voxel points distributed in 29 different visual regions using linear interpolation methods to unify the data dimension to the embedding space dimension, and then combine the feature vectors of the different visual regions to form a 29*Num visual region feature sequence.

[0072] Step 3.2: Input the visual region feature sequence into the GRU gated recurrent unit. The GRU integrates the visual region feature sequence into a single feature vector and takes the Num-dimensional hidden features of the final stage of the GRU as the visual region feature vector.

[0073] In this embodiment, the step size of the GRU is the same as the number of visual regions in the brain, the number of layers is 1, and the size of the input layer and the output layer is the same, both being Num.

[0074] Step 4. The whole-brain information encoder converts response signals from all areas of the brain, excluding the visual regions, into latent feature space vectors to obtain feature vectors from non-visual regions:

[0075] Step 4.1: Similar to step 3.1, perform upsampling or downsampling operations on voxel points distributed in 151 different non-visual regions based on the linear interpolation method to unify the data dimension to the embedding space dimension. Then, sort the response signals of different non-visual regions according to the non-visual region number to form a 151*Num non-visual region feature sequence.

[0076] Step 4.2: Initialize the linear fully connected layers q, k, v, and the weight matrices of each linear fully connected layer are W. W W K W V Each weight matrix has a dimension of Num*Num, and each linear fully connected layer includes the same number of weight matrices, all set to H. The 29*Num visual region feature sequences are input into fully connected layers k and v, respectively, along with H weight matrices W. K W V Multiplying them yields H 29*Num K and V matrices; the 151*Num non-visual region feature sequence is then multiplied by the H weight matrices W. Q Multiplying them together yields H Q matrices of size 151*Num;

[0077] Step 4.3: Multiply the H Q matrices by the transposes of the H K matrices, and then multiply by the H V matrices to obtain H 151*Num Z matrices. Concatenate the H Z matrices row by row and input them into a fully connected layer to change the dimensions, resulting in a 151*Num Z' matrix. Average the Z' matrix column by column to obtain Num-dimensional hidden features as feature vectors for non-visual regions.

[0078] 5. The feature vectors obtained in steps 1, 2, 3, and 4 are fused into a single feature using a feature fusion processor. Different features are then generated based on the progressive transfer learning training strategy and fed into the decoder.

[0079] Step 5.1: Add the visual region feature vectors and non-visual region feature vectors from Steps 3 and 4 to obtain the whole-brain region feature vector A. Then, add feature vector A to the image feature vector I from Step 2 using a weight value α, i.e., α*I+(1-α)*A, where the value of α gradually decreases from 1 to 0. Finally, denote this feature as B and add it to the vector corresponding to the cue word part in the embedded vector sequence from Step 1 to obtain the input of the GPT decoder.

[0080] Step 5.2: The training strategy of progressive transfer learning can be divided into three stages: In the first stage, the value of α is 1, and the feature B is I; in the second stage, the value of α decreases from 1 to 0 in an arithmetic sequence, and the feature B is α*I+(1-α)*A; in the third stage, the value of α is 0, and the feature B is A.

[0081] Step 6. The language generator obtains the text prediction vector through the GPT decoder (preferably the GPT-2 decoder) in Step 5, and transforms it into text describing the image stimuli based on a preset word embedding matrix:

[0082] Step 6.1: Depending on the training stage, the input features to the GPT decoder will differ, with the initial input sequence being the output features of the feature fusion unit. The input vector passes through all layers of the GPT decoder, producing a single vector. This vector is scored according to the model's vocabulary, and then the token with the highest probability is output.

[0083] Step 6.2: Add the token output from the previous step to the current input sequence of the decoder, allowing the decoder to make the next prediction. Repeat this process until the token at the end of the sentence becomes the end-of-sentence marker, finally obtaining the text prediction vector.

[0084] Preferably, the embedding vector length of each token in GPT-2 is 768. The probability of generating a token is obtained by multiplying the output text prediction vector with the lm_head word embedding matrix of the language model's lm_head component to obtain a list of predicted probabilities for the next word, i.e., a list of logits. The maximum value among these logits is selected as the generated token.

[0085] The lm_head module generates a probability distribution for predicting the next word based on the input text prediction vector. This prediction is based on the mapping between the input vector and the model's internal state. An lm_head typically includes one or more fully connected layers that transform the input vector into a higher-dimensional representation used to generate the predicted probabilities. In addition to linear layers, lm_heads often include extra components such as normalization layers or attention mechanisms to enhance the model's representational power and the accuracy of the generated probabilities. These components can be tailored and optimized according to the specific task and model architecture.

[0086] In one embodiment, the loss function for the GPT-based category and text decoding model is set as follows:

[0087] 1) Calculate the cross-entropy loss L CE :

[0088]

[0089] Where N is the number of target tokens to be predicted, and g i It is an index of the vocabulary corresponding to manually annotated natural language words in the training samples, p i It is the predicted probability of generating the token.

[0090] 2) Calculate the loss using cosine similarity:

[0091]

[0092] Where A′ is the feature obtained by the image encoder, i.e., the image feature vector I, and B′ is the feature vector of the whole brain region (referred to as brain feature) obtained by adding the visual region feature vector and the non-visual region feature vector in steps 3 and 4 above.

[0093] The obtained value Cosine Similarity(A,B)∈[-1,1] is used to evaluate the similarity between two features. When it is 1, it means that the two features have the same direction, and when it is 0, it means that the two features have no similarity.

[0094] 3) Calculate the final loss:

[0095] L = L CE -Cosine Similarity(A,B)

[0096] This approach allows for the alignment of fMRI and image features during the optimization process, while also improving the model's prediction accuracy.

[0097] The GPT-based progressive transfer training decoding method provided in this invention solves the problem that multiple decoding tasks cannot be performed in a single decoding model, and introduces effective global information from the brain, thereby improving decoding accuracy and enhancing the model's generalization ability.

[0098] Example

[0099] In one embodiment, the present invention provides a progressive transfer learning decoding method based on GPT, which can simultaneously decode category information and text information. The method proposed in this embodiment includes the following implementation steps:

[0100] Step S1: Construct the image encoding module.

[0101] Step S101: Extract image features using ten different pre-trained models: Bert text encoder, Clip image and text encoder, Dino image encoder, Gemma text encoder, Gpt text encoder, Llama text encoder, Swin image encoder, Vgg image encoder, and Vit image encoder, resulting in ten feature vectors (x1, x2, ..., x68) of size 768. 10 ).

[0102] Step S102: Combine the ten feature vectors obtained in step S101 with the learnable parameter vector (θ). (1) ,θ (2) ,...,θ (10) The features are multiplied separately and then added together to obtain a sequence of feature vectors of size 768.

[0103] Step S2: Construct the Prompt encoding module.

[0104] Step S201: Design the Prompt input text, which contains the subject number and the decoding task (category decoding and text decoding), separated by the # symbol. For example, when decoding subject number 01, the Prompt would be "0#01#Decode subject01's fMRI captions:", where the first number is the sample number and the number between the two # symbols is the subject number.

[0105] Step S202: Input the Prompt text obtained in step S201 into the Tokenizer, which converts it into a Token sequence, with each Token corresponding to a numerical index according to the vocabulary. Input this index sequence into the model, which uses these index sequences to distinguish and encode different Tokens.

[0106] Step S203: Multiply the Token sequence with the Word Embedding Matrix (WTE) to obtain a 9*768 word embedding vector sequence.

[0107] Step S204: Multiply the Token sequence obtained in step S203 with the preset position embedding matrix (WPE) to calculate the position-encoded position embedding vector sequence.

[0108] Step S3: Construct the visual encoding module.

[0109] Step S301: Perform upsampling or downsampling operations on voxel points distributed in 29 different visual regions using a linear interpolation method to unify the dimension size with the dimension of the position embedding vector sequence. Then, sort the feature vectors of different visual regions according to the visual region number to form a 29*768 visual region feature sequence.

[0110] Step S302: Input the visual region feature sequence into the GRU gated recurrent unit. The GRU integrates the 29 visual region feature sequences into a single feature vector and takes the 1*768-dimensional hidden feature of the final stage of the GRU as the visual region feature vector.

[0111] Step S4: Construct the whole-brain coding module.

[0112] Step S401: Similar to step S3, perform upsampling or downsampling operations on voxel points distributed in 151 different non-visual regions using a linear interpolation method to make the dimension size consistent with the dimension of the position embedding vector sequence. Then, sort the response signals of the different non-visual regions according to the non-visual region number to form a 151*768 non-visual region feature sequence.

[0113] Step S402: Initialize the linear fully connected layers q, k, v, and the weight matrices of each linear fully connected layer are W. Q W K W V Each weight matrix has a dimension of 768*768, and each linear fully connected layer includes the same number of weight matrices, all set to 8. The 29*768 visual region feature sequences are input into the fully connected layers k and v, respectively, along with H weight matrices W. K W V Multiplying them yields eight 29*768 K and V matrices; the 151*768 non-visual region feature sequence is then multiplied by the eight weight matrices W. Q Multiplying them together yields eight Q matrices of 151*768 each.

[0114] Step S403: Multiply the 8 Q matrices by the transposes of the 8 K matrices, and then multiply by the 8 V matrices to obtain 8 Z matrices of 151*768. Concatenate the 8 Z matrices row by row and input them into a fully connected layer to change the dimensions, resulting in a Z' matrix of 151*768. Average the Z' matrix column by column to obtain 768-dimensional hidden features as feature vectors for non-visual regions.

[0115] Step S5: Build the language generator.

[0116] Step S501: Depending on the training stage, the input features to the GPT-2 decoder will vary. The input vector passes through all layers of the decoder, producing a single vector. This vector is scored according to the model's vocabulary, and then the token with the highest probability is output.

[0117] Step S502: Add the token output from step S501 to the input sequence, allowing the model to make the next prediction. Repeat this process until the token at the end of the sentence is the terminator "<|endoftext|>", finally obtaining the text prediction vector.

[0118] Step S6: Training Phase

[0119] Step S601: Obtain stimulus images, brain fMRI signals, and manually labeled image categories and text descriptions from the training set to form a sample group.

[0120] Step S602: Add the visual region feature vectors and non-visual region feature vectors from steps S3 and S4 to obtain a 768-dimensional feature vector for the whole brain region. Then, multiply the whole brain region feature vector by 1-α using a weight value α, multiply the 768-dimensional image feature vector from step S1 by α, and finally add them together. The value of α gradually decreases from 1 to 0 with each training iteration. Finally, add this feature vector to the vector corresponding to the Prompt part of the embedded vector sequence in step S2 to obtain the input of the GPT decoder.

[0121] Step S603: Use attention masking to mask the sentence, obtain the preceding text of the sentence word by word, score the tokens according to the preceding text and the corresponding fMRI signals, and select the token with the highest probability to generate the following sentence word by word, so as to obtain the distribution of the predicted following text.

[0122] Step S604: Based on the set total loss L, use the AdamW optimization algorithm to update the corresponding weights of the parameters of the entire model.

[0123] The specific optimization parameters for AdamW are as follows: the step size ε defaults to 0.00015; the exponential decay rates ρ1 and ρ2 for moment estimation default to 0.9 and 0.999 respectively; and the numerical stability constant δ defaults to 10. -8 The weight decay coefficient λ is set to 10 by default. -2 Initialize the parameters θ with a normal distribution of mean 0 and variance 0.001; initialize the first and second moment variables as s = 0, r = 0; initialize the time step to t = 0; each time there are m context sample pairs {(x (1) ,y (1) ),(x (2) ,y (2) ),…,(x (m) ,y (m) The following updates will be made:

[0124] Calculate the gradient: Where x (i) For input, f(x) (i) ;θ) represents the predicted output, L(f(x) (i) ;θ),y (i) () represents the loss function for the i-th sample;

[0125] Update time step: t = t + 1;

[0126] Update the biased first moment estimate: s = ρ1s + (1-ρ1)g, where ρ1 represents the exponential decay rate of the first moment estimate (default is 0.9), and s represents the first moment variable;

[0127] Update the partial second moment estimate: r = ρ2r + (1-ρ2)g⊙g, where ρ2 represents the exponential decay rate of the second moment estimate (default is 0.999), and r represents the second moment variable;

[0128] Correcting the deviation of the first moment: in: The variable representing the modified first moment. This represents ρ1 raised to the power of t;

[0129] Correcting the deviation of the second moment: in: The variable representing the modified second moment. This represents ρ² raised to the power of t;

[0130] Calculation update: Where: Δ θ This represents the amount of change in parameter θ;

[0131] Weight parameter update: θ = θ + Δ θ .

[0132] Step S7: Testing phase.

[0133] Step S701: Obtain brain fMRI signals and corresponding manually labeled image categories and text descriptions from the test set to form a sample group. The test set size is 2770, and these data will not appear in the training set.

[0134] Step S702: Based on the training phase, a Prompt module, a visual encoding module, a whole-brain encoding module, and a language generation module are obtained. Similar to the training phase, the brain fMRI signals from the test set are input into the visual encoding module and the whole-brain encoding module. Then, the Prompt module, composed of manually labeled image categories and text descriptions, is input into the Prompt module to generate the corresponding text. If the Prompt is a category decoding, the model will output the predicted category label, where the confusion matrix of the predicted category is as follows: Figure 2 As shown; if Prompt is text decoding, the model will output the predicted text description as follows: Figure 3 As shown, the average prediction accuracy for category decoding reached 67.6%, far exceeding the random level of 8.33% for 12 categories.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0136] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A progressive transfer learning decoding method based on GPT, characterized in that, Includes the following steps: Step 1: Construct a GPT-based category and text decoding model based on the functional magnetic resonance signals collected from the subjects while they are viewing visually perceiving images. The model includes: a text encoder, an image encoder, a visual information encoder, a whole-brain information encoder, a feature fusion unit, and a language generator. in, The text encoder is used to embed the image description text containing cue words and the cue word positions of the visually perceived image viewed by the subject into the latent feature space to obtain the text latent feature vector; An image encoder is used to embed multiple image feature vectors of the visually perceived image viewed by the subject into a latent feature space to obtain image latent feature vectors; wherein, each image feature vector is extracted through a pre-trained image feature extraction model; Image feature vectors include ten types: semantic representation model Bert text features, Clip image features, Clip text features, Dino image features, Gemma text features, Gpt text features, Llama text features, Swin image features, Vgg image features, and Vit image features. Image encoders are based on learnable parameter vectors We assign weights to each type of image feature vector, where M represents the number of types of image feature vectors. This represents the weight of the m-th image feature vector, where m = 1, 2 And each The initial values ​​are all the same; based on weights The image latent feature vectors output by the image encoder are obtained by weighted fusion of the M types of image feature vectors. The visual information encoder uses a recurrent neural network to map the activity signals of the brain's visual region in the acquired functional magnetic resonance imaging signals to the latent feature space, thereby obtaining the visual latent feature vector. The whole-brain information encoder uses the activity signals of the visual region of the brain as context and maps the response signals of the whole brain other than the visual region of the brain to the latent feature space based on the multi-head attention mechanism to obtain non-visual latent feature vectors. The feature fusion unit is used to fuse textual latent feature vectors, image latent feature vectors, visual latent feature vectors, and non-visual latent feature vectors to generate a fused feature vector. The language generator takes the fused feature vector output by the feature fusion ... Specifically, the feature fusion unit is: Add the visual latent feature vector to the non-visual region feature vector to obtain the feature vector A of the whole brain region; Based on image latent feature vectors The weight α is then used to weight and fuse it with the feature vector A to obtain the feature vector. ; Then, feature vector B is added to the vector corresponding to the prompt word part in the latent feature vector of the text to generate a fused feature vector; The progressive transfer learning training strategy is divided into three stages based on the number of training rounds. The initial stage is the first stage. When the preset threshold of the first training round is reached, the second stage is entered. When the preset threshold of the second training round is reached, the third stage is entered. In the first stage, the weight α is set to feature 1; in the second stage, the weight α decreases from 1 to 0 in an arithmetic sequence; in the third stage, the weight α is set to 0. Step 2: Train the constructed GPT-based category and text decoding model based on the set training loss function. During training, the feature fusioner uses a progressive transfer learning training strategy to generate different fused feature vectors. When the preset training convergence conditions are met, the category and text decoding models for visual neural decoding are obtained.

2. The method as described in claim 1, characterized in that, The text encoder is specifically: The input image description text containing prompt words is segmented into words. Then, the number of each segmented word is queried in a pre-set vocabulary for encoding. The numbers are combined into a word sequence of length L, where L is a user-defined value. Multiply the sequence by the preset word embedding matrix to obtain the L*Num word embedding vector sequence, where Num represents the dimension of the word embedding vector; Multiply the word sequence by the position embedding matrix of the prompt word to obtain the position-encoded position embedding vector sequence, which has a dimension of L*Num; The latent feature vector of the text is obtained by linearly adding the word embedding vector sequence and the position embedding vector sequence.

3. The method as described in claim 1, characterized in that, The visual information encoder is specifically: Based on the adopted brain region segmentation template, the voxel points distributed in multiple different visual regions are subjected to upsampling or downsampling operations using linear interpolation. The data dimension of the collected activity signals is unified to the latent feature space dimension to obtain the feature vector of each visual region. Then, based on the feature vectors of all visual regions, an N1*Num visual region feature sequence is formed, where N1 represents the number of visual regions in the brain region segmentation template and Num represents the dimension of the word embedding vector. The visual region feature sequence is input into a recurrent neural network, which integrates the visual region feature sequence into a single feature vector. The Num-dimensional hidden features in the final stage of the recurrent neural network are taken as the visual latent feature vector.

4. The method as described in claim 3, characterized in that, The whole-brain information encoder is specifically: Based on the adopted brain region segmentation template, the voxel points distributed in different non-visual regions are subjected to upsampling or downsampling operations using linear interpolation. The data dimension of the collected activity signals is unified to the latent feature space dimension to obtain the feature vectors of each non-visual region. Then, based on the feature vectors of all non-visual regions, an N2*Num non-visual region feature sequence is formed, where N2 represents the number of non-visual regions in the brain region segmentation template. Initialize three fully connected linear layers q, k, and v to represent the query, key, and value, respectively. The weight matrices of each fully connected layer are as follows: , , Each weight matrix has a dimension of Num*Num, and each linear fully connected layer includes the same number of weight matrices, all set to H. The N1*Num visual region feature sequences are input into the fully connected layers k and v, respectively, along with the H weight matrices. , Multiplying them together yields H N1*Num units. Matrix and Matrix; combining N2*Num non-visual region feature sequences with H weight matrices. Multiplying them together yields H N2*Num units. matrix; Combine the H Q matrices with the H... The transpose of the matrix is ​​multiplied, and then multiplied with H V matrices to obtain H N2*Num Z matrices; the H Z matrices are concatenated row by row and then input into a fully connected layer to adjust the dimensions to obtain an N2*Num Z' matrix; the matrix Z' is then averaged column by column to obtain a Num-dimensional non-visual region feature vector.

5. The method as described in claim 1, characterized in that, The language generator is specifically: (1) The fused feature vector is passed through the decoder of the GPT model to obtain a text prediction vector for descriptive words that characterize the visual stimuli of the image; Then, based on the pre-set vocabulary list used to generate words, each word in the text prediction vector is scored to obtain the prediction probability of each word. Then, the word with the highest probability is selected and output to obtain the current generated word. (2) The generated words are added to the current input sequence of the decoder of the GPT model so that the decoder can make the next prediction; Repeat the processing steps (1) and (2) until the word at the end of the sentence is a preset terminator, and obtain the text prediction vector based on all generated words.

6. The method as described in claim 5, characterized in that, The prediction probability of each word is obtained by multiplying the output text prediction vector with the lm_head word embedding matrix of the lm_head component of the GPT language model to obtain a list of prediction probabilities for the next word, and selecting the maximum value as the generated word.

7. The method as described in claim 1, characterized in that, In step 2, the training loss function is set to Where L is the total loss of the GPT-based category and text decoding model. For the model's cross-entropy loss, For feature cosine similarity loss, , Preset weights for the two losses; Cross-entropy loss Where N is the number of target words to be predicted. It is an index of the vocabulary corresponding to words in the manually annotated natural language in the training samples. It is the predicted probability of the generated word; Feature cosine similarity loss ,in, It is the latent feature vector of the image output by the image encoder. These are feature vectors for the entire brain region.

Images