Progressive migration training decoding method based on GPT

Abstract

The invention discloses a progressive migration training decoding method based on GPT, and belongs to the field of optic nerve decoding. According to the method, a GPT-based category and text decoding model is constructed based on functional magnetic resonance signal data recorded when a subject watches a large number of visual perception images, and the method comprises an image encoder used for embedding image feature vectors extracted by adopting a pre-training model into a hidden feature space; prompt containing task information and position codes are embedded into a hidden feature space; the visual information encoder is used for mapping a brain visual region voxel signal to a hidden feature space by using a GRU; the whole-brain information encoder is used for guiding the selection of voxel signals of other regions of the brain by taking a voxel signal of a visual region of the brain as a context, and mapping global information to a hidden feature space; and the language generator fuses all the hidden features and selects the Token with the highest probability to generate the text of the corresponding content. According to the method, the decoding accuracy is improved, and the generalization ability of the model is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of visual neural decoding, and in particular to a GPT-based progressive migration training decoding method. Background Art

[0002] Visual neural decoding has been gaining attention in recent decades. Many researchers have spent a lot of effort to explore the various mechanisms of neural decoding, and these mechanisms can be divided into three categories according to the decoding type: visual stimulus classification, visual stimulus recognition, and visual stimulus reconstruction. Among them, the study of visual stimulus classification was initially conducted by Haxby et al. as early as 2001. They used the MVPA method based on fMRI (Functional Magnetic Resonance Imaging) signals to classify the 8 different categories presented to the participants. Yargholi and Hossein-Zadeh used an enhanced naive Bayes classifier to decode the fMRI signals in response to handwritten digits into the correct category.

[0003] These early neural decoding methods for classification typically use classifiers to learn the relationship between brain activity and a predefined set of labels, making it impossible to apply these models to new categories. The work of Horikawa and Kamitani breaks through this limitation. They proposed to decode arbitrary object categories by learning a linear regression function that converts fMRI patterns into convolutional neural network features of the viewed image. By comparing the correlation coefficient between the predicted image features and the average visual features of various categories, the new image category can be estimated. However, the performance of this method depends significantly on a large amount of paired stimulus and response data. With the continuous updating and iteration of deep learning technology, emerging new technologies such as VAE, Transformer and other models have been applied to the field of neural decoding. Du et al. proposed BraVL, a VAE-based model divided into two parts, multimodal joint modeling and mutual information regularization, to learn the relationship between brain activity, visual stimuli and Wikipedia descriptions of categories. The learned potential representation is then classified using a support vector machine. In addition, Liu proposed the BrainCLIP model in 2023, which aims to train a mapping network to convert fMRI patterns into representations that are well aligned with the CLIP embedding space, decoding visual information by combining visual and text supervision. Their study confirmed the importance of multimodal information in visual decoding.

[0004] However, previous studies still face many problems. First, there is a lack of research on multi-task decoding. Previous studies often focus on a single task and ignore the connection between tasks. Second, with the advancement of technology, previous models cannot achieve ideal performance and lack cross-modal generalization capabilities. Summary of the invention

[0005] The present invention provides a progressive transfer training decoding method based on GPT (an autoregressive language model based on Transformer architecture), which is used to realize multi-task decoding involving category and language decoding in biomedical image brain decoding. The present invention can solve the lack of cross-modal generalization ability in the prior art and improve model performance through progressive transfer learning.

[0006] The technical solution adopted by the present invention is:

[0007] A progressive migration training decoding method based on GPT, the method comprising the following steps:

[0008] Step 1: construct a GPT-based category and text decoding model based on the functional magnetic resonance signals collected when the subjects watch visual perception images. The model includes: a text encoder, an image encoder, a visual information encoder, a whole-brain information encoder, a feature fusion device, and a language generator;

[0009] in,

[0010] A text encoder is used to embed the image description text containing the prompt words (i.e., the image description text in natural language) and the prompt word position encoding of the visual perception image viewed by the subject into the latent feature space to obtain a text latent feature vector;

[0011] An image encoder is used to embed multiple image feature vectors of the visual perception image viewed by the input subject into a latent feature space to obtain an image latent feature vector; wherein each image feature vector is extracted by a pre-trained image feature extraction model;

[0012] The visual information encoder uses a recurrent neural network to map the activity signals of the visual area of ​​the brain in the collected functional magnetic resonance signals to the latent feature space to obtain the visual latent feature vector;

[0013] The whole-brain information encoder uses the activity signals of the visual area of ​​the brain as the context and maps the response signals of the whole brain except the visual area of ​​the brain to the latent feature space based on the multi-head attention mechanism to obtain the non-visual latent feature vector;

[0014] The feature fuser is used to fuse the text latent feature vector, image latent feature vector, visual latent feature vector and non-visual latent feature vector to generate a fused feature vector; that is, in the present invention, the dimension of the feature vector in the latent feature space can be customized based on the actual scene, while the dimension of the feature vector extracted by the text encoder, image encoder, visual information encoder and whole-brain information encoder remains consistent.

[0015] The language generator, whose initial input sequence is the fused feature vector output by the feature fuser, obtains the text prediction vector of the current input fused feature vector through the decoder of the GPT model, and then converts the text prediction vector into text describing the visual stimulus of the image based on the preset word embedding matrix;

[0016] Step 2: Based on the set training loss function, the constructed GPT-based category and text decoding model is trained, and during the training, the feature fuser adopts a training strategy of progressive transfer learning to generate different fused feature vectors;

[0017] When the preset training convergence conditions are reached, the category and text decoding models for visual neural decoding are obtained.

[0018] Furthermore, the text encoder is specifically:

[0019] The input image description text containing the prompt word is segmented according to the preset vocabulary for encoding to obtain a word sequence of a specified length; that is, the image description text is segmented, and then the corresponding number in the vocabulary is queried for each segmented word, and the numbers are combined into a sequence to obtain a word sequence of length L (custom value);

[0020] Multiply the sequence by the preset word embedding matrix (WTE) to calculate the word embedding vector sequence of L*Num, where Num represents the dimension of the word embedding vector;

[0021] Multiply the word sequence by the position embedding matrix (WPE) of the prompt word to calculate the position embedding vector sequence after position encoding, whose dimension is also L*Num; WPE is used to describe the order of the prompt word in the sentence, so that the model can understand the order information of the prompt word;

[0022] The word embedding vector sequence and the position embedding vector sequence are linearly added to obtain the fused sentence embedding vector sequence, that is, the text latent feature vector.

[0023] Furthermore, the image encoder is based on a learnable parameter vector (θ (1) ,θ (2) ,...,θ (M) ), assign weights to each image feature vector, where M represents the number of image feature vectors, θ (m) represents the weight of the mth image feature vector, m = 1, 2, ..., M, where θ (m) The initial values ​​of θ are the same. When training the GPT-based category and text decoding model, each time a sample is trained, the loss function is used to calculate θ. (1) ,θ (2) ,...,θ (M) Update; based on weight θ (m)The M types of image feature vectors are weightedly fused to obtain an image latent feature vector output by the image encoder (described as image fusion feature I).

[0024] Furthermore, the image feature vectors include ten types: semantic representation model Bert text features, Clip (Contrastive Language-Image Pre-training) image features, Clip text features, Dino (DETR with Improved deNoising AnchOr boxes) image features, Gemma text features, Gpt text features, Llama (Large Language Model Meta) text features, Swin (a visual model based on Transformer architecture) image features, Vgg (Visual Geometry Group Network) image features, and Vit (Vision Transformer) image features, that is, the features extracted by the backbone network of the existing mainstream text and / or image processing models.

[0025] Furthermore, the visual information encoder is specifically:

[0026] Based on the adopted brain area division template, the voxel points distributed in multiple different visual areas are up- or down-sampled by linear interpolation, and the data dimension of the collected activity signal is unified to the dimension of the latent feature space to obtain the feature vector of each visual area. Then, based on the feature vectors of all visual areas, a visual area feature sequence of N1*Num is formed, where N1 represents the number of visual areas in the brain area division template;

[0027] The visual area feature sequence is input into a recurrent neural network (preferably a gated recurrent unit GRU), the visual area feature sequence is integrated into a single feature vector through the recurrent neural network, and the Num-dimensional hidden features in the last stage of the recurrent neural network are taken as the visual latent feature vector.

[0028] Furthermore, when the step size of the recurrent neural network is the same as the number of visual areas in the brain, the number of layers is 1, and the size of the input layer and the output layer is the same, both Num.

[0029] Furthermore, the whole brain information encoder is specifically:

[0030] Based on the adopted brain area division template, the voxel points distributed in different non-visual areas are up- or down-sampled by linear interpolation, and the data dimension of the collected activity signal is unified to the dimension of the latent feature space to obtain the feature vector of each non-visual area; then, based on the feature vectors of all non-visual areas, a non-visual area feature sequence of N2*Num is formed, where N2 represents the number of non-visual areas in the brain area division template;

[0031] Initialize three linear fully connected layers q, k, v to represent query, key, and value respectively. The weight matrices of each linear fully connected layer are W Q , W K , W V ; The dimension of each weight matrix is ​​Num*Num, and the number of weight matrices included in each linear fully connected layer is the same, all set to H; the N1*Num visual area feature sequence is input into the fully connected layer k, v, and H weight matrices W K , W V Multiply them together to get H N1*Num K matrices and V matrices; multiply the N2*Num non-visual area feature sequences with the H weight matrices W Q Multiply them together to get H N2*Num Q matrices;

[0032] Multiply the transpose of H Q matrices with H K matrices, and then multiply them with H V matrices to obtain H N2*Num Z matrices; concatenate the H Z matrices by rows and input them into the fully connected layer to adjust the dimension to obtain a N2*Num Z' matrix; then average the matrix Z' by columns to obtain the Num-dimensional non-visual area feature vector.

[0033] Furthermore, the feature fuser is specifically:

[0034] Add the visual latent feature vector to the feature vector of the non-visual area to obtain the feature vector A of the whole brain area;

[0035] Based on the weight α of the image latent feature vector (i.e., the image fusion feature I), it is weightedly fused with the feature vector A to obtain the feature vector B = α*I + (1-α)*A;

[0036] Then add the feature vector B to the vector corresponding to the prompt word part in the text latent feature vector to generate a fused feature vector to obtain the input of the GPT decoder;

[0037] Among them, the weight α gradually decreases from 1 to 0, and its decreasing step speed is negatively correlated with the size of the training round, that is, the larger the training round, the slower the step speed. If the training round is too small, the step speed needs to be limited to prevent insufficient model learning; if the training round is sufficient, there is no need to limit the step speed.

[0038] Furthermore, the training strategy of progressive transfer learning is divided into three stages based on training rounds. For example, two training round thresholds can be set. The initial stage is the first stage. When the first training round threshold is reached, it is considered to have reached the second stage. When the second training round threshold is reached, it is considered to have reached the third stage. In the first stage, the value of weight α is set to feature 1, and feature B is I at this time. In the second stage, the value of weight α decreases from 1 to 0 according to an arithmetic progression. At this time, feature B is α*I+(1-α)*A. In the third stage, the value of weight α is set to 0, and feature B is feature A at this time.

[0039] The language generator takes as input the fused feature vector output by the feature fuser, obtains the text prediction vector of the current input fused feature vector through the decoder of the GPT model, and then converts the text prediction vector into text describing the visual stimulus of the image based on the preset word embedding matrix;

[0040] Furthermore, the language generator is specifically:

[0041] (1) Passing the fused feature vector through all network layers of the decoder of the GPT model, outputting a text prediction vector of the description word used to represent the visual stimulus of the image;

[0042] Then, score each word in the vector according to the preset vocabulary for generating vocabulary (i.e., the vocabulary based on the category and text decoding model of GPT, which can be customized by the user), obtain the predicted probability of each word, and then select the word with the highest probability to output, that is, obtain the currently predicted generated word;

[0043] (2) Add the generated word output in (1) to the current input sequence of the decoder of the GPT model, and let the decoder make the next prediction;

[0044] Repeat the processes (1) and (2) until the token at the end of the sentence is the preset end symbol, and obtain the text prediction vector based on all the generated words.

[0045] Furthermore, the prediction probability of each word is obtained by multiplying the output text prediction vector with the lm_head word embedding matrix of the lm_head component of the GPT language model to obtain the prediction probability list of the next word, i.e., the logits list, and selecting the maximum value as the generated token.

[0046] Furthermore, in step 2, the training loss function is set to:

[0047] L=β1L CE -β2Cosine Similarity(A′,B′)

[0048] Where L is the total loss of the GPT-based category and text decoding model, L CE is the cross entropy loss of the model, Cosine Similarity(A′,B′) is the feature cosine similarity loss, β1 and β2 are the preset weights of the two losses;

[0049] Cross Entropy Loss Among them, N is the number of target tokens to be predicted, g i is the index of the vocabulary corresponding to the words in the natural language manually annotated in the training sample, p i is the predicted probability of generating a token;

[0050] Feature cosine similarity loss Among them, A′ is the image latent feature vector output by the image encoder, and B′ is the feature vector of the whole brain area. The obtained value Cosine Similarity(A′,B′)∈[-1,1] is used to evaluate the similarity of two features. When it is 1, it means that the two features have the same direction, and when it is 0, it means that the two features have no similarity. Therefore, in the optimization process, both fMRI and image features can be aligned and the prediction accuracy of the model can be improved.

[0051] The technical solution provided by the present invention brings at least the following beneficial effects:

[0052] The present invention adopts a training strategy based on progressive transfer learning, and uses a text encoder, an image encoder, a visual information encoder, a whole-brain information encoder, and a language generator to perform multi-task decoding of natural images, including category classification and text decoding. The obtained category accuracy is high, and the text description is more consistent with natural language. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 A schematic diagram of the structure of a GPT-based category and text decoding model used in an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of a confusion matrix result of category decoding according to an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of text decoding results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0058] The embodiment of the present invention provides a progressive transfer training decoding method based on GPT, referring to the figure, the method first constructs a GPT-based category and text decoding model based on the functional magnetic resonance signal collected when the subject is watching the visual perception image; secondly, the constructed GPT-based category and text decoding model is trained based on the set training loss function, and during the training, the feature fuser adopts the training strategy of progressive transfer learning to generate different fused feature vectors; when the preset training convergence condition is reached (such as reaching the maximum number of training iterations or the convergence of the loss function value, etc.), the category and text decoding model for visual neural decoding is obtained. The model constructed by the method includes: a text encoder, an image encoder, a visual information encoder, a whole-brain information encoder, a feature fusion device and a language generator; wherein the text encoder is used to embed the image description text (i.e., the image description text in natural language) containing the prompt words and the prompt word position encoding of the visual perception image viewed by the subject into the latent feature space to obtain the text latent feature vector; the image encoder is used to embed multiple image feature vectors of the input visual perception image viewed by the subject into the latent feature space to obtain the image latent feature vector; wherein each image feature vector is extracted by a pre-trained image feature extraction model; the visual information encoder uses a recurrent neural network to extract the collected functional magnetic The activity signal of the visual area of ​​the brain in the resonance signal is mapped to the latent feature space to obtain the visual latent feature vector; the whole brain information encoder uses the activity signal of the visual area of ​​the brain as the context, and maps the response signal of the whole brain except the visual area of ​​the brain to the latent feature space based on the multi-head attention mechanism to obtain the non-visual latent feature vector; the feature fusion device is used to fuse the text latent feature vector, image latent feature vector, perception latent feature vector and non-visual latent feature vector to generate a fused feature vector; that is, in the present invention, the dimension of the feature vector of the latent feature space can be customized based on the actual scene, while the dimension of the feature vector extracted by the text encoder, image encoder, visual information encoder and whole brain information encoder remains consistent. The language generator, whose input is the fused feature vector output by the feature fusion device, obtains the text prediction vector of the current input fused feature vector through the decoder of the GPT model, and then converts the text prediction vector into text describing the visual stimulation of the image based on the preset word embedding matrix;

[0059] In one embodiment, the embodiment of the present invention constructs a GPT-based category and text decoding model, specifically comprising the following steps:

[0060] Step 1. The text encoder converts the text containing the prompt and the natural language describing the image into a vector in the latent feature space. The vector sequence contains position information:

[0061] Step 1.1: Segment the input text containing Prompt according to the vocabulary to obtain a word sequence, that is, query the corresponding numbers in the vocabulary, and combine the numbers into a sequence to obtain a Token sequence of length L;

[0062] Step 1.2: Multiply the Token sequence by the preset word embedding matrix (WTE) to calculate the L*Num word embedding vector sequence, where Num represents the dimension of the word embedding vector;

[0063] Step 1.3: Multiply the word sequence obtained in step 1.1 by the position embedding matrix (WPE) of Prompt to calculate the position embedding vector sequence after position encoding;

[0064] Step 1.4: Linearly add the word embedding vector sequence and the position embedding vector sequence to obtain the fused sentence embedding vector sequence.

[0065] Step 2. The image encoder converts the stimulus image viewed by the subject into an image feature vector that can be input to the model through ten different pre-trained architectures, and adds the ten features through learnable parameters;

[0066] Step 2.1: Input the input image into ten different pre-trained models to obtain Bert text features, Clip image and text features, Dino image features, Gemma text features, Gpt text features, Llama text features, Swin image features, Vgg image features, and Vit image features, including most of the mainstream pre-trained models;

[0067] Step 2.2: Pass a learnable parameter vector (θ (1) ,θ (2) ,...,θ (10) ), assign weights to each feature, with an initial value of 1. Through model learning, the weights of useless features are reduced and the weights of effective features are increased. Each time a sample is trained, these ten parameters are updated;

[0068] Step 2.3: Multiply these ten features with the ten weight parameters respectively, and add them up to get the final output feature I.

[0069] Furthermore, the length of the image feature vector I is the same as the input vector length of GPT-2, which is 768.

[0070] Step 3. The visual information encoder converts the brain visual area activity signal obtained based on magnetic resonance imaging when viewing the visual perception image into a latent feature space vector:

[0071] Step 3.1: Perform up- or down-sampling operations on the voxel points distributed in 29 different visual areas based on the linear interpolation method, unify the data dimension to the embedding space dimension, and then combine the feature vectors of different visual areas into a 29*Num visual area feature sequence;

[0072] Step 3.2: Input the visual area feature sequence into the GRU gated recurrent unit. GRU integrates the visual area feature sequence into a single feature vector and takes the Num-dimensional hidden features in the last stage of GRU as the visual area feature vector.

[0073] In this embodiment, the step size of GRU is the same as the number of visual areas in the brain, the number of layers is 1, and the size of the input layer and the output layer is the same, both Num.

[0074] Step 4. The whole-brain information encoder converts the response signals in the whole brain except the visual area into latent feature space vectors to obtain the feature vectors of the non-visual area:

[0075] Step 4.1: Similar to step 3.1, up- or down-sampling operations are performed on the voxel points distributed in 151 different non-visual areas based on the linear interpolation method, and the data dimension is unified to the embedding space dimension. Then, the response signals of different non-visual areas are sorted according to the non-visual area numbers to form a 151*Num non-visual area feature sequence;

[0076] Step 4.2: Initialize the linear fully connected layers q, k, v, and the weight matrices of each linear fully connected layer are W W , W K , W V ; The dimension of each weight matrix is ​​Num*Num, and the number of weight matrices included in each linear fully connected layer is the same, all set to H; the 29*Num visual area feature sequence is input into the fully connected layer k, v, and H weight matrices W K , W V Multiply them together to get H 29*Num K matrices and V matrices; multiply the 151*Num non-visual area feature sequence with H weight matrices W Q Multiply them to get H 151*Num Q matrices;

[0077] Step 4.3: Multiply the transpose of the H Q matrices with the H K matrices, and then multiply them with the H V matrices to obtain H 151*Num Z matrices. Concatenate the H Z matrices by row and input them into the fully connected layer to change the dimension to obtain a 151*Num Z' matrix; average the Z' matrix by column to obtain the Num-dimensional hidden features as the non-visual area feature vector.

[0078] Step 5: The feature vectors obtained in steps 1, 2, 3, and 4 are fused into one feature through the feature fuser, and different features are generated according to the training strategy of progressive transfer learning and sent to the decoder:

[0079] Step 5.1: Add the feature vectors of the visual area and the feature vectors of the non-visual area in steps 3 and 4 to obtain the feature vector A of the whole brain area. Then add the feature vector A and the image feature vector I in step 2 by the weight value α, that is, α*I+(1-α)*A, where the α value gradually decreases from 1 to 0. Finally, this feature is recorded as B and added to the vector corresponding to the prompt word part in the embedded vector sequence in step 1 to obtain the input of the GPT decoder.

[0080] Step 5.2: The training strategy of progressive transfer learning can be divided into three stages: in the first stage, the α value is 1, and the feature B is I; in the second stage, the α value decreases from 1 to 0 in an arithmetic progression, and the feature B is α*I+(1-α)*A; in the third stage, the α value is 0, and the feature B is A.

[0081] Step 6. The language generator obtains the text prediction vector through the GPT decoder (preferably the GPT-2 decoder) in step 5, and converts it into text describing the image stimulus based on the preset word embedding matrix:

[0082] Step 6.1: Depending on the training stage, the input features to the GPT decoder will be different, and its initial input sequence is the output features of the feature fuser. The input vector passes through all layers of the GPT decoder to produce a vector. The vector is scored according to the model's vocabulary, and the token with the highest probability is selected for output.

[0083] Step 6.2: Add the token output in the previous step to the current input sequence of the decoder, and let the decoder make the next prediction. Repeat this process until the token at the end of the sentence is the terminator, and finally get the text prediction vector.

[0084] Preferably, the embedding vector length of each token in GPT-2 is 768. The probability of generating a token is based on the multiplication of the output text prediction vector and the lm_head word embedding matrix of the lm_head component of the language model to obtain the predicted probability list of the next word, i.e., the logits list, and the maximum value is selected as the generated token.

[0085] Among them, lm_head can generate a probability distribution for predicting the next word based on the input text prediction vector. This prediction is based on the mapping relationship between the input vector and the internal state of the model. lm_head usually includes one or more fully connected layers, which can convert the input vector into a higher-dimensional representation and use it to generate prediction probabilities. In addition to linear layers, lm_head also usually includes some additional components, such as normalization layers or attention mechanisms, to enhance the representation ability of the model and the accuracy of the generated probability. These components can be adjusted and optimized according to specific tasks and model architectures.

[0086] In one embodiment, the loss function of the GPT-based category and text decoding model is set to:

[0087] 1) Calculate the cross entropy loss L CE :

[0088]

[0089] Among them, N is the number of target tokens to be predicted, g i is the index of the vocabulary corresponding to the words in the natural language manually annotated in the training sample, p i is the predicted probability of generating a token.

[0090] 2) Calculate the loss by cosine similarity:

[0091]

[0092] Among them, A′ is the feature obtained by the image encoder, that is, the image feature vector I, and B′ is the feature vector of the whole brain area (referred to as brain feature) obtained by adding the visual area feature vector and the non-visual area feature vector in the above steps 3 and 4.

[0093] The obtained value Cosine Similarity(A,B)∈[-1,1] is used to evaluate the similarity between two features. When it is 1, it means that the two features have the same direction, and when it is 0, it means that the two features have no similarity.

[0094] 3) Calculate the final loss:

[0095] L=L CE -Cosine Similarity(A,B)

[0096] In this way, the fMRI and image features can be aligned during the optimization process, and the prediction accuracy of the model can be improved.

[0097] The GPT-based progressive transfer training decoding method provided in the embodiment of the present invention solves the problem that multiple decoding tasks cannot be performed in one decoding model, introduces effective global information of the brain, improves decoding accuracy, and enhances the generalization ability of the model.

[0098] Example

[0099] In one embodiment, the present invention provides a GPT-based progressive transfer training decoding method, which can simultaneously decode category information and text information. The method of this embodiment includes the following implementation steps:

[0100] Step S1: construct an image encoding module.

[0101] Step S101: Ten pre-trained models are used to extract image features, namely Bert text encoder, Clip image and text encoder, Dino image encoder, Gemma text encoder, Gpt text encoder, Llama text encoder, Swin image encoder, Vgg image encoder, and Vit image encoder, to obtain ten feature vectors of size 768 (x1, x2, ..., x 10 ).

[0102] Step S102: Compare the ten feature vectors obtained in step S101 with the learnable parameter vector (θ (1) ,θ (2) ,...,θ (10) ) are multiplied respectively and finally added together to obtain a feature vector sequence of size 768.

[0103] Step S2: Construct a Prompt encoding module.

[0104] Step S201: Design a prompt input text, which includes the subject number and decoding task (category decoding and text decoding). The subject number and decoding task are directly separated by the # symbol. For example, when decoding subject number 01, the prompt is "0#01#Decode subject01's fMRI captions:", where the first number is the sample number, and the number between the two # symbols is the subject number.

[0105] Step S202: Input the Prompt text obtained in step S201 into the Tokenizer, and convert it into a Token sequence. Each Token in the vocabulary has a corresponding digital index. This index sequence is input into the model, and the model uses these index sequences to distinguish different Tokens for encoding.

[0106] Step S203: Multiply the Token sequence by the word embedding matrix (WTE) to obtain a 9*768 word embedding vector sequence.

[0107] Step S204: multiply the Token sequence obtained in step S203 by the preset position embedding matrix (WPE) to calculate a position embedding vector sequence after position encoding.

[0108] Step S3: Construct a visual encoding module.

[0109] Step S301: Perform up- or down-sampling operations on voxel points distributed in 29 different visual areas based on the linear interpolation method to unify the dimension size and position embedding vector sequence, and then sort the feature vectors of different visual areas according to the visual area number to form a 29*768 visual area feature sequence.

[0110] Step S302: Input the visual area feature sequence into the GRU gated recurrent unit. The GRU integrates the 29 visual area feature sequences into a single feature vector, and takes the 1*768-dimensional hidden feature of the last stage of the GRU as the visual area feature vector.

[0111] Step S4: Construct the whole-brain encoding module.

[0112] Step S401: Similar to step S3, up- or down-sampling operations are performed on the voxel points distributed in 151 different non-visual areas based on the linear interpolation method to unify the dimension size and the dimension of the position embedding vector sequence, and then the response signals of different non-visual areas are sorted according to the non-visual area numbers to form a 151*768 non-visual area feature sequence.

[0113] Step S402: Initialize the linear fully connected layers q, k, v, and the weight matrices of each linear fully connected layer are W Q , W K , W V ; The dimension of each weight matrix is ​​768*768, and the number of weight matrices included in each linear fully connected layer is the same, which is set to 8; The 29*768 visual area feature sequence is input into the fully connected layer k, v, and H weight matrices W respectively. K , W V Multiply them together to get 8 29*768 K matrices and V matrices; multiply the 151*768 non-visual area feature sequence with the 8 weight matrices W Q Multiply them together to get 8 151*768 Q matrices.

[0114] Step S403: Multiply the 8 Q matrices by the transpose of the 8 K matrices, and then multiply them by the 8 V matrices to obtain 8 151*768 Z matrices. Concatenate the 8 Z matrices by row and input them into the fully connected layer to change the dimension to obtain a 151*768 Z' matrix; average the Z' matrix by column to obtain a 768-dimensional hidden feature as the non-visual area feature vector.

[0115] Step S5: Construct a language generator.

[0116] Step S501: Depending on the training stage, the input features to the GPT-2 decoder will be different. The input vector passes through all layers of the decoder to produce a vector. The vector is scored according to the vocabulary of the model, and the token with the highest probability is selected for output.

[0117] Step S502: Add the Token outputted from step S501 to the input sequence, and let the model make the next prediction. Repeat this process until the Token at the end of the sentence is the end symbol "<|endoftext|>", and finally obtain the text prediction vector.

[0118] Step S6: Training phase

[0119] Step S601: Acquire stimulus images, brain fMRI signals, and manually labeled image categories and text descriptions in a training set to form a sample group.

[0120] Step S602: Add the visual area feature vector and the non-visual area feature vector in step S3 and step S4 to obtain a 768-dimensional feature vector for the whole brain area. Then, multiply the whole brain area feature vector by 1-α through the weight value α, multiply the 768-dimensional image feature vector in step S1 by α, and finally add them together, where the α value gradually decreases from 1 to 0 with the iteration of training. Finally, add this feature to the vector corresponding to the Prompt part in the embedded vector sequence in step S2 to obtain the input of the GPT decoder.

[0121] Step S603: Use the attention mask to mask the sentence, obtain the context of the sentence word by word, score the token according to the context and the corresponding fMRI signal, and select the token with the highest probability to generate the following sentence word by word to obtain the distribution of the predicted following sentence.

[0122] Step S604: According to the set total loss L, use the AdamW optimization algorithm to update the corresponding weights of the parameters of the entire model.

[0123] The specific optimization parameters of AdamW are as follows: the default step size ε is 0.00015; the default exponential decay rates ρ1 and ρ2 of the moment estimate are 0.9 and 0.999 respectively; the default small constant δ for numerical stability is 10 -8 ; The weight decay coefficient λ defaults to 10 -2 ; Initialize the parameter θ with a normal distribution with a mean of 0 and a variance of 0.001; Initialize the first-order and second-order moment variables to s = 0, r = 0; Initialize the time step to t = 0; Each time there are m context sample pairs {(x (1) ,y (1) ),(x (2) ,y (2) ),…,(x (m) ,y (m) )} Make the following updates:

[0124] Compute the gradient: where x (i) is the input, f(x (i) ; θ) represents the predicted output, L(f(x (i) ;θ),y (i) ) represents the loss function of the i-th sample;

[0125] Update time step: t=t+1;

[0126] Update biased first-order moment estimate: s = ρ1s + (1-ρ1)g, where ρ1 represents the exponential decay rate of the first-order moment estimate (the default is 0.9), and s represents the first-order moment variable;

[0127] Update the biased second-order moment estimate: r = ρ2r + (1-ρ2)g⊙g, where ρ2 represents the exponential decay rate of the second-order moment estimate (the default is 0.999), and r represents the second-order moment variable;

[0128] Correct the deviation of the first moment: in: represents the modified first-order moment variable, represents ρ1 to the power of t;

[0129] Correct the deviation of the second moment: in: represents the modified second-order moment variable, represents ρ2 raised to the power of t;

[0130] Compute updates: Where: Δ θ represents the change in parameter θ;

[0131] Weight parameter update: θ = θ + Δ θ .

[0132] Step S7: Testing phase.

[0133] Step S701: Obtain brain fMRI signals and corresponding manually annotated image categories and text descriptions in the test set to form a sample group. The size of the test set is 2770, and these data do not appear in the training set.

[0134] Step S702: According to the training phase, the Prompt module, visual encoding module, whole-brain encoding module, and language generation module are obtained. Similar to the training phase, the brain fMRI signal in the test set is input into the visual encoding module and the whole-brain encoding module. Then the Prompt formed by the manually annotated image category and text description is input into the Prompt module, and finally the corresponding text generation is formed. If the Prompt is a category decoding, the model will output the predicted category label, where the confusion matrix of the predicted category is as follows Figure 2 As shown; if Prompt is a text decoder, the model will output the predicted text description as follows Figure 3 The average prediction accuracy of the category decoding reached 67.6%, which is much higher than the random level of 8.33% for the 12 categories.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0136] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A progressive transfer training decoding method based on GPT, characterized in that: The following steps are involved: Step 1: construct a GPT-based category and text decoding model based on the functional magnetic resonance signals collected when the subjects watch visual perception images. The model includes: a text encoder, an image encoder, a visual information encoder, a whole-brain information encoder, a feature fusion device, and a language generator; in, A text encoder is used to encode and embed the image description text containing the prompt words and the position of the prompt words of the visual perception image viewed by the subject into a latent feature space to obtain a text latent feature vector; An image encoder is used to embed multiple image feature vectors of the visual perception image viewed by the input subject into a latent feature space to obtain an image latent feature vector; wherein each image feature vector is extracted by a pre-trained image feature extraction model; The visual information encoder uses a recurrent neural network to map the activity signals of the visual area of ​​the brain in the collected functional magnetic resonance signals to the latent feature space to obtain the visual latent feature vector; The whole-brain information encoder uses the activity signals of the visual area of ​​the brain as the context and maps the response signals of the whole brain except the visual area of ​​the brain to the latent feature space based on the multi-head attention mechanism to obtain the non-visual latent feature vector; A feature fusion device is used to fuse text latent feature vectors, image latent feature vectors, visual latent feature vectors and non-visual latent feature vectors to generate a fused feature vector; The language generator, whose initial input sequence is the fused feature vector output by the feature fuser, obtains the text prediction vector of the current input fused feature vector through the decoder of the GPT model, and then converts the text prediction vector into text describing the visual stimulus of the image based on the preset word embedding matrix; Step 2: Based on the set training loss function, the constructed GPT-based category and text decoding model is trained, and during the training, the feature fuser adopts a training strategy of progressive transfer learning to generate different fused feature vectors; When the preset training convergence conditions are reached, the category and text decoding models for visual neural decoding are obtained.

2. The method according to claim 1, characterized in that The text encoder is: Segment the input image description text containing the prompt word, then query the number of each segmented word in the vocabulary table in the preset vocabulary table for encoding, and combine the numbers into a word sequence of length L, where L is a custom value; Multiply the sequence by the preset word embedding matrix to calculate the word embedding vector sequence of L*Num, where Num represents the dimension of the word embedding vector; Multiply the word sequence by the position embedding matrix of the prompt word to calculate the position embedding vector sequence after position encoding, whose dimension is L*Num; Linearly add the word embedding vector sequence and the position embedding vector sequence to obtain the text latent feature vector.

3. The method according to claim 1, characterized in that The image encoder is based on a learnable parameter vector (θ (1) ,θ (2) ,...,θ (M) ), assign weights to each image feature vector, where M represents the number of image feature vectors, θ (m) represents the weight of the mth image feature vector, m = 1, 2, ..., M, and each θ (m) The initial values ​​of are the same; Based on the weight θ (m) The M types of image feature vectors are weightedly fused to obtain the image latent feature vector output by the image encoder.

4. The method according to claim 1, characterized in that The visual information encoder is specifically: Based on the adopted brain area division template, the voxel points distributed in multiple different visual areas are up- or down-sampled by linear interpolation, and the data dimension of the collected activity signal is unified to the dimension of the latent feature space to obtain the feature vector of each visual area. Then, based on the feature vectors of all visual areas, a visual area feature sequence of N1*Num is formed, where N1 represents the number of visual areas in the brain area division template, and Num represents the dimension of the word embedding vector; The visual area feature sequence is input into the recurrent neural network, the visual area feature sequence is integrated into a single feature vector through the recurrent neural network, and the Num-dimensional hidden features in the last stage of the recurrent neural network are taken as the visual latent feature vector.

5. The method according to claim 4, characterized in that The whole brain information encoder is specifically: Based on the adopted brain area division template, the voxel points distributed in different non-visual areas are up- or down-sampled by linear interpolation, and the data dimension of the collected activity signal is unified to the dimension of the latent feature space to obtain the feature vector of each non-visual area; then, based on the feature vectors of all non-visual areas, a non-visual area feature sequence of N2*Num is formed, where N2 represents the number of non-visual areas in the brain area division template; Initialize three linear fully connected layers q, k, v to represent query, key, and value respectively. The weight matrices of each linear fully connected layer are W Q , W K , W V ; The dimension of each weight matrix is ​​Num*Num, and the number of weight matrices included in each linear fully connected layer is the same, all set to H; the N1*Num visual area feature sequence is input into the fully connected layer k, v, and H weight matrices W K , W V Multiply them together to get H N1*Num K matrices and V matrices; multiply the N2*Num non-visual area feature sequences with the H weight matrices W Q Multiply them together to get H N2*Num Q matrices; Multiply the transpose of H Q matrices with H K matrices, and then multiply them with H V matrices to obtain H N2*Num Z matrices; concatenate the H Z matrices by rows and input them into the fully connected layer to adjust the dimension to obtain a N2*Num Z' matrix; then average the matrix Z' by columns to obtain the Num-dimensional non-visual area feature vector.

6. The method according to claim 1, characterized in that The feature fuser is specifically: Add the visual latent feature vector to the feature vector of the non-visual area to obtain the feature vector A of the whole brain area; Based on the weight α of the image latent feature vector I, it is weightedly fused with the feature vector A to obtain the feature vector B = α*I + (1-α)*A; Then add the feature vector B to the vector corresponding to the prompt word part in the text latent feature vector to generate a fused feature vector; The training strategy of progressive transfer learning is divided into three stages based on the training rounds. The first stage is the initial stage. When the preset threshold of the first training round is reached, the second stage is entered. When the preset threshold of the second training round is reached, the third stage is entered. In the first stage, the value of weight α is set to feature 1; In the second stage, the value of weight α gradually decreases from 1 to 0; in the third stage, the value of weight α is set to 0.

7. The method according to claim 6, characterized in that In the second stage, the value of weight α decreases from 1 to 0 in an arithmetic progression.

8. The method according to claim 1, characterized in that The language generator is specifically: (1) Passing the fused feature vector through the decoder of the GPT model to obtain a text prediction vector of the description word used to represent the visual stimulus of the image; Then, score each word in the text prediction vector according to the preset vocabulary table for generating vocabulary, obtain the prediction probability of each word, and then select the word with the highest probability to output, and obtain the current generated word; (2) The generated word is added to the current input sequence of the decoder of the GPT model, allowing the decoder to make the next prediction; Repeat the processes (1) and (2) until the word at the end of the sentence is the preset end symbol, and obtain the text prediction vector based on all the generated words.

9. The method according to claim 8, characterized in that The prediction probability of each word is obtained by multiplying the output text prediction vector with the lm_head word embedding matrix of the lm_head component of the GPT language model to obtain a list of prediction probabilities for the next word, and selecting the maximum value as the generated word.

10. The method according to claim 1, characterized in that In step 2, the training loss function is set as: L=β1L CE -β2CosineSimilarity(A′,B′) Where L is the total loss of the GPT-based category and text decoding model, L CE is the cross entropy loss of the model, CosineSimilarity(A′,B′) is the feature cosine similarity loss, β1 and β2 are the preset weights of the two losses; Cross Entropy Loss Among them, N is the number of target words to be predicted, g i is the index of the vocabulary corresponding to the words in the natural language manually annotated in the training sample, p i is the predicted probability of the generated word; Feature cosine similarity loss Among them, A′ is the image latent feature vector output by the image encoder, and B′ is the feature vector of the whole brain area.

Images