A gpt-based multi-task visual perception decoding method

Abstract

The application discloses a kind of multi-task visual perception decoding methods based on GPT, belong to biomedical image brain decoding in language decoding technical field.This application is based on the functional magnetic resonance signal data under a large number of visual perception images to establish the language decoding model based on GPT model, it includes: Prompt embedding module, Prompt containing task information is embedded into latent feature space;Visual coding module, using GRU will be encoded to latent feature space with visual related area voxel signal;Network coding module, visual related area voxel signal is used as context, guide the selection of all other areas of brain voxel signal, and encode global information to latent feature space;Language generation module, fusion latent feature and identify latent feature to generate text corresponding to content.The application solves the problem that multiple decoding tasks cannot be performed in one decoding model, and introduces effective global information of the brain, which can realize the practical application of multi-task language generation brain-computer interface scenario.

Description

Technical Field

[0001] This method belongs to the interdisciplinary field of biomedical engineering, visual neuroscience and artificial intelligence, and specifically involves a multi-task visual perception decoding method based on GPT (a natural language processing (NLP) model based on the Transformer architecture). Background Technology

[0002] Language decoding based on functional magnetic resonance imaging (fMRI) initially relied on image captioning techniques. In 2018, Matsuo et al. proposed a method for generating captions for visually perceived images using fMRI data, based on the image captioning framework. This method consisted of two stages: 1) first, converting the fMRI data into image features, and 2) then generating text describing the stimulus image (i.e., the visually perceived image) from the image features. Subsequently, in 2020, Takada et al. proposed a method for generating captions directly from fMRI data, where the fMRI data was first transformed into an unsupervised text latent space and then fed into a Long Short-Term Memory (LSTM) network to directly generate text. However, grammatical and linguistic inconsistencies still existed between the decoded text and the corresponding stimulus image content. Subsequently, in 2021, Huang et al. proposed a progressive transfer language decoding model based on fMRI, generating text describing stimulus images from multiple visual cortical response activities.

[0003] Currently, existing language decoding methods generally suffer from two problems: 1) The cortical regions on which decoding relies are selected empirically from a small number of areas, ignoring the contributions of other cortical regions to brain information encoding. 2) Previous decoding methods only achieve a single decoding task and cannot simultaneously implement multiple decoding tasks in a single model. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a multi-task visual perception decoding method based on GPT, which solves the problems that existing technologies cannot perform decoding of multiple tasks simultaneously and do not utilize global information from other areas of the brain for decoding.

[0005] The technical solution adopted in this invention is as follows:

[0006] A multi-task visual perception decoding method based on GPT, comprising: performing the following steps in a multi-task visual perception and decoding model including a text encoder, a visual information encoder, a whole-brain information encoder, and a language generator:

[0007] Step 1: The text encoder embeds the text containing the prompt word into the latent feature space to obtain a sequence of sentence embedding vectors containing positional embedding information.

[0008] Step 2: The visual information encoder converts the visual region response signal obtained from magnetic resonance imaging when viewing the visual perception image into a latent feature space vector to obtain the visual region feature vector.

[0009] Step 3: The whole-brain information encoder converts the response signals in the whole brain, excluding the visual areas, into latent feature space vectors to obtain feature vectors of non-visual areas.

[0010] Step 4: The language generator outputs text prediction vectors based on the natural language processing model GPT decoder (e.g., a decoder composed of GPT-2 Decoder Blocks), and outputs descriptive text of the visually perceived image based on a preset word embedding matrix.

[0011] The input to the GPT decoder is: the visual region feature vector and the non-visual region feature vector are added together to obtain the whole brain feature vector; then the whole brain feature vector is added to the sentence embedding vector corresponding to the prompt word part in the sentence embedding vector sequence, and the resulting latent feature sequence is the input to the GPT decoder.

[0012] Furthermore, in step 1, the text encoder's encoding method is as follows:

[0013] Step 1.1: Perform tokenization on the input text containing prompt words according to the vocabulary. After tokenization, a sequence of grammatical units (i.e., token sequence) in the form of an integer sequence of encoded text information is obtained.

[0014] Step 1.2: Multiply the sequence of syntactic units with the preset word embedding matrix (WT) to calculate the L*Num word embedding vector sequence, where L represents the length of the syntactic unit sequence and Num represents the dimension of the word embedding vector;

[0015] Step 1.3: Multiply the Token sequence obtained in Step 1.1 with the preset position embedding matrix (WPE) to calculate the L*Num position embedding vector sequence; where the word embedding matrix and the position embedding matrix have the same dimension;

[0016] Step 1.4: Linearly add the word embedding vector sequence and the position embedding vector sequence to obtain the fused L*Num sentence embedding vector sequence. Num is also used to represent the vector dimension of the sentence embedding vector sequence.

[0017] Furthermore, in step 2, the encoding method of the visual information encoder is as follows:

[0018] Step 2.1: Perform upsampling or downsampling operations on the visual region response signals from M different visual regions using the linear interpolation method to unify the data dimension to the embedding space dimension. Then, sort the obtained visual region response signals from different visual regions according to the visual region number to form an M*Num visual region feature sequence. The feature dimension of the visual region feature sequence is consistent with the vector dimension Num of the sentence embedding vector sequence.

[0019] Step 2.2: Input the visual region feature sequence into the GRU gated recurrent unit. The GRU integrates the visual region feature sequence into a single feature vector and takes the Num-dimensional hidden features of the final stage of the GRU as the visual region feature vector.

[0020] Furthermore, the encoding method of the whole-brain information encoder is as follows:

[0021] Step 3.1: Perform upsampling or downsampling operations on the response signals from N different non-visual regions using the linear interpolation method to unify the data dimension to the embedding space dimension. Then, sort the response signals from the different non-visual regions according to the non-visual region number to form an N*Num non-visual region feature sequence.

[0022] Step 3.2: Initialize the linear fully connected layers q, k, v, and the weight matrices of each linear fully connected layer are W. Q W K W V Each weight matrix has a dimension of Num*Num, and each linear fully connected layer contains the same number of weight matrices, all set to H.

[0023] The visual region feature sequences of M*Num are input into fully connected layers k and v, respectively, and are respectively coupled with H weight matrices W. K W V Multiplying them together yields H M*Num K and V matrices;

[0024] The N*Num non-visual region feature sequence is combined with H weight matrices W Q Multiplying them together yields H N*Num Q matrices;

[0025] Step 3.3: Multiply the H Q matrices by the transposes of the H K matrices, and then multiply them by the H V matrices to obtain H N*Num Z matrices;

[0026] The H Z matrices are concatenated row by row and then fed into a fully connected layer to change the dimensions, resulting in an N*Num Z' matrix. The Z' matrix is ​​then averaged column by column to obtain Num-dimensional hidden features, which are used as feature vectors for non-visual regions.

[0027] Furthermore, in step 4, outputting the descriptive text for the visually perceived image based on the preset word embedding matrix specifically includes:

[0028] The text prediction vector output by the GPT decoder is multiplied by the lm_head word embedding matrix of the lm_head component of the language model to obtain a list of predicted probabilities (logits) for the next word corresponding to the current text prediction vector. The maximum value in the prediction list is selected as the prediction token. The prediction token is then converted into the corresponding word and added to the end of the input sentence.

[0029] If the word at the end of the sentence is not an end marker, then input a new sentence of length L+1 into the multi-task visual perception and decoding model, and return to step 1; if the word at the end of the sentence is an end marker, then end the generation and processing of the descriptive text of the visual perception image, and output the current sentence.

[0030] Furthermore, in step 2, the step size of the gated recurrent unit is consistent with the number of (brain) visual regions divided, the number of layers is 1, and the size of the input layer and the output layer are consistent, both being Num-dimensional.

[0031] Furthermore, the parameter training of the multi-task visual perception and decoding model includes:

[0032] Acquire training data, which includes the categories of stimulus images (visual perception images), manually annotated text descriptions, and response signals from all brain regions evoked by each visual perception image;

[0033] The preceding text of a sentence is obtained word by word using attention mask, and the following text is predicted word by word based on the preceding text to generate text, thus obtaining the distribution of the predicted following text;

[0034] The cross-entropy loss L is calculated based on the predicted distribution of the text and the tag text:

[0035]

[0036] Where N represents the number of target words to be predicted, g i p represents the vocabulary index corresponding to the words in the manually labeled sentences of the training samples. i This represents the predicted probability of words generated by the language generator;

[0037] The model parameters of the multi-task visual perception and decoding model are trained and updated based on the loss function L and a pre-set optimization algorithm (such as the AdamW optimization algorithm).

[0038] Furthermore, the training and updating of model parameters for the multi-task visual perception and decoding model based on the loss function L and a pre-defined optimization algorithm includes:

[0039] (1) Initialize parameters, including learning rate α, step size ε, exponential decay rate ρ1 for first moment estimation, exponential decay rate ρ2 for second moment estimation, constant δ, weight decay coefficient λ, model parameter θ, first moment variable s, second moment variable r, time step t, batch size m.

[0040] Each m context sample pair {(x (1) ,y (1) ),(x (2) ,y (2) ),…,(x (m) ,y (m) The following updates were made this time:

[0041] (2) Iteratively update the parameters of the multi-task visual perception and decoding model based on the loss function L:

[0042] Calculate the gradient: Where, f(x) (i) ;θ) represents the input of the i-th sample x (i) The predicted output, L(f(x) (i) ;θ),y (i) ) represents the loss function for the i-th sample, the symbol ← indicates assignment, and g represents the gradient;

[0043] Update time step: t←t+1;

[0044] Update the partial first-order moment estimate: s←ρ1s+(1-ρ1)g;

[0045] Update the partial second moment estimate: r←ρ2r+(1-ρ2)g⊙g;

[0046] Correcting the deviation of the first moment: in, The first moment variable represents the correction. This represents ρ1 raised to the power of t;

[0047] Correcting the deviation of the second moment: in, The variable representing the modified second moment. This represents ρ² raised to the power of t;

[0048] Calculate the update amount of the model parameters:

[0049] Update model parameters: θ = θ + Δθ;

[0050] (3) Determine whether the convergence condition is met (such as the value of time step t reaching the preset upper limit or the value of loss function L converging). If yes, stop; otherwise, continue to execute step (2).

[0051] The technical solution provided by this invention brings at least the following beneficial effects:

[0052] This invention establishes a multi-task whole-brain information language decoding model based on the GPT model, which performs multi-level language decoding on complex natural images. The decoded category information has high accuracy, and the decoded language description reflects accurate language information and has a high similarity to human-described language. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the language reconstruction process according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the model structure of the multi-task visual perception and decoding model according to an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0057] This invention provides a multi-task visual perception decoding method based on GPT, which is a multi-task whole-brain information language decoding technology based on the GPT model. This invention is based on functional magnetic resonance imaging (fMRI) signal data from viewing a large number of natural images (visual perception images) and establishes a language decoding model based on the GPT model. The proposed model includes the following four modules: 1. A Prompt embedding module, which uses a trainable language encoding matrix and a position encoding matrix to embed the Prompt containing task information into the latent feature space; 2. A visual encoding module, which uses a GRU to encode voxel signals from visually relevant regions into the latent feature space; 3. A network encoding module, which uses a multi-head cross-attention mechanism to use voxel signals from visually relevant regions as context to guide the selection of voxel signals from all other brain regions and encodes global information into the latent feature space; 4. A language generation module, which fuses and identifies the latent features extracted from the above three modules to generate corresponding text. This invention solves the problem that multiple decoding tasks cannot be performed in a single decoding model and introduces effective global information from the brain, enabling practical applications in multi-task language-generative brain-computer interface scenarios.

[0058] refer to Figure 1 , Figure 2 The specific implementation steps of the multi-task visual perception decoding method based on GPT provided in this embodiment of the invention include:

[0059] A. Construct the Prompt encoding module:

[0060] Step A1: Enter a standard formatted Prompt text containing the following information: subject number, decoding task (category decoding or language decoding). For example, to perform language decoding on subject number 01, the Prompt should be designed as "subject 01fMRIcaption decoding:", which will be used as an example below.

[0061] Step A2: Input the Prompt text obtained in Step A1 into the Tokenizer, converting it into a 9-character token sequence. For the model, a token is a digital representation. Each token is associated with a unique numeric ID, which the model uses to distinguish different tokens. During training, the model learns how to map text to these numeric IDs, enabling it to encode and decode new text.

[0062] Step A3: Use the Token sequence obtained in Step A2 as the input for this step. First, input the WTE word embedding matrix (lookup table) to obtain a 9*768 word embedding vector sequence.

[0063] Step A4: Use the Token sequence obtained in Step A2 as the input for this step. Input the WPE position embedding matrix (lookup table) to obtain a 9*768 position embedding vector sequence.

[0064] B. Constructing the visual encoding module:

[0065] Step B1: Perform upsampling or downsampling operations on the response signal data from 29 different visual regions using a linear interpolation method to unify the data dimension to the embedding space dimension. Then, sort the signals from the different visual regions according to the visual region number to form a 29*768 visual region feature sequence.

[0066] Step B2: Input the 29*768 visual region feature sequence obtained in step B1 into the GRU gated recurrent unit. The GRU integrates the feature vector sequence into a single feature vector and takes the 1*768-dimensional hidden feature of the last stage of the GRU as the visual region feature vector.

[0067] Steps B1 and B2 convert the visual region signal into a latent feature space vector.

[0068] C. Constructing a whole-brain coding module:

[0069] Step C1: Perform upsampling or downsampling operations on the response signal data from 151 different non-visual regions using a linear interpolation method to unify the data dimension to 768 dimensions. Then, sort the response signals from different non-visual regions according to the non-visual region number to form a 151*768 non-visual region feature sequence.

[0070] Step C2: Initialize the linear fully connected layers q, k, and v, with an input dimension of 768 and an output dimension of 8*768 = 6144. Input the 29*768 visual region feature sequence obtained in step B1 into the fully connected layers k and v respectively to obtain matrices K and V of 29*6144. Input the 151*768 visual region feature sequence obtained in step C1 into the fully connected layer q to obtain the Q matrix of 151*6144.

[0071] Step C3: Multiply the 151*6144 Q matrix obtained in Step C2 by the transpose of the corresponding 29*6144 K matrix (6144*29), then pass it through a softmax layer, and multiply it by the 29*6144 V matrix to obtain a 151*6144 Z matrix; initialize the fully connected layer z with an input dimension of 6144 and an output dimension of 768; input the 151*6144 Z matrix row by row into the fully connected layer to change its dimensions, resulting in a 151*768 Z matrix; average the Z matrix column by column to obtain a 768-dimensional hidden feature as the feature vector for the non-visual region.

[0072] D. Constructing the language generation module:

[0073] Step D1: Add the 768-dimensional visual region feature vector obtained in step B2 to the 768-dimensional non-visual region feature vector obtained in step C3 to obtain a 768-dimensional whole-brain feature vector.

[0074] Step D2: Add the 768-dimensional whole-brain feature vector obtained in step D1 to the 9*768 sentence embedding vector corresponding to the Prompt part in the sentence embedding vector sequence obtained in step A4, to obtain a 9*768 latent feature sequence.

[0075] Step D3: Input the 9*768 latent feature sequence obtained in step D2 into the decoder consisting of 12 stacked GPT-2 Decoder Blocks, and output the prediction vector;

[0076] Step D4: Multiply the predicted vector obtained in step D3 with the lm_head word embedding matrix to obtain a list of logits. Select the maximum value in the logits list as the predicted token. Convert the predicted token into the corresponding word and add it to the end of the input sentence.

[0077] Step D5: If the word corresponding to the last token is not the end-of-text symbol "<|endoftext|>", input a new sentence of length 10 (9+1) into the model and return to step A1; if the word corresponding to the last token is the end-of-text symbol "<|endoftext|>", end text generation.

[0078] E. Training Phase:

[0079] Step E1: Acquire training data, which includes the category of stimulus images, manually annotated text descriptions, and response signals from all brain regions evoked by each stimulus image.

[0080] Step E2: Use Attention Mask to sequentially obtain the preceding text of the sentence word by word, and predict the following text word by word based on the preceding text and the corresponding fMRI signal to generate text, thus obtaining the distribution of the predicted following text.

[0081] Step E3: Calculate the cross-entropy loss L based on the predicted distribution of the text and the tag text, as follows:

[0082]

[0083] Where N represents the number of target words to be predicted; g i p represents the vocabulary index of words in manually annotated sentences in the training samples (converted to one-hot labels during calculation). i This represents the list of logits generated by the language generator;

[0084] Step E4: Based on the loss function L, use the AdamW optimization algorithm to update the corresponding weights of the parameters of the entire model.

[0085] The specific optimization parameters for AdamW are as follows: the step size ε defaults to 0.00015; the exponential decay rates ρ1 and ρ2 for moment estimation default to 0.9 and 0.999 respectively; and the numerical stability constant δ defaults to 10. -8 The weight decay coefficient λ is set to 10 by default. -2 Initialize the parameters θ with a normal distribution of mean 0 and variance 0.001; initialize the first and second moment variables as s = 0, r = 0; initialize the time step to t = 0; each time there are m context sample pairs {(x (1) ,y (1) ),(x (2) ,y (2) ),…,(x (m) ,y (m) The following updates will be made:

[0086] Calculate the gradient: Where: f(x) (i) ;θ) represents the input x (i) The predicted output, L(f(x) (i) ;θ),y (i) ) represents the loss function for the i-th sample, the symbol ← indicates assignment, and g represents the gradient;

[0087] Update time step: t←t+1;

[0088] Update the partial first moment estimate: s←ρ1s+(1-ρ1)g, where: ρ1 represents the exponential decay rate of the first moment estimate (default is 0.9), and s represents the first moment variable;

[0089] Update the partial second moment estimate: r←ρ2r+(1-ρ2)g⊙g, where: ρ2 represents the exponential decay rate of the second moment estimate (default is 0.999), and r represents the second moment variable;

[0090] Correcting the deviation of the first moment: in: The first moment variable represents the correction. This represents ρ1 raised to the power of t;

[0091] Correcting the deviation of the second moment: in: The variable representing the modified second moment. This represents ρ² raised to the power of t;

[0092] Calculation update: Where: Δ θ This represents the amount of change in parameter θ;

[0093] Weight parameter update: θ←θ+Δ θ ;

[0094] F. Testing Phase:

[0095] Step F1: Collect test data, including: data from Microsoft Common Objects in The dataset consists of 1000 color natural stimulus images cropped to 425*425 pixels from the Context (COCO) dataset, along with 5 to 6 manually annotated text descriptions of the image content. It also includes visual cortex signals (including 29 cortical regions: 'V1', 'MST', 'V6', 'V2', 'V3', 'V4', 'V8', 'V3A', 'V7', 'IPS1', 'FFC', 'V3B', 'LO1', 'LO2', 'PIT', 'MT', 'PCV', 'STV', 'PH', 'DVT', 'V6A', 'VMV1', 'VMV3', 'V4t', 'FST', 'V3CD', 'LO3', 'VMV2', and 'VVC') and signals from the remaining 151 non-visual cortex regions, based on HCP-MMP1 atlas partitioning. Test data is not included in the training data.

[0096] Step F2: Based on the training phase, a model is obtained that includes a Prompt encoding module, a visual encoding module, a whole-brain encoding module, and a language generation module. The voxel signals of the visual region corresponding to the test image are input into the visual encoding module, while the voxel signals of both the visual and non-visual regions corresponding to the test image are input into the whole-brain encoding module. The Prompt text is then input into the Prompt encoding module. Based on the Prompt content, the model generates the corresponding text. If the Prompt instruction is a category decoding, the model outputs the predicted category label of the visual stimulus; if the Prompt instruction is a language decoding, the model outputs the predicted text description of the visual stimulus. This process achieves multi-task decoding based on whole-brain signals. The prediction accuracy of category decoding reaches 63.63%, far exceeding the random level of 8.33% for 12-class classification.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0098] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A multi-task visual perception decoding method based on GPT, characterized in that, Perform the following steps in a multi-task visual perception and decoding model that includes a text encoder, a visual information encoder, a whole-brain information encoder, and a language generator: Step 1: The text encoder embeds the text containing the prompt words into the latent feature space to obtain a sequence of sentence embedding vectors containing positional embedding information. Step 2: The visual information encoder converts the visual region response signal obtained from magnetic resonance imaging when viewing the visual perception image into a latent feature space vector to obtain the visual region feature vector. Step 3: The whole-brain information encoder converts the response signals in the whole brain, excluding the visual areas, into latent feature space vectors to obtain feature vectors of non-visual areas. Step 4: The language generator outputs text prediction vectors based on the GPT decoder of the natural language processing model, and outputs descriptive text of the visually perceived image based on the preset word embedding matrix. The input to the GPT decoder is: the visual region feature vector and the non-visual region feature vector are added together to obtain the whole brain feature vector; then the whole brain feature vector is added to the sentence embedding vector corresponding to the prompt word part in the sentence embedding vector sequence, and the resulting latent feature sequence is the input to the GPT decoder. The encoding method of the whole-brain information encoder is as follows: Step 3.1: Perform upsampling or downsampling operations on the response signals from N different non-visual regions using a linear interpolation method to unify the data dimension to the embedding space dimension. Then, sort the response signals from the different non-visual regions according to the non-visual region number to form an N*Num non-visual region feature sequence; where Num is a preset value, consistent with the vector dimension of the sentence embedding vector sequence and the visual region feature vector sequence. Step 3.2: Initialize the linear fully connected layers q, k, v, and the weight matrices of each linear fully connected layer are as follows: , , Each weight matrix has a dimension of Num*Num, and each linear fully connected layer contains the same number of weight matrices, all set to H. The visual region feature sequences of M*Num are input into fully connected layers k and v, respectively, and are respectively coupled with H weight matrices. , Multiplying them together yields H M*Num units. Matrix and matrix; Combine the N*Num non-visual region feature sequences with H weight matrices Multiplying them together yields H N*Num units. matrix; Step 3.3: Combine the H Q matrices with the H... Multiply the transposes of the matrices, and then multiply them by H V matrices to obtain H N*Num Z matrices; The H Z matrices are concatenated row by row and then fed into a fully connected layer to change the dimensions, resulting in an N*Num Z' matrix. The Z' matrix is ​​then averaged column by column to obtain Num-dimensional hidden features, which are used as feature vectors for non-visual regions. In step 4, the descriptive text for the visually perceived image, based on a pre-set word embedding matrix, specifically includes: The text prediction vector output by the GPT decoder is multiplied by the lm_head word embedding matrix of the lm_head component of the language model to obtain a list of predicted probabilities for the next word corresponding to the current text prediction vector. The maximum value in the prediction list is selected as the prediction word token, and then the prediction word token is converted into the corresponding word and added to the end of the input sentence. If the word at the end of the sentence is not an end marker, then input a new sentence of length L+1 into the multi-task visual perception and decoding model, and return to step 1; if the word at the end of the sentence is an end marker, then end the generation and processing of the descriptive text of the visual perception image, and output the current sentence.

2. The method as described in claim 1, characterized in that, In step 1, the text encoder's encoding method is as follows: Step 1.1: Perform word segmentation on the input text containing prompt words according to the vocabulary. After word segmentation, a sequence of grammatical units in the form of an integer sequence of encoded text information is obtained. Step 1.2: Multiply the sequence of syntactic units with the preset word embedding matrix to calculate the L*Num word embedding vector sequence, where L represents the length of the syntactic unit sequence and Num represents the dimension of the word embedding vector; Step 1.3: Multiply the Token sequence obtained in Step 1.1 with the preset position embedding matrix to calculate the L*Num position embedding vector sequence; where the word embedding matrix and the position embedding matrix have the same dimension; Step 1.4: Linearly add the word embedding vector sequence and the position embedding vector sequence to obtain the fused L*Num sentence embedding vector sequence.

3. The method as described in claim 1, characterized in that, In step 2, the encoding method of the visual information encoder is as follows: Step 2.1: Perform upsampling or downsampling operations on the visual region response signals from M different visual regions using the linear interpolation method to unify the data dimension to the embedding space dimension. Then, sort the obtained visual region response signals from different visual regions according to the visual region number to form an M*Num visual region feature sequence. The feature dimension of the visual region feature sequence is consistent with the vector dimension Num of the sentence embedding vector sequence. Step 2.2: Input the visual region feature sequence into the gated recurrent unit (GRU). The GRU integrates the visual region feature sequence into a single feature vector and takes the Num-dimensional hidden features from the final stage of the GRU as the visual region feature vector.

4. The method as described in claim 1, characterized in that, In step 2, the step size of the gated loop unit is the same as the number of visual regions divided, the number of layers is 1, and the size of the input layer and the output layer are the same.

5. The method as described in claim 1, characterized in that, The parameter training of the multi-task visual perception and decoding model includes: Acquire training data, which includes the categories of visual perception images, manually annotated text descriptions, and response signals from all brain regions evoked by each visual perception image. Using masked attention, we sequentially obtain the preceding text of a sentence word by word, and predict the following sentence word by word based on the preceding text to generate text, thus obtaining the distribution of the predicted following text; Calculate the cross-entropy loss based on the predicted distribution of the text and the tag text. : ; in, This indicates the number of target words that need to be predicted. This represents the vocabulary index corresponding to the words in the manually labeled sentences of the training samples. This represents the predicted probability of words generated by the language generator; Based on loss function The model parameters of the multi-task visual perception and decoding model are trained and updated using a pre-defined optimization algorithm.

6. The method as described in claim 5, characterized in that, Based on loss function The training and updating of model parameters for the multi-task visual perception and decoding model using pre-defined optimization algorithms includes: (1) Initialize parameters, including learning rate Step length The exponential decay rate estimated by the first moment The exponential decay rate estimated by the second moment ,constant Weight decay coefficient Model parameters First-order moment variables Second-order moment variables Time step t, batch size m; Each m context sample pairs The following updates have been made: (2) Based on loss function Iterative updates to the parameters of the multi-task visual perception and decoding model: Calculate the gradient: ;in, This indicates that the i-th sample is input. The predicted output at that time Let represent the loss function for the i-th sample, with the symbol . This indicates assignment. Represents the gradient; Update time step: ; Update the partial first-order moment estimate: ; Updated partial second-order moment estimate: ; Correcting the deviation of the first moment: ,in, The first moment variable represents the correction. express The power of t; Correcting the deviation of the second moment: ,in, The variable representing the modified second moment. express The power of t; Calculate the update amount of the model parameters: ; Update model parameters: ; (3) Determine whether the convergence condition is met (such as the value of time step t reaching the preset upper limit or the value of loss function L converging). If yes, stop; otherwise, continue to execute step (2).

Images