Multi-granularity brain semantic alignment method based on large model

Abstract

The invention discloses a multi-granularity brain semantic alignment method based on a large model, which comprises the following steps of: generating a set of sentences with the same semantics and different syntaxes on the aspect of sentence granularity based on a word order disruption operation of a reserved stroke; extracting sentence characterization in the set by using the coding function of the large language model to the sentences, and averaging to obtain a sentence meaning characterization vector; and through a linear fitting model, sentence meaning characterization is used to predict the intensity of functional magnetic resonance of each position of the human brain, and positions related to sentence meaning understanding in the human brain are obtained. In terms of word granularity, constructing an unsupervised probability statistical model, and introducing an alignment relationship and an output rate to describe an activation relationship between words and a human brain interval; and performing parameter iteration by using an expectation maximization method, training the model until convergence, and obtaining the alignment probability of the words to the brain region. According to the method, modeling is carried out on two granularities of sentences and words respectively, an experiment is carried out on a parallel data set of text-brain activity records, and brain region distribution of human brain semantic understanding under multiple granularities is obtained.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary technical field of computer natural language processing and neuroscience, and in particular to a multi-granularity brain semantic alignment method based on a large model. Background Art

[0002] The human brain's processing of natural language is one of the topics that continues to be explored in many fields such as neuroscience and cognitive linguistics. Although it started early, there are still many problems to be explored in this topic. In the past period of time, the fields of computer and artificial intelligence have developed rapidly, and the natural language processing direction has produced many milestone methods and models. In recent years, some work has used methods or models in the field of natural language processing (especially the large language models that have developed rapidly in recent years) to assist in the study of the language mechanism of the human brain, and has achieved many results. This includes the study of the semantic understanding process of the human brain.

[0003] With the help of various language models, a representation vector of a word or a sentence is generated, and then linear regression is used to predict the functional magnetic resonance imaging (fMRI) intensity of the human brain from the representation vector, and finally the Pearson correlation coefficient is used to evaluate the prediction effect. This process is called "brain encoding". Language models at the word level include word2vec, ELMO, etc., and language models at the sentence level include GPT-2, LLaMA, etc. Use these models to generate representation vectors from words and sentences, and then use brain encoding technology to map them to the fMRI image of the human brain.

[0004] However, this method is somewhat rough for sentence meaning representation mapping, because the vectors in the language model contain not only the sentence meaning information in the discourse, but also other information such as syntactic structure. Adding an adversarial goal to the model to minimize the performance of the representation vector in grammatical tasks can extract semantic representation. However, the vector obtained under this adversarial goal does not perform well in brain encoding tasks, indicating that its similarity with the brain has decreased. On the other hand, for word meaning, the vector representation of the word can be extracted from the model and then brain encoded, but the current language model gradually pays less attention to the improvement of word representation methods, which has hindered the development of research in this area. At the same time, the lack of parallel corpus data sets of brain imaging at the word level makes it difficult to conduct supervised alignment model training. Summary of the invention

[0005] The present invention provides a multi-granularity brain semantic alignment method based on a large model, so as to conduct a systematic multi-granularity exploration of the semantic understanding process of the human brain. The semantic understanding process is divided into the understanding of the meaning of each word in a sentence and the understanding of the sentence meaning of the overall combined semantics. Sentence meanings need to be effectively represented and extracted, and word meanings require new alignment modeling ideas. The present invention provides alignment modeling methods for sentence meanings and word meanings to human brain fMRI (functional magnetic resonance imaging) images, respectively. The combination of the two can obtain the distribution characteristics of semantics of different granularities in the human brain, thereby providing a possible path for hierarchical comprehensive analysis, which is easy to operate and highly scalable.

[0006] The embodiment of the present invention provides a multi-granularity brain semantic alignment method based on a large model, comprising the following steps:

[0007] In terms of sentence meaning, the sentence meaning representation vector is extracted: each sentence in the data set is shuffled while preserving the word order, and the core action, implementer and receiver of the action are retained during the shuffle, and a set of sentences with similar sentence meanings but different grammatical structures are generated. The sentences in this set are input into the large language model, and vectors are extracted from its hidden layer. The vectors are averaged to obtain the sentence meaning representation vector of the sentence;

[0008] Mapping of sentence meaning representation to the human brain: The sentence meaning representation vector will be mapped to each voxel of the functional magnetic resonance imaging image of the human brain through brain coding technology. If the nuclear magnetic intensity on a voxel is predicted by the sentence meaning representation vector to be higher than the set threshold, it means that this voxel is related to the sentence meaning comprehension function;

[0009] In terms of word meaning, the alignment model is modeled and simplified: a probability model is established for the alignment relationship between each word in the data set text and each brain region in the human brain. At the same time, for the alignment relationship of "one word may cause reactions in multiple brain regions", the "output rate" model is used to make assumptions about the established model to simplify the model;

[0010] Training of the alignment model: The obtained model is converted into an optimization problem and trained on a parallel dataset. The trained model includes the probability of alignment between each word and brain region.

[0011] Optionally, in one embodiment of the present invention, extracting the sentence meaning representation vector specifically includes:

[0012] (1) Determine the words corresponding to the action elements of the sentence. The action is the core action of the sentence, the doer and the receiver of the action;

[0013] (2) The word order of each non-dynamic word in the sentence is changed, and the position after the change is determined by a random variable and the original position, thereby obtaining a sentence with a disrupted word order;

[0014] (3) Repeat step (2) to obtain multiple sentences with shuffled word order;

[0015] (4) Input the set of sentences obtained in step (3) into the large language model and extract the representation vector from the hidden layer of the model;

[0016] (5) Take the average of the set of language vectors obtained in step (4) to obtain the sentence meaning representation vector of the original sentence.

[0017] Optionally, in one embodiment of the present invention, the mapping of sentence meaning representation to the human brain specifically includes:

[0018] (1) Preprocessing of sentence meaning representation vectors and human brain functional magnetic resonance imaging sequences, including principal component analysis to unify dimensions and downsampling of functional magnetic resonance imaging;

[0019] (2) Train a ridge regression model and use the sentence meaning representation vector of each sentence in the dataset to predict the functional magnetic resonance imaging image of the human brain corresponding to this sentence, that is, the magnetic resonance intensity at each voxel;

[0020] (3) The Pearson correlation coefficient was used to evaluate the prediction effect of the ridge regression model, and the brain areas with predictions higher than the set threshold were those related to sentence meaning processing.

[0021] Optionally, in one embodiment of the present invention, the modeling and simplification of the alignment model specifically includes:

[0022] (1) Use probability models and alignment relationships to describe the process of words triggering brain activity. That is, given multiple words, the probability model describes the conditional probability of the brain areas they activate.

[0023] (2) Introducing output rate to describe the situation where “one word can trigger responses in multiple brain regions”;

[0024] (3) Use the chain rule to disassemble the model and propose hypotheses to simplify the model.

[0025] Optionally, in one embodiment of the present invention, the training of the alignment model specifically includes:

[0026] (1) The Lagrange multiplier method is introduced to transform the model into an unconstrained optimization problem;

[0027] (2) Use the expectation maximization algorithm to iterate parameters, calculate the expectation in the E step, and iterate the parameters to maximize the probability in the M step;

[0028] (3) Repeat steps E and M in step (2) until the parameters converge.

[0029] Optionally, in one embodiment of the present invention, the training of the alignment model specifically includes:

[0030] In a period of time, that is, a functional magnetic resonance imaging measurement time slice, assume that m words are mapped to n brain areas that produce responses, and these words are denoted as S = {w1, w2, ... w m}, the responding brain area is B = {r1, r2, ... r n}, there is an alignment relationship This process of "multiple words triggering responses in multiple brain regions" is described as:

[0031]

[0032] For each word w in S i , may activate brain regions, called output rate, recorded as Their positions in B are recorded as After introducing the output rate, the model is rewritten as:

[0033]

[0034] in, Then make the following five assumptions:

[0035]

[0036] The final model is:

[0037]

[0038] p0+p1=1.

[0039] Optionally, in one embodiment of the present invention, in the training of the alignment model, the constrained optimization problem is converted into a Lagrangian unconstrained problem:

[0040]

[0041] Among them, θ is the target parameter, and the optimal parameter is calculated using the expected maximum algorithm. In the E step, the expected frequency is used for calculation, and the following expected frequency formula is defined:

[0042]

[0043] Among them, δ(x,y) indicates whether x and y appear at the same time, if yes, it is 1, otherwise it is 0. In the M step, the parameter estimation is performed for the training set of K samples, and the parameter estimation formula is:

[0044]

[0045] The multi-granularity brain semantic alignment method based on a large model in an embodiment of the present invention has the following beneficial effects:

[0046] It provides a systematic (involving both sentence meaning and word meaning) semantic alignment modeling framework. The sentence meaning representation extraction method is simpler to operate than previous methods, and experiments show that it has good effectiveness. It innovatively proposes an unsupervised word-brain region alignment model, which improves the process of language mapping to the human brain, has good scalability, and provides the possibility of statistical analysis for different linguistic classifications, solving the problem that the previous models of the prior art use word embedding technology and can only perform statistical analysis on the mathematical characteristics of vectors.

[0047] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0049] Figure 1 A flowchart of a multi-granularity brain semantic alignment method based on a large model according to an embodiment of the present invention;

[0050] Figure 2 A flowchart of extracting a sentence meaning representation vector according to an embodiment of the present invention;

[0051] Figure 3 Schematic diagram of the correspondence between brain regions and vocabulary according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0053] Figure 1 The present invention provides a flowchart of a multi-granularity brain semantic alignment method based on a large model according to an embodiment of the present invention.

[0054] like Figure 1 As shown, the multi-granularity brain semantic alignment method based on a large model includes the following steps:

[0055] In terms of sentence meaning, (1) Extract sentence meaning representation vectors. The word order of each sentence in the dataset is shuffled while preserving the “motion” to generate a set of sentences with similar sentence meanings but different grammatical structures. The sentences in this set are input into a large language model, and vectors are extracted from its hidden layer. The vectors are averaged to obtain the sentence meaning representation vector;

[0056] (2) Mapping of sentence meaning representation to the human brain. The sentence meaning representation vector will be mapped to each voxel of the fMRI image of the human brain through brain coding technology. If the nuclear magnetic intensity on a voxel is well predicted by the sentence meaning representation vector, it means that this voxel is related to the sentence meaning comprehension function;

[0057] Specifically, the human brain and semantic alignment framework based on multi-granularity modeling includes two granularities: sentence meaning and word meaning.

[0058] Extraction of sentence meaning representation. In this step, the word order of each sentence in the data set is shuffled while preserving the "action", generating a set of sentences with similar sentence meanings but different grammatical structures. The sentences in this set are input into a large language model such as LLaMA3, and vectors are extracted from each hidden layer. The vectors are averaged to obtain the sentence meaning representation vector. The so-called "action" refers to the core action in the sentence and its doer and patient (such as "I ate a cake with a fork last night.", the core action is "ate", the doer is "I", and the patient is "cake").

[0059] Mapping of representation vectors to the human brain. In this step, the sentence meaning representation vector obtained in the previous step will be mapped to each voxel of the fMRI image of the human brain through brain coding technology (i.e., linear regression + Pearson correlation coefficient). If the magnetic intensity on a voxel is well predicted by the sentence meaning representation vector, it means that this voxel is related to the sentence meaning comprehension function.

[0060] The word meaning granularity includes the following steps:

[0061] In terms of word meaning, (1) Modeling and simplification of alignment models. A probability model is established for the alignment relationship between each word in the data set text and each brain region in the human brain. At the same time, for the alignment relationship where "one word may cause reactions in multiple brain regions", the "output rate" model is used. After that, reasonable assumptions are made on the established model to simplify the model;

[0062] (2) Training of the alignment model. The obtained model is converted into an optimization problem and trained on a parallel dataset. The trained model includes the probability of alignment between each word and brain region. Relevant statistical analysis can be performed based on this, such as the preference relationship between words of different parts of speech and specific brain regions.

[0063] Optionally, in one embodiment of the present invention, the method for extracting a sentence meaning representation vector comprises the following steps:

[0064] (1) Determine the words that correspond to the “action” elements in the sentence (i.e., the core action of the sentence, the doer and the recipient of the action);

[0065] (2) Change the word order of each non-dynamic word in the sentence, and the position after the change is determined by a random variable and the original position. This step results in a sentence with a disrupted word order.

[0066] (3) Repeat step (2) to obtain multiple sentences with shuffled word order.

[0067] (4) Input the set of sentences obtained in step (3) into a large language model (LLaMA3 is used in the present invention) and extract the representation vector from the hidden layer of the model.

[0068] (5) Take the average of the set of language vectors obtained in step (4) to obtain the sentence meaning representation vector of the original sentence.

[0069] Optionally, in one embodiment of the present invention, mapping the sentence meaning representation to the human brain includes the following steps:

[0070] (1) Preprocessing the sentence meaning representation vector and human brain fMRI sequence, including principal component analysis to unify the dimension and fMRI downsampling;

[0071] (2) Train a ridge regression model and use the sentence meaning representation vector of each sentence in the dataset to predict the human brain fMRI image (i.e., the magnetic resonance intensity at each voxel) corresponding to this sentence;

[0072] (3) The Pearson correlation coefficient was used to evaluate the prediction effect of the ridge regression model. The brain areas with good prediction effect were those related to sentence meaning processing.

[0073] Optionally, in one embodiment of the present invention, the modeling and simplification of the alignment model includes the following steps:

[0074] (1) Use probability models and alignment relationships to describe the process of words triggering brain activity. That is, given multiple words, the probability model describes the conditional probability of the brain areas they activate;

[0075] (2) Introducing output rate to describe the situation where “one word can trigger responses in multiple brain regions”;

[0076] (3) Use the chain rule to disassemble the model and propose hypotheses to simplify the model.

[0077] Optionally, in one embodiment of the present invention, the training of the alignment model includes the following steps:

[0078] (1) The Lagrange multiplier method is introduced to transform the model into an unconstrained optimization problem;

[0079] (2) Use the EM algorithm (Expectation Maximization Algorithm) to iterate parameters. Find the expectation in the E step and iterate the parameters to maximize the probability in the M step;

[0080] (3) Repeat steps E and M in step (2) until the parameters converge.

[0081] Optionally, in one embodiment of the present invention, in the modeling and simplification of the alignment model, within a period of time (i.e., within an fMRI measurement time slice), it is assumed that m words are mapped to n brain regions that produce responses, and these words are recorded as S = {w1, w2, ... w n}, the responding brain area is B = {r1, r2, ... r n}. There is an alignment relationship This process of "multiple words triggering responses in multiple brain regions" is described as:

[0082]

[0083] For each word w in S i , may activate brain regions (called output rate), recorded as Their positions in B are recorded as After introducing the output rate, the model is rewritten as:

[0084]

[0085] in, Then make the following five assumptions:

[0086]

[0087] The final model is:

[0088]

[0089] p0+p1=1.

[0090] Optionally, in one embodiment of the present invention, in the training of the alignment model, the constrained optimization problem is converted into a Lagrangian unconstrained problem:

[0091]

[0092] Among them, θ is the target parameter, and the optimal parameter is calculated using the expected maximum algorithm. In the E step, the expected frequency is used for calculation, and the following expected frequency formula is defined:

[0093]

[0094] Among them, δ(x,y) indicates whether x and y appear at the same time, if yes, it is 1, otherwise it is 0. In the M step, the parameter estimation is performed for the training set of K samples, and the parameter estimation formula is:

[0095]

[0096] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0097] In order to conduct a systematic and multi-granular exploration of the semantic understanding process of the human brain, Figure 1 As shown, the framework of the invention is divided into two parts: a sentence meaning part, which includes the extraction and mapping of sentence meaning representations; and a word meaning part, which includes the modeling and training of alignment relationships.

[0098] Sentence meaning part:

[0099] (1) Extraction of sentence meaning representation

[0100] The overall sentence meaning extraction method is as follows Figure 2 As shown, in this method, for each sentence, the individual words that make up its "action process" are first determined, that is, the core action in the sentence and its doer and patient. The relative order of these words will remain unchanged to keep the meaning of the sentence unchanged (for example, "The dog bit the boy." If the relative order becomes "The boy bit thedog.", the semantics is changed).

[0101] For non-"dynamic" words in the sentence, each word will take an integer p that conforms to the normal distribution, and the word will be adjusted to the position before or after it (because p can be positive or negative, the corresponding position can be swapped before or after). In this way, a sentence with similar meaning to the original sentence but different grammatical structure is obtained. Repeat the operation multiple times on each original sentence to obtain multiple sentences with similar semantics but different syntax. For a sentence S = {w1, w2, w3, ... w n}(w i , i∈[1,n] is a word in the sentence) to shuffle the word order:

[0102] Predication is the set of dynamic elements in S.

[0103]

[0104] These sentences are input into the language model, and the representation vectors of the sentences are obtained from the hidden layer of the model. The representation vectors of a group of sentences with similar semantics but different syntactic features are averaged to obtain the sentence meaning representation vector of this group of sentences (that is, the original sentences).

[0105] (2) Mapping of representation vectors to the human brain

[0106] Here we describe in detail the voxel-level brain encoding process after obtaining the sentence meaning representation.

[0107] Hypothesis Representation M is the number of words corresponding to X, and d is the vector dimension. For example, the vector dimension corresponding to the GPT-2 model is 768. The fMRI vectors corresponding to these words N represents the number of time slices (TR). Ridge regression is used to fit both:

[0108] argmin f ∑(Yf(X)) 2 +λ‖f‖ 2

[0109] Where λ is the regularization coefficient.

[0110] The fitting effect is called brain score in brain coding, which is the Pearson correlation coefficient between the fMRI predicted value and the true value:

[0111]

[0112] In addition, since the dimensions of X and Y are different and there is some unimportant information in X, principal component analysis will be used to unify the dimensions during the fitting process.

[0113] Meaning part:

[0114] (1) Modeling and simplification of alignment relationships

[0115] The model describes a scenario where, when measuring fMRI data, the subject hears several words within a period of time (i.e., within an fMRI measurement time slice), and these words cause some brain regions to respond. Assume that m words are mapped to n brain regions that respond, and these words ("S" represents external stimulus, stimuli) are S = {w1,w2,…w m}, the responding brain area is B = {r1, r2, ... r n}. They have an alignment relationship like Figure 3 The goal is to explore this alignment relationship. Note that the NULL symbol is introduced here to describe the possibility that a word does not cause a response in the brain area and non-text elements cause a response in some brain areas.

[0116] First, we model this mapping process. Here, we use the idea of ​​a probabilistic model, that is, within a specific time slice, the mapping of external stimulus S to brain area B is a probabilistic problem. That is, given an external stimulus S, the probability of some brain areas B producing a response is recorded as P(B|S). The alignment relationship is that the mapping first traverses each possible alignment relationship, and each alignment method corresponds to the probability P(B,a|S) of generating a correct mapping under this alignment relationship. Each alignment relationship is independent of each other. According to the addition criterion, the mapping probability is:

[0117]

[0118] A word may correspond to multiple brain areas, and a brain area may be triggered by multiple words. This is a "many-to-many" problem, which makes the number of "traversing each possible alignment relationship" very large. In order to simplify the subsequent modeling, it is split into a two-way "one-to-many" problem. That is, a word or non-text factor can cause reactions in multiple brain areas, while a brain area cannot be activated by multiple words. This alignment relationship is symmetrical. In the subsequent training phase, two trainings (forward and reverse) can be performed, and the resulting alignment can be taken as the union. The alignment relationship representation is rewritten here: a = {w i →r j |i∈[1,m],j∈[1,n],w i Activate j}.

[0119] For each word w in S i , may activate brain regions (called output rate), recorded as Their positions in B are recorded as A <τ,π> pair corresponds to a<B,a> Yes, and one<B,a> Yes <τ,π> pairs. Translate the above alignment model:

[0120]

[0121] in The above equation is decomposed by the chain rule. Without any assumptions, both sides of the equation are strictly equal. However, in order to simplify the model, some assumptions need to be made to simplify it:

[0122] 1. Assume that the output rate is only related to the current word, that is

[0123] 2. Assume that the probability of a brain area being activated by a word is only related to this word, that is

[0124] 3. Since the position order in the brain region set has no practical meaning, we assume that the brain region positions are evenly distributed. i The set of activated brain regions τ i , after the locations of k brain regions have been determined, there are positions, i.e.

[0125] 4. Assume that the After the non-empty aligned brain regions are generated, the position of the empty aligned brain region is generated behind each brain region with probability p1. It follows the binomial distribution,

[0126] 5. Assume that after determining the brain regions and their locations activated by non-empty words, the remaining brain regions are generated by empty words. After k empty words have been placed, there are still positions, and are evenly distributed, but

[0127] There are models:

[0128]

[0129] p0+p1=1

[0130] (2) Model optimization and training

[0131] The above constrained optimization problem is transformed into a Lagrangian unconstrained problem, where θ is the parameter in the optimization objective, i.e., p0, p1, f(r|w):

[0132]

[0133] Since the above formula has no analytical solution, the EM algorithm (Expectation Maximization Algorithm) is used for parameter iteration (using the old parameters to calculate the current new parameters, and then replacing the old parameters with the new parameters). The EM algorithm is divided into the E step and the M step. In the E step, the expected frequency is used for calculation, and the following expected frequency formula is defined:

[0134]

[0135] Among them, δ(x,y) indicates whether x and y appear at the same time, if yes, it is 1, otherwise it is 0. In the M step, the parameter estimation is performed for the training set of K samples, and the parameter estimation formula is:

[0136]

[0137] The new values ​​of each parameter can be calculated according to the above formula. Repeat steps E and M until the parameter values ​​converge. Finally, the alignment probability f(r j |w i ).

[0138] The multi-granularity brain semantic alignment method based on a large model proposed in an embodiment of the present invention includes: at the sentence granularity, generating a set of sentences with the same semantics but different syntax based on a word order shuffling operation that retains the action sequence; using a large language model to extract the sentence representations in the set of encoding functions of the sentences and then averaging them to obtain a sentence meaning representation vector; using a linear fitting model, using sentence meaning representations to predict the intensity of functional magnetic resonance at individual positions in the human brain, and obtaining positions in the human brain related to sentence meaning understanding. At the word granularity, an unsupervised probabilistic statistical model is constructed, and the alignment relationship and output rate are introduced to describe the activation relationship between words and brain regions of the human brain; the expectation maximization method is used to iterate parameters, and the model is trained until convergence to obtain the alignment probability of words to brain regions. The present invention models the sentence and word granularities separately, and conducts experiments on a parallel data set of text-brain activity records to obtain the brain region distribution of human brain semantic understanding at multiple granularities.

[0139] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0140] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0141] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

Claims

1. A multi-granularity brain semantic alignment method based on a large model, characterized in that: The following steps are involved: In terms of sentence meaning, the sentence meaning representation vector is extracted: each sentence in the data set is shuffled while preserving the word order, and the core action, implementer and receiver of the action are retained during the shuffle, and a set of sentences with similar sentence meanings but different grammatical structures are generated. The sentences in this set are input into the large language model, and vectors are extracted from its hidden layer. The vectors are averaged to obtain the sentence meaning representation vector of the sentence; Mapping of sentence meaning representation to the human brain: The sentence meaning representation vector will be mapped to each voxel of the functional magnetic resonance imaging image of the human brain through brain coding technology. If the nuclear magnetic intensity on a voxel is predicted by the sentence meaning representation vector to be higher than the set threshold, it means that this voxel is related to the sentence meaning comprehension function; In terms of word meaning, the alignment model is modeled and simplified: a probability model is established for the alignment relationship between each word in the data set text and each brain region in the human brain. At the same time, for the alignment relationship of "one word may cause reactions in multiple brain regions", the "output rate" model is used to make assumptions about the established model to simplify the model; Training of the alignment model: The obtained model is converted into an optimization problem and trained on a parallel dataset. The trained model includes the probability of alignment between each word and brain region.

2. The method according to claim 1, characterized in that The extraction of sentence meaning representation vector specifically includes: (1) Determine the words corresponding to the "action" element of the sentence. The action is the core action of the sentence, the doer and the receiver of the action; (2) The word order of each non-dynamic word in the sentence is changed, and the position after the change is determined by a random variable and the original position, thereby obtaining a sentence with a disrupted word order; (3) Repeat step (2) to obtain multiple sentences with shuffled word order; (4) Input the set of sentences obtained in step (3) into the large language model and extract the representation vector from the hidden layer of the model; (5) Take the average of the set of language vectors obtained in step (4) to obtain the sentence meaning representation vector of the original sentence.

3. The method according to claim 1, characterized in that The mapping of sentence meaning representation to the human brain specifically includes: (1) Preprocessing of sentence meaning representation vectors and human brain functional magnetic resonance imaging sequences, including principal component analysis to unify dimensions and downsampling of functional magnetic resonance imaging; (2) Train a ridge regression model and use the sentence meaning representation vector of each sentence in the dataset to predict the functional magnetic resonance imaging image of the human brain corresponding to this sentence, that is, the magnetic resonance intensity at each voxel; (3) The Pearson correlation coefficient was used to evaluate the prediction effect of the ridge regression model, and the brain areas with predictions higher than the set threshold were those related to sentence meaning processing.

4. The method according to claim 1, characterized in that: The modeling and simplification of the alignment model specifically include: (1) Use probability models and alignment relationships to describe the process of words triggering brain activity. That is, given multiple words, the probability model describes the conditional probability of the brain areas they activate. (2) Introducing output rate to describe the situation where "one word can trigger responses in multiple brain regions"; (3) Use the chain rule to disassemble the model and propose hypotheses to simplify the model.

5. The method according to claim 1, characterized in that The training of the alignment model specifically includes: (1) The Lagrange multiplier method is introduced to transform the model into an unconstrained optimization problem; (2) Use the expectation maximization algorithm to iterate parameters, calculate the expectation in the E step, and iterate the parameters to maximize the probability in the M step; (3) Repeat steps E and M in step (2) until the parameters converge.

6. The method according to claim 5, characterized in that The training of the alignment model specifically includes: In a period of time, that is, a functional magnetic resonance imaging measurement time slice, assume that m words are mapped to n brain areas that produce responses, and these words are denoted as S = {w1, w2, ... w m }, the responding brain area is B = {r1, r2, ... r n }, there is an alignment relationship This process of "multiple words triggering responses in multiple brain regions" is described as: For each word w in S i , may activate brain regions, called output rate, recorded as Their positions in B are recorded as After introducing the output rate, the model is rewritten as: in, Then make the following five assumptions: The final model is: p0+p1=1.

7. The method according to claim 5, characterized in that In the training of the alignment model, the constrained optimization problem is transformed into a Lagrangian unconstrained problem: Among them, θ is the target parameter, and the optimal parameter is calculated using the expected maximum algorithm. In the E step, the expected frequency is used for calculation, and the following expected frequency formula is defined: Among them, δ(x,y) indicates whether x and y appear at the same time, if yes, it is 1, otherwise it is 0. In the M step, the parameter estimation is performed for the training set of K samples, and the parameter estimation formula is:

Images