Neural semantic encoding and decoding analysis method and system

By combining training regression model and probe task, the text representation model is adjusted, which solves the problem of poor certainty in the neural semantic codec technology, and improves the accuracy of text analysis and model robustness.

CN112232084BActive Publication Date: 2024-11-12INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202011102971.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-10-15
Publication Date
2024-11-12
Estimated Expiration
2040-10-15

AI Technical Summary

Technical Problem

In the existing neural semantic codec technology, it is impossible to effectively determine the codec relationship, resulting in unstable accuracy of text processing.

Method used

By training the regression model to fit the mapping relationship between brain nerve activation level and the vector representation of text stimulation, establish a text representation model, and quantify the language features through probe task, and further adjust the text representation model through ablation task to improve the analysis task accuracy.

Benefits of technology

Improve the accuracy of text analysis tasks, enhance the robustness of text representation models, and ensure efficient performance when performing analysis tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112232084B_ABST
    Figure CN112232084B_ABST
Patent Text Reader

Abstract

The present invention relates to a neural semantic coding and decoding analysis method and system, the neural semantic coding and decoding analysis method comprising: training a regression model to fit the mapping relationship between brain nerve activation level and vector representation of text stimulus, and establishing a text representation model; quantitatively analyzing the ability of the text representation model to describe various language features through probe tasks, and obtaining probe task performance; adjusting the text representation model through ablation tasks according to the probe task performance; and performing analysis tasks based on sentence vectors generated by the adjusted text representation model. The present invention trains a regression model to fit the mapping relationship between brain nerve activation level and vector representation of text stimulus, and establishes a text representation model; quantitatively analyzing the ability of the text representation model to describe various language features through probe tasks; further confirming the robustness of the text representation model through ablation tasks, and adjusting the text representation model accordingly, thereby improving the accuracy when performing analysis tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of text processing, and in particular to a neural semantic encoding and decoding analysis method and system based on probe tasks and ablation tests. Background Art

[0002] In recent years, neurosemantic encoding and decoding technology has developed rapidly. Neurosemantic encoding simulates the language cognitive working mechanism of brain nerves by predicting the human brain's response to specific language stimuli, while neurosemantic decoding reveals the content of the text read or heard by people by analyzing human brain activities. At present, the brain imaging signals used in neurosemantic encoding and decoding mainly include neurophysiological signals, such as EEG, and neuroimaging signals, such as magnetic resonance imaging. Among them, functional magnetic resonance imaging (fMRI) has the advantages of non-invasiveness and no radiation exposure, which makes it widely used in this field.

[0003] The basic paradigm of existing neural encoding and decoding is to fit the mapping relationship between brain neural activity and the quantitative representation of stimuli. Therefore, under the premise that the brain imaging paradigm is specified as fMRI, the accuracy of neural encoding and decoding depends largely on how to establish the quantitative representation of stimuli. In recent years, driven by machine learning technology, text representation methods have been continuously iterating and evolving. This provides a large number of text quantitative representation models with different structures and encoding features for neural encoding and decoding candidates. Since it is impossible to effectively determine the specific encoding and decoding, the accuracy of encoding and decoding for text processing is unstable. Summary of the invention

[0004] In order to solve the above problems in the prior art, that is, to quickly determine the codec and improve the accuracy of the text analysis task, the purpose of the present invention is to provide a neural semantic codec analysis method and system.

[0005] In order to solve the above technical problems, the present invention provides the following solutions:

[0006] A neural semantic encoding and decoding analysis method, the neural semantic encoding and decoding analysis method comprising:

[0007] The regression model is trained to fit the mapping relationship between the brain neural activation level and the vector representation of the text stimulus, and a text representation model is established;

[0008] The probe task is used to quantify the ability of the parsing text representation model to describe various language features and obtain the probe task performance;

[0009] Based on the performance of the probe task, the text representation model is adjusted through the ablation task;

[0010] Perform analysis tasks based on the sentence vectors generated by the adjusted text representation model.

[0011] Optionally, the text representation model includes a neural encoder model and a neural decoder model;

[0012] The training regression model fits the mapping relationship between the cranial nerve activation level and the vector representation of the text stimulus to establish a text representation model, specifically including:

[0013] Acquire a training sample set, wherein the training samples in the training sample set include functional magnetic resonance imaging samples and corresponding text stimuli, voxel matrices, and sentence representation matrices;

[0014] Initializing a regression model according to the dimensions of the functional magnetic resonance image and the corresponding stimulus representation, and setting network parameters;

[0015] Based on the initialized regression model and network parameters, the neural encoder model and the neural decoder model are obtained according to the voxel matrix and the sentence representation matrix respectively.

[0016] Optionally, obtaining a neural encoder model according to the voxel matrix and the sentence representation matrix specifically includes:

[0017] By minimizing the first objective function, we obtain the regression coefficient W of the neural encoder model. e :

[0018] The first objective function is:

[0019] Among them, λ is the regularization parameter, x i represents the voxel matrix, Z e represents the sentence representation matrix, N E Represents the number of training samples, N V Represents the number of voxels, N D Indicates the dimensionality of sentence representation; represents the second norm, represents a norm;

[0020] Based on the initialized regression model, according to the regression coefficient W of the neural encoder model e , determine the neural encoder model, which learns to predict brain activity from sentence representations.

[0021] Optionally, obtaining a neural decoder model according to the voxel matrix and the sentence representation matrix specifically includes:

[0022] By minimizing the second objective function, we obtain the regression coefficient W of the neural decoder model. d :

[0023] The second objective function is:

[0024] Among them, λ is the regularization parameter, N E Represents the number of training samples, N V Represents the number of voxels, N D represents the dimension of the sentence representation, X represents the voxel matrix, z i represents the sentence vector matrix, represents the second norm, represents a norm;

[0025] Based on the initialized regression model, according to the regression coefficient W of the neural decoder model d , determine a neural decoder model; the neural decoder model is performed by estimating semantic vectors directly from voxels, with each dimension predicted by a separate regression model.

[0026] Optionally, the probe task includes at least one of a semantic probe task, a syntactic probe task and a morphological probe task;

[0027] The probe task quantifies the ability of the text representation model to describe various language features, and obtains the probe task performance, which specifically includes:

[0028] The ability of the parsing text representation model to describe semantic features is quantified through semantic probe tasks. The ability of the parsing text representation model to describe syntactic structure features is quantified through syntactic probe tasks. The ability of the parsing text representation model to describe syntactic morphological features is quantified through morphological probe tasks.

[0029] Optionally, adjusting the text representation model through an ablation task according to the probe task performance specifically includes:

[0030] Correlate the probe task performance with the accuracy of the neural encoder model and the neural decoder model to obtain the correlation results;

[0031] According to the association result, the damaged text representation model generates sentence vector representations through the damaged text representation model based on the training corpus, trains the neural decoder model and the neural encoder model, and determines the accuracy of the neural decoder model and the neural encoder model;

[0032] The text representation model is adjusted according to the accuracy.

[0033] Optionally, associating the probe task performance with the accuracy of the neural encoder model and the neural decoder model to obtain the association result specifically includes:

[0034] Calculate the correlation coefficients between the performance of the semantic probe task, syntactic probe task, and morphological probe task and the accuracy of the neural encoder model and the neural decoder model, and the significance level corresponding to the correlation coefficient.

[0035] In order to solve the above technical problems, the present invention also provides the following solution: a neural semantic encoding and decoding analysis system, the neural semantic encoding and decoding analysis system comprising:

[0036] A training unit, used for training a regression model to fit the mapping relationship between the brain nerve activation level and the vector representation of text stimuli, and establishing a text representation model;

[0037] The parsing unit is used to quantify the ability of the parsing text representation model to describe various language features through probe tasks to obtain the probe task performance.

[0038] An adjustment unit, used to adjust the text representation model through an ablation task according to the performance of the probe task;

[0039] The execution unit is used to perform the analysis task based on the sentence vectors generated by the adjusted text representation model.

[0040] In order to solve the above technical problems, the present invention also provides the following solution: a neural semantic encoding and decoding analysis system, comprising:

[0041] Processor; and

[0042] a memory arranged to store computer executable instructions which, when executed, cause the processor to:

[0043] The regression model is trained to fit the mapping relationship between the brain neural activation level and the vector representation of the text stimulus, and a text representation model is established;

[0044] The probe task is used to quantify the ability of the parsing text representation model to describe various language features and obtain the probe task performance;

[0045] Based on the performance of the probe task, the text representation model is adjusted through the ablation task;

[0046] Perform analysis tasks based on the sentence vectors generated by the adjusted text representation model.

[0047] To solve the above technical problems, the present invention also provides the following solution: a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device performs the following operations:

[0048] The regression model is trained to fit the mapping relationship between the brain neural activation level and the vector representation of the text stimulus, and a text representation model is established;

[0049] The probe task is used to quantify the ability of the parsing text representation model to describe various language features and obtain the probe task performance;

[0050] Based on the performance of the probe task, the text representation model is adjusted through the ablation task;

[0051] Perform analysis tasks based on the sentence vectors generated by the adjusted text representation model.

[0052] According to the embodiments of the present invention, the present invention discloses the following technical effects:

[0053] The present invention establishes a text representation model by training a regression model to fit the mapping relationship between the level of brain nerve activation and the vector representation of text stimuli. The ability of the text representation model to describe various language features is quantified and analyzed through probe tasks. The robustness of the text representation model is further confirmed through ablation tasks, and the text representation model is adjusted accordingly, thereby improving the accuracy when performing analysis tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of the neural semantic encoding and decoding analysis method of the present invention;

[0055] Figure 2 It is a schematic diagram of the module structure of the neural semantic encoding and decoding analysis system of the present invention;

[0056] Figure 3 It is a schematic diagram of the hardware framework of the neural semantic encoding and decoding analysis system of the present invention;

[0057] Figure 4 It is a schematic diagram of the module structure of a computer system with a computer-readable storage medium.

[0058] Explanation of symbols:

[0059] Training unit—1, parsing unit—2, adjustment unit—3, execution unit—4. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0061] The purpose of the present invention is to provide a neural semantic encoding and decoding analysis method, which establishes a text representation model by training a regression model to fit the mapping relationship between the brain neural activation level and the vector representation of text stimuli, and quantifies the ability of the text representation model to describe various language features through probe tasks; further confirms the robustness of the text representation model through ablation tasks, and adjusts the text representation model accordingly, thereby improving the accuracy when performing analysis tasks.

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] like Figure 1 As shown, the neural semantic encoding and decoding analysis method of the present invention includes:

[0064] Step 100: training a regression model to fit the mapping relationship between the brain nerve activation level and the vector representation of the text stimulus, and establishing a text representation model;

[0065] Step 200: quantify the ability of the parsing text representation model to describe various language features through the probe task to obtain the probe task performance;

[0066] Step 300: According to the performance of the probe task, adjust the text representation model through the ablation task;

[0067] Step 400: Perform an analysis task based on the sentence vectors generated by the adjusted text representation model.

[0068] Among them, the text representation model includes a neural encoder model and a neural decoder model.

[0069] In step 100, the training regression model fits the mapping relationship between the cranial nerve activation level and the vector representation of the text stimulus to establish a text representation model, which specifically includes:

[0070] Step 110: Acquire a training sample set, wherein the training samples in the training sample set include functional magnetic resonance image samples and corresponding text stimuli, voxel matrices, and sentence representation matrices.

[0071] Preferably, after obtaining the training sample set, the training sample set is preprocessed, which specifically includes:

[0072] The functional magnetic resonance imaging (FMRI) images to be decoded were obtained, and the FSL method (FSL is a neuroimaging processing software) was used for time alignment and head motion correction. The FLIRT method was used to convert the corrected data into the standard space MNI-152 (i.e., spatial normalization) to obtain the preprocessed images.

[0073] Decoding the preprocessed image by a decoding model based on elastic synaptic gates to obtain a first representation; the first representation is a vector representation of the stimulus to which the subject is subjected when acquiring the functional magnetic resonance image;

[0074] Encoding textual stimuli into numerical representations using different text representation models.

[0075] Step 120: Initialize the regression model according to the dimensions of the functional magnetic resonance image and the corresponding stimulus representation, and set the network parameters.

[0076] Step 130: Based on the initialized regression model and network parameters, a neural encoder model and a neural decoder model are obtained according to the voxel matrix and the sentence representation matrix, respectively.

[0077] The step of obtaining a neural encoder model based on the voxel matrix and the sentence representation matrix specifically includes:

[0078] By minimizing the first objective function, we obtain the regression coefficient W of the neural encoder model. e :

[0079] The first objective function is:

[0080] Among them, λ is the regularization parameter, Z e is the sentence representation matrix, N E Represents the number of training samples, N V Represents the number of voxels, N D represents the dimension of sentence representation; x i represents the voxel matrix, represents the second norm, represents a norm;

[0081] Based on the initialized regression model, according to the regression coefficient W of the neural encoder model e , determine the neural encoder model, which learns to predict brain activity from sentence representations.

[0082] The neural decoder model is obtained according to the voxel matrix and the sentence representation matrix, specifically including:

[0083] By minimizing the second objective function, we obtain the regression coefficient W of the neural decoder model. d :

[0084] The second objective function is:

[0085] Among them, λ is the regularization parameter, N E represents the number of training samples, NV Represents the number of voxels, N D represents the dimension of the sentence representation, X represents the voxel matrix, z i represents the sentence vector matrix, represents the second norm, represents a norm;

[0086] Based on the initialized regression model, according to the regression coefficient W of the neural decoder model d , determine a neural decoder model; the neural decoder model is performed by estimating semantic vectors directly from voxels, with each dimension predicted by a separate regression model.

[0087] In this embodiment, λ is 0.1; in the calculation process of the neural decoder model and the neural encoder model, a stochastic gradient descent optimization algorithm is used, and the learning rate is 0.01.

[0088] Furthermore, in step 100, it also includes: verifying the encoding and decoding accuracy of the text representation model through a test set.

[0089] In this embodiment, both neural encoding and decoding are evaluated through paired matching tasks. The following takes neural decoding as an example to illustrate the paired matching task:

[0090] The decoding model optimized on the training set decodes a semantic vector on the brain map of the test set and calculates the correlation coefficient between the semantic vector and the corresponding real sentence vector. If the decoded semantic vector is more similar to its real sentence vector than its other random sentence vector, it is considered a successful match. For each possible sentence pair S in the test set, i and S j , Z Si and Z Sj Indicates S i and S j The corresponding real sentence vector D Si and D Sj represents the decoded semantic vector. If

[0091]

[0092] The pair matching is successful.

[0093] In step 200, the probe task includes at least one of a semantic probe task, a syntactic probe task, and a morphological probe task.

[0094] The ability of the text representation model to describe various language features by quantitative analysis using probe tasks is specifically to complete different downstream tasks based on the sentence vectors generated by the text representation model. The downstream tasks are a set of evaluation tasks, in which each subtask requires the model to encode linguistic features including but not limited to grammatical, semantic, and morphological features.

[0095] Preferably, the ability of the text representation model to describe various language features through quantitative analysis of the probe task to obtain the probe task performance specifically includes:

[0096] The ability of the parsing text representation model to describe semantic features is quantified through semantic probe tasks. The ability of the parsing text representation model to describe syntactic structure features is quantified through syntactic probe tasks. The ability of the parsing text representation model to describe syntactic morphological features is quantified through morphological probe tasks.

[0097] Among them, (1) the ability of the parsing text representation model to describe semantic features is quantified through the semantic probing task, which specifically includes:

[0098] Obtain a corpus containing a set number of sentences; manually annotate the semantic similarity between each pair of sentences in the corpus; after the annotation is completed, calculate the similarity of each pair of sentence vectors based on the sentence vectors generated by the text representation model; finally, compare the correlation between the manual annotation results and the sentence vector calculation results.

[0099] In this embodiment, STS and SICK corpora are used. Preferred implementation methods for calculating sentence vector similarity include but are not limited to cosine similarity and Pearson's correlation. Preferred implementation methods for comparing the correlation between manually annotated similarity and sentence vector calculated similarity include but are not limited to Pearson's correlation and Spearman correlation.

[0100] (2) The ability of the parsing text representation model to describe syntactic structural features is quantified through the syntactic probe task, including:

[0101] Measures the degree to which the syntactic tree of a sentence can be reconstructed from its vector representation.

[0102] The preferred implementation method of "rebuilding the syntax tree" is: specify a natural language text corpus, and use a syntax analysis tool to generate a syntax tree for each sentence in the corpus. i , let w1, ..., w nRepresents the words in the sentence. A text representation model m can encode the word into a word vector m(w j ). Fit a transfer matrix b so that

[0103]

[0104] Among them, w j and w k Represents any two words in a sentence. The parse tree of the sentence represents the separation w j and w k The number of edges in B is essentially a measure of syntactic distance. Once B is fitted on the training set, it can be applied to the test set to predict the syntactic distance between any two words in a test set sentence. For a sentence containing n words, this prediction can result in an n×n distance matrix. An undirected syntactic analysis tree can be reconstructed on this distance matrix using the minimum spanning tree algorithm. The accuracy of the tree reconstruction is measured by the unlabeled attachment score.

[0105] (3) The ability of the parsing text representation model to describe syntactic morphological features is quantified through morphological probe tasks, including:

[0106] Evaluate how well the length of sentences can be predicted from vector representations generated by a text representation model.

[0107] Length is considered to be a morphological feature of the sentence because it can be obtained without the need for semantic knowledge. The implementation method of this embodiment is to specify the corpus as Toronto Book Corpus and divide the sentences in the corpus into multiple categories according to the number of words. On this corpus, a three-layer forward neural network using only sentence vectors as input is trained to complete the sentence length classification task. Stochastic gradient descent optimization is used in training, and the learning rate is 0.001. The accuracy of the obtained model on the test set is reported as the performance of the corresponding text representation on the morphological probe task.

[0108] In step 300, adjusting the text representation model through the ablation task according to the probe task performance specifically includes:

[0109] Step 310: Correlate the probe task performance with the accuracy of the neural encoder model and the neural decoder model respectively to obtain correlation results.

[0110] The step of associating the probe task performance with the accuracy of the neural encoder model and the neural decoder model to obtain an association result specifically includes:

[0111] Calculate the correlation coefficients between the performance of semantic probe tasks, syntactic probe tasks, and morphological probe tasks and the accuracy of the neural encoder model and neural decoder model, and the significance level corresponding to the correlation coefficients. This is used to see whether the ability of the sentence model to encode a certain language feature is significantly related to its encoding performance and encoding and decoding accuracy.

[0112] Step 320: According to the association result, the damaged text representation model generates sentence vector representations based on the training corpus through the damaged text representation model, trains the neural decoder model and the neural encoder model, and determines the accuracy of the neural decoder model and the neural encoder model.

[0113] Among them, the method of syntactic ablation is to debug the pre-trained representation model on sentences with completely disrupted word order. This debugging reduces the ability of the representation model to encode syntactic structure. In this type of ablation test, only the word order in the sentence is disrupted, and the original words are not changed, added or deleted. Therefore, even if the order is disrupted, the words contained in it can still reflect the semantic information of the original sentence to a certain extent.

[0114] The method of semantic ablation: the pre-trained representation model is debugged on a machine without changing the word order, but all nouns or verbs are replaced with other random nouns or verbs. This replacement will not cause syntactic errors in the sentence, and its syntactic structure is still feasible, but there are grammatical distortions. Using this corpus to debug the pre-trained sentence representation model will reduce its ability to encode fine-grained grammatical information. In particular, for some distributed models that learn the representation of corresponding language units by modeling the relationship between words, sentences, phrases and their context, ablation will greatly reduce the model's ability to encode semantic relationships between different language units.

[0115] After obtaining the structure of the ablation experiment, it can be determined whether the ability of the damaged text representation model to encode a certain language feature has an impact on its neural encoding and decoding accuracy. Combined with the results of the probe task experiment, it can be determined whether encoding a certain language feature significantly affects the performance of the text representation model on the neural encoding and decoding task.

[0116] Step 330: Adjust the text representation model according to the accuracy.

[0117] To prove the effectiveness of the decoding method of the present invention, it is compared with the decoding models of other models, such as Figure 3 To explain what differences in the features they capture lead to differences in their accuracy in predicting the brain, we performed three probe tasks, correlated probe task scores with encoding performance, and Figure 3 The results are described in [a].

[0118] like Figure 3 As shown in [a], the morphological probe task scores were not significantly correlated with encoding in any brain region in the language network (p>0.01). This means that differences in the ability of sentence embeddings to describe sentence length cannot reliably explain their different encoding performance in the language network. The syntactic probe task scores were correlated with encoding accuracy in the lateral parietal area (LPar), left posterior temporal gyrus (LPTG), and left inferior frontal gyrus (LIFGorb), with p<0.01 for these three brain regions. This means that in these three brain regions, the unlabeled attachment score (UAS) of the dependency parse tree reconstructed from the sentence embeddings of different text representation models accounted for a large part of the differences in their encoding accuracy. The semantic probe scores were significantly correlated with the matching accuracy of all brain regions in the language network (p<0.01 for each brain region) except LaMFG (p>0.01).

[0119] like Figure 3 As shown in [a], we found that the semantic probe scores of different text representation models were significantly correlated with the encoding accuracy of most brain regions in the language atlas. To further verify this finding, we selected the BERT text representation model that reliably encodes brain activity in the language network and performed ablation tests. We adjusted the syntactic or semantic representation of the model on the corresponding ablation dataset to see which interferences would significantly affect the encoding accuracy of the text representation model in the language network. Figure 3 As shown in [b], the perturbation of the ablation task produced different results in different brain regions. Relative to the unadjusted baseline, in 8 of the 11 brain regions, BERT adjusted by the semantic ablation task reduced matching accuracy (all p<0.01).

[0120] In addition, the present invention also provides a neural semantic encoding and decoding analysis system. Figure 2 As shown, the neural semantic encoding and decoding analysis system of the present invention includes a training unit 1, a parsing unit 2, an adjustment unit 3 and an execution unit 4.

[0121] Specifically, the training unit 1 is used to train a regression model to fit the mapping relationship between the brain nerve activation level and the vector representation of the text stimulus, and establish a text representation model;

[0122] The parsing unit 2 is used to quantify the ability of the parsing text representation model to describe various language features through the probe task to obtain the probe task performance.

[0123] The adjustment unit 3 is used to adjust the text representation model through the ablation task according to the probe task performance;

[0124] The execution unit 4 is used to perform the analysis task based on the sentence vector generated by the adjusted text representation model.

[0125] Furthermore, the present invention also provides a neural semantic encoding and decoding analysis system, comprising:

[0126] Processor; and

[0127] a memory arranged to store computer executable instructions which, when executed, cause the processor to:

[0128] The regression model is trained to fit the mapping relationship between the brain neural activation level and the vector representation of the text stimulus, and a text representation model is established;

[0129] The probe task is used to quantify the ability of the parsing text representation model to describe various language features and obtain the probe task performance;

[0130] Based on the performance of the probe task, the text representation model is adjusted through the ablation task;

[0131] Perform analysis tasks based on the sentence vectors generated by the adjusted text representation model.

[0132] The present invention also provides the following solution: a computer-readable storage medium storing one or more programs, wherein when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device performs the following operations:

[0133] The regression model is trained to fit the mapping relationship between the brain neural activation level and the vector representation of the text stimulus, and a text representation model is established;

[0134] The probe task is used to quantify the ability of the parsing text representation model to describe various language features and obtain the probe task performance;

[0135] Based on the performance of the probe task, the text representation model is adjusted through the ablation task;

[0136] Perform analysis tasks based on the sentence vectors generated by the adjusted text representation model.

[0137] Compared with the prior art, the neural semantic coding and decoding analysis system and computer-readable storage medium of the present invention have the same beneficial effects as the above-mentioned neural semantic coding and decoding analysis method, which will not be repeated here.

[0138] Below Figure 4 , which is an example to illustrate a computer system with a computer-readable storage medium. Figure 4The server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0139] like Figure 4 As shown in , the computer system includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503. Various programs and data required for system operation are also stored in the RAM 503. The CPU 501, ROM 502 and RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0140] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.

[0141] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the method of the present invention are executed. It should be noted that the above-mentioned computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more conductors, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device, or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0142] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A neural semantic encoding and decoding analysis method, characterized in that: The neural semantic encoding and decoding analysis method comprises: The regression model is trained to fit the mapping relationship between the brain neural activation level and the vector representation of the text stimulus, and a text representation model is established; The probe task is used to quantify the ability of the parsing text representation model to describe various language features and obtain the probe task performance; Based on the performance of the probe task, the text representation model is adjusted through the ablation task; Perform analysis tasks based on the sentence vectors generated by the adjusted text representation model; The text representation model includes a neural encoder model and a neural decoder model; The training regression model fits the mapping relationship between the cranial nerve activation level and the vector representation of the text stimulus to establish a text representation model, specifically including: Acquire a training sample set, wherein the training samples in the training sample set include functional magnetic resonance imaging samples and corresponding text stimuli, voxel matrices, and sentence representation matrices; After obtaining the training sample set, preprocessing is also included: obtaining the functional magnetic resonance image to be decoded, and using the FSL method to perform time alignment and head motion correction, and using the FLIRT method to perform spatial standardization on the corrected data to obtain a preprocessed image; decoding the preprocessed image through a decoding model based on elastic synaptic gates to obtain a first representation; the first representation is a vector representation of the stimulus received by the subject when the functional magnetic resonance image is collected; using different text representation models to encode the text stimulus into a numerical representation; Initializing a regression model according to the dimensions of the functional magnetic resonance image and the corresponding stimulus representation, and setting network parameters; Based on the initialized regression model, the neural encoder model and the neural decoder model are obtained according to the voxel matrix and the sentence representation matrix respectively.

2. The neural semantic encoding and decoding analysis method according to claim 1, characterized in that: The neural encoder model is obtained according to the voxel matrix and the sentence representation matrix, specifically including: By minimizing the first objective function, we obtain the regression coefficient W of the neural encoder model. e : The first objective function is: Among them, λ is the regularization parameter, Z e represents the sentence representation matrix, N E Represents the number of training samples, N V Represents the number of voxels, N D represents the dimension of sentence representation; x i represents the voxel matrix, represents the binary norm, and ||·||1 represents the mononomial norm Based on the initialized regression model, according to the regression coefficient W of the neural encoder model e , determine the neural encoder model, which learns to predict brain activity from sentence representations.

3. The neural semantic encoding and decoding analysis method according to claim 1, characterized in that: The neural decoder model is obtained according to the voxel matrix and the sentence representation matrix, specifically including: By minimizing the second objective function, we obtain the regression coefficient W of the neural decoder model. d : The second objective function is: Among them, λ is the regularization parameter, N E Represents the number of training samples, N V Represents the number of voxels, N D represents the dimension of the sentence representation, X represents the voxel matrix, z i represents the sentence representation matrix, represents the second norm, ||·||1 represents the first norm; Based on the initialized regression model, according to the regression coefficient W of the neural decoder model d , determine the neural decoder model; the neural decoder model is performed by estimating the semantic vector directly from the voxels, and each dimension is predicted by a separate regression model.

4. The neural semantic encoding and decoding analysis method according to claim 1, characterized in that: The probe task includes at least one of a semantic probe task, a syntactic probe task and a morphological probe task; The probe task quantifies the ability of the text representation model to describe various language features, and obtains the probe task performance, which specifically includes: The ability of the parsing text representation model to describe semantic features is quantified through semantic probe tasks; the ability of the parsing text representation model to describe syntactic structure features is quantified through syntactic probe tasks; and the ability of the parsing text representation model to describe syntactic morphological features is quantified through morphological probe tasks.

5. The neural semantic encoding and decoding analysis method according to claim 4, characterized in that: According to the performance of the probe task, the text representation model is adjusted through the ablation task, specifically including: Correlate the probe task performance with the accuracy of the neural encoder model and the neural decoder model to obtain the correlation results; According to the association result, the damaged text representation model generates sentence vector representations through the damaged text representation model based on the training corpus, trains the neural decoder model and the neural encoder model, and determines the accuracy of the neural decoder model and the neural encoder model; The text representation model is adjusted according to the accuracy.

6. The neural semantic encoding and decoding analysis method according to claim 5, characterized in that: The step of associating the probe task performance with the accuracy of the neural encoder model and the neural decoder model to obtain an association result specifically includes: Calculate the correlation coefficients between the performance of the semantic probe task, syntactic probe task, and morphological probe task and the accuracy of the neural encoder model and the neural decoder model, and the significance level corresponding to the correlation coefficient.

7. A neural semantic encoding and decoding analysis system, characterized in that: The neural semantic encoding and decoding analysis system comprises: A training unit, used for training a regression model to fit the mapping relationship between the brain nerve activation level and the vector representation of text stimuli, and establishing a text representation model; The parsing unit is used to quantify the ability of the parsing text representation model to describe various language features through probe tasks to obtain the probe task performance. An adjustment unit, used to adjust the text representation model through an ablation task according to the performance of the probe task; An execution unit, configured to perform an analysis task based on the sentence vectors generated by the adjusted text representation model; The text representation model includes a neural encoder model and a neural decoder model; The training regression model fits the mapping relationship between the cranial nerve activation level and the vector representation of the text stimulus to establish a text representation model, specifically including: Acquire a training sample set, wherein the training samples in the training sample set include functional magnetic resonance imaging samples and corresponding text stimuli, voxel matrices, and sentence representation matrices; After obtaining the training sample set, preprocessing is also included: obtaining the functional magnetic resonance image to be decoded, and using the FSL method to perform time alignment and head motion correction, and using the FLIRT method to perform spatial standardization on the corrected data to obtain a preprocessed image; decoding the preprocessed image through a decoding model based on elastic synaptic gates to obtain a first representation; the first representation is a vector representation of the stimulus received by the subject when the functional magnetic resonance image is collected; using different text representation models to encode the text stimulus into a numerical representation; Initializing a regression model according to the dimensions of the functional magnetic resonance image and the corresponding stimulus representation, and setting network parameters; Based on the initialized regression model, the neural encoder model and the neural decoder model are obtained according to the voxel matrix and the sentence representation matrix respectively.

8. A neural semantic encoding and decoding analysis system, comprising: processor; as well as a memory arranged to store computer executable instructions which, when executed, cause the processor to: The regression model is trained to fit the mapping relationship between the brain neural activation level and the vector representation of the text stimulus, and a text representation model is established; The probe task is used to quantify the ability of the parsing text representation model to describe various language features and obtain the probe task performance; Based on the performance of the probe task, the text representation model is adjusted through the ablation task; Perform analysis tasks based on the sentence vectors generated by the adjusted text representation model; The text representation model includes a neural encoder model and a neural decoder model; The training regression model fits the mapping relationship between the cranial nerve activation level and the vector representation of the text stimulus to establish a text representation model, specifically including: Acquire a training sample set, wherein the training samples in the training sample set include functional magnetic resonance imaging samples and corresponding text stimuli, voxel matrices, and sentence representation matrices; After obtaining the training sample set, preprocessing is also included: obtaining the functional magnetic resonance image to be decoded, and using the FSL method to perform time alignment and head motion correction, and using the FLIRT method to perform spatial standardization on the corrected data to obtain a preprocessed image; decoding the preprocessed image through a decoding model based on elastic synaptic gates to obtain a first representation; the first representation is a vector representation of the stimulus received by the subject when the functional magnetic resonance image is collected; using different text representation models to encode the text stimulus into a numerical representation; Initializing a regression model according to the dimensions of the functional magnetic resonance image and the corresponding stimulus representation, and setting network parameters; Based on the initialized regression model, the neural encoder model and the neural decoder model are obtained according to the voxel matrix and the sentence representation matrix respectively.

9. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform the following operations: The regression model is trained to fit the mapping relationship between the brain neural activation level and the vector representation of the text stimulus, and a text representation model is established; The probe task is used to quantify the ability of the parsing text representation model to describe various language features and obtain the probe task performance; Based on the performance of the probe task, the text representation model is adjusted through the ablation task; Perform analysis tasks based on the sentence vectors generated by the adjusted text representation model; The text representation model includes a neural encoder model and a neural decoder model; The training regression model fits the mapping relationship between the cranial nerve activation level and the vector representation of the text stimulus to establish a text representation model, specifically including: Acquire a training sample set, wherein the training samples in the training sample set include functional magnetic resonance imaging samples and corresponding text stimuli, voxel matrices, and sentence representation matrices; After obtaining the training sample set, preprocessing is also included: obtaining the functional magnetic resonance image to be decoded, and using the FSL method to perform time alignment and head motion correction, and using the FLIRT method to perform spatial standardization on the corrected data to obtain a preprocessed image; decoding the preprocessed image through a decoding model based on elastic synaptic gates to obtain a first representation; the first representation is a vector representation of the stimulus received by the subject when the functional magnetic resonance image is collected; using different text representation models to encode the text stimulus into a numerical representation; Initializing a regression model according to the dimensions of the functional magnetic resonance image and the corresponding stimulus representation, and setting network parameters; Based on the initialized regression model, the neural encoder model and the neural decoder model are obtained according to the voxel matrix and the sentence representation matrix respectively.

Citation Information

Patent Citations

  • Generating natural language representations of mental content from functional brain images

    CN108135520A

  • Encoder-decoder framework pre-training method for neural machine translation

    CN111382580A

  • Generating natural language representations of mental content from functional brain images

    US20190380657A1

  • Semantic analysis method and apparatus, and storage medium

    US20200065389A1

  • System and method for deep mind analysis

    US20200222010A1