Neural image event attribute joint extraction method based on deep learning

Abstract

The invention discloses a neural image event attribute joint extraction method based on deep learning. The invention provides a lightweight neural image event extraction joint model by fusing an advanced adversarial learning technology and a deep learning algorithm, and the model is composed of a bidirectional long-short term memory network (BiLSTM), a conditional random field (CRF), a Sigmoid activation function and a free adversarial training (FreeAT) strategy. In view of the characteristics that the complexity of a nerve image event argument is relatively high and the event structure is relatively simple, the model adopts a BiLSTM + CRF + Sigmoid architecture, and the design concept of'key recognition of the argument and simplification of role extraction 'is realized. The architecture enables the model to rapidly achieve convergence on a small-scale training set by simplifying model parameters. According to the method, an adversarial training mechanism based on FreeAT is introduced, and the overfitting problem is relieved. According to the method, the accuracy is improved, and the generalization ability of the model under the resource limited condition is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of neuroimaging text mining. Background Art

[0002] The openness and FAIR (Findable, Accessible, Interoperable, and Reusable) sharing of neuroimaging data[1] has been widely recognized and has always been a concern in the field of computational neuroscience. From fMRIDC[2], OpenfMRI[3] to OpenNeuro[4], a series of public neuroimaging databases have been built and have played an important role in brain and intelligence research. Taking the INDI database as an example, as of March 22, 2017, Google Scholar search results showed that 913 papers explicitly pointed out the use of INDI data, with a total of 20,697 citations[5]. In addition to sharing original experimental data, research information sharing has also become an important way to share neuroimaging data[6][7]. BrainMap (http: / / www.brainmap.org / ) collects published information on brain activation areas from neuroimaging articles and supports coordinate-based meta-analysis[7]. Neurosynth (https: / / www.neurosynth.org / ) provides a dedicated tool that automatically extracts brain activation area coordinates and research topic information from published neuroimaging articles and shares them through a unified platform [8]. NeuroVault (http: / / neurovault.org) collects non-threshold statistical graphs and related information from different computational neurosciences to support graph-based meta-analysis [9]. These neuroimaging research information sharing efforts enrich the content and methods of neuroimaging data sharing and effectively support brain and intelligence research in computational neuroscience. For example, according to statistics, as of September 2024, the number of papers based on the Neurosynth platform has reached 14,371. Related research involves brain mechanisms of cognitive states

[10] , brain region functional analysis

[11] , large-scale brain structure-function mapping

[12] , etc.

[0003] Neuroimaging text mining can identify important information in neuroimaging texts, especially in neuroimaging articles, and provides a practical method for automatically extracting the provenance information of the publication source. In recent years, related research has mainly focused on identifying neuroimaging entities, hierarchical terminology systems, and interest spans. Since the extracted information is mainly some words or phrases, lacking rich relationships, especially non-category relationships, these studies can only obtain some scattered key points in the research process and cannot provide a vivid description of the process. Therefore, it is necessary to develop a new neuroimaging text mining technology for the construction of neuroimaging evidence for research sharing. Summary of the invention

[0004] The present invention proposes a method for joint extraction of neuroimaging event attributes based on deep learning. The present invention adopts an architecture that combines long short-term memory network (BiLSTM), conditional random field (CRF) and Sigmoid to achieve the design concept of accurately identifying key arguments and simplifying the role extraction process. This architecture significantly improves the convergence speed on a small-scale training set by optimizing model parameters, thereby improving training efficiency. In addition, this study innovatively introduced an adversarial training strategy based on Free-AT, which effectively overcomes the overfitting phenomenon that is prone to occur when lightweight models learn on small training sets. Through this innovative design, the present invention not only significantly improves the accuracy of neuroimaging event extraction, but also significantly enhances the generalization ability of the model in a resource-constrained environment, bringing major technological progress to the field of neuroimaging data analysis.

[0005] The creativity of the present invention is mainly reflected in:

[0006] (1) The present invention obtains a deep adversarial learning framework for a joint extraction model of neuroimaging events and attributes based on adversarial training, which aims to extract specific events from neuroimaging data. The architecture combines BiLSTM, CRF and Sigmoid to achieve the design concept of focusing on identifying arguments and simplifying role extraction. This architecture enables the model to converge quickly on a small-scale training set by streamlining model parameters, thereby improving training efficiency.

[0007] (2) This paper applies the adversarial learning strategy based on Free Adversarial Training (Free-AT) to the event joint extraction task, significantly improving the event extraction accuracy in a small sample learning environment. Through this optimization, the proposed model shows higher practical value in the field of computational neuroscience, especially in the absence of high-quality annotated data. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a method for joint extraction of neural image event attributes based on deep learning of the present invention. DETAILED DESCRIPTION

[0009] The specific implementation modes of the present invention are further described in detail with reference to the accompanying drawings and examples.

[0010] Step 1: Before implementing text vectorization, data set collection and corpus annotation are required. The specific process is as follows:

[0011] Dataset collection: The data of the corpus comes from the journal PLOS ONE. We searched the literature by screening the keywords in the article abstracts, including "fMRI", "functional magnetic resonance imaging" or "functional MRI", and "functional MRI", and limited the search time range to July 18, 2014 to July 18, 2019. In the end, we collected 677 relevant papers. Using these papers, we used the rule matching method to extract event mentions. The specific processing flow is as follows:

[0012] Corpus annotation: The collected data sets are annotated into 9 arguments, namely brain region (BRI), cognitive function (COG), research participants (STP), stimulus response pattern (SEN), task (TAS), data acquisition equipment (ACQ), acquisition object (AOB), analysis tool or method (TOL) and analysis result (RLT). Annotated in the form of BIO. During the annotation process, the annotation of trigger words and parameters adopts the "B / I / O" category abbreviation mode. Among them, B represents the beginning of the entity. I represents the middle or end, and O represents others, which is used to mark irrelevant characters.

[0013] After the text is annotated, the word vectorization is realized. Through the Glove word vector model, the words and word vectors are combined to achieve high-quality vector representation of the input text. The goal of word vector learning is to capture the semantic attributes and language relationships between words and encode this information into a new vector space. Therefore, the word2vec model is used to obtain the vector representation of words. The text vectorization layer encodes the sentence into a vector that can express the semantics of the text as the input of the model. That is, for a given sentence:

[0014] s=[w1,w2,…,w n ] (1)

[0015] where w i represents the word vector, n represents the number of words, and the word vector w i The encoding method is:

[0016] w i =[w x ,wc ] (2)

[0017] where w x Represents word vector, w c Represents a word vector.

[0018] Step 2: Realize the prediction of event elements by extracting features from the vector constructed in step 1; a bidirectional long short-term memory network (BiLSTM) and a conditional random field (CRF) (BiLSTM-CRF) model are used to predict potential event elements, including arguments and trigger words.

[0019] First, the word vector w formed in step 1 i Input to BiLSTM to model the context information of the sentence, so as to more accurately identify and classify the key elements in the event. The process is concretely as follows:

[0020]

[0021] where w i represents the word vector, and represents the output of the forward hidden layer and the backward hidden layer, h i Represents the output of the hidden layer corresponding to the word.

[0022] Secondly, the context information of the sentence is used to accurately predict the category and boundary of the event element. The continuous tags of each event element composed of the BIO (Begin-Inside-Outside) annotation strategy are used to calculate and predict each word w using CRF. i A score is assigned to each specific event element, namely:

[0023] score (e) (h i )=W e f(U e h i ),i∈[1,n] (4)

[0024] The superscript e indicates the prediction type of the event element, n indicates the number of words in the sentence, and f(*) indicates the activation function. p is the number of event element label type sets Y, l is the width of LSTM, and d is the size of the LSTM hidden layer. The CRF score of the linear chain of score is:

[0025]

[0026] Among them, yi ∈Y is w i Tags, Represents a given label y i In the word group w i The prediction score of Represents y e i to y e i+1 Finally, the probability of a given token sequence on all possible token sequences of the sentence input is defined as:

[0027]

[0028] where y i ～e Represents w i All possible labels Pr represent the probability values ​​of a given tag sequence in the sentence over all possible tag sequences input in the sentence sentence.

[0029] During training, the cross entropy loss is minimized:

[0030]

[0031] Where W e ,U e is a set of matrix parameters, Pr(y i |w i ; θ) represents the calculation of w i and the true label y i The cross loss.

[0032] Step 3: Identify the role of the attribute. Use the sigmoid function to identify the relationship between a trigger word and two or more arguments. In the role identification layer, we use the sigmoid function to identify the role of the attribute. Relation extraction is a one-to-one mapping between entities, while the identification of argument roles is a one-to-many relationship, because a trigger word usually corresponds to multiple arguments in an event. In view of this, this paper constructs the role identification task as a multi-head selection problem

[68] , that is, extracting the association between a trigger word and two or more arguments.

[0033] The state of the last hidden layer output of LSTM is input into the sigmoid function, and the role is marked r k The word w i and w j The prediction scores between are:

[0034] score (r) (h i ,h j ,rk )=W r f(U r h i +V r h j ) (8)

[0035] The superscript r represents role recognition, f(*) represents the activation function, and w j represents the word vector, h i Represents the corresponding w i The output state of the LSTM output layer, l is the width of LSTM, d is the size of the LSTM hidden layer, and word w j The head is w i , and the role mark between them is r k The probability of

[0036] Pr(head=w j ,tag=r k |w i ,θ)=σ(score (r) (h i ,h j ,r k )) (9)

[0037] where θ is h i ,h j ,r k Parameter set, σ(*) is the sigmoid function, head represents w j Is it w i Head, tag means w i and w j For r k The relationship between the role labels, where Pr(head=w j ,tag=r k |w i ,θ) represents the j As w i The head of the character between them is marked as r k The probability value of .

[0038] During training, the cross entropy loss is minimized:

[0039]

[0040] where y i,j is the true value of the vector, r k is the role mark, m (m<n) is w i During decoding, the most likely head and role are selected using threshold-based prediction.

[0041] Finally, for the joint event extraction task, the final target is calculated as L NER +L RAR This study minimizes the objective function in the joint model training process.

[0042] Step 4: In order to improve the generalization and robustness of the model, a continuously enhanced small disturbance is input to the joint model through an adversarial learning mechanism. After interference adversarial training, the output distribution is consistent with the original distribution. Therefore, the Free-AT perturbation strategy is adopted for training, and K consecutive iterations are performed on each sample to find the optimal interference method. The value range of K is set according to the batch of model samples. In the process of calculating the interference, the gradient information of the previous step is reused. By dividing the total number of iterations by K, the calculation speed problem caused by multiple internal iterations is effectively alleviated. The definition of perturbation is:

[0043]

[0044] where r adv Represents the disturbance, each iteration, the model parameter W r ,U,V r will change, is the current model parameter W r ,U r ,V r Copy of, is the perturbation parameter, a is the parameter factor, which is set according to the size of the data set, and D is the word vector w i The dimension of is the number of internal iterations represents the cross loss function of step 3, Represents the derivation of the cross loss function.

[0045] References

[0046] [1]Abrams,MB,Bjaalie,JG,Das,S.,Egan,GF,and Martone,ME(2021).Astandards organization for open and fair neuroscience:the internationalneuroinformatics coordinating facility.Neuroinformatics 1–12.doi:10.31219 / 6osf.io / 3rt9b.

[0047] [2]Van Horn,J.D.,Grethe,J.S.,Kostelec,P.,Woodward,J.B.,Aslam,J.A.,Rus,D.,et al.(2001).The functional magnetic resonance imaging data center(fMRIDC):the challenges and rewards of large-scaledatabasing of neuroimagingstudies.Philos.Trans.R.Soc.Lond.B Biol.Sci.1412,1323–1339.doi:10.1098 / rstb.2001.0916.

[0048] [3]Poldrack,R.A.,and Gorgolewski,K.J.(2015).Openfmri:open sharing oftask fMRI data.NeuroImage 144,259–261.doi:10.1016 / j.neuroimage.2015.05.073.

[0049] [4]Gorgolewski,K.J.,Esteban,O.,Schaefer,G.,Wandell,B.A.,and Poldrack,R.A.(2017).OpenNeuro-a free online platform for sharing and analysis ofneuroimaging data.Poster at the23rd Annual Meeting of the Organization forHuman Brain Mapping(OHBM)2017,Vancouver,BC,1677.

[0050] [5]Milham,M.P.,Cameron,C.R.,Son,J.J.,Fleischmann,M.,Clucas,J.,Xu,H.,et al.(2018).Assessment of the impact of shared brain imaging data on thescientific literature.Nat.Commun.9:2818.doi:10.1038 / s41467-018-04976-1.

[0051] [6]Poline,J.B.,Breeze,J.L.,Ghosh,S.,Gorgolewski,K.,Halchenko,Y.O.,Hanke,M.,et al.(2012).Data sharing in neuroimagingresearch.Front.Neuroinform.6:9.doi:10.3389 / fninf.2012.00009.

[0052] [7]Laird,A.R.,Lancaster,J.L.,and Fox,P.T.(2005).BrainMap:the socialevolution of a human brain mapping database.Neuroinformatics 3,65–78.doi:10.1385 / ni:3:1:065.

[0053] [8]Yarkoni,T.,Poldrack,R.A.,Nichols,T.E.,Van Essen,D.C.,and Wager,T.D.(2011).Large-scale automated synthesis of human functional neuroimagingdata.Nat.Methods 8,665–670.doi:10.1038 / nmeth.1635.

[0054] [9]Gorgolewski,K.J.,Varoquaux,G.,Rivera,G.,Schwartz,Y.,Sochat,V.V.,Ghosh,S.S.,et al.(2016).NeuroVault.org:a repository for sharing unthresholdedstatistical maps,parcellations,and atlases of the human brain.Neuroimage 124,1242–1244.doi:10.1016 / j.neuroimage.2015.04.016.

[0055]

[10] Alcalá-López,D.,Smallwood,J.,Jefferies,E.,Van Overwalle,F.,Vogeley,K.,Mars,R.B.,et al.(2017).Computing the social brain connectome crosssystems and states.Cereb.Cortex 28,2207–2232.doi:10.1093 / cercor / .

[0056]

[11] Genon,S.,Reid,A.,Li,H.,Fan,L.,Müller,V.I.,Cieslik,E.C.,et al.(2018).The heterogeneity of the left dorsal premotor cortex evidenced bymultimodal connectivity-based parcellation and functionalcharacterization.Neuroimage 170,400–411.doi:10.1016 / j.neuroimage.2017.02.034.

[0057]

[12] Sheng,Y.,Chen,J.,He,X.,Xu,Z.,and Lin,S.(2020).Atopic learningpipeline for curating brain cognitive researches.IEEE Access 8,191758–191774.doi:10.1109 / ACCESS.2020.3032173.

Claims

1. A method for joint extraction of neuroimaging event attributes based on deep learning, characterized in that The following steps are involved: Step 1: Implement text vectorization of word vectors. Through the GloVe word vector model, words and word vectors are combined to achieve vector representation of the input text. The goal of word vector learning is to capture the semantic attributes and linguistic relationships between words and encode this information into a new vector space. Therefore, the word2vec model is used to obtain the vector representation of words. The text vectorization layer encodes sentences into vectors that can express the semantics of the text as the input of the model. That is, for a given sentence s=[w1,w2,…,w n ],w i represents the word vector, n represents the number of words in the sentence, and the word vector w i The encoding method is w i =[w x ,w c ], where w x Represents word vector, w c Represents a word vector. Step 2: Realize the prediction of event elements by extracting features from the vector constructed in step 1; a bidirectional long short-term memory network (BiLSTM) and a conditional random field (CRF) (BiLSTM-CRF) model are used to predict potential event elements, including event arguments and event trigger words. Step 3: Identify the role of the attribute. Use the sigmoid function to identify the relationship between a trigger word and two or more arguments. Input the state of the last hidden layer output of the LSTM into the sigmoid function, and the role label r k The word w i and w j The prediction scores between are: score (r) (h i ,h j ,r k )=W r f(U r h i +V r h j ) The superscript r represents role recognition, f(*) represents the activation function, and w j represents the word vector, h i Represents the corresponding w i The output state of the LSTM output layer, h j Represents the corresponding w j The output state of the LSTM output layer, l is the width of LSTM, d is the size of the LSTM hidden layer, and word w i The head is w j , and the role mark between them is r k The probability is: Pr(head=w j ,tag=r k |w i ,θ)=σ(score (r) (h i ,h j ,r k )) where θ is W r ,U r ,V r Matrix parameter set, σ(*) is the sigmoid function, head represents w j Is it w i Head, tag means w i and w j Between r k The relationship between the role labels, where Pr(head=w j ,tag=r k |w i ,θ) represents w j As w i The head of the two characters is marked as r k The probability value of . During training, the cross entropy loss is minimized: where y i,j is the true value of the vector, r k is the role mark, m,n (m<n) is w i The number of related head roles, relation represents the real role label r of the model k , Pr(head=y i,j ,relation=r k |w i ) represents the calculation of w i The real character is marked as r k With w i The actual head tag y i,j During decoding, the most likely head and character are selected using threshold-based predictions. Finally, for the joint event extraction task, the final target is calculated as L NER +L RAR Minimize the objective function during joint model training. Step 4: In order to improve the generalization and robustness of the model, interference is added to the joint model through the adversarial learning mechanism. After interference adversarial training, the output distribution is consistent with the original distribution. Therefore, the Free-AT perturbation strategy is adopted for training, and K consecutive iterations are performed on each sample to find the optimal interference method. The value range of K is set according to the batch of model samples. In the process of calculating interference, the gradient information of the previous step is reused. By dividing the total number of iterations by K, the calculation speed problem caused by multiple internal iterations is effectively alleviated. The definition of perturbation is: where r adv Represents the disturbance, each iteration, the model parameter W r ,U r ,V r will change, is the current model parameter W r ,U r ,V r of the copy, is the perturbation parameter, a is the parameter factor, which is set according to the size of the data set, and D is the word vector w i The dimension of is the number of internal iterations, represents the cross loss function of step 3, ▽ w Represents the derivation of the cross loss function.

2. The method according to claim 1: The process of predicting the event element described in step 2 is as follows: First, the word vector w formed in step 1 i Input to BiLSTM to model the context information of the sentence, so as to more accurately identify and classify the key elements in the event. The process is concretely as follows: where w i represents the word vector, and represents the output of the forward hidden layer and the backward hidden layer, h i Represents the output of the hidden layer words. Secondly, the context information of the sentence is used to accurately predict the category and boundary of the event element. The continuous tags of each event element composed of the BIO (Begin-Inside-Outside) annotation strategy are used to calculate and predict each word w using CRF. i A score is assigned to each specific event element, namely: score (e) (h i )=W e f(U e h i ),i∈[1,n] The superscript e indicates the prediction type of the event element, n indicates the number of words in the sentence, and f(*) indicates the activation function. p is the number of event element label type sets Y, l is the width of LSTM, and d is the size of the LSTM hidden layer. The CRF score of the linear chain of score is: in, y e i ∈Y is w i Tags, Represents a given label y e i In the word group w i The prediction score of Represents y e i to y e i+1 Finally, the probability of a given token sequence on all possible token sequences of the sentence input is defined as: where y ～e i Represents w i All possible labels. Pr represents the probability value of a given tag sequence in the sentence on all possible tag sequences input in the sentence sentence. During training, the cross entropy loss is minimized: where θ is W e ,U e Matrix parameter set, Pr(y i |w i ; θ) represents the calculation of w i and the true label y i The cross loss.

Images