A biaffine-based biomedical document structuring method

Abstract

The present invention discloses a biomedical document structuring method based on double affine, comprising the steps of obtaining a biomedical document data set and performing data preprocessing on the data set; using the preprocessed data to construct a preliminary model for biomedical document classification through a double affine method; using the constructed preliminary model to complete parameter update of the preliminary model through adversarial training to construct a final document classification model; using the constructed final document classification model to complete classification and structuring processing of the biomedical document; the present invention directly calculates the classification score of each span through double affine, which can reduce the time overhead of the model; and the classification efficiency and classification accuracy of the present invention are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a biomedical document structuring method based on biaffine. Background Art

[0002] The large amount of literature produced by biomedical research every day plays a vital role in evidence-based medicine. However, the amount of biomedical literature is huge and is increasing rapidly every day. It is becoming increasingly difficult to find "useful" information in such a large number of biomedical literature. Therefore, an efficient and automated system is needed to structure the biomedical literature to simplify and accelerate the information retrieval process.

[0003] like Figure 1 The figure shows an example of structured processing of biomedical literature. The example classifies the sentences and divides the literature into four parts, including "Background", "Methods", "Results" and "Conclusions". The structured biomedical literature can help doctors find the required information accurately and effectively. Currently, there are two common methods for structured processing of documents, namely sequence labeling method and span-based method.

[0004] Sequence labeling methods classify individual sentences in a document individually, ignoring the semantic structure of the document, which limits their performance. Span-based methods first enumerate the spans consisting of adjacent sentences in the document, and then classify the spans. Span-based methods need to enumerate all possible spans. At the same time, the number of spans will increase exponentially with the increase in sequence length, resulting in a large amount of time spent on enumerating spans. In addition, span-based methods are usually insensitive to span boundaries, which often leads to sentences at the boundaries being misclassified.

[0005] In summary, the current structured methods cannot efficiently and automatically complete the structured processing of biomedical documents. The time required for document classification through modeling is too long, and the classification accuracy of documents is not high. The classification accuracy needs to be improved, and the time complexity needs to be further optimized. Summary of the Invention

[0006] The object of the present invention is to provide a biomedical document structuring method based on biaffine with improved classification efficiency and increased classification accuracy.

[0007] The biaffine-based biomedical document structuring method provided by the present invention comprises the following steps:

[0008] S1. Obtain a biomedical document dataset and perform data preprocessing on the dataset;

[0009] S2. Using the data preprocessed in step S1, construct a preliminary model for biomedical document classification using a biaffine method;

[0010] S3. Using the preliminary model constructed in step S2, update the parameters of the preliminary model through adversarial training to construct the final document classification model;

[0011] S4. Use the final document classification model constructed in step S3 to complete the classification and structured processing of biomedical documents.

[0012] The step S1 of obtaining a biomedical document dataset and performing data preprocessing on the dataset specifically includes:

[0013] The biomedical document dataset obtained is a public dataset;

[0014] Data preprocessing specifically includes: providing “sentence-level” annotation data in the dataset, {S1, S2, …, S i ,,S n} is a document containing n sentences. After sentence-level annotation, sentence-label pairs {(S1,y1),(S2,y2),…,(S i ,y i ),…,(S n ,y n )}, where n is the number of sentences contained in the document, S i is the i-th sentence in the document, y i is the category label of the i-th sentence;

[0015] Convert sentence-level annotations to “span-level annotations”, specifically: convert an annotated document containing n sentences {(S1,y1),(S2,y2),…,(S i ,y i ),…,(S n ,y n )} are merged into a "span". If a sentence has different labels from the sentences on the left and right, then this sentence forms a "span" of length 1. A triple (s, e, y) is used to represent a "span", where s represents the subscript of the first sentence in the span in the document, e represents the subscript of the last sentence in the span in the document, and y represents the label category of the span. After the above transformation, the annotated documents {(S1, y1), (S2, y2), ..., (S i ,y i),…,(S n ,y n )} is represented as m span triples {(s1,e1,y1),(s2,e2,y2),…,(s i ,e i ,y i ),…,(s m ,e m ,y m )}, where m is the number of spans obtained after processing, and m≤n; s i is the subscript of the first sentence in the i-th span in the original document; e i is the subscript of the last sentence in the i-th span in the original document; y i is the category label of the i-th span;

[0016] Step S2 uses the data pre-processed in step S1 to construct a preliminary model of biomedical documents using a biaffine method, specifically including:

[0017] (2-1) Encoding of individual statements

[0018] (1) Encoding of words:

[0019] Using the data obtained after step S1 and processing, for the i-th sentence S in a document in the dataset i , BioBERT is used to encode the word and the encoding process is performed using the following formula:

[0020] H i =BioBERT(S i )

[0021] ={h i1 ,h i2 ,…,h iw}

[0022] Among them, BioBERT is a pre-trained language model proposed for the biomedical field; It is sentence S i The encoded representation of all words in ; It's S i The encoding representation of the jth word in , represents the set of all real numbers, d is the number of dimensions of the word encoding vector, and w is the sentence S i The number of words contained in ;

[0023] (2) Encoding the sentence:

[0024] The obtained word encoding representation is mapped into key vectors and value vectors. The mapping process is expressed by the following formula:

[0025] K i =H i W k

[0026] V i =H i W v

[0027] in, To transform the i-th sentence S i The matrix formed by mapping the encoding of all words in H into key vectors; i For sentence S i The encoded representation of all words in ; The mapping matrix that maps word encoding representations to key vectors is the parameter to be learned in the classification model constructed in step S4; The mapping matrix that maps word encoding representations to value vectors is the parameter to be learned in the classification model constructed in step S4; To transform the i-th sentence S i The matrix formed by mapping the encoding of all words in the value vector; the matrix K i The jth row K in ij Represents sentence S i The key vector of the jth word in; the matrix V i The jth row in V ij Represents sentence S i The value vector of the jth word in ;

[0028] Calculate the weight of each word and obtain the encoded representation of the sentence by weighting the encoded representation of each word. The processing process is expressed by the following formula:

[0029]

[0030] r i =α i ·V i

[0031] in, For the i-th sentence S i The weight of each word in ; Softmax(·) represents the normalized exponential function; is the query vector, which is the parameter to be learned in the constructed classification model; For the i-th sentence S i The encoding representation of ;

[0032] (2-2) Encoding of context statements:

[0033] The context sentence encoding process is performed through the Bidirectional Long Short-Term Memory (BiLSTM) network, and the encoding representation of all sentences obtained in step (2-1) is defined as R={r1,r2,…,r i ,…,r n}, is the i-th sentence S in the document i The encoding representation of R is n, where n is the number of sentences in the document. By inputting R into the bidirectional long short-term memory network, the context encoding representation of the sentence is calculated. The calculation process is expressed by the following formula:

[0034] C=BiLSTM(R)

[0035] ={c1,c2,…,c n}

[0036] in, Encode the context representation of all sentences in the document; d r The number of dimensions used to encode the sentence context; is the context encoding representation of the i-th sentence;

[0037] (2-3) Constructing a preliminary model of biomedical documents through the biaffine method:

[0038] 1) Use the data after encoding processing in step (2-2) to perform mapping processing:

[0039] The sentence S i The context encoding representation is mapped into "start representation" and "end representation", and the mapping process is expressed by the following formula:

[0040] s i =c i W start +b start

[0041] e i =c i W end +b end

[0042] in, is the start representation of the i-th sentence in the document; is the context encoding representation of the i-th sentence in the document; The weights of the linear transformation that maps the context representation of the sentence to the “start representation”; The bias of the linear transformation that maps the context representation of the sentence to the "start representation"; is the end representation of the i-th sentence in the document; The weight of the linear transformation that maps the context representation of the sentence to the "end representation"; The bias of the linear transformation that maps the context representation of the sentence to the “end representation”; d s is the number of dimensions of the end representation and the beginning representation of the sentence; W start 、b start 、W end 、b end These are all parameters to be learned for the constructed document classification model;

[0043] 2) Use biaffine to calculate the classification score of each span:

[0044] The classification score for each span is calculated using the following formula:

[0045]

[0046] in, is the classification score of the span (i, j) from the beginning of the i-th sentence to the end of the j-th sentence in the document. It is a vector of length c, where each component represents the score of classifying the span into the corresponding category, and c represents the number of labels. is the classification score calculated in the forward direction, from left to right, i≤j; The classification score is calculated in the reverse direction, from right to left, with i>j; it is expressed by the following formula and The calculation process:

[0047]

[0048]

[0049] in, is the weight of the biaffine transformation, and is the parameter to be learned of the constructed document classification model; is the start representation of the i-th sentence in the document; is the end representation of the jth sentence in the document;

[0050] Step S3 uses the preliminary model constructed in step S2 to update the parameters of the preliminary model through adversarial training to construct the final document classification model, specifically including:

[0051] (3-1) Calculate the loss on a single span:

[0052] Assuming that the true label of span (i, j) is y, the following formula is used to express the calculation process of the classification model's loss for span (i, j):

[0053] p i,j =Softmax(score i,j )

[0054] p j,i =Softmax(score j,i )

[0055] ce(i,j)=-log(p i,j (y))

[0056] ce(j,i)=-log(p j,i (y))

[0057]

[0058] Among them, Softmax represents the normalized exponential function; p i,j is the classification score of span (i, j) by Softmax i,j Calculate the obtained classification probability; ce(i,j) is the cross entropy loss between the classification probability of span (i,j) and the true label; p j,i is the classification score of span (i, j) calculated by Softmax based on reverse bi-radiation j,i Calculate the obtained classification probability; ce(j,i) is the cross entropy loss of the classification probability and the true label of the span (i,j) calculated by reverse biradiation; loss(i,j) is the loss of the classification model in the span (i,j);

[0059] (3-2) Calculate the overall loss of the classification model:

[0060] The overall loss of the model is obtained by averaging the losses of all true spans, and the calculation formula is as follows:

[0061]

[0062] Where φ(S) is the set of all real spans in the document sentence sequence S; |φ(S)| is the number of spans included;

[0063] (3-3) Calculate the adversarial loss:

[0064] Calculate the overall loss of the model span The gradient on the BioBERT word embedding layer and the perturbation added to the word embedding layer based on the gradient are calculated as follows;

[0065]

[0066]

[0067] EMB=EMB+P adv

[0068] Among them, g is the overall loss of the model span Gradients for the word embedding layer; Indicates partial derivative of word embedding layer; P adv is the perturbation calculated based on the gradient; ε∈[0,1] is a hyperparameter that controls the size of the perturbation; |g| is the second norm of g, and EMB is the word embedding layer in BioBERT;

[0069] Based on the above calculation results, complete the above step S2 again, and recalculate the overall loss of the model according to steps (3-1) and (3-2) after completion. The calculated overall loss is defined as "adversarial loss" and recorded as loss adv ;

[0070] (3-4) Calculate the final loss of the model:

[0071] The final loss of the model is divided into two parts: the overall model loss and the adversarial loss. The calculation process is expressed by the following formula:

[0072] loss=loss span +λ*loss adv

[0073] Among them, λ is used to balance the overall loss of the model span and adversarial loss adv Hyperparameters of

[0074] (3-5) Update model parameters:

[0075] Restore the value of the word embedding layer EMB in BioBERT to the value before adding the perturbation in step (3-3), calculate the gradient of the model's final loss with respect to all learnable parameters in the model, and update the model parameters using the gradient descent method to construct the final document classification model;

[0076] Step S4 uses the final document classification model constructed in step S3 to complete the classification and structuring of biomedical documents, specifically including:

[0077] The dynamic programming algorithm is used to decode the labels in the model, thereby completing the classification and structuring of biomedical documents. Specifically, it includes:

[0078] Define the variable n as the length of the document sentence sequence or the number of sentences contained in the document; scorei,j is the calculated classification score of the span; alpha(i) is an intermediate variable in the dynamic programming algorithm, which is used to save the maximum classification score of all classification schemes of the first i sentences in the document calculated during the decoding process, and the value range of i is 0≤i≤n; path(i) is an intermediate variable in the dynamic programming algorithm, which is used to save the starting position of the span ending with the i-th sentence decoded during the decoding process, and the value range of i is 0≤i≤n; tag(i) is an intermediate variable in the dynamic programming algorithm, which is used to save the label category of the span ending with the i-th sentence decoded during the decoding process, and the value range of i is 0≤i≤n;

[0079] The algorithm operation process specifically includes:

[0080] (4-1) Let alpha(0) = 0, j = 1;

[0081] (4-2) Calculate the maximum classification score of all classification schemes for the first j sentences in the document and save it to alpha(j). The calculation formula is as follows:

[0082]

[0083] Where A(·) represents the classification score of the span calculated in (2-3) of step S2; A(i,j,k) represents the classification score of span (i,j) i,j The kth component of , or the score that classifies span (i, j) into the kth category;

[0084] (4-3) Record the solution with the largest classification score among all the classification solutions of the first j sentences and save it in path(j) and tag(j). The calculation formula is as follows:

[0085]

[0086] path(j)=i

[0087] tag(j)=k

[0088] Where i is the starting position of the span ending with j in the solution with the largest classification score; k is the label category of the span;

[0089] (4-4) Add 1 to the value of j;

[0090] (4-5) If j is less than n, repeat the above steps (4-2), (4-3), and (4-4) until j is greater than n;

[0091] (4-6) Let e ​​= n, where e is the end position of the current decoding span; n is the length of the document, or the number of sentences contained in the document; define X{·} as the set that stores the decoding results;

[0092] (4-7) Let s = path(e), t = tag(e), where s is the starting position of the current decoding span and t is the tag category of the current decoding span;

[0093] (4-8) Add (s, e, t) to X, and let e = s-1;

[0094] (4-9) If e is greater than 0, repeat the above steps (4-7) and (4-8) until e is less than or equal to 0, and then repeat the steps;

[0095] (4-10) Obtain the final classification result X of the sentence in the document = {(s v ,e v ,t v )|1≤v≤m}, where m is the number of spans included in the optimal decoding solution, s v is the starting position of the vth span, e v is the end position of the vth span, t v is the category label of the v-th span;

[0096] The biaffine-based biomedical document structuring method provided by the present invention includes directly classifying spans in a sequence; directly calculating the classification score of each span through biaffine without enumerating all possible spans in the sequence, thereby reducing the time overhead of the model; introducing a dynamic programming algorithm in the label decoding stage to improve the model's classification accuracy for sentences on span boundaries; the classification efficiency of the method of the present invention is improved and the classification accuracy is increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 This is an example of biomedical document sentence classification in the PubMed 20k RCT dataset used by the method of the present invention.

[0098] Figure 2 Schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0099] like Figure 2 The figure shows a schematic flow chart of the method of the present invention: The biaffine-based biomedical document structuring method provided by the present invention comprises the following steps:

[0100] S1. Obtain a biomedical document dataset and perform data preprocessing on the dataset; specifically, the biomedical document datasets obtained are public datasets, including two types: (1) PubMed 200k RCT dataset: The dataset includes abstracts of randomized controlled trials (RCT) obtained from the PubMed database, and the sentences in the abstracts are divided into five categories, including: background, objective, methods, results, and conclusions; (2) PubMed-PICO dataset: The dataset includes abstracts from the PubMed database, and the sentences in the abstracts are divided into seven categories, including: objective (Aim), participants (P), intervention (I), outcome (O), methods (M), results (R), and conclusion (C).

[0101] Data preprocessing specifically includes: providing “sentence-level” annotation data in the dataset, {S1, S2, …, S i ,…,S n} is a document containing n sentences. After sentence-level annotation, sentence-label pairs {(S1,y1),(S2,y2),…,(S i ,y i ),…,(S n ,y n )}, where n is the number of sentences contained in the document, S i is the i-th sentence in the document, y i is the category label of the i-th sentence;

[0102] Convert sentence-level annotations to “span-level annotations”, specifically: convert an annotated document containing n sentences {(S1,y1),(S2,y2),…,(S i ,y i ),…,(S n ,y n )} are merged into a "span". If a sentence has different labels from the sentences on the left and right, then this sentence forms a "span" of length 1. A triple (s, e, y) is used to represent a "span", where s represents the subscript of the first sentence in the span in the document, e represents the subscript of the last sentence in the span in the document, and y represents the label category of the span. After the above transformation, the annotated documents {(S1, y1), (S2, y2), ..., (Si ,y i ),…,(S n ,y n )} is represented as m span triples {(s1,e1,y1),(s2,e2,y2),…,(s i ,e i ,y i ),…,(s m ,e m ,y m )}, where m is the number of spans obtained after processing, and m≤n; s i is the subscript of the first sentence in the i-th span in the original document; e i is the subscript of the last sentence in the i-th span in the original document; y i is the category label of the i-th span;

[0103] S2. Using the data preprocessed in step S1, construct a preliminary model for biomedical document classification using a biaffine method; specifically, the following steps are involved:

[0104] (2-1) Encoding of individual statements

[0105] (1) Encoding of words:

[0106] Using the data obtained after step S1 and processing, for the i-th sentence S in a document in the dataset i , BioBERT is used to encode the word and the encoding process is performed using the following formula:

[0107] H i =BioBERT(S i )

[0108] ={h i1 ,h i2 ,…,h iw}

[0109] Among them, BioBERT is a pre-trained language model proposed for the biomedical field; It is sentence S i The encoded representation of all words in ; It's S i The encoding representation of the jth word in , represents the set of all real numbers, d is the number of dimensions of the word encoding vector, and w is the sentence S i The number of words contained in ;

[0110] (2) Encoding the sentence:

[0111] The obtained word encoding representation is mapped into key vectors and value vectors. The mapping process is expressed by the following formula:

[0112] K i =H i W k

[0113] V i =H i W v

[0114] in, To transform the i-th sentence S i The matrix formed by mapping the encoding of all words in H into key vectors; i For sentence S i The encoded representation of all words in ; The mapping matrix that maps word encoding representations to key vectors is the parameter to be learned in the classification model constructed in step S4; The mapping matrix that maps word encoding representations to value vectors is the parameter to be learned in the classification model constructed in step S4; To transform the i-th sentence S i The matrix formed by mapping the encoding of all words in the value vector; the matrix K i The jth row K in ij Represents sentence S i The key vector of the jth word in; the matrix V i The jth row in V ij Represents sentence S i The value vector of the jth word in ;

[0115] Calculate the weight of each word and obtain the encoded representation of the sentence by weighting the encoded representation of each word. The processing process is expressed by the following formula:

[0116]

[0117] r i =α i ·V i

[0118] in, For the i-th sentence S i The weight of each word in ; Softmax(·) represents the normalized exponential function; is the query vector, which is the parameter to be learned in the constructed classification model; For the i-th sentence S i The encoding representation of ;

[0119] (2-2) Encoding of context statements:

[0120] The context sentence encoding process is performed through the Bidirectional Long Short-Term Memory (BiLSTM) network, and the encoding representation of all sentences obtained in step (2-1) is defined as R={r1,r2,…,r i ,…,r n}, is the i-th sentence S in the document i The encoding representation of R is n, where n is the number of sentences in the document. By inputting R into the bidirectional long short-term memory network, the context encoding representation of the sentence is calculated. The calculation process is expressed by the following formula:

[0121] C=BiLSTM(R)

[0122] ={c1,c2,…,c n}

[0123] in, Encode the context representation of all sentences in the document; d r The number of dimensions used to encode the sentence context; is the context encoding representation of the i-th sentence;

[0124] (2-3) Constructing a preliminary model of biomedical documents through the biaffine method:

[0125] 1) Use the data after encoding processing in step (2-2) to perform mapping processing:

[0126] The sentence S i The context encoding representation is mapped into "start representation" and "end representation", and the mapping process is expressed by the following formula:

[0127] s i =c i W start +b start

[0128] e i =c i W end +b end

[0129] in, is the start representation of the i-th sentence in the document; is the context encoding representation of the i-th sentence in the document; The weights of the linear transformation that maps the context representation of the sentence to the “start representation”; The bias of the linear transformation that maps the context representation of the sentence to the "start representation"; is the end representation of the i-th sentence in the document; The weight of the linear transformation that maps the context representation of the sentence to the "end representation"; The bias of the linear transformation that maps the context representation of the sentence to the “end representation”; d s is the number of dimensions of the end representation and the beginning representation of the sentence; W start 、b start 、W end 、b end These are all parameters to be learned for the constructed document classification model;

[0130] 2) Use biaffine to calculate the classification score of each span:

[0131] The classification score for each span is calculated using the following formula:

[0132]

[0133] in, is the classification score of the span (i, j) from the beginning of the i-th sentence to the end of the j-th sentence in the document. It is a vector of length c, where each component represents the score of classifying the span into the corresponding category, and c represents the number of labels. is the classification score calculated in the forward direction, from left to right, i≤j; The classification score is calculated in the reverse direction, from right to left, with i>j; it is expressed by the following formula and The calculation process:

[0134]

[0135]

[0136] in, is the weight of the biaffine transformation, and is the parameter to be learned of the constructed document classification model; is the start representation of the i-th sentence in the document; is the end representation of the jth sentence in the document;

[0137] S3. Using the preliminary model constructed in step S2, update the parameters of the preliminary model through adversarial training to construct the final document classification model; specifically,

[0138] (3-1) Calculate the loss on a single span:

[0139] Assuming that the true label of span (i, j) is y, the following formula is used to express the calculation process of the classification model's loss for span (i, j):

[0140] p i,j =Softmax(score i,j )

[0141] p j,i =Softmax(score j,i )

[0142] ce(i,j)=-log(p i,j (y))

[0143] ce(j,i)=-log(p j,i (y))

[0144]

[0145] Among them, Softmax represents the normalized exponential function; p i,j is the classification score of span (i, j) by Softmax i,j Calculate the obtained classification probability; ce(i,j) is the cross entropy loss between the classification probability of span (i,j) and the true label; p j,i is the classification score of span (i, j) calculated by Softmax based on reverse bi-radiation j,i Calculate the obtained classification probability; ce(j,i) is the cross entropy loss of the classification probability and the true label of the span (i,j) calculated by reverse biradiation; loss(i,j) is the loss of the classification model in the span (i,j);

[0146] (3-2) Calculate the overall loss of the classification model:

[0147] The overall loss of the model is obtained by averaging the losses of all true spans, and the calculation formula is as follows:

[0148]

[0149] Where φ(S) is the set of all real spans in the document sentence sequence S; |φ(S)| is the number of spans included;

[0150] (3-3) Calculate the adversarial loss:

[0151] Calculate the overall loss of the model span The gradient on the BioBERT word embedding layer and the perturbation added to the word embedding layer based on the gradient are calculated as follows;

[0152]

[0153]

[0154] EMB=EMB+P adv

[0155] Among them, g is the overall loss of the model span Gradients for the word embedding layer; Indicates partial derivative of word embedding layer; P adv is the perturbation calculated based on the gradient; ε∈[0,1] is a hyperparameter that controls the size of the perturbation; |g| is the second norm of g, and EMB is the word embedding layer in BioBERT;

[0156] Based on the above calculation results, complete the above step S2 again, and recalculate the overall loss of the model according to steps (3-1) and (3-2) after completion. The calculated overall loss is defined as "adversarial loss" and recorded as loss adv ;

[0157] (3-4) Calculate the final loss of the model:

[0158] The final loss of the model is divided into two parts: the overall model loss and the adversarial loss. The calculation process is expressed by the following formula:

[0159] loss=loss span +λ*loss adv

[0160] Among them, λ is used to balance the overall loss of the model span and adversarial loss adv Hyperparameters of

[0161] (3-5) Update model parameters:

[0162] Restore the value of the word embedding layer EMB in BioBERT to the value before adding the perturbation in step (3-3), calculate the gradient of the model's final loss with respect to all learnable parameters in the model, and update the model parameters using the gradient descent method to construct the final document classification model;

[0163] S4. Using the final document classification model constructed in step S3, complete the classification and structural processing of biomedical documents; specifically, including:

[0164] The dynamic programming algorithm is used to decode the labels in the model, thereby completing the classification and structuring of biomedical documents. Specifically, it includes:

[0165] Define the variable n as the length of the document sentence sequence or the number of sentences contained in the document; scorei,j is the calculated classification score of the span; alpha(i) is an intermediate variable in the dynamic programming algorithm, which is used to save the maximum classification score of all classification schemes of the first i sentences in the document calculated during the decoding process, and the value range of i is 0≤i≤n; path(i) is an intermediate variable in the dynamic programming algorithm, which is used to save the starting position of the span ending with the i-th sentence decoded during the decoding process, and the value range of i is 0≤i≤n; tag(i) is an intermediate variable in the dynamic programming algorithm, which is used to save the label category of the span ending with the i-th sentence decoded during the decoding process, and the value range of i is 0≤i≤n;

[0166] The algorithm operation process specifically includes:

[0167] (4-1) Let alpha(0) = 0, j = 1;

[0168] (4-2) Calculate the maximum classification score of all classification schemes for the first j sentences in the document and save it to alpha(j). The calculation formula is as follows:

[0169]

[0170] Where A(·) represents the classification score of the span calculated in (2-3) of step S2; A(i,j,k) represents the classification score of span (i,j) i,j The kth component of , or the score that classifies span (i, j) into the kth category;

[0171] (4-3) Record the solution with the largest classification score among all the classification solutions of the first j sentences and save it in path(j) and tag(j). The calculation formula is as follows:

[0172]

[0173] path(j)=i

[0174] tag(j)=k

[0175] Where i is the starting position of the span ending with j in the solution with the largest classification score; k is the label category of the span;

[0176] (4-4) Add 1 to the value of j;

[0177] (4-5) If j is less than n, repeat the above steps (4-2), (4-3), and (4-4) until j is greater than n;

[0178] (4-6) Let e ​​= n, where e is the end position of the current decoding span; n is the length of the document, or the number of sentences contained in the document; define X{·} as the set that stores the decoding results;

[0179] (4-7) Let s = path(e), t = tag(e), where s is the starting position of the current decoding span and t is the tag category of the current decoding span;

[0180] (4-8) Add (s, e, t) to X, and let e = s-1;

[0181] (4-9) If e is greater than 0, repeat the above steps (4-7) and (4-8) until e is less than or equal to 0, and then repeat the steps;

[0182] (4-10) Obtain the final classification result X of the sentence in the document = {(s v ,e v ,t v )|1≤v≤m}, where m is the number of spans included in the optimal decoding solution, s v is the starting position of the vth span, e v is the end position of the vth span, t v is the category label of the v-th span;

[0183] In order to verify the effectiveness of the proposed method, experiments were conducted on two benchmark datasets, PubMed 20k RCT and PubMed-PICO. The experimental results are shown in Tables 1 and 2, respectively.

[0184] Table 1: Model results on the PubMed 20k RCT dataset

[0185]

[0186] Table 1 shows the performance of the method of the present invention and the comparative method on the PubMed 20k RCT dataset. The PubMed 20k RCT dataset has a total of five labels: Background, Objective, Methods, Results, and Conclusions. The span-level F1 and sentence-level F1 were calculated. For the span-level F1, the results for different categories were calculated separately. From the experimental results in Table 1, it can be seen that the method provided by the present invention outperforms the comparative method in both span-level F1 and sentence-level F1. The overall span F1 is 2.2 higher than the best result of the comparative method, and the overall sentence F1 is 0.7 higher than the best result of the comparative method.

[0187] Table 2: Model results on the PubMed-PICO dataset

[0188]

[0189] Table 2 shows the performance of the proposed method and the comparative method on the PubMed-PICO dataset, focusing on the performance of the model on P elements, I elements, and O elements. For each category, the precision, recall, and F1 value are calculated respectively.

[0190] The experimental results in Table 2 show that the method of the present invention is more than 1 percentage point higher than the best result of the comparison method in F1 of the three types of elements;

[0191] To further verify the role of each component in the method of the present invention, an ablation experiment was conducted on the PubMed 20k RCT dataset, and the experimental results are shown in Table 3; among them, "original" refers to the complete version of the method of the present invention; "removing the attention mechanism" refers to removing the attention mechanism in the method of the present invention and directly averaging the encoded representations of the words obtained from BioBERT as the encoded representation of the sentence; "removing reverse biaffine" refers to removing the process of calculating the span classification score by using the biaffine transformation in the reverse direction in the method of the present invention; "removing adversarial training" refers to removing the process of calculating the adversarial loss in the method of the present invention and using only the overall loss of the model. span To calculate the gradient and update the model parameters; "removing global optimization" means removing the dynamic programming algorithm used for label decoding in the method of the present invention and using a greedy strategy for label decoding. The decoding method is as follows:

[0192]

[0193] in, The labels predicted by the model;

[0194] Table 3 shows the experimental results of the ablation experiment on the PubMed 20k RCT dataset, as follows:

[0195] Table 3: Ablation experiments on the PubMed 20k RCT dataset

[0196] method Span F1 original 86.7 Remove dynamic programming algorithm 86.4 Remove the attention mechanism 86.1 Remove inverse biaffine 86.0 Remove adversarial training 85.8

[0197] From the experimental results in Table 3, it can be seen that removing any part of the method of the present invention will lead to a decrease in the effect of the method; thus, it can be determined that each step in the method of the present invention has an improving effect on the final effect.

Claims

1. A biomedical document structuring method based on biaffine, comprising the following steps: S1. Obtain a biomedical document dataset and perform data preprocessing on the dataset; specifically, the following steps are required: The biomedical document dataset obtained is a public dataset; Data preprocessing specifically includes: providing "sentence level" annotation data in the dataset, {S1, S2, ..., S i ,…,S n } is a document containing n sentences. After sentence-level annotation, sentence-label pairs {(S1,y1),(S2,y2),…,(S i ,y i ),…,(S n ,y n )}, where n is the number of sentences contained in the document, S i is the i-th sentence in the document, y i is the category label of the i-th sentence; Convert sentence-level annotations to "span-level" annotations, specifically including: converting an annotated document containing n sentences {(S1,y1),(S2,y2),…,(S i ,y i ),…,(S n ,y n )} are merged into a "span". If a sentence has different labels from the sentences on the left and right, then this sentence forms a "span" of length 1. A "span" is represented by a triple (s,e,y), where s represents the subscript of the first sentence in the span in the document, e represents the subscript of the last sentence in the span in the document, and y represents the label category of the span. After the above transformation, the annotated documents {(S1,y1),(S2,y2),…,(S i ,y i ),…,(S n ,y n )} is represented as m span triples {(s1,e1,y1),(s2,e2,y2),…,(s i ,e i ,y i ),…,(s m ,e m ,y m )}, where m is the number of spans obtained after processing, and m≤n; s i is the subscript of the first sentence in the i-th span in the original document; e i is the subscript of the last sentence in the i-th span in the original document; y i is the category label of the i-th span; S2. Using the data preprocessed in step S1, construct a preliminary model for biomedical document classification using a biaffine method; S3. Using the preliminary model constructed in step S2, update the parameters of the preliminary model through adversarial training to construct the final document classification model; S4. Use the final document classification model constructed in step S3 to complete the classification and structured processing of biomedical documents.

2. A biomedical document structuring method based on biaffine according to claim 1, characterized in that Step S2 uses the data pre-processed in step S1 to construct a preliminary model of biomedical documents using a biaffine method, specifically including: (2-1) Processing the words in the sentence to obtain the encoded representation of the words, and obtaining the encoded representation of the sentence through weighted processing; (2-2) Using the sentence encoding representation obtained in step (2-1), obtain the encoding representation of the context sentence through BiLSTM processing; (2-3) Using the context sentence encoding representation obtained in step (2-2), a preliminary model of biomedical documents is constructed using the biaffine method.

3. A biomedical document structuring method based on biaffine according to claim 2, characterized in that The encoding process of the individual sentences specifically includes: (1) Encoding of words: Using the data obtained after step S1 and processing, for the i-th sentence S in the dataset i , BioBERT is used to encode the word and the encoding process is performed using the following formula: H i =BioBERT(S i ) ={h i1 ,h i2 ,…,h iw } Among them, BioBERT is a pre-trained language model proposed for the biomedical field; It is sentence S i The encoded representation of all words in ; It's S i The encoding representation of the jth word in , represents the set of all real numbers, d is the number of dimensions of the word encoding vector, and w is the sentence S i The number of words contained in ; (2) Encoding the sentence: The obtained word encoding representation is mapped into key vectors and value vectors. The mapping process is expressed by the following formula: K i =H i ·W k V i =H i ·W v in, To transform the i-th sentence S i The matrix formed by mapping the encoding of all words in H into key vectors; i For sentence S i The encoded representation of all words in ; The mapping matrix that maps word encoding representations to key vectors is the parameter to be learned in the classification model constructed in step S4; The mapping matrix that maps word encoding representations to value vectors is the parameter to be learned in the classification model constructed in step S4; To transform the i-th sentence S i The matrix formed by mapping the encoding of all words in the value vector; the matrix K i The jth row K in ij Represents sentence S i The key vector of the jth word in; the matrix V i The jth row in V ij Represents sentence S i The value vector of the jth word in ; Calculate the weight of each word and obtain the encoded representation of the sentence by weighting the encoded representation of each word. The processing process is expressed by the following formula: r i ＝α i ·V i in, For the i-th sentence S i The weight of each word in ; Softmax(·) represents the normalized exponential function; is the query vector, which is the parameter to be learned in the constructed classification model; For the i-th sentence S i The encoding representation of .

4. A biomedical document structuring method based on biaffine according to claim 3, characterized in that The encoding process of the context sentence specifically includes: The context sentence encoding process is performed through the bidirectional long short-term memory network, and the encoding representation of all sentences obtained in step (2-1) is defined as is the i-th sentence S in the document i The encoding representation of R is n, where n is the number of sentences in the document. By inputting R into the bidirectional long short-term memory network, the context encoding representation of the sentence is calculated. The calculation process is expressed by the following formula: C=BiLSTM(R) ={c1,c2,…,c n } in, Encode the context representation of all sentences in the document; d r The number of dimensions used to encode the sentence context; is the context encoding representation of the i-th sentence.

5. A biomedical document structuring method based on biaffine according to claim 4, characterized in that The construction of a preliminary model of biomedical documents by the biaffine method specifically includes: 1) Use the data after encoding processing in step (2-2) to perform mapping processing: The sentence S i The context encoding representation is mapped into "start representation" and "end representation". The mapping process is expressed by the following formula: s i =c i ·W start +b start e i =c i ·W end +b end in, is the start representation of the i-th sentence in the document; is the context encoding representation of the i-th sentence in the document; is the weight of the linear transformation that maps the context representation of the sentence to the "start representation"; The bias of the linear transformation that maps the context representation of the sentence to the "start representation"; is the end representation of the i-th sentence in the document; is the weight of the linear transformation that maps the context representation of the sentence to the "end representation"; The bias of the linear transformation that maps the context representation of the sentence to the "end representation"; d s is the number of dimensions of the end representation and the beginning representation of the sentence; W start 、b start 、W end 、b end These are all parameters to be learned for the constructed document classification model; 2) Use biaffine to calculate the classification score of each span: The classification score for each span is calculated using the following formula: in, is the classification score of the span (i, j) from the beginning of the i-th sentence to the end of the j-th sentence in the document. It is a vector of length c, where each component represents the score of classifying the span into the corresponding category, and c represents the number of labels. is the classification score calculated in the forward direction, from left to right, i≤j; The classification score is calculated in the reverse direction, from right to left, with i>j; it is expressed by the following formula and The calculation process: in, is the weight of the biaffine transformation, and is the parameter to be learned of the constructed document classification model; is the start representation of the i-th sentence in the document; is the end representation of the jth sentence in the document.

6. A biomedical document structuring method based on biaffine according to claim 5, characterized in that Step S3 uses the preliminary model constructed in step S2 to complete parameter updates of the preliminary model through adversarial training to construct the final document classification model, specifically including: (3-1) Calculate the loss on a single span: Assuming that the true label of span (i, j) is y, the following formula is used to express the calculation process of the classification model's loss for span (i, j): p i,j =Softmax(score i,j ) p j,i =Softmax(score j,i ) ce(i,j)=-log(p i,j (y)) ce(j,i)=-log(p j,i (y)) Among them, Softmax represents the normalized exponential function; p i,j is the classification score of span (i, j) by Softmax i,j Calculate the obtained classification probability; ce(i,j) is the cross entropy loss between the classification probability of span (i,j) and the true label; p j,i is the classification score of span (i, j) calculated by Softmax based on reverse bi-radiation j,i Calculate the obtained classification probability; ce(j,i) is the cross entropy loss of the classification probability and the true label of the span (i,j) calculated by reverse biradiation; loss(i,j) is the loss of the classification model in the span (i,j); (3-2) Calculate the overall loss of the classification model: The overall loss of the model is obtained by averaging the losses of all true spans, and the calculation formula is as follows: Where φ(S) is the set of all real spans in the document sentence sequence S; φ(S)| is the number of spans included; (3-3) Calculate the adversarial loss: Calculate the overall loss of the model span The gradient on the BioBERT word embedding layer and the perturbation added to the word embedding layer based on the gradient are calculated as follows; EMB=EMB+P adv Among them, g is the overall loss of the model span Gradients for the word embedding layer; Indicates partial derivative of word embedding layer; P adv is the perturbation calculated based on the gradient; ε∈[0,1] is a hyperparameter that controls the size of the perturbation; |g| is the second norm of g, and EMB is the word embedding layer in BioBERT; Based on the above calculation results, complete the above step S2 again, and after completion, recalculate the overall loss of the model according to steps (3-1) and (3-2). The calculated overall loss is defined as "adversarial loss" and recorded as loss adv ; (3-4) Calculate the final loss of the model: The final loss of the model is divided into two parts: the overall model loss and the adversarial loss. The calculation process is expressed by the following formula: loss=loss span +λ*loss adv Among them, λ is used to balance the overall loss of the model span and adversarial loss adv Hyperparameters of (3-5) Update model parameters: Restore the value of the word embedding layer EMB in BioBERT to the value before adding the perturbation in step (3-3), calculate the gradient of the model's final loss for all learnable parameters in the model, and update the model parameters through the gradient descent method to build the final document classification model.

7. A biomedical document structuring method based on biaffine according to claim 6, characterized in that Step S4 uses the final document classification model constructed in step S3 to complete the classification and structuring of biomedical documents, specifically including: The dynamic programming algorithm is used to decode the labels in the model, thereby completing the classification and structuring of biomedical documents. Specifically, it includes: Define the variable n as the length of the document sentence sequence or the number of sentences contained in the document; score i,j is the calculated classification score of the span; alpha(i) is an intermediate variable in the dynamic programming algorithm, which is used to save the maximum classification score of all classification schemes of the first i sentences in the document calculated during the decoding process, and the value range of i is 0≤i≤n; path(i) is an intermediate variable in the dynamic programming algorithm, which is used to save the starting position of the span ending with the i-th sentence decoded during the decoding process, and the value range of i is 0≤i≤n; tag(i) is an intermediate variable in the dynamic programming algorithm, which is used to save the label category of the span ending with the i-th sentence decoded during the decoding process, and the value range of i is 0≤i≤n; The algorithm operation process specifically includes: (4-1) Let alpha(0) = 0, j = 1; (4-2) Calculate the maximum classification score of all classification schemes for the first j sentences in the document and save it to alpha(j). The calculation formula is as follows: Where A(·) represents the classification score of the span calculated in (2-3) of step S2; A(i,j,k) represents the classification score of span (i,j) i,j The kth component of , or the score that classifies span (i, j) into the kth category; (4-3) Record the solution with the largest classification score among all the classification solutions of the first j sentences and save it in path(j) and tag(j). The calculation formula is as follows: path(j)=i tag(j)=k Where i is the starting position of the span ending with j in the solution with the largest classification score; k is the label category of the span; (4-4) Add 1 to the value of j; (4-5) If j is less than n, repeat the above steps (4-2), (4-3), and (4-4) until j is greater than n; (4-6) Let e ​​= n, where e is the end position of the current decoding span; n is the length of the document, or the number of sentences contained in the document; define X{·} as the set that stores the decoding results; (4-7) Let s = path(e), t = tag(e), where s is the starting position of the current decoding span and t is the tag category of the current decoding span; (4-8) Add (s, e, t) to X, and let e = s-1; (4-9) If e is greater than 0, repeat the above steps (4-7) and (4-8) until e is less than or equal to 0, and then repeat the steps; (4-10) Obtain the final classification result X of the sentence in the document = {(s v ,e v ,t v )|1≤v≤m}, where m is the number of spans included in the optimal decoding solution, s v is the starting position of the vth span, e v is the end position of the vth span, t v is the category label of the v-th span.

Images