A method for constructing a transferable language model based on electronic medical record text

Abstract

This paper proposes a transferable language model based on electronic medical record text. This model uses a medical knowledge base to separate electronic medical record text into two main components: templates and terminology. Pattern Attention is first used to independently model the natural language templates, and then KG Cross Attention is used to integrate the corresponding medical terminology. This allows the model to model the medical record text using the medical knowledge base, thereby achieving cross-specialty information extraction. To make the model more adaptable to electronic medical record text, the present invention designed three pre-training tasks. This pre-training method can significantly reduce the difficulty of model transfer within similar specialties.

Description

Technical Field

[0001] The present invention relates to a language model, and in particular to a method for constructing a transferable language model for electronic medical records. Background Art

[0002] The text of electronic medical records contains the patient's symptoms, test results, and the doctor's description of the diagnosis and treatment process based on basic data such as symptoms and physical and chemical indicators. This important information is stored in unstructured information and cannot be understood and processed by computers.

[0003] Due to the confidentiality of medical data and the specialized nature of medical terminology, researchers need to minimize the amount of annotation required. However, existing models can typically only be used for one domain at a time, and switching domains requires re-annotating the data, which is time-consuming and labor-intensive. Furthermore, medical record text itself has certain specificities and is typically constructed using a "template + terminology" approach. For example, the text for a gastric cancer surgical procedure: "Exploration: liver, gallbladder, pancreas, spleen, small and large intestines are normal, and the lesion is located on the anterior wall of the lesser curvature of the stomach" can be decomposed into a template: "Exploration [body organ] [abnormality], lesion located on [body organ]" combined with the terms: "liver," "gallbladder," "pancreas," "spleen," "large and small intestines," "anterior wall of the lesser curvature of the stomach," and "no abnormality." Furthermore, within similar specialties, the templates are nearly identical, with only the terminology being replaced. For example, the text for a colorectal cancer surgical procedure: "Exploration: liver, gallbladder, pancreas, spleen, no obvious abnormalities, extensive adhesions of the small intestine in the abdominal cavity." Establishing an information extraction model that separates templates from specialized terminology can significantly reduce the difficulty of model migration within similar specialties. Summary of the Invention

[0004] To address the shortcomings of existing language models, this paper provides a method for constructing a transferable language model based on electronic medical record text. This method separates electronic medical record text into two main components: templates and terminology. This allows the model to leverage the medical knowledge base to separate the medical record text for modeling, thereby enabling cross-specialty information extraction. Furthermore, the use of an unsupervised pre-training approach reduces the need for manually annotated data, making the model more easily transferable across electronic medical record texts from different specialties.

[0005] The present invention adopts the following technical solutions:

[0006] 1. The main idea of ​​the method for constructing a transferable language model based on electronic medical record text is as follows:

[0007] S1: Template term separator, using the medical knowledge base as a dictionary, matches the corresponding term parts from the medical text S, replaces the matched terms, and generates a text template and a set of professional terms.

[0008] S2: Template term encoder, which inputs text template and professional terminology and outputs the fused vector representation.

[0009] S3: Pre-training downstream task layer. This paper uses three methods for pre-training: template word filling, term restoration, and negation consideration. The output of the pre-training stage is the loss of the downstream task, and the output of the fine-tuning stage is the fused vector representation E l+1 .

[0010] 2. Specifically, in step S1, the template term separator method includes the following steps:

[0011] S11: Input is medical record text S Doc , field name S key , the input and output are field values ​​S value , different processing methods are determined according to the field type. Specifically, tasks can be divided into the following three categories:

[0012] S12: Using the dictionary tree matching algorithm, with the medical knowledge base KG as the dictionary, match the corresponding term part from the medical text S, and then replace it to generate the text template S pattern and professional terminology set S KG .

[0013] 3. Specifically in step S2, the template term encoder method includes the following steps:

[0014] S21: Template term encoder uses PattenAttention and KG CrossAttention to capture the contextual semantic information of the template and the association information between the template and the knowledge base.

[0015] S22: Perform a nonlinear transformation using the FNN layer to obtain the fused vector representation E1+1. The specific formula is as follows, with the number of layers l∈{x|1≤x≤12}.

[0016] SelfAttention(X)=ln(mult_head h＝12 (X, X, X, MASK) + X)

[0017] KG CrossAttention(X,K)=ln(mult_head h＝12 (X, K, K, MASK) + X)

[0018] E l+1 =FFN(KG CrossAttention(SelfAttention(E l ), K, MASK))

[0019] E l=layer_normal(add([x i ] s-max ,[p i ] s_max ))

[0020]

[0021] Among them, E l is the initial vector, which comes from the text X after word vector mapping [x i ] s_max And the corresponding position code [p i ] s_max K is the professional term set S KG Each word in MASK is a mask matrix that controls the attention range of each word. It is used in KG Cross Attention to allow the template to focus only on the vector representation of the corresponding replacement position.

[0022] 4. Specifically in step S3, the template term decoder method includes the following steps:

[0023] S31: Template word filling task, used to learn context-sensitive representations for each character in the template. Specifically, 15% of normal characters are randomly selected from the template, replaced with [MASK] with an 80% probability, 10% are replaced with other characters in the vocabulary, and 10% remain unchanged. These characters are then restored through the downstream task layer.

[0024] S32: Term restoration task, used to learn the connection between the slots in the template and the filled terms. Specifically, 10% of the terms are randomly selected from the term set, replaced with [MASK] with an 80% probability, 10% are replaced with other terms from the knowledge base, and 10% remain unchanged. These terms are then restored through the downstream task layer. Furthermore, medical records often contain some structured information, which can be inferred from the descriptive text. This paper performs the same replacement operation on the structured terms in this information, forcing the model to learn the inference relationship between the descriptive text and the structured text.

[0025] S33: Semantic tendency judgment task, the specific approach is: use predefined rules to match paragraphs containing negative meanings from the corpus as negative examples, randomly select some other paragraphs as positive examples, where the maximum sampling number of positive examples is limited to the number of negative examples, and then predict the tendency of the paragraph through the downstream task layer.

[0026] S34: Use three tasks for joint pre-training. The output of the pre-training stage is the loss of the downstream task, and the output of the fine-tuning stage is the fused vector representation E l+1 . BRIEF DESCRIPTION OF THE DRAWINGS

[0027] After reading the detailed description of the present invention with reference to the accompanying drawings, readers will have a clearer understanding of the various aspects of the present invention.

[0028] Figure 1 A schematic diagram of a transferable language model based on electronic medical record text according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] To provide a more detailed and complete understanding of the technical content disclosed in this application, reference is made to the accompanying drawings and the following various specific embodiments of the present invention. In the accompanying drawings, the same reference numerals represent the same or similar components. However, those skilled in the art will appreciate that the embodiments provided below are not intended to limit the scope of the present invention. Furthermore, the accompanying drawings are for illustrative purposes only and are not drawn to scale.

[0030] The specific embodiments of various aspects of the present invention are further described in detail below with reference to the accompanying drawings.

[0031] Figure 1 A schematic diagram of a transferable language model based on electronic medical record text according to an embodiment of the present invention is shown.

[0032] S1: Template term separator, using the medical knowledge base as a dictionary, matches the corresponding term parts from the medical text S, replaces the matched terms, and generates a text template and a set of professional terms.

[0033] S2: Template term encoder, which inputs text template and professional terminology and outputs the fused vector representation.

[0034] S3: Pre-training the downstream task layer. The present invention uses three methods: template word filling, term restoration, and negation consideration for pre-training.

[0035] 2. Specifically, in step S1, constructing a transferable language model based on electronic medical record text includes the following steps:

[0036] S11: Build a transferable language model based on electronic medical record text. The model structure is mainly divided into a template term separator, a template term encoder, and a pre-trained downstream task layer.

[0037] S12: The template term separator uses a dictionary tree matching algorithm, takes the medical knowledge base KG as a dictionary, matches the corresponding term part from the medical text S, and then replaces it to generate a text template S pattern and professional terminology set S KG .

[0038] For example, the text of the gastric cancer surgery process: "Exploration: liver, gallbladder, pancreas, spleen, no abnormalities found in the large and small intestines, the lesion is located in the anterior wall of the lesser curvature of the stomach" can be decomposed into a template: "Exploration [body organ] [abnormality], the lesion is located in [body organ]" and a combination of the terms: "liver", "gallbladder", "pancreas", "spleen", "large and small intestines", "anterior wall of the lesser curvature of the stomach" and "no abnormalities found".

[0039] 3. Specifically, in step S2, constructing the template term encoder includes the following steps:

[0040] S21: Template term encoder uses PattenAttention and KG CrossAttention to capture the contextual semantic information of the template and the association information between the template and the knowledge base.

[0041] S22: Use the FNN layer to perform a nonlinear transformation to obtain the fused vector representation E1+1. The specific formula is as follows, with the number of layers l∈{x|1≤x≤12}.

[0042] SelfAttention(X)=ln(mult_head h＝12 (X, X, X, MASK) + X)

[0043] KG CrossAttention(X,K)=ln(mult_head h＝12 (X, K, K, MASK) + X)

[0044] E l+1 =FFN(KG CrossAttention(SelfAttention(E l ), K, MASK))

[0045] E l =layer_normal(add([x i ] s-max ,[p i ] s_max ))

[0046]

[0047] Among them, E l is the initial vector, which comes from the text X after word vector mapping [x i ] s_max And the corresponding position code [p i ] s_max K is the professional term set S KG Each word in MASK is a mask matrix that controls the attention range of each word. It is used in KG CrossAttention to allow the template to focus only on the vector representation of the corresponding replacement position.

[0048] 4. Specifically, in step S3, the model pre-training includes the following steps:

[0049] S31: Template word filling task, used to learn context-sensitive representations of each character in the template. Specifically, 15% of normal characters are randomly selected from the template, replaced with [MASK] with 80% probability, 10% are replaced with other characters in the vocabulary, and 10% remain unchanged. The character is then restored through the downstream task layer. The specific calculation formula is as follows:

[0050] H mlm =ln(relu(W mlm D l+1 +b mlm ))

[0051]

[0052] This task enables the model to generate corresponding representations for each word in the input text based on the context in an unsupervised manner.

[0053] S32: Term restoration task, used to learn the connection between the slots in the template and the terms filled in. Specifically, 10% of the terms are randomly selected from the term set, replaced with [MASK] with an 80% probability, 10% are replaced with other terms from the knowledge base, and 10% remain unchanged. These terms are then restored through the downstream task layer. Furthermore, medical records often contain some structured information, which can be inferred from the descriptive text. This paper performs the same replacement operation on the structured terms in this information, forcing the model to learn the inference relationship between the descriptive text and the structured text.

[0054] The overall calculation formula for the term reduction task is as follows:

[0055] H tmlm =ln(relu(W tmlm D l+1 +b tmlm ))

[0056]

[0057] This task targets the fact that electronic medical record texts contain a large number of terms, enabling the model to learn based on the characteristics of the terms and enhancing the model's performance when dealing with electronic medical record texts.

[0058] S33: Semantic tendency judgment task. The specific approach is: Use predefined rules to match paragraphs containing negative meanings from the corpus as negative examples, randomly select some other paragraphs as positive examples, where the maximum number of positive examples sampled is limited to the number of negative examples, and then predict the tendency of the paragraph through the downstream task layer. The specific calculation formula is as follows:

[0059]

[0060] P(paraispositive|Hp)=sigmod(W neg H p'’ +b neg )

[0061] loss neg =-∑ p∈positive para log(P(para is positive|H p ))-∑ p∈negative para log(1-P(parais positive|H p ))

[0062] Since the attention mechanism tends to capture the connection between words, it tends to ignore the overall contextual information. The requirements for negative context in medicine are much higher than in general fields. Therefore, this task is used to improve the model's understanding of negative semantics.

[0063] S34: Use three tasks to jointly perform pre-training. The final total pre-training loss is the weighted sum of the three tasks. The formula is as follows:

[0064] Loss = loss mlm +losst mlm +loss neg

[0065] This paper proposes a transferable language model based on electronic medical record text. This model uses a medical knowledge base to separate electronic medical record text into two major components: templates and terminology. It first uses PatternAttention to independently model the natural language templates, and then uses KG Cross Attention to integrate the corresponding medical terminology. This allows the model to model the medical record text using the medical knowledge base, thereby completing cross-specialty information extraction. To make the model more adaptable to electronic medical record text, the paper designs three pre-training tasks. This pre-training method can significantly reduce the difficulty of model transfer within similar specialties.

[0066] The above are merely examples of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention without creative work are included in the scope of protection of the present invention.

Claims

1. A method for constructing a transferable language model based on electronic medical record text, characterized in that: The steps include: S1: Template term separator, which uses the medical knowledge base as a dictionary, matches the corresponding term parts from the medical text S, replaces the matched terms, and generates a text template and a set of professional terms; S2: Template term encoder, which encodes the text template and professional terminology to obtain the vector representation E after the fusion of text and terminology l+1 ; The template term encoder includes the following steps: S21: The template term encoder uses PattenAttention and KG CrossAttention to capture the contextual semantic information of the template and establish the association information between the text template and the knowledge base. The specific formula is as follows; SelfAttention(X)=ln(mult_head h＝12 (X,X,X,MASK)+X) KGCrossAttention(X,K)=ln(mult_head h＝12 (X,K,K,MASK)+X) S22: Use the FNN layer to perform a nonlinear transformation on the information captured in S21 to obtain the fused vector representation E l +1 , the specific formula is as follows: l∈{x|1≤x≤12}, l is the number of layers E l+1 =FFN(KGCrossAttention(SelfAttention(E l ),K,MASK)) E l =layer_normal(add([x i ] s-max ,[p i ] s_max )) Among them, E l is the initial vector, which comes from the text X after word vector mapping [x i ] s_max And the corresponding position code [p i ] s_max ; K is the professional term set S KG Each word in MASK is a mask matrix that controls the attention range of each word and is used in KG Cross Attention to allow the template to focus only on the vector representation of the corresponding replacement position. S3: Pre-training downstream task layer, which will use three tasks: template word filling, term restoration, and negation consideration for pre-training. The output is the loss of downstream tasks for model training and optimization.

2. The method for constructing a transferable language model based on electronic medical record text according to claim 1, characterized in that: In step S1, the template term separator includes the following steps: S11: Input is medical record text S Doc , field name S key , the output is the field value S value , determine different processing methods according to the field type; S12: Using the dictionary tree matching algorithm, with the medical knowledge base KG as the dictionary, match the corresponding term part from the medical text S, and then replace it to generate the text template S pattern and professional terminology set S KG .

Images