Electronic medical record symptom feature extraction method and system based on multi-modal enhancement
By constructing a multimodal enhanced electronic medical record symptom feature extraction method, and using BERT and BILSTM sequence modeling layers combined with CRF decoding layers, the accuracy problem of symptom recognition in Chinese electronic medical records was solved, enabling early diagnosis of drug-induced liver injury and acute kidney injury.
Patent Information
- Application Number
- CN202410989701.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-23
AI Technical Summary
Existing symptom recognition methods for Chinese electronic medical records cannot accurately identify symptoms in the diagnosis of specific diseases such as drug-induced liver injury. They suffer from problems such as polysemy of words, homonymy of multiple words, and complex sentence structures, resulting in poor diagnostic performance.
A symptom feature extraction method based on multimodal enhancement of electronic medical records is adopted. By constructing a symptom feature pre-identification model, combining BERT input layer, BILSTM sequence modeling layer and CRF decoding layer, symptom feature identification is performed using multimodal feature data, including data annotation, model training and fine-tuning. Masked Language Model and Next Sentence Prediction are used for training, and symptom features are extracted by combining multimodal feature data.
It enables rapid and accurate identification of symptom features in electronic medical records, transforms unstructured data into structured data, improves the accuracy of symptom feature extraction, and is suitable for the early diagnosis of drug-induced liver injury and acute kidney injury.
Smart Images

Figure CN118940760A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of entity naming and recognition, and in particular to a method and system for extracting symptom features of electronic medical records based on multimodal enhancement. Background Art
[0002] Named entity recognition refers to the task of extracting specific types of entities from complex structured, unstructured and semi-structured data. It provides basic support for natural language processing tasks such as text structuring, relationship extraction, text summarization and machine translation, and is a hot issue in natural language processing. Using named entity recognition technology, medical entities and structured medical records can be efficiently obtained, providing support for medical knowledge graphs, health intelligent question and answer, intelligent auxiliary diagnosis and treatment, knowledge subscription and other services, and promoting the development of smart medicine.
[0003] In the medical field, Electronic Medical Records (EMRs) are important information resources that contain a large amount of medical expertise. However, due to the huge amount of information in EMRs and the complex format, most of which exists in the form of unstructured data, it is difficult for doctors to find key information in a short time, which affects the diagnosis and treatment of drug-induced liver injury. In recent years, natural language processing technology has made significant progress in the medical field, such as named entity recognition (NER) and relationship extraction. Among them, named entity recognition aims to identify and classify entities with specific meanings from unstructured text, and plays an important role in a variety of natural language processing tasks.
[0004] At present, named entity recognition (NER) for Chinese electronic medical records is to identify medical information entities with independent or specific meanings from massive electronic medical record data, such as six types of entities including drugs, anatomical parts, diagnosis, examination, inspection, and surgery, and to perform sequence annotation and standardization. However, in the symptom diagnosis of a specific disease (such as DILI), the current popular entity annotation method has the defect of not being able to intuitively reflect specific symptoms. Chinese text also has the characteristics of polysemy, multiple words with one meaning, complex sentence structure, and multiple omissions. The part of speech and meaning of the same word at different positions in the sentence may be completely different. The existing Chinese NER method is not very applicable to the problem of symptom recognition in Chinese electronic medical records. Summary of the invention
[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for extracting symptom features of electronic medical records based on multimodal enhancement.
[0006] To achieve the above purpose, the technical solution provided by the present invention is:
[0007] A method for extracting symptom features of electronic medical records based on multimodal enhancement, comprising:
[0008] Annotate the symptom characteristics closely related to the target disease in the electronic medical record to obtain annotated data;
[0009] Constructing a symptom feature pre-recognition model, and training the constructed symptom feature pre-recognition model based on the labeled data;
[0010] Fine-tune the trained symptom feature pre-recognition model, and use the fine-tuned symptom feature pre-recognition model to perform symptom feature recognition to obtain the symptom feature pre-recognition result
[0011] Collect multimodal feature data;
[0012] Combine the collected multimodal feature data with the symptom feature pre-recognition results In combination, the NLP method based on multimodal enhancement is used to extract the symptom feature recognition results.
[0013] Furthermore, the symptom characteristics closely related to the target disease in the electronic medical record are annotated, including:
[0014] For the patient's electronic medical records, regular expressions were used to search for keywords of symptom characteristics when obtaining data, and the data was cleaned and desensitized to obtain data 1;
[0015] Perform BIO annotation on data 1 to obtain annotated data;
[0016] BIO annotations include "BX", "IX", and "O"; "BX" means that the fragment where the element is located is of type X and the element is at the beginning of this fragment, "IX" means that the fragment where the element is located is of type X and the element is in the middle of this fragment, and "O" means that it does not belong to any type.
[0017] Furthermore, the constructed symptom feature pre-recognition model includes a BERT input layer, a BILSTM sequence modeling layer, and a CRF decoding layer;
[0018] in,
[0019] The BERT input layer consists of the Embedding layer and the Transformer Encoder; the Embedding layer consists of Positional Embedding, Sentence Embedding, and Token Embedding; Positional Embedding enables the model to utilize the order of the sequence by adding the position information of the sequence marker; the formula is as follows:
[0020]
[0021]
[0022] In formulas (1) and (2), pos refers to the position of a word in a sentence. The value range of this position is from 0 to the position value of the last token in the sentence. i refers to the dimension number of the word vector, and d refers to the representation dimension of the word.
[0023] In the Transformer Encoder, dot product is used for similarity calculation to learn the word dependency within the sentence and capture the internal structure of the sentence; the formula is as follows:
[0024]
[0025] In formula (3), Q, K, V are the word vector matrices obtained by performing three linear transformations on the word vector, and dk is the dimension of the input vector;
[0026] In the BILSTM sequence modeling layer, the word vector X obtained by the BERT input layer is received, and the context information is bidirectionally extracted through the forget gate, input gate, and output gate:
[0027] The forget gate selectively forgets some information:
[0028] f t =σ(W f ·[h t-1 , X]+b f ) (4)
[0029] The input gate adds new information:
[0030] i t =σ(W i ·[h t-1 ,X]+b i )
[0031]
[0032] Update current status:
[0033]
[0034] The output gate outputs the part that determines the output:
[0035] O t =σ(W o ·[h t-1 , X]+b o )
[0036] ht =O t *tanh(C t ) (7)
[0037] In the above formula, h t-1 is the hidden state of the text at time t-1; σ is the Sigmoid activation function; b is the bias vector; tanh is the hyperbolic tangent function; W is the model parameter matrix, · represents the dot product of two vectors, and * represents the cross product of two vectors;
[0038] In the CRF decoding layer, the result h obtained by the BILSTM sequence modeling layer t Decoding; this includes:
[0039] Constraints are used to ensure that the final extraction result is valid. The combination of all labels constitutes the sum of the paths, and the path with the highest score needs to be found, that is, the constraints between the labels. The score of each path is calculated by formula (8), where The transfer score matrix is a parameter of the model and is updated with each training iteration; is the emission score matrix, which is the output of the BILSTM sequence modeling layer h t ; X is the observation sequence, whose value is {x1,x2,...,x n}, is the input of the symptom feature pre-recognition model; extract the sequence label y = h t ={y1,y2,...,y n}, which is the output h of the BILSTM sequence modeling layer t ; The label path corresponding to the maximum value of the path score score(X,y) is the final output, and various named entities are output according to the labels; the output format is:
[0040]
[0041] Furthermore, the BERT input layer trains the symptom feature pre-recognition model by designing Masked Language Model and Next SentencePrediction;
[0042] Masked Language Model: Randomly mask or replace any word or phrase in a sentence, and then the model extracts it based on the context;
[0043] Next Sentence Prediction: Put two sentences together, and the model extracts whether the two sentences are adjacent to each other based on the original text.
[0044] Furthermore, when fine-tuning the trained symptom feature pre-recognition model, the trained symptom feature pre-recognition model is used to extract the test set in the labeled data and return the extraction result; the model evaluation indicators use precision P, recall R and F1 value, where F1 value is a balance between precision and recall, and the effect of symptom feature recognition is based on F1 value;
[0045] The symptom feature pre-identification model extracts symptom features from electronic medical records. The extraction results directly output by the model are in BIO format. After data processing of the extraction results, the symptom feature extraction results based on medical records are obtained, and the results are converted into structured data. Each symptom feature is binary data, with 0 representing "no" or "false" and 1 representing "yes" or "true".
[0046] Furthermore, the collected multimodal feature data includes the patient's laboratory data, medication data, and medical history data.
[0047] Furthermore, the collected multimodal feature data and the symptom feature pre-identification results are combined Combined with the NLP method based on multimodal enhancement, the symptom feature recognition results are extracted, including:
[0048] The binary symptom feature pre-recognition results are combined with the obtained multimodal feature data to obtain the multimodal feature data and X, the annotation result is y, and the machine learning algorithm is used to mirror the combined data set for training and extraction, and the feature recognition result S of the extracted symptom n is n for:
[0049]
[0050] In formula (9), f is a classifier; F i is the multimodal feature data, representing the vector of patient i.
[0051] To achieve the above-mentioned purpose, the present invention further provides an electronic medical record symptom feature extraction system based on multimodal enhancement, which is used to implement the above-mentioned electronic medical record symptom feature extraction method based on multimodal enhancement, including a data annotation module, a symptom feature pre-recognition model construction and training module, a model fine-tuning module, a symptom feature pre-recognition module, a multimodal feature data collection module, and a symptom feature recognition module;
[0052] in,
[0053] The data annotation module is used to annotate the symptom characteristics closely related to the target disease in the electronic medical record to obtain annotated data;
[0054] When the data annotation module performs data annotation, it searches for keywords of symptom characteristics using regular expressions in the patient's electronic medical records when acquiring data, and cleans and desensitizes the data to obtain data 1;
[0055] Perform BIO annotation on data 1 to obtain annotated data;
[0056] BIO annotations include "BX", "IX", and "O"; "BX" means that the element belongs to the X type and the element is at the beginning of this segment, "IX" means that the element belongs to the X type and the element is in the middle of this segment, and "O" means that it does not belong to any type;
[0057] The symptom feature pre-recognition model construction and training module is used to construct a symptom feature pre-recognition model and train the constructed symptom feature pre-recognition model based on the labeled data;
[0058] The constructed symptom feature pre-recognition model includes BERT input layer, BILSTM sequence modeling layer, and CRF decoding layer;
[0059] in,
[0060] The BERT input layer consists of the Embedding layer and the Transformer Encoder; the Embedding layer consists of Positional Embedding, Sentence Embedding, and Token Embedding; Positional Embedding enables the model to utilize the order of the sequence by adding the position information of the sequence marker; the formula is as follows:
[0061]
[0062]
[0063] In formulas (1) and (2), pos refers to the position of a word in a sentence. The value range of this position is from 0 to the position value of the last token in the sentence. i refers to the dimension number of the word vector, and d refers to the representation dimension of the word.
[0064] In the Transformer Encoder, dot product is used for similarity calculation to learn the word dependency within the sentence and capture the internal structure of the sentence; the formula is as follows:
[0065]
[0066] In formula (3), Q, K, and V are the word vector matrices obtained by performing three linear transformations on the word vectors, and d k is the dimension of the input vector;
[0067] In the BILSTM sequence modeling layer, the word vector X obtained by the BERT input layer is received, and the context information is bidirectionally extracted through the forget gate, input gate, and output gate:
[0068] The forget gate selectively forgets some information:
[0069] f t =σ(W f ·[h t-1 ,X]+b f ) (4)
[0070] The input gate adds new information:
[0071] i t =σ(W i ·[h t-1 , X]+b i )
[0072]
[0073] Update current status:
[0074]
[0075] The output gate outputs the part that determines the output:
[0076] O t =σ(W o ·[h t-1 ,X]+b o )
[0077] h t =O t *tanh(C t ) (7)
[0078] In the above formula, h t-1 is the hidden state of the text at time t-1; σ is the Sigmoid activation function; b is the bias vector; tanh is the hyperbolic tangent function; W is the model parameter matrix, · represents the dot product of two vectors, and * represents the cross product of two vectors;
[0079] In the CRF decoding layer, the result h obtained by the BILSTM sequence modeling layer t Decoding; this includes:
[0080] Constraints are used to ensure that the final extraction result is valid. The combination of all labels constitutes the sum of the paths, and the path with the highest score needs to be found, that is, the constraints between the labels. The score of each path is calculated by formula (8), where The transfer score matrix is a parameter of the model and is updated with each training iteration; is the emission score matrix, which is the output of the BILSTM sequence modeling layer h t ; X is the observation sequence, whose value is {x1,x2,...,x n}, is the input of the symptom feature pre-recognition model; extract the sequence label y = h t ={y1,y2,...,y n}, which is the output h of the BILSTM sequence modeling layer t ; The label path corresponding to the maximum value of the path score score(X,y) is the final output, and various named entities are output according to the labels; the output format is:
[0081]
[0082] The BERT input layer trains the symptom feature pre-recognition model by designing the Masked Language Model and Next Sentence Prediction;
[0083] Masked Language Mode l: Randomly mask or replace any word or phrase in a sentence, and then the model extracts it based on the context;
[0084] Next Sentence Prediction: Put two sentences together, and the model extracts whether the two sentences are adjacent to each other based on the original text;
[0085] The model fine-tuning module is used to fine-tune the trained symptom feature pre-recognition model;
[0086] When fine-tuning the trained symptom feature pre-recognition model, the trained symptom feature pre-recognition model is used to extract the test set in the labeled data and return the extraction result; the model evaluation indicators use precision rate P, recall rate R and F1 value, among which F1 value is a balance between precision rate and recall rate, and the effect of symptom feature recognition is based on F1 value;
[0087] The symptom feature pre-identification model extracts symptom features from electronic medical records. The extraction results directly output by the model are in BIO format. After data processing of the extraction results, the symptom feature extraction results based on medical records are obtained and converted into structured data. Each symptom feature is binary data, with 0 representing "no" or "false" and 1 representing "yes" or "true".
[0088] The symptom feature pre-recognition module is used to perform symptom feature recognition using the fine-tuned symptom feature pre-recognition model to obtain a symptom feature pre-recognition result.
[0089] The multimodal feature data collection module is used to collect multimodal feature data;
[0090] The collected multimodal feature data include patients’ laboratory data, medication data, and medical history data;
[0091] The symptom feature recognition module is used to collect the multimodal feature data and the symptom feature pre-recognition results. Combined with the NLP method based on multimodal enhancement, the symptom feature recognition results are extracted, including:
[0092] The binary symptom feature pre-recognition results are combined with the obtained multimodal feature data to obtain the multimodal feature data and X, the annotation result is y, and the machine learning algorithm is used to mirror the combined data set for training and extraction, and the feature recognition result S of the extracted symptom n is n for:
[0093]
[0094] In formula (9), f is a classifier; F i is the multimodal feature data, representing the vector of patient i.
[0095] Compared with the prior art, the principles and advantages of this technical solution are as follows:
[0096] 1. Faced with electronic medical record data with huge amounts of information and complex formats, most of which exist in the form of unstructured data, the symptom feature pre-recognition model adopted by this technical solution can quickly and accurately identify the entity of each character, and on this basis, convert the unstructured data into medical structured data (each symptom is binary data).
[0097] 2. Collect multimodal feature data and symptom feature pre-identification results to more specifically determine the patient's physical condition In combination, the symptom feature recognition results are extracted using the NLP method based on multimodal enhancement, which can greatly improve the accuracy of electronic medical record symptom feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0098] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the services required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0099] Figure 1A connection block diagram of an electronic medical record symptom feature extraction system based on multimodal enhancement according to the present invention;
[0100] Figure 2 This is a principle flow chart of a method for extracting symptom features from electronic medical records based on multimodal enhancement of the present invention (data annotation and model building parts are omitted);
[0101] Figure 3 This is the evaluation result diagram of the drug-induced liver injury test set;
[0102] Figure 4 This is the evaluation result of the acute kidney injury test set. DETAILED DESCRIPTION
[0103] The present invention will be further described below in conjunction with specific embodiments:
[0104] like Figure 1 As shown, the electronic medical record symptom feature extraction system based on multimodal enhancement described in this embodiment includes a data annotation module, a symptom feature pre-recognition model construction and training module, a model fine-tuning module, a symptom feature pre-recognition module, a multimodal feature data collection module, and a symptom feature recognition module;
[0105] in,
[0106] The data annotation module is used to annotate the symptom features closely related to the target disease in the electronic medical record to obtain the annotated data; the symptom feature pre-recognition model construction and training module is used to construct the symptom feature pre-recognition model and train the constructed symptom feature pre-recognition model based on the annotated data; the model fine-tuning module is used to fine-tune the trained symptom feature pre-recognition model; the symptom feature pre-recognition module is used to perform symptom feature recognition using the fine-tuned symptom feature pre-recognition model to obtain the symptom feature pre-recognition results The multimodal feature data collection module is used to collect multimodal feature data; the symptom feature recognition module is used to combine the collected multimodal feature data with the symptom feature pre-recognition results. In combination, the NLP method based on multimodal enhancement is used to extract the symptom feature recognition results.
[0107] like Figure 2 As shown, the working principle of this embodiment includes the following steps:
[0108] S1. Annotate the symptom characteristics closely related to the target disease in the electronic medical record through the data annotation module to obtain annotated data;
[0109] For the patient's electronic medical records, regular expressions were used to search for keywords of symptom characteristics when obtaining data, and the data was cleaned and desensitized to obtain data 1;
[0110] Perform BIO annotation on data 1 to obtain annotated data;
[0111] BIO annotations include "BX", "IX", and "O"; "BX" means that the fragment where the element is located is of type X and the element is at the beginning of this fragment, "IX" means that the fragment where the element is located is of type X and the element is in the middle of this fragment, and "O" means that it does not belong to any type.
[0112] Some annotation samples are shown in Table a:
[0113] Suffer from Those V. Lack of force , B-Weakness I-Weakness I-Weakness I-Weakness I-Weakness O
[0114] Table a Annotation example
[0115] S2, constructing a symptom feature pre-recognition model through a symptom feature pre-recognition model construction and training module, and training the constructed symptom feature pre-recognition model based on the labeled data;
[0116] The constructed symptom feature pre-recognition model includes BERT input layer, BILSTM sequence modeling layer, and CRF decoding layer;
[0117] in,
[0118] The BERT input layer consists of the Embedding layer (embedding) and the Encoder of the Transformer (model); the Embedding layer consists of Positional Embedding (position embedding, which is an encoding method in the BERT model, used to assign different embedding vectors to each position in the input sequence), Sentence Embedding (sentence embedding, an important output in the BERT model, obtained by summarizing or aggregating the embeddings of all tokens in the input sequence) and Token Embedding (token embedding, which is the representation of each input token in BERT; in the BERT model, the input text is first divided into tokens, and then each token is embedded as a vector representation; token embedding is one of the inputs of the BERT model, which contains the semantic information of each token); Positional Embedding enables the model to utilize the order of the sequence by adding the position information of the sequence token; the formula is as follows:
[0119]
[0120]
[0121] In formulas (1) and (2), pos refers to the position of a word in a sentence. The value range of this position is from 0 to the position value of the last token in the sentence. i refers to the dimension number of the word vector, and d refers to the representation dimension of the word.
[0122] In the Transformer Encoder, the main idea is to use the self-attention mechanism instead of the traditional RNN, and use the dot product for similarity calculation to learn the word dependency within the sentence and capture the internal structure of the sentence; the formula is as follows:
[0123]
[0124] In formula (3), Q, K, and V are the word vector matrices obtained by performing three linear transformations on the word vectors, and d k is the dimension of the input vector;
[0125] The output of the BERT model contains two main word vectors: word-level word vectors and sentence-level word vectors.
[0126] Specifically, for each word in the input sequence, the BERT model generates its corresponding word vector, which contains the semantic information of the word in the context. The word vector at the sentence level is obtained by encoding special tags, which comprehensively represents the semantic information of the entire sequence.
[0127] The BERT model obtains the word vector X through the pre-training stage and the fine-tuning stage.
[0128] Pretraining: In the pretraining phase, the BERT model is trained using large-scale unlabeled text data. This phase includes two main tasks: MLM and NSP. Through the training of these two tasks, the BERT model will learn context-dependent representations at the word level and sentence level.
[0129] Masked Language Model (MLM): Randomly mask or replace any word or phrase in a sentence, and then the model extracts it based on the context; in the MLM task, some words or tokens in the input text sequence will be randomly masked (replaced with special masking tokens). Then, the model needs to predict these masked words or tokens. In this way, the BERT model can learn context-related word representations and has a certain semantic understanding ability.
[0130] Next Sentence Prediction (NSP): Put two sentences together, and the model extracts whether the two sentences are adjacent to each other based on the original text; in the NSP task, the BERT model needs to determine whether the two input sentences are adjacent in the order of the original text. The model predicts whether they are adjacent through the embedded representation of the input sentences. This task helps the BERT model learn sentence-level semantic relationships and context understanding capabilities.
[0131] Fine-tuning: In the fine-tuning stage, the BERT model is further trained using labeled task-specific data. For example, named entity recognition data. In the fine-tuning stage, the parameters of the BERT model are adjusted according to the specific task through supervised training on the task data to adapt to the needs of the task.
[0132] After the pre-training and fine-tuning phases are completed, the parameters of the BERT model are fixed, and the model can be used to encode new input sequences to obtain word-level word vectors X. These word vectors X capture the semantic information of the word in context and can be used for downstream natural language processing tasks such as text classification, named entity recognition, sentence similarity, etc.
[0133] In the BILSTM sequence modeling layer, the word vector X obtained by the BERT input layer is received, and the context information is bidirectionally extracted through the forget gate, input gate, and output gate:
[0134] The forget gate selectively forgets some information:
[0135] f t =σ(W f ·[h t-1 ,X]+b f ) (4)
[0136] The input gate adds new information:
[0137] i t =σ(W i ·[h t-1 , X]+b i )
[0138]
[0139] Update current status:
[0140]
[0141] The output gate outputs the part that determines the output:
[0142] O t =σ(W o·[h t-1 ,X]+b o )
[0143]
[0144] In the above formula, h t-1 is the hidden state of the text at time t-1; σ is the Sigmoid activation function; b is the bias vector; tanh is the hyperbolic tangent function; W is the model parameter matrix, · represents the dot product of two vectors, and * represents the cross product of two vectors;
[0145] In the CRF decoding layer, the result h obtained by the BILSTM sequence modeling layer t Decoding; this includes:
[0146] Constraints are used to ensure that the final extraction result is valid. The combination of all labels constitutes the sum of the paths, and the path with the highest score needs to be found, that is, the constraints between the labels. The score of each path is calculated by formula (8), where The transfer score matrix is a parameter of the model and is updated with each training iteration; is the emission score matrix, which is the output of the BILSTM sequence modeling layer h t ; X is the observation sequence, whose value is {x1,x2,...,x n}, is the input of the symptom feature pre-recognition model; extract the sequence label y = h t ={y1,y2,...,y n}, which is the output h of the BILSTM sequence modeling layer t ; The label path corresponding to the maximum value of the path score score(X,y) is the final output, and various named entities are output according to the labels; the output format is:
[0147]
[0148] S3, fine-tuning the trained symptom feature pre-recognition model through the model fine-tuning module, and using the fine-tuned symptom feature pre-recognition model through the symptom feature pre-recognition module to perform symptom feature recognition, and obtaining the symptom feature pre-recognition result
[0149] When fine-tuning the trained symptom feature pre-recognition model, the trained symptom feature pre-recognition model is used to extract the test set in the labeled data and return the extraction result; the model evaluation indicators use precision rate P, recall rate R and F1 value, among which F1 value is a balance between precision rate and recall rate, and the effect of symptom feature recognition is based on F1 value;
[0150] The symptom feature pre-identification model extracts symptom features from electronic medical records. The extraction results directly output by the model are in BIO format. After data processing of the extraction results, the symptom feature extraction results based on medical records are obtained, and the results are converted into structured data. Each symptom feature is binary data, with 0 representing "no" or "false" and 1 representing "yes" or "true".
[0151] S4, collecting multimodal feature data through a multimodal feature data collection module;
[0152] In the electronic medical record, in addition to the doctor's text data, there are also a lot of patient data (such as laboratory tests, drugs, medical history, etc.). These data together constitute multimodal feature data. Since the patient's diagnosis and treatment information in the sample report is stored in different tables, after collecting the data, it is necessary to integrate the data of multiple information table reports of the patient.
[0153] For the model used in the embodiment, data in the medical field usually has the problem of data sparsity, and data collection and storage often encounter problems such as irregular collection, data noise, missing data, and errors. Therefore, the extracted data needs to be preprocessed to solve these problems. After data preprocessing, the following feature data can be extracted and summarized:
[0154] Laboratory tests: creatinine, lymphocyte ratio, blood routine;
[0155] Drugs: levofloxacin, moxifloxacin, latamoxef;
[0156] Medical history: diabetes, malignant tumor, chronic kidney disease;
[0157] S5. The collected multimodal feature data and the symptom feature pre-recognition results are combined through the symptom feature recognition module In combination, the NLP method based on multimodal enhancement is used to extract the symptom feature recognition results.
[0158] The symptom feature pre-recognition model developed only by transfer learning methods may not meet the application requirements of actual diagnosis and is not enough to be put into clinical diagnosis. Conventional NLP fine-tuning methods are used, but the improvement is not obvious. Therefore, in the method described in the embodiment, considering the characteristics and unique features of the task, that is, the specific symptoms of the patient, it is necessary to significantly improve the accuracy. This is the motivation for the behavior-enhanced BERT model.
[0159] The goal of this step is to use the collected multimodal feature data of patients to enhance the symptom feature pre-recognition model. The advantage of this multimodal enhancement method is that it can use all available medical record information to obtain accurate symptom prediction results. The traditional electronic medical record large language model only uses the medical record text itself; however, the generation of medical record text is closely related to the patient's laboratory tests, drug use, and medical history. These data of the patient can predictively infer the patient's symptoms of a certain type of disease. Multimodal data can be represented as a vector F of patient i i , so the final predicted result S of the patient's symptom n n for:
[0160]
[0161] Where f is a classifier, such as a random forest or XGBoost model. Formula (9) can be interpreted as taking the results of the symptom feature pre-identification model and multimodal data as feature inputs and then outputting new feature extraction results. The specific steps are as follows:
[0162] 1) Combine the binary symptom feature pre-recognition results with the obtained multimodal feature data to X, the labeled result is y, and the machine learning algorithm is used to mirror the training and extraction of the combined data set;
[0163] 2) The model evaluation indicators include precision P, recall R and F1 value, among which the F1 value is a balance between precision and recall. Therefore, the effect of the multimodal enhanced NLP method is mainly based on the F1 value.
[0164] To demonstrate the effectiveness and superiority of the method of the present invention, the electronic medical record symptom feature extraction method based on multimodal enhancement of the present invention is applied to the medical records of two drug-induced injury patients to test the feature extraction results:
[0165] Drug-induced liver injury:
[0166] Drug-induced liver injury (DILI) refers to liver damage caused by taking various prescription or over-the-counter drug preparations, their metabolites and even excipients. According to statistics, the incidence of DILI in China is as high as 23.8 people per 100,000 people. And it has been on the rise in recent years. Missed diagnosis and misdiagnosis will have a variety of adverse effects on the treatment and prognosis of the patient's original disease. Active early warning of early liver injury signals, accurate diagnosis, timely discontinuation of suspicious drugs and relevant intervention measures are the key links in the diagnosis and treatment of DILI.
[0167] The 1000 electronic medical records collected were annotated and divided into training set, validation set, and test set in a ratio of 8:1:1. The annotation rules for the symptom characteristics of drug-induced liver injury are shown in Table 1.
[0168] The names of the six symptom features that need to be identified and the number of entities in the dataset are shown in Table 2. The text example annotations are shown in Table 3.
[0169] Symptom feature entity category Start position tag Middle or end position label pregnancy B-Pregnancy I-Pregnancy anemia B-Anemia I-Anemia Fatigue B-Weakness I-Weakness Loss of appetite B - Poor Appetite I-Poor Appetite fever B-Fever I-Fever nausea B-Nausea I-Nausea
[0170] Table 1 Symptoms and characteristics of drug-induced liver damage: labeling rules
[0171] Entity Name Number of entities Pregnancy 720 Anemia 565 Weakness 610 Poor Appetite 895 Fever 944 Nausea 711 total 4445
[0172] Table 2 Statistics of the number of entities in the drug-induced liver injury dataset
[0173]
[0174]
[0175] Table 3 Text example annotation
[0176] This experiment uses the trained model to make predictions on the test set and accurately counts all entity labels in the character dimension for three evaluation indicators. The performance is evaluated using precision, recall, and F1 score indicators. Precision represents the ratio of the number of samples that actually belong to a certain class to the number of samples predicted to be in that class among the samples predicted by the model. It is an indicator for a specific category and reflects the model's prediction accuracy for that category. Recall, which quantifies the proportion of true positive predictions among all actual positive instances, evaluates the sensitivity of the model in detecting positive cases. The F1 score combines precision and recall to provide a balanced measure of the model's performance. By incorporating these indicators into the evaluation, a comprehensive understanding of the model's feature extraction performance is obtained.
[0177] The evaluation results of the drug-induced liver injury test set are as follows Figure 3 As shown, the calculation unit of the result is a single word, which is calculated by Figure 3 From the statistics of the number of entities in the dataset, we can see that the feature extraction results of the entities “pregnancy” and “fever” are poor, and the numerical deviations are large. The main reason may be that when labeling electronic medical records, it is difficult to unify the standards because “pregnancy” and “fever” involve a wide range of concepts.
[0178] The BIO result data extracted and output by the model is truncated according to the medical records, and the prediction indicators are calculated based on the medical records. The indicators based on the medical records no longer consider the B and I of each entity separately, but only whether the medical record prediction result correctly points out the patient's symptoms. After that, the medical record prediction results are converted into structured data, and each symptom is binary data with 0 representing "no" or "false" and 1 representing "yes" or "true".
[0179] The binary text medical record data is combined with the multimodal feature data obtained in step S4, and the xgboost machine learning algorithm is used to optimize the symptom feature pre-recognition results, with the multimodal feature data and binary symptom data as X and the annotation result as y. The evaluation indicators of the multimodal enhanced NLP method are shown in Table 4.
[0180] Multimodality precision recall f1-score support anemia 0.923 0.960 0.941 173 Fatigue 0.950 0.905 0.927 173 Loss of appetite 0.963 1.000 0.981 173 nausea 1.000 0.917 0.957 173
[0181] Table 4 Evaluation indicators of multimodal enhanced NLP methods
[0182] Acute Kidney Injury:
[0183] Acute kidney injury is the most common but easily overlooked disease in clinical practice. Its early diagnosis and treatment are crucial to the prevention and treatment of kidney damage. At present, renal failure has been internationally recognized as acute kidney injury. The basic starting point of this study is to combine early diagnosis of the disease with early intervention. Early diagnosis of acute kidney injury can prevent the condition of patients with kidney damage from deteriorating to the stage of renal failure, thereby reducing the mortality rate of the disease, alleviating the medical burden, and improving the quality of life of patients with the disease.
[0184] The 1000 medical records collected were annotated and divided into training set, validation set, and test set in a ratio of 8:1:1. The annotation rules for the symptom characteristics of renal damage are shown in Table 5.
[0185] The names of the six symptom features that need to be identified and the number of entities in the dataset are shown in Table 6. The text example annotations are shown in Table 7.
[0186] Entity Class Start position tag Middle or end position label Drinking B-Drinking I-Drinking Smoking B-Smoking I-Smoking Mechanical ventilation B-Mechanical Ventilation I-MechanicalVentilation shock B-Shock I-Shock respiratory failure B-Respiratory Failure I-Respiratory Failure thrombus B-Thrombus I-Thrombus
[0187] Table 5 Symptoms and characteristics of acute kidney injury
[0188] Entity Name Number of entities Mechanical ventilation 12075 Drinking 25 Smoking 20 shock 613 respiratory failure 1095 thrombus 75 total 13903
[0189] Table 6 Statistics of the number of entities in the acute kidney injury dataset
[0190] Text Sequence Entity Tags Entity Class drink B-Drinking Drinking liquor I-Drinking Drinking history O ten O Year O have O Remain O , O through O often O sex O suck B-Smoking Smoking cigarette I-Smoking Smoking , O cigarette B-Smoking Smoking age I-Smoking Smoking 2 O 0 O Year O Left O right O
[0191] Table 7 Text example annotation
[0192] This experiment uses the trained model to predict the test set and accurately calculates the three evaluation indicators for all entity tags in the character dimension. The test set evaluation results are as follows: Figure 4 As shown, the calculation unit of the result is a single word, which is calculated by Figure 4 From the statistics of the number of entities in the dataset, we can see that the more annotations (support) a certain entity has, that is, the more samples the model can use for learning, the better the entity prediction results obtained by fine-tuning. Since there are too few relevant entity data for "drinking", "smoking" and "thrombosis" in the training, the values of the three evaluation indicators are all zero, so these three types of entities are not put into the next step of evaluation.
[0193] The BIO result data output by the model prediction is truncated according to the medical record, and the prediction index is calculated based on the medical record. The index based on the medical record no longer considers the B and I of each entity separately, but only whether the medical record prediction result correctly points out the patient's symptoms. After that, the medical record prediction result is converted into structured data, and each symptom is binary data with 0 representing "no" or "false" and 1 representing "yes" or "true".
[0194] The binary text medical record data is combined with the multimodal feature data obtained in step 4, and the xgboost machine learning algorithm is used to optimize the symptom feature pre-recognition results, with the multimodal feature data and binary symptom data as X and the manual annotation results as y. The evaluation indicators of the multimodal enhanced NLP method are shown in Table 8.
[0195] precision recall f1-score support Mechanical ventilation 0.972 0.993 0.982 173 shock 0.950 0.905 0.927 173 respiratory failure 1.000 0.923 0.960 173
[0196] Table 8 Evaluation indicators of multimodal enhanced NLP methods
[0197] The embodiments described above are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made according to the shape and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for extracting symptom features from electronic medical records based on multimodal enhancement, characterized in that: include: Annotate the symptom characteristics closely related to the target disease in the electronic medical record to obtain annotated data; Constructing a symptom feature pre-recognition model, and training the constructed symptom feature pre-recognition model based on the labeled data; Fine-tune the trained symptom feature pre-recognition model, and use the fine-tuned symptom feature pre-recognition model to perform symptom feature recognition to obtain the symptom feature pre-recognition result Collect multimodal feature data; Combine the collected multimodal feature data with the symptom feature pre-recognition results In combination, the NLP method based on multimodal enhancement is used to extract the symptom feature recognition results.
2. According to claim 1, a method for extracting electronic medical record symptom features based on multimodal enhancement is characterized in that: Annotate the symptom characteristics in the electronic medical record that are closely related to the target disease, including: For the patient's electronic medical records, regular expressions were used to search for keywords of symptom characteristics when obtaining data, and the data was cleaned and desensitized to obtain data 1; Perform BIO annotation on data 1 to obtain annotated data; BIO annotations include "BX", "IX", and "O"; "BX" means that the element's fragment is of type X and the element is at the beginning of the fragment, "IX" means that the element's fragment is of type X and the element is in the middle of the fragment, and "O" means that it does not belong to any type.
3. The method for extracting symptom features of electronic medical records based on multimodal enhancement according to claim 1, characterized in that: The constructed symptom feature pre-recognition model includes BERT input layer, BILSTM sequence modeling layer, and CRF decoding layer; in, The BERT input layer consists of the Embedding layer and the Transformer Encoder; the Embedding layer consists of Positional Embedding, Sentence Embedding, and Token Embedding; Positional Embedding enables the model to utilize the order of the sequence by adding the position information of the sequence markers; the formula is as follows: In formulas (1) and (2), pos refers to the position of a word in a sentence. The value range of this position is from 0 to the position value of the last token in the sentence. i refers to the dimension number of the word vector, and d refers to the representation dimension of the word. In the Transformer Encoder, dot product is used for similarity calculation to learn the word dependency within the sentence and capture the internal structure of the sentence; the formula is as follows: In formula (3), Q, K, and V are the word vector matrices obtained by performing three linear transformations on the word vectors, and d k is the dimension of the input vector; In the BILSTM sequence modeling layer, the word vector X obtained by the BERT input layer is received, and the context information is bidirectionally extracted through the forget gate, input gate, and output gate: The forget gate selectively forgets some information: f t =σ(W f ·[h t-1 ,X]+b f ) (4) The input gate adds new information: Update current status: The output gate outputs the part that determines the output: In the above formula, h t-1 is the hidden state of the text at time t-1; σ is the Sigmoid activation function; b is the bias vector; tanh is the hyperbolic tangent function; W is the model parameter matrix, · represents the dot product of two vectors, and * represents the cross product of two vectors; In the CRF decoding layer, the result h obtained by the BILSTM sequence modeling layer t Decoding; this includes: Constraints are used to ensure that the final extraction result is valid. The combination of all labels constitutes the sum of the paths, and the path with the highest score needs to be found, that is, the constraints between the labels. The score of each path is calculated by formula (8), where The transfer score matrix is a parameter of the model and is updated with each training iteration; is the emission score matrix, which is the output of the BILSTM sequence modeling layer h t ; X is the observation sequence, whose value is {x1,x2,...,x n }, is the input of the symptom feature pre-recognition model; extract the sequence label y = h t ={y1,y2,...,y n }, which is the output h of the BILSTM sequence modeling layer t ; The label path corresponding to the maximum value of the path score score(X,y) is the final output, and various named entities are output according to the labels; the output format is:
4. The method for extracting symptom features of electronic medical records based on multimodal enhancement according to claim 3 is characterized in that: The BERT input layer trains the symptom feature pre-recognition model by designing the Masked Language Model and Next SentencePrediction; Masked Language Model: Randomly mask or replace any word or phrase in a sentence, and then the model extracts it based on the context; Next Sentence Prediction: Put two sentences together, and the model extracts whether the two sentences are adjacent to each other based on the original text.
5. The method for extracting electronic medical record symptom features based on multimodal enhancement according to claim 1, characterized in that: When fine-tuning the trained symptom feature pre-recognition model, the trained symptom feature pre-recognition model is used to extract the test set in the labeled data and return the extraction result; the model evaluation indicators use precision rate P, recall rate R and F1 value, among which F1 value is a balance between precision rate and recall rate, and the effect of symptom feature recognition is based on F1 value; The symptom feature pre-identification model extracts symptom features from electronic medical records. The extraction results directly output by the model are in BIO format. After data processing of the extraction results, the symptom feature extraction results based on medical records are obtained, and the results are converted into structured data. Each symptom feature is binary data, with 0 representing "no" or "false" and 1 representing "yes" or "true".
6. The method for extracting electronic medical record symptom features based on multimodal enhancement according to claim 5, characterized in that: The collected multimodal feature data include the patient's laboratory data, medication data, and medical history data.
7. The method for extracting electronic medical record symptom features based on multimodal enhancement according to claim 6, characterized in that: Combine the collected multimodal feature data with the symptom feature pre-recognition results Combined with the NLP method based on multimodal enhancement, the symptom feature recognition results are extracted, including: The binary symptom feature pre-recognition results are combined with the obtained multimodal feature data to obtain the multimodal feature data and X, the annotation result is y, and the machine learning algorithm is used to mirror the combined data set for training and extraction, and the feature recognition result S of the extracted symptom n is n for: In formula (9), f is a classifier; F i is the multimodal feature data, representing the vector of patient i.
8. A system for extracting symptom features of electronic medical records based on multimodal enhancement, used to implement the method for extracting symptom features of electronic medical records based on multimodal enhancement as described in any one of claims 1 to 7, characterized in that: It includes data annotation module, symptom feature pre-identification model construction and training module, model fine-tuning module, symptom feature pre-identification module, multimodal feature data collection module, and symptom feature identification module; in, The data annotation module is used to annotate the symptom characteristics closely related to the target disease in the electronic medical record to obtain annotated data; When the data annotation module performs data annotation, it searches for keywords of symptom characteristics using regular expressions in the patient's electronic medical records when acquiring data, and cleans and desensitizes the data to obtain data 1; Perform BIO annotation on data 1 to obtain annotated data; BIO annotations include "BX", "IX", and "O"; "BX" means that the element belongs to the X type and the element is at the beginning of the segment, "IX" means that the element belongs to the X type and the element is in the middle of the segment, and "O" means that it does not belong to any type; The symptom feature pre-recognition model construction and training module is used to construct a symptom feature pre-recognition model and train the constructed symptom feature pre-recognition model based on the labeled data; The constructed symptom feature pre-recognition model includes BERT input layer, BILSTM sequence modeling layer, and CRF decoding layer; in, The BERT input layer consists of the Embedding layer and the Transformer Encoder; the Embedding layer consists of Positional Embedding, Sentence Embedding, and Token Embedding; Positional Embedding enables the model to utilize the order of the sequence by adding the position information of the sequence markers; the formula is as follows: In formulas (1) and (2), pos refers to the position of a word in a sentence. The value range of this position is from 0 to the position value of the last token in the sentence. i refers to the dimension number of the word vector, and d refers to the representation dimension of the word. In the Transformer Encoder, dot product is used for similarity calculation to learn the word dependency within the sentence and capture the internal structure of the sentence; the formula is as follows: In formula (3), Q, K, and V are the word vector matrices obtained by performing three linear transformations on the word vectors, and d k is the dimension of the input vector; In the BILSTM sequence modeling layer, the word vector X obtained by the BERT input layer is received, and the context information is bidirectionally extracted through the forget gate, input gate, and output gate: The forget gate selectively forgets some information: f t =σ(W f ·[h t-1 ,X]+b f ) (4) The input gate adds new information: i t =σ(W i ·[h t-1 ,X]+b i ) Update current status: The output gate outputs the part that determines the output: The t =σ(W o ·[h t-1 ,X]+b o ) h t =O t *tanh(C t ) (7) In the above formula, h t-1 is the hidden state of the text at time t-1; σ is the Sigmoid activation function; b is the bias vector; tanh is the hyperbolic tangent function; W is the model parameter matrix, · represents the dot product of two vectors, and * represents the cross product of two vectors; In the CRF decoding layer, the result h obtained by the BILSTM sequence modeling layer t Decoding; this includes: Constraints are used to ensure that the final extraction result is valid. The combination of all labels constitutes the sum of the paths, and the path with the highest score needs to be found, that is, the constraints between the labels. The score of each path is calculated by formula (8), where The transfer score matrix is a parameter of the model and is updated with each training iteration; is the emission score matrix, which is the output of the BILSTM sequence modeling layer h t ; X is the observation sequence, whose value is {x1,x2,...,x n }, is the input of the symptom feature pre-recognition model; extract the sequence label y = h t ={y1,y2,...,y n }, which is the output h of the BILSTM sequence modeling layer t ; The label path corresponding to the maximum value of the path score score(X,y) is the final output, and various named entities are output according to the labels; the output format is: The BERT input layer trains the symptom feature pre-recognition model by designing the Masked Language Model and Next Sentence Prediction; Masked Language Model: Randomly mask or replace any word or phrase in a sentence, and then the model extracts it based on the context; Next Sentence Prediction: Put two sentences together, and the model extracts whether the two sentences are adjacent to each other based on the original text; The model fine-tuning module is used to fine-tune the trained symptom feature pre-recognition model; When fine-tuning the trained symptom feature pre-recognition model, the trained symptom feature pre-recognition model is used to extract the test set in the labeled data and return the extraction result; the model evaluation indicators use precision rate P, recall rate R and F1 value, among which F1 value is a balance between precision rate and recall rate, and the effect of symptom feature recognition is based on F1 value; The symptom feature pre-identification model extracts symptom features from electronic medical records. The extraction results directly output by the model are in BIO format. After data processing of the extraction results, the symptom feature extraction results based on medical records are obtained and converted into structured data. Each symptom feature is binary data, with 0 representing "no" or "false" and 1 representing "yes" or "true"; The symptom feature pre-recognition module is used to perform symptom feature recognition using the fine-tuned symptom feature pre-recognition model to obtain a symptom feature pre-recognition result. The multimodal feature data collection module is used to collect multimodal feature data; The collected multimodal feature data include patients’ laboratory data, medication data, and medical history data; The symptom feature recognition module is used to collect the multimodal feature data and the symptom feature pre-recognition results. Combined with the NLP method based on multimodal enhancement, the symptom feature recognition results are extracted, including: The binary symptom feature pre-recognition results are combined with the obtained multimodal feature data to obtain the multimodal feature data and X, the annotation result is y, and the machine learning algorithm is used to mirror the combined data set for training and extraction, and the feature recognition result S of the extracted symptom n is n for: In formula (9), f is a classifier; F i is the multimodal feature data, representing the vector of patient i.
Citation Information
Patent Citations
Information acquisition method and system based on multi-modal interaction
CN112069484A
In-hospital death risk prediction method based on multi-mode electronic medical record data
CN117219268A
Text generation method and device, equipment and storage medium
CN117473954A
Method and System for Multi-Modal Fusion Model
US20180189572A1
