An electronic medical record symptom feature extraction method and system based on multi-modal enhancement

Abstract

The application discloses a kind of based on multimodal enhancement electronic medical record symptom feature extraction method and system, method includes: the symptom feature closely related to target disease in electronic medical record is marked, and obtains labeled data;Symptom feature pre-identification model is constructed, and the symptom feature pre-identification model obtained by construction is trained based on labeled data;The fine-tuning of trained symptom feature pre-identification model is carried out, and the symptom feature pre-identification model after fine-tuning is used to carry out symptom feature recognition, and obtain symptom feature pre-identification result collection multimodal feature data;The multimodal feature data obtained by collection is combined with symptom feature pre-identification result, using the NLP method based on multimodal enhancement, symptom feature recognition result is extracted.The application can greatly improve the accuracy of electronic medical record symptom feature extraction.

Description

Technical Field

[0001] This invention relates to the technical field of entity naming recognition, and in particular to a method and system for extracting symptom features from electronic medical records based on multimodal enhancement. Background Technology

[0002] Named entity recognition (NER) is the task of extracting specific types of entities from complex structured, unstructured, and semi-structured data. It provides fundamental support for natural language processing tasks such as text structuring, relation extraction, text summarization, and machine translation, and is a hot topic in NLP. Using NER technology, medical entities and structured medical records can be efficiently obtained, supporting services such as medical knowledge graphs, intelligent health question answering, intelligent assisted diagnosis, and knowledge subscription, thus promoting the development of smart healthcare.

[0003] In the medical field, Electronic Medical Records (EMRs) serve as crucial information resources, containing a wealth of medical expertise. However, due to the sheer volume and complex formats of information within EMRs, and the fact that much of it exists in unstructured data, doctors often struggle to quickly locate key information, impacting the diagnosis and treatment of drug-induced liver injury. In recent years, Natural Language Processing (NLP) technology has made significant progress in the medical field, such as in Named Entity Recognition (NER) and relation extraction. Among these, NER aims to identify and classify entities with specific meanings from unstructured text, playing a vital role in various NLP tasks.

[0004] Currently, Named Entity Recognition (NER) for Chinese electronic medical records (EMRs) identifies medical information entities with independent or specific meanings from massive amounts of EMR data. These entities fall into six categories: drugs, anatomical locations, diagnoses, examinations, tests, and surgeries. These entities are then sequence-annotated and standardized. However, in diagnosing specific symptoms of a particular disease (such as DILI), current popular entity annotation methods have the limitation of failing to intuitively reflect specific symptoms. Furthermore, Chinese text exhibits characteristics such as polysemy, multiple words with the same meaning, complex sentence structures, and frequent ellipsis. The same words in different positions within a sentence may have completely different parts of speech and meanings. Therefore, existing Chinese NER methods are poorly suited for symptom recognition problems in Chinese EMRs. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for extracting symptom features from electronic medical records based on multimodal enhancement.

[0006] To achieve the above objectives, the technical solution provided by this invention is as follows:

[0007] A method for extracting symptom features from electronic medical records based on multimodal enhancement, comprising:

[0008] Annotate the symptom features closely related to the target disease in electronic medical records to obtain labeled data;

[0009] A symptom feature pre-identification model was constructed, and the constructed symptom feature pre-identification model was trained based on labeled data;

[0010] The trained symptom feature pre-identification model is fine-tuned, and then used to identify symptom features to obtain the symptom feature pre-identification results.

[0011] Collect multimodal feature data;

[0012] The collected multimodal feature data and symptom feature pre-identification results will be used to... By combining these methods and utilizing a multimodal augmented NLP approach, symptom feature recognition results were extracted.

[0013] Furthermore, symptom features closely related to the target disease in the electronic medical record are annotated, including:

[0014] For patients' electronic medical records, regular expressions were used to search for keywords of symptom characteristics when acquiring data, and the data was cleaned and desensitized to obtain data 1;

[0015] Perform BIO annotation on data 1 to obtain annotated data;

[0016] BIO annotations include "BX", "IX", and "O"; "BX" indicates that the element is in a fragment of type X and the element is at the beginning of the fragment, "IX" indicates that the element is in a fragment of type X and the element is in the middle of the fragment, and "O" indicates that it does not belong to any type.

[0017] Furthermore, the constructed symptom feature pre-identification model includes a BERT input layer, a BILSTM sequence modeling layer, and a CRF decoding layer;

[0018] in,

[0019] The BERT input layer consists of an embedding layer and a Transformer encoder; the embedding layer comprises positional embedding, sentence embedding, and token embedding; positional embedding incorporates the positional information of sequence tokens, enabling the model to utilize the sequence order; the formula is as follows:

[0020]

[0021] In equations (1) and (2), pos refers to the position of a character in a sentence. The value of this position ranges from 0 to the position value of the last marker in the sentence. i refers to the dimension index of the character vector, and d refers to the representation dimension of the character.

[0022] In the Transformer's Encoder, dot products are used to calculate similarity, thereby learning word dependencies within a sentence and capturing its internal structure; the formula is as follows:

[0023]

[0024] In equation (3), Q, K, and V are the character vector matrices obtained by performing three linear transformations on the character vectors, and d k The dimension of the input vector;

[0025] In the BILSTM sequence modeling layer, the word vector X obtained from the BERT input layer is received, and bidirectional high-dimensional feature extraction of contextual information is performed through the forget gate, input gate, and output gate.

[0026] The forgetting gate selectively forgets some information:

[0027] f t =σ(W f ·[h t-1 ,X]+b f (4)

[0028] Add new information to the input gate:

[0029] i t =σ(W i ·[h t-1 ,X]+b i )

[0030]

[0031] Update current status:

[0032]

[0033] The output gate determines the output part:

[0034] O t =σ(W o ·[h t-1 ,X]+b o )

[0035] h t =O t*tanh(C t (7)

[0036] In the above formula, h t-1 Let be the hidden state of the text at time t-1; σ be the Sigmoid activation function; b be the bias vector; tanh be the hyperbolic tangent function; W be the model parameter matrix, where · represents the dot product of two vectors and * represents the cross product of two vectors;

[0037] In the CRF decoding layer, the result h obtained from the BILSTM sequence modeling layer... t Decoding is performed; this includes:

[0038] Constraints are used to ensure the validity of the final extraction result; the combination of all labels constitutes the sum of paths, and the path with the highest score needs to be found, i.e., the constraints between labels; the score of each path is calculated by formula (8), where The transition score matrix is ​​a parameter of the model that is updated during training iterations. The emission fraction matrix is ​​the output h of the BILSTM sequence modeling layer. t X is the observation sequence with values ​​{x1, x2, ..., x}. n} is the input to the symptom feature pre-identification model; extract sequence labels y = h t ={y1,y2,...,y n}, that is, the output h of the BILSTM sequence modeling layer t The label path corresponding to the maximum value of the path score(X,y) is the final output. Various named entities are then output based on the labels. The output format is:

[0039]

[0040] Furthermore, the BERT input layer trains a symptom feature pre-identification model by designing a Masked Language Model and Next SentencePrediction;

[0041] Masked Language Model: Randomly mask or replace any word or phrase in a sentence, and then the model extracts it based on the context;

[0042] Next Sentence Prediction: The model puts two sentences together and extracts whether these two sentences are adjacent sentences based on the original text.

[0043] Furthermore, when fine-tuning the trained symptom feature pre-identification model, the trained symptom feature pre-identification model is used to extract the test set from the labeled data, and the extraction results are returned. The model evaluation metrics are precision (P), recall (R), and F1 score, where the F1 score is a balance between precision and recall, and the effectiveness of symptom feature identification is based on the F1 score.

[0044] The symptom feature pre-identification model extracts symptom features from electronic medical records. The model outputs the extraction results in BIO format. After data processing, the symptom feature extraction results are obtained in units of medical records. The results are then converted into structured data, where each symptom feature is a binary data element, with 0 representing "no" or "false" and 1 representing "yes" or "true".

[0045] Furthermore, the collected multimodal feature data includes patients' laboratory data, medication data, and medical history data.

[0046] Furthermore, the collected multimodal feature data is compared with the symptom feature pre-identification results. By combining and utilizing NLP methods based on multimodal enhancement, symptom feature recognition results are extracted, specifically including:

[0047] The binarized symptom feature pre-identification results are combined with the obtained multimodal feature data, using multimodal feature data and Let X be the model number and y be the labeling result. A machine learning algorithm is used to train and extract features from this combined dataset, resulting in the feature recognition result S for symptom n. n for:

[0048]

[0049] In formula (9), f is a classifier; F i For multimodal feature data, let represent the vector of patient i.

[0050] To achieve the above objectives, the present invention further provides an electronic medical record symptom feature extraction system based on multimodal enhancement, used to implement the above-mentioned electronic medical record symptom feature extraction method based on multimodal enhancement, including a data annotation module, a symptom feature pre-identification model construction and training module, a model fine-tuning module, a symptom feature pre-identification module, a multimodal feature data collection module, and a symptom feature recognition module;

[0051] in,

[0052] The data annotation module is used to annotate symptom features closely related to the target disease in electronic medical records to obtain annotated data;

[0053] When the data annotation module performs data annotation, it uses regular expressions to search for keywords of symptom characteristics when acquiring data from patients' electronic medical records, and cleans and desensitizes the data to obtain data 1;

[0054] Perform BIO annotation on data 1 to obtain annotated data;

[0055] BIO tags include "BX", "IX", and "O"; "BX" indicates that the element is in a segment of type X and the element is at the beginning of the segment, "IX" indicates that the element is in a segment of type X and the element is in the middle of the segment, and "O" indicates that it does not belong to any type.

[0056] The symptom feature pre-identification model construction and training module is used to construct a symptom feature pre-identification model and train the constructed symptom feature pre-identification model based on labeled data.

[0057] The constructed symptom feature pre-identification model includes a BERT input layer, a BILSTM sequence modeling layer, and a CRF decoding layer;

[0058] in,

[0059] The BERT input layer consists of an embedding layer and a Transformer encoder; the embedding layer comprises positional embedding, sentence embedding, and token embedding; positional embedding incorporates the positional information of sequence tokens, enabling the model to utilize the sequence order; the formula is as follows:

[0060]

[0061] In equations (1) and (2), pos refers to the position of a character in a sentence. The value of this position ranges from 0 to the position value of the last marker in the sentence. i refers to the dimension index of the character vector, and d refers to the representation dimension of the character.

[0062] In the Transformer's Encoder, dot products are used to calculate similarity, thereby learning word dependencies within a sentence and capturing its internal structure; the formula is as follows:

[0063]

[0064] In equation (3), Q, K, and V are the character vector matrices obtained by performing three linear transformations on the character vectors, and d k The dimension of the input vector;

[0065] In the BILSTM sequence modeling layer, the word vector X obtained from the BERT input layer is received, and bidirectional high-dimensional feature extraction of contextual information is performed through the forget gate, input gate, and output gate.

[0066] The forgetting gate selectively forgets some information:

[0067] f t =σ(W f ·[h t-1 ,X]+b f (4)

[0068] Add new information to the input gate:

[0069] i t =σ(W i ·[h t-1 ,X]+b i )

[0070]

[0071] Update current status:

[0072]

[0073] The output gate determines the output part:

[0074] O t =σ(W o ·[h t-1 ,X]+b o )

[0075] h t =O t *tanh(C t (7)

[0076] In the above formula, h t-1 Let be the hidden state of the text at time t-1; σ be the Sigmoid activation function; b be the bias vector; tanh be the hyperbolic tangent function; W be the model parameter matrix, where · represents the dot product of two vectors and * represents the cross product of two vectors;

[0077] In the CRF decoding layer, the result h obtained from the BILSTM sequence modeling layer... t Decoding is performed; this includes:

[0078] Constraints are used to ensure the validity of the final extraction result; the combination of all labels constitutes the sum of paths, and the path with the highest score needs to be found, i.e., the constraints between labels; the score of each path is calculated by formula (8), where The transition score matrix is ​​a parameter of the model that is updated during training iterations. The emission fraction matrix is ​​the output h of the BILSTM sequence modeling layer. t X is the observation sequence with values ​​{x1, x2, ..., x}. n} is the input to the symptom feature pre-identification model; extract sequence labels y = h t ={y1,y2,...,y n}, that is, the output h of the BILSTM sequence modeling layer t The label path corresponding to the maximum value of the path score(X,y) is the final output. Various named entities are then output based on the labels. The output format is:

[0079]

[0080] The BERT input layer trains a symptom feature pre-identification model by designing a Masked Language Model and Next Sentence Prediction.

[0081] Masked Language Model: Randomly mask or replace any word or phrase in a sentence, and then the model extracts it based on the context;

[0082] Next Sentence Prediction: The model puts two sentences together and extracts whether these two sentences are adjacent sentences based on the original text.

[0083] The model fine-tuning module is used to fine-tune the trained symptom feature pre-identification model;

[0084] When fine-tuning the trained symptom feature pre-identification model, the trained symptom feature pre-identification model is used to extract the test set from the labeled data and the extraction results are returned. The model evaluation metrics are precision (P), recall (R), and F1 score. The F1 score is a balance between precision and recall, and the effectiveness of symptom feature identification is based on the F1 score.

[0085] The symptom feature pre-identification model extracts symptom features from electronic medical records. The model outputs the extraction results in BIO format. After data processing, the symptom feature extraction results are obtained in units of medical records. The results are then converted into structured data, where each symptom feature is a binary data element, with 0 representing "no" or "false" and 1 representing "yes" or "true".

[0086] The symptom feature pre-identification module is used to perform symptom feature identification using the fine-tuned symptom feature pre-identification model to obtain symptom feature pre-identification results.

[0087] The multimodal feature data collection module is used to collect multimodal feature data;

[0088] The collected multimodal feature data includes patients' laboratory data, medication data, and medical history data;

[0089] The symptom feature recognition module is used to combine the collected multimodal feature data with the symptom feature pre-recognition results. By combining and utilizing NLP methods based on multimodal enhancement, symptom feature recognition results are extracted, specifically including:

[0090] The binarized symptom feature pre-identification results are combined with the obtained multimodal feature data, using multimodal feature data and Let X be the model number and y be the labeling result. A machine learning algorithm is used to train and extract features from this combined dataset, resulting in the feature recognition result S for symptom n. n for:

[0091]

[0092] In formula (9), f is a classifier; F i For multimodal feature data, let represent the vector of patient i.

[0093] Compared with existing technologies, the principles and advantages of this technical solution are as follows:

[0094] 1. Faced with electronic medical record data that is massive in volume and complex in format, and mostly exists in the form of unstructured data, the symptom feature pre-identification model adopted in this technical solution can quickly and accurately identify each character as an entity, and on this basis, transform the unstructured data into medical structured data (each symptom is binary data).

[0095] 2. Combine the collected multimodal feature data to more specifically determine the patient's physical condition with the symptom feature pre-identification results. By combining these methods and using NLP based on multimodal enhancement to extract symptom features, the accuracy of symptom feature extraction from electronic medical records can be greatly improved. Attached Figure Description

[0096] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the services required in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0097] Figure 1This is a connection block diagram of an electronic medical record symptom feature extraction system based on multimodal enhancement according to the present invention;

[0098] Figure 2 This is a flowchart illustrating the principle of a multimodal enhancement-based electronic medical record symptom feature extraction method of the present invention (data annotation and model building parts are omitted);

[0099] Figure 3 This is a graph showing the evaluation results of the drug-induced liver injury test set;

[0100] Figure 4 This is a graph showing the evaluation results of the acute kidney injury test set. Detailed Implementation

[0101] The present invention will be further described below with reference to specific embodiments:

[0102] like Figure 1 As shown in this embodiment, an electronic medical record symptom feature extraction system based on multimodal enhancement includes a data annotation module, a symptom feature pre-identification model construction and training module, a model fine-tuning module, a symptom feature pre-identification module, a multimodal feature data collection module, and a symptom feature recognition module.

[0103] in,

[0104] The data annotation module is used to annotate symptom features closely related to the target disease in electronic medical records, obtaining annotated data; the symptom feature pre-identification model construction and training module is used to construct a symptom feature pre-identification model and train the constructed symptom feature pre-identification model based on the annotated data; the model fine-tuning module is used to fine-tune the trained symptom feature pre-identification model; the symptom feature pre-identification module is used to perform symptom feature identification using the fine-tuned symptom feature pre-identification model, obtaining the symptom feature pre-identification results. The multimodal feature data collection module is used to collect multimodal feature data; the symptom feature recognition module is used to compare the collected multimodal feature data with the symptom feature pre-recognition results. By combining these methods and utilizing a multimodal augmented NLP approach, symptom feature recognition results were extracted.

[0105] like Figure 2 As shown, the working principle of this embodiment includes the following steps:

[0106] S1. The data annotation module is used to annotate the symptom features closely related to the target disease in the electronic medical record to obtain annotated data;

[0107] For patients' electronic medical records, regular expressions were used to search for keywords of symptom characteristics when acquiring data, and the data was cleaned and desensitized to obtain data 1;

[0108] Perform BIO annotation on data 1 to obtain annotated data;

[0109] BIO annotations include "BX", "IX", and "O"; "BX" indicates that the element is in a fragment of type X and the element is at the beginning of the fragment, "IX" indicates that the element is in a fragment of type X and the element is in the middle of the fragment, and "O" indicates that it does not belong to any type.

[0110] Some examples of annotations are shown in Table a:

[0111] Suffering The Lawsuit lack force ， B-Weakness I-Weakness I-Weakness I-Weakness I-Weakness O

[0112] Example of table a annotation

[0113] S2. Construct a symptom feature pre-identification model through the symptom feature pre-identification model construction and training module, and train the constructed symptom feature pre-identification model based on labeled data;

[0114] The constructed symptom feature pre-identification model includes a BERT input layer, a BILSTM sequence modeling layer, and a CRF decoding layer;

[0115] in,

[0116] The BERT input layer consists of an embedding layer and a Transformer (model) encoder. The embedding layer comprises Positional Embedding (an encoding method in the BERT model used to assign a different embedding vector to each position in the input sequence), Sentence Embedding (an important output of the BERT model, obtained by summarizing or aggregating the embeddings of all tokens in the input sequence), and Token Embedding (a representation of each input token in BERT; in the BERT model, the input text is first segmented into tokens, and then each token is embedded as a vector representation; token embedding is one of the inputs of the BERT model, containing the semantic information of each token). Positional Embedding incorporates the positional information of the sequence tokens, allowing the model to utilize the sequence order; the formula is as follows:

[0117]

[0118] In equations (1) and (2), pos refers to the position of a character in a sentence. The value of this position ranges from 0 to the position value of the last marker in the sentence. i refers to the dimension index of the character vector, and d refers to the representation dimension of the character.

[0119] The core idea of ​​the Transformer's Encoder is to replace the traditional RNN with a self-attention mechanism, using dot products for similarity calculation to learn word dependencies within a sentence and capture its internal structure; the formula is as follows:

[0120]

[0121] In equation (3), Q, K, and V are the character vector matrices obtained by performing three linear transformations on the character vectors, and d k The dimension of the input vector;

[0122] The output of the BERT model contains two main types of word vectors: word-level word vectors and sentence-level word vectors.

[0123] Specifically, for each character in the input sequence, the BERT model generates its corresponding character vector, which contains the semantic information of the character in its context. Sentence-level character vectors are obtained by encoding special markers, which provide a comprehensive representation of the semantic information of the entire sequence.

[0124] The BERT model obtains the word vector X through a pre-training phase and a fine-tuning phase.

[0125] Pretraining: In the pretraining phase, the BERT model is trained using large-scale unlabeled text data. This phase includes two main tasks: MLM and NSP. Through training on these two tasks, the BERT model learns context-sensitive representations at the word and sentence levels.

[0126] Masked Language Model (MLM): Randomly masks or replaces any words or phrases within a sentence, and then the model extracts them based on the context. In the MLM task, some words or tokens in the input text sequence are randomly masked (replaced with special masking markers). The model then needs to predict these masked words or tokens. In this way, the BERT model can learn context-dependent word representations and has a certain semantic understanding capability.

[0127] Next Sentence Prediction (NSP): This task places two sentences together, and the model determines whether these two sentences are consecutive based on the original text. In the NSP task, the BERT model needs to determine whether two input sentences are adjacent in the order they appear in the original text. The model predicts their adjacency using the embeddings of the input sentences. This task helps the BERT model learn sentence-level semantic relationships and contextual understanding.

[0128] Fine-tuning: In the fine-tuning phase, the BERT model is further trained using labeled, task-specific data. For example, named entity recognition data. Through supervised training on task data, the parameters of the BERT model are adjusted to suit the specific needs of the task.

[0129] After the pre-training and fine-tuning phases are completed, the parameters of the BERT model are fixed, and the model can be used to encode new input sequences to obtain word-level word vectors X. These word vectors X capture the semantic information of words in context and can be used for downstream natural language processing tasks such as text classification, named entity recognition, and sentence similarity.

[0130] In the BILSTM sequence modeling layer, the word vector X obtained from the BERT input layer is received, and bidirectional high-dimensional feature extraction of contextual information is performed through the forget gate, input gate, and output gate.

[0131] The forgetting gate selectively forgets some information:

[0132] f t =σ(W f ·[h t-1 ,X]+b f (4)

[0133] Add new information to the input gate:

[0134] i t =σ(W i ·[h t-1 ,X]+b i )

[0135]

[0136] Update current status:

[0137]

[0138] The output gate determines the output part:

[0139] O t =σ(W o ·[h t-1 ,X]+b o )

[0140] h t =O t *tanh(C t (7)

[0141] In the above formula, h t-1Let be the hidden state of the text at time t-1; σ be the Sigmoid activation function; b be the bias vector; tanh be the hyperbolic tangent function; W be the model parameter matrix, where · represents the dot product of two vectors and * represents the cross product of two vectors;

[0142] In the CRF decoding layer, the result h obtained from the BILSTM sequence modeling layer... t Decoding is performed; this includes:

[0143] Constraints are used to ensure the validity of the final extraction result; the combination of all labels constitutes the sum of paths, and the path with the highest score needs to be found, i.e., the constraints between labels; the score of each path is calculated by formula (8), where The transition score matrix is ​​a parameter of the model that is updated during training iterations. The emission fraction matrix is ​​the output h of the BILSTM sequence modeling layer. t X is the observation sequence with values ​​{x1, x2, ..., x}. n} is the input to the symptom feature pre-identification model; extract sequence labels y = h t ={y1,y2,...,y n}, that is, the output h of the BILSTM sequence modeling layer t The label path corresponding to the maximum value of the path score(X,y) is the final output. Various named entities are then output based on the labels. The output format is:

[0144]

[0145] S3. The trained symptom feature pre-identification model is fine-tuned using the model fine-tuning module, and then the symptom feature pre-identification module is used to perform symptom feature identification to obtain the symptom feature pre-identification results.

[0146] When fine-tuning the trained symptom feature pre-identification model, the trained symptom feature pre-identification model is used to extract the test set from the labeled data and the extraction results are returned. The model evaluation metrics are precision (P), recall (R), and F1 score. The F1 score is a balance between precision and recall, and the effectiveness of symptom feature identification is based on the F1 score.

[0147] The symptom feature pre-identification model extracts symptom features from electronic medical records. The model outputs the extraction results in BIO format. After data processing, the symptom feature extraction results are obtained in units of medical records. The results are then converted into structured data, where each symptom feature is a binary data element, with 0 representing "no" or "false" and 1 representing "yes" or "true".

[0148] S4. Collect multimodal feature data through the multimodal feature data collection module;

[0149] In addition to doctors' text data, electronic medical records contain a wealth of patient data (such as lab results, medication information, and medical history). This data collectively constitutes multimodal feature data. Since patient medical information in sample reports is stored in different tables, data integration is required from these multiple information tables after data collection.

[0150] For the model used in the implementation example, medical data often suffers from sparsity. Data collection and storage frequently encounter problems such as irregular collection, data noise, missing data, and errors. Therefore, the extracted data needs to be preprocessed to address these issues. After data preprocessing, the following characteristic data can be extracted and summarized:

[0151] Laboratory tests: creatinine, lymphocyte ratio, and complete blood count;

[0152] Medications: Levofloxacin, Moxifloxacin, Latamoxef;

[0153] Medical history: diabetes, malignant tumor, chronic kidney disease;

[0154] S5. The collected multimodal feature data is compared with the symptom feature pre-identification results through the symptom feature recognition module. By combining these methods and utilizing a multimodal augmented NLP approach, symptom feature recognition results were extracted.

[0155] Symptom feature pre-identification models developed solely using transfer learning methods may not meet the application needs of actual diagnosis and are insufficient for clinical use. Conventional NLP fine-tuning methods are employed, but the improvement is not significant. Therefore, in the method described in this embodiment, considering the characteristics and unique features of the task—namely, the patient's specific symptoms—it is necessary to significantly improve accuracy. This is the motivation behind the proposed behavior-enhanced BERT model.

[0156] The goal of this step is to enhance the symptom feature pre-identification model using collected patient multimodal feature data. The advantage of this multimodal enhancement method is that it can utilize all available medical record information to obtain accurate symptom prediction results. Traditional electronic medical record large language models only use the medical record text itself; however, the generation of medical record text is closely related to the patient's laboratory tests, medication use, and medical history, and this data can predictively infer symptoms of a certain type of disease. Multimodal data can be represented as a vector F of patient i. i Therefore, the final predictive result S of the patient's symptom n n for:

[0157]

[0158] Where f is a classifier, such as a random forest or XGBoost model. Formula (9) can be interpreted as taking the results of the symptom feature pre-identification model and multimodal data as feature inputs, and then outputting new feature extraction results. The specific steps are as follows:

[0159] 1) Combine the binary symptom feature pre-identification results with the obtained multimodal feature data, using the multimodal feature data and Let X be the labeled result and y be the annotation result. Use machine learning algorithms to train and extract the combined dataset in a mirror image.

[0160] 2) The model evaluation metrics are precision (P), recall (R), and F1 score. The F1 score is a balance between precision and recall, so the effectiveness of multimodal augmented NLP methods is mainly based on the F1 score.

[0161] To demonstrate the effectiveness and superiority of the method described in this invention, the multimodal enhancement-based electronic medical record symptom feature extraction method described in this invention is applied to the medical records of two patients with drug-induced injuries to verify the feature extraction results:

[0162] Drug-induced liver injury:

[0163] Drug-induced liver injury (DILI) refers to liver damage induced in patients by the use of various prescription or over-the-counter drug preparations, their metabolites, and even excipients. Proactive early warning signs of liver injury, accurate diagnosis, timely discontinuation of suspected drugs, and implementation of relevant interventions are crucial aspects of the diagnosis and treatment of DILI.

[0164] The collected 1000 electronic medical records were labeled and divided into training, validation, and test sets in an 8:1:1 ratio. The labeling rules for the symptom characteristics of drug-induced liver injury are shown in Table 1.

[0165] Table 2 shows the names of the six symptom features to be identified and the number of entities in the dataset. Example text annotations are shown in Table 3.

[0166] Symptom characteristics entity category Start position label Middle or end position tags pregnancy B-Pregnancy I-Pregnancy anemia B-Anemia I-Anemia fatigue B-Weakness I-Weakness Loss of appetite B-PoorAppetite I-PoorAppetite fever B-Fever I-Fever nausea B-Nausea I-Nausea

[0167] Table 1. Rules for labeling the symptom characteristics of drug-induced liver injury.

[0168] Entity Name Entity count Pregnancy 720 Anemia 565 Weakness 610 PoorAppetite 895 Fever 944 Nausea 711 total 4445

[0169] Table 2. Entity count statistics for the drug-induced liver injury dataset.

[0170]

[0171]

[0172] Table 3. Example Text Annotations

[0173] This experiment utilizes a trained model to predict on a test set and performs precise statistical analysis on all entity labels across the character dimension using three metrics. Precision, recall, and F1 score are used to evaluate performance. Precision represents the proportion of samples predicted as belonging to a certain class out of the total number of predicted samples. It is a metric specific to a particular category, reflecting the model's accuracy in predicting that category. Recall quantifies the proportion of true positive predictions among all actual positive instances, assessing the model's sensitivity in detecting positive cases. The F1 score combines precision and recall, providing a balanced measure of model performance. By incorporating these metrics into the evaluation, a comprehensive understanding of the model's feature extraction performance is achieved.

[0174] The results of the drug-induced liver injury test set evaluation are as follows: Figure 3 As shown, the unit of calculation for this result is a single word, calculated by... Figure 3 As shown in Table 2, which contains statistics on the number of entities in the dataset, the feature extraction results for the entities "pregnancy" and "fever" are poor, with significant numerical deviations. The main reason is likely that the broad concepts of "pregnancy" and "fever" make it difficult to standardize the labeling process in electronic medical records.

[0175] The BIO results extracted from the model are truncated according to medical records, and predictive indicators are statistically analyzed on a case-by-case basis. Indicators on a case-by-case basis no longer consider the B and I of each entity separately; they only indicate whether the prediction result for that case is correct and thus indicate the patient's symptoms. Then, the case prediction results are transformed into structured data, where each symptom is represented by binary data, with 0 representing "no" or "false" and 1 representing "yes" or "true".

[0176] The binary medical record data is combined with the multimodal feature data obtained in step S4, and the XGBoost machine learning algorithm is used to optimize the symptom feature pre-identification results. The multimodal feature data and binary symptom data are represented as X, and the labeled results as y. The evaluation metrics for the multimodal enhanced NLP method are shown in Table 4.

[0177] Multimodal precision recall f1-score support anemia 0.923 0.960 0.941 173 fatigue 0.950 0.905 0.927 173 Loss of appetite 0.963 1.000 0.981 173 nausea 1.000 0.917 0.957 173

[0178] Table 4 Evaluation metrics for multimodal enhanced NLP methods

[0179] Acute kidney injury:

[0180] Acute kidney injury (AKI) is one of the most common yet easily overlooked diseases in clinical practice, and its early diagnosis and treatment are crucial for preventing and treating kidney damage. Currently, kidney failure is internationally recognized as AKI. The basic premise of this research is to combine early diagnosis with early intervention. Early diagnosis of AKI can prevent patients from progressing to kidney failure, thereby reducing mortality, alleviating the medical burden, and improving the quality of life for patients with this disease.

[0181] The collected 1000 medical records were labeled and divided into training, validation, and test sets in an 8:1:1 ratio. The labeling rules for the symptom characteristics of kidney damage are shown in Table 5.

[0182] Table 6 shows the names of the six symptom features to be identified and the number of entities in the dataset. Example text annotations are shown in Table 7.

[0183] Entity Category Start position label Middle or end position tags Drinking B-Drinking I-Drinking Smoking B-Smoking I-Smoking Mechanical ventilation B-MechanicalVentilation I-MechanicalVentilation shock B-Shock I-Shock respiratory failure B-RespiratoryFailure I-RespiratoryFailure thrombus B-Thrombus I-Thrombus

[0184] Table 5. Rules for labeling the symptom characteristics of acute kidney injury.

[0185]

[0186]

[0187] Table 6. Entity count statistics for the acute kidney injury dataset.

[0188] text sequence Entity label Entity Category drink B-Drinking Drinking liquor I-Drinking Drinking history O ten O Year O have O Remain O ， O through O often O sex O suck B-Smoking Smoking cigarette I-Smoking Smoking ， O cigarette B-Smoking Smoking age I-Smoking Smoking 2 O 0 O Year O Left O right O

[0189] Table 7 Examples of Text Annotations

[0190] This experiment utilizes a trained model to predict on the test set and performs precise statistical analysis on all entity labels across three evaluation metrics at the character dimension. The test set evaluation results are as follows: Figure 4 As shown, the unit of calculation for this result is a single word, calculated by... Figure 4 As shown in Table 6, which contains statistics on the number of entities in the dataset, the more support labels an entity has, the more samples the model can use for learning, and the better the final entity prediction results after fine-tuning. Since there was too little relevant entity data for "drinking," "smoking," and "thrombosis" in the training, and all three evaluation metrics were zero, these three entities were not included in the next evaluation step.

[0191] The BIO results output by the model are truncated according to medical records, and the prediction indicators are statistically analyzed on a case-by-case basis. Indicators on a case-by-case basis no longer consider the B and I of each entity separately; they only indicate whether the prediction result for that case is correct and thus indicate the patient's symptoms. Then, the medical record prediction results are transformed into structured data, where each symptom is represented by binary data, with 0 representing "no" or "false" and 1 representing "yes" or "true".

[0192] The binary medical record data is combined with the multimodal feature data obtained in step four, and the XGBoost machine learning algorithm is used to optimize the symptom feature pre-identification results. Let the multimodal feature data and binary symptom data be X, and the manually labeled results be y. The evaluation metrics for the multimodal enhanced NLP method are shown in Table 8.

[0193] precision recall f1-score support Mechanical ventilation 0.972 0.993 0.982 173 shock 0.950 0.905 0.927 173 respiratory failure 1.000 0.923 0.960 173

[0194] Table 8 Evaluation metrics for multimodal augmented NLP methods

[0195] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Therefore, any changes made in accordance with the shape and principle of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for extracting symptom features from electronic medical records based on multimodal enhancement, characterized in that, include: Annotate the symptom features closely related to the target disease in electronic medical records to obtain labeled data; A symptom feature pre-identification model was constructed, and the constructed symptom feature pre-identification model was trained based on labeled data; The trained symptom feature pre-identification model is fine-tuned, and then used to identify symptom features to obtain the symptom feature pre-identification results. Collect multimodal feature data; The collected multimodal feature data and symptom feature pre-identification results will be used to... By combining these methods and utilizing a multimodal augmentation-based NLP approach, symptom feature recognition results were extracted. When fine-tuning the trained symptom feature pre-identification model, the trained symptom feature pre-identification model is used to extract the test set from the labeled data and the extraction results are returned. The model evaluation metrics are precision (P), recall (R), and F1 score. The F1 score is a balance between precision and recall, and the effectiveness of symptom feature identification is based on the F1 score. The symptom feature pre-identification model extracts symptom features from electronic medical records. The model outputs the extraction results in BIO format. After data processing, the symptom feature extraction results are obtained in units of medical records. The results are then converted into structured data, where each symptom feature is a binary data element, with 0 representing "no" or "false" and 1 representing "yes" or "true".

2. The method for extracting symptom features from electronic medical records based on multimodal enhancement according to claim 1, characterized in that, Annotate symptom features in electronic medical records that are closely related to the target disease, including: For patients' electronic medical records, regular expressions were used to search for keywords of symptom characteristics when acquiring data, and the data was cleaned and desensitized to obtain data 1; Perform BIO annotation on data 1 to obtain annotated data; BIO tags include "BX", "IX", and "O"; "BX" indicates that the element is in a segment of type X and the element is at the beginning of the segment, "IX" indicates that the element is in a segment of type X and the element is in the middle of the segment, and "O" indicates that it does not belong to any type.

3. The method for extracting symptom features from electronic medical records based on multimodal enhancement according to claim 1, characterized in that, The constructed symptom feature pre-identification model includes a BERT input layer, a BILSTM sequence modeling layer, and a CRF decoding layer; in, The BERT input layer consists of an embedding layer and a Transformer encoder. The embedding layer comprises Position Embedding, Sentence Embedding, and Token Embedding. Position Embedding incorporates the positional information of sequence tokens, enabling the model to utilize the sequence order. The formula is as follows: In equations (1) and (2), pos refers to the position of a character in a sentence. The value of this position ranges from 0 to the position value of the last marker in the sentence. i refers to the dimension index of the character vector, and d refers to the representation dimension of the character. In the Transformer's Encoder, dot products are used to calculate similarity, thereby learning word dependencies within a sentence and capturing its internal structure; the formula is as follows: In equation (3), Q, K, and V are the character vector matrices obtained by performing three linear transformations on the character vectors, and d k The dimension of the input vector; In the BILSTM sequence modeling layer, the word vector X obtained from the BERT input layer is received, and bidirectional high-dimensional feature extraction of contextual information is performed through the forget gate, input gate, and output gate. The forgetting gate selectively forgets some information: f t =σ(W f ·[h t-1 ,X]+b f ) (4) Add new information to the input gate: Update current status: The output gate determines the output part: In the above formula, h t-1 Let be the hidden state of the text at time t-1; σ be the sigmoid activation function; b be the bias vector; tanh be the hyperbolic tangent function; W ... f W i W c W o All are model parameter matrices, where · represents the dot product of two vectors and * represents the cross product of two vectors; In the CRF decoding layer, the result h obtained from the BILSTM sequence modeling layer... t Decoding is performed; this includes: Constraints are used to ensure the validity of the final extraction result; the combination of all labels constitutes the sum of paths, and the path with the highest score needs to be found, i.e., the constraints between labels; the score of each path is calculated by formula (8), where The transition score matrix is ​​a parameter of the model that is updated during training iterations. The emission fraction matrix is ​​the output h of the BILSTM sequence modeling layer. t X is the observation sequence with values ​​{x1, x2, ..., x}. n } is the input to the symptom feature pre-identification model; extract sequence labels y = h t ={y1,y2,...,y n }, that is, the output h of the BILSTM sequence modeling layer t The label path corresponding to the maximum value of the path score(X,y) is the final output. Various named entities are then output based on the labels. The output format is:

4. The method for extracting symptom features from electronic medical records based on multimodal enhancement according to claim 3, characterized in that, The BERT input layer trains a symptom feature pre-identification model by designing a Masked Language Model and Next Sentence Prediction. Masked Language Model: Randomly mask or replace any word or phrase in a sentence, and then the model extracts it based on the context; Next Sentence Prediction: The model puts two sentences together and extracts whether these two sentences are adjacent sentences based on the original text.

5. The method for extracting symptom features from electronic medical records based on multimodal enhancement according to claim 1, characterized in that, The collected multimodal feature data includes patients' laboratory data, medication data, and medical history data.

6. The method for extracting symptom features from electronic medical records based on multimodal enhancement according to claim 5, characterized in that, The collected multimodal feature data and symptom feature pre-identification results will be used to... By combining and utilizing NLP methods based on multimodal enhancement, symptom feature recognition results are extracted, specifically including: The binarized symptom feature pre-identification results are combined with the obtained multimodal feature data, using multimodal feature data and Let X be the model number and y be the labeling result. A machine learning algorithm is used to train and extract features from the combined dataset, resulting in the feature recognition result S for symptom n. n for: In formula (9), f is a classifier; F i For multimodal feature data, let represent the vector of patient i.

7. A symptom feature extraction system for electronic medical records based on multimodal enhancement, used to implement the symptom feature extraction method for electronic medical records based on multimodal enhancement as described in any one of claims 1-6, characterized in that, It includes a data annotation module, a symptom feature pre-identification model building and training module, a model fine-tuning module, a symptom feature pre-identification module, a multimodal feature data collection module, and a symptom feature recognition module; in, The data annotation module is used to annotate symptom features closely related to the target disease in electronic medical records to obtain annotated data; When the data annotation module performs data annotation, it uses regular expressions to search for keywords of symptom characteristics when acquiring data from patients' electronic medical records, and cleans and desensitizes the data to obtain data 1; Perform BIO annotation on data 1 to obtain annotated data; BIO tags include "BX", "IX", and "O"; "BX" indicates that the element is in a segment of type X and the element is at the beginning of the segment, "IX" indicates that the element is in a segment of type X and the element is in the middle of the segment, and "O" indicates that it does not belong to any type. The symptom feature pre-identification model construction and training module is used to construct a symptom feature pre-identification model and train the constructed symptom feature pre-identification model based on labeled data. The constructed symptom feature pre-identification model includes a BERT input layer, a BILSTM sequence modeling layer, and a CRF decoding layer; in, The BERT input layer consists of an embedding layer and a Transformer encoder. The embedding layer comprises Position Embedding, Sentence Embedding, and Token Embedding. Position Embedding incorporates the positional information of sequence tokens, enabling the model to utilize the sequence order. The formula is as follows: In equations (1) and (2), pos refers to the position of a character in a sentence. The value of this position ranges from 0 to the position value of the last marker in the sentence. i refers to the dimension index of the character vector, and d refers to the representation dimension of the character. In the Transformer's Encoder, dot products are used to calculate similarity, thereby learning word dependencies within a sentence and capturing its internal structure; the formula is as follows: In equation (3), Q, K, and V are the character vector matrices obtained by performing three linear transformations on the character vectors, and d k The dimension of the input vector; In the BILSTM sequence modeling layer, the word vector X obtained from the BERT input layer is received, and bidirectional high-dimensional feature extraction of contextual information is performed through the forget gate, input gate, and output gate. The forgetting gate selectively forgets some information: f t =σ(W f ·[h t-1 ,X]+b f ) (4) Add new information to the input gate: i t =σ(W i ·[h t-1 ,X]+b i ) Update current status: The output gate determines the output part: The t =σ(W o ·[h t-1 ,X]+b o ) h t =O t *fishy(C) t ) (7) In the above formula, h t-1 Let be the hidden state of the text at time t-1; σ be the Sigmoid activation function; b be the bias vector; tanh be the hyperbolic tangent function; W be the model parameter matrix, where · represents the dot product of two vectors and * represents the cross product of two vectors; In the CRF decoding layer, the result h obtained from the BILSTM sequence modeling layer... t Decoding is performed; this includes: Constraints are used to ensure the validity of the final extraction result; the combination of all labels constitutes the sum of paths, and the path with the highest score needs to be found, i.e., the constraints between labels; the score of each path is calculated by formula (8), where The transition score matrix is ​​a parameter of the model that is updated during training iterations. The emission fraction matrix is ​​the output h of the BILSTM sequence modeling layer. t X is the observation sequence with values ​​{x1, x2, ..., x}. n } is the input to the symptom feature pre-identification model; extract sequence labels y = h t ={y1,y2,...,y n }, that is, the output h of the BILSTM sequence modeling layer t The label path corresponding to the maximum value of the path score(X,y) is the final output. Various named entities are then output based on the labels. The output format is: The BERT input layer trains a symptom feature pre-identification model by designing a Masked Language Model and Next Sentence Prediction. Masked Language Model: Randomly mask or replace any word or phrase in a sentence, and then the model extracts it based on the context; Next Sentence Prediction: The model puts two sentences together and extracts whether these two sentences are adjacent sentences based on the original text. The model fine-tuning module is used to fine-tune the trained symptom feature pre-identification model; When fine-tuning the trained symptom feature pre-identification model, the trained symptom feature pre-identification model is used to extract the test set from the labeled data and the extraction results are returned. The model evaluation metrics are precision (P), recall (R), and F1 score. The F1 score is a balance between precision and recall, and the effectiveness of symptom feature identification is based on the F1 score. The symptom feature pre-identification model extracts symptom features from electronic medical records. The model outputs the extraction results in BIO format. After data processing, the symptom feature extraction results are obtained in units of medical records. The results are then converted into structured data, where each symptom feature is a binary data element, with 0 representing "no" or "false" and 1 representing "yes" or "true". The symptom feature pre-identification module is used to perform symptom feature identification using the fine-tuned symptom feature pre-identification model to obtain symptom feature pre-identification results. The multimodal feature data collection module is used to collect multimodal feature data; The collected multimodal feature data includes patients' laboratory data, medication data, and medical history data; The symptom feature recognition module is used to combine the collected multimodal feature data with the symptom feature pre-recognition results. By combining and utilizing NLP methods based on multimodal enhancement, symptom feature recognition results are extracted, specifically including: The binarized symptom feature pre-identification results are combined with the obtained multimodal feature data, using multimodal feature data and Let X be the model number and y be the labeling result. A machine learning algorithm is used to train and extract features from the combined dataset, resulting in the feature recognition result S for symptom n. n for: In formula (9), f is a classifier; F i For multimodal feature data, let represent the vector of patient i.

Images