Medical field risk prediction system based on NLP technology

Abstract

The medical field risk prediction system based on the NLP technology is composed of a medical data structuring processing part and a medical risk prediction part, and the data structuring processing part comprises the steps that S100, data cleaning and model pre-training are carried out, a Bert model is finely adjusted through collected medical text data, and a language model MedBert suitable for the medical field is obtained; s200: information extraction: extracting related entities, relationships and attribute information in the text through a pre-trained large language model, and structuring data; s300, standardization of the medical terms, mapping the extracted information to the standard medical terms through multi-strategy calling and negative sample training, and the risk prediction part comprises: S400, carrying out fine tuning pre-training on a TableBert model by using structured data, and extracting feature information of a patient from the data; and S500, through the risk prompt words and the extracted features, using a classifier to carry out probability prediction on corresponding risks, and solving the problems of low utilization rate and high application difficulty of unstructured data in the medical field.

Description

Technical Field

[0001] The present invention relates to the technical field of medical risk prediction based on computer technology, and in particular to a medical field risk prediction system based on NLP technology. Background Art

[0002] In recent years, the medical industry has faced a series of challenges such as the growing patient burden, shortage of medical resources, and inefficient medical services. At the same time, with the digitization of medical information and the rise of big data, the scale of medical data has also increased dramatically. Medical data covers a wealth of information, including patient medical records, doctor's records, diagnosis reports, etc. By effectively utilizing these text data, a deep understanding of medical information can be achieved, thereby improving the diagnosis and treatment level of patients, reducing the workload of medical staff, and achieving better allocation of medical resources. In this context, artificial intelligence and machine learning methods have been widely used in the medical field. A lot of medical data is mined and analyzed through artificial intelligence to assist in disease treatment. Especially in the field of imaging, there are currently many models and applications.

[0003] In recent years, natural language processing (NLP) technology has also made great progress. Especially since the advent of the Transformers model based on the self-attention mechanism, large language models have made breakthrough progress in many fields and have also been put into practice in many applications.

[0004] However, the use of NLP technology in the medical field still has great limitations, mainly due to several application difficulties. The first is the diversity of text data. Text data in the medical field is very diverse, including patient medical records, doctor's records, research papers, etc. The diversity and complexity of these data sources make processing and analysis more difficult, and the data needs to be structured and standardized to make better use of the data. The second is medical professionalism and terminology. The medical field is highly professional and contains a large number of medical terms and professional terms. NLP models need to have an in-depth understanding of these terms to accurately process medical texts, which requires a lot of medical knowledge, which makes it difficult for general NLP models to be directly applied to the medical field.

[0005] Therefore, there is an urgent need to develop and design a medical risk prediction system based on NLP technology to solve the above problems. Summary of the invention

[0006] In response to the above technical problems, the present invention provides a medical risk prediction system based on NLP technology to solve the problems of low utilization rate and high application difficulty of unstructured data in the medical field. Patients or hospitals can upload electronic medical records, physical examination reports and other information. After the system parses the files, it will give a predicted probability for common risks through a model. In addition, users can also get predictions of specific risks by asking questions.

[0007] To achieve the above object, the technical solution of the present invention is as follows:

[0008] The medical field risk prediction system based on NLP technology includes a data preprocessing module, a structured information extraction module, a terminology standardization processing module, a pre-trained model fine-tuning module and a risk prediction module, and is characterized by being specifically composed of five steps:

[0009] S100: The data preprocessing module is used for data cleaning and model pre-training. The collected medical text data is used to fine-tune the Bert model to obtain the language model MedBert suitable for the medical field. Bert refers to BidirectionalEncoder Representations from Transformers, which is a bidirectional encoding representation based on the Transformers architecture.

[0010] S200: The structured information extraction module is used to structure the text data, extract entity, relationship and attribute information through the model, obtain structured text information, and filter irrelevant information;

[0011] S300: The term standardization processing module is used for medical term standardization, mapping a given medical entity to a medical standard dictionary to reduce ambiguity;

[0012] S400: The pre-trained model fine-tuning module uses the processed data to fine-tune the pre-trained Bert model and extract hidden features from the processed data;

[0013] S500: The risk prediction module predicts the risk probability based on the prompt words, and can fine-tune new risk types using a small amount of data.

[0014] During data cleaning in S100, the data is cleaned and filtered to remove duplicate data, sensitive words, etc., and then a word segmentation model suitable for the medical field is constructed using the BPE method, so that the model learns the terminology and proper nouns in the medical field, and the training results are integrated with the word segmentation model of the Bert model to generate a word vector, wherein BPE refers to Byte Pair Encoding, an unsupervised word segmentation algorithm based on statistics,

[0015] The extraction in S200 is performed by fine-tuning the pre-trained MedBert model using data with entity and relationship annotations, and then jointly extracting entities and relationships by using the fine-tuned model.

[0016] The terminology standardization in S300 is to standardize the extracted information through the medical knowledge base SNOMED CT, screen relevant standard terms through multi-strategy recall + negative sample training, perform semantic relevance training, sort the recalled data according to semantics, and select the closest standardized terms.

[0017] In S400, a TableBert model suitable for text tables is used to extract features from the text data structured and standardized in S200 and S300, and a maximum pooling layer is used to perform feature fusion on the output sequence to obtain a feature vector of the table. When the structured data is input, the header text is also encoded, and the position encoding of the Bert model is replaced and input into the TableBert model, and the model is pre-trained using mask prediction.

[0018] The S500 outputs the feature vector obtained by the S400, connects it to the classifier for training, and outputs the risk probability. The model is fine-tuned using the data annotated with the risks. The risks to be predicted are input into the model together with the table data as prompt words. The TableBert model parameters are adjusted to improve the accuracy of risk prediction. After the model training is completed, the user can obtain the risk probability by inputting text information and risk prompt words.

[0019] Specifically, the data include electronic medical records, physical examination forms, consultation records and other data. Some of the data come from annotated public data sets, such as CBLUE, CMeEE, etc., and there are also unlabeled data crawled from public websites such as Haodaifu.

[0020] Beneficial effects of the present invention:

[0021] The present invention builds its own medical professional vocabulary and integrates it into the BPE word segmenter, which greatly improves the recognition ability of professional and rare words in the medical field. The BPE+medical vocabulary method can effectively process professional terms and new words that appear in data sets in specific fields, thereby improving the model's understanding and processing capabilities of these words.

[0022] The present invention solves the problems of low utilization rate and high application difficulty of unstructured data in the medical field. Patients or hospitals can upload electronic medical records, physical examination reports and other information. After the system parses the files, it will give a predicted probability for common risks through a model. In addition, users can also get predictions of specific risks by asking questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the process of the medical field risk prediction system based on NLP technology of the present invention;

[0024] Figure 2 Example diagram of entity data annotation;

[0025] Figure 3 The following is a schematic diagram of the prediction process of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings and tables. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Example 1

[0028] The present invention consists of two parts: data processing and risk prediction. The medical field risk prediction system based on NLP technology includes data preprocessing, structured information extraction module, terminology standardization processing module, pre-trained model fine-tuning module and risk prediction module. The specific process steps are as follows:

[0029] S100: First, we use the BPE model to build a word segmenter to split common words into smaller subwords, and enhance the ability to identify unregistered and rare words. At the same time, in order to enhance the recognition of professional words in the medical field, we built a medical professional word library and integrated it into the BPE word segmenter, which greatly improved the recognition of professional words and rare words in the medical field. The BPE+medical word library method can effectively process professional terms and new words that appear in data sets in specific fields, thereby improving the model's understanding and processing capabilities of these words.

[0030] After the word segmentation is completed, the word segmentation results are merged into the word segmentation of the pre-trained model, and then input into the language model for structured processing. For text data, the traditional CNN (Convolutional Neural Network) captures global information through windows, but it is difficult to capture long-distance dependency problems. Although RNN (Reccurent Neural Network) can solve this problem, it is easy to cause gradient vanishing and gradient explosion problems in long sequences. In order to solve these two problems, the large language model adopts the Transformers architecture. Transformers allow the model to directly view all positions in the input sequence and process global information in a single step. The present invention uses the pre-trained Bert model for fine-tuning to obtain the language model MedBert suitable for the medical field. In it, Bert is a sequence encoding model based on the self-attention Transformer architecture. The attention output gives a weighted representation of the input, which can capture the long-distance dependency relationship within the sequence. After calculation through the self-attention mechanism, it is output to the feedforward neural network, which contains two layers of linear transformation and a ReLU activation function. The ReLU function form is formula (1):

[0031] ReLU(x)=max(x,0) (1),

[0032] S200: The structured processing of text is completed through the joint extraction of entities and relationships. There are a large number of entity overlaps and nestings in the medical field text. For example, in "bladder, renal colic", "bladder pain" and "renal colic" are symptoms that need to be extracted, but because it is a non-continuous text, traditional sequence labeling is difficult to achieve the ideal extraction effect, and the errors in entity extraction will accumulate to the subsequent relationship and attribute extraction. The present invention uses a multi-head labeling method to construct a labeling matrix, and explicitly labels the possible head and tail entity fragments for each potential relationship prediction. Through the joint extraction method, the MedBert model is fine-tuned in a supervised manner. At this point, the data structuring part is completed. Through the fine-tuned model, key information such as symptoms and body parts in texts such as electronic medical records and physical examination reports can be identified, and then the unstructured text data can be converted into a structured template to facilitate subsequent risk prediction.

[0033] S300: There are a large number of professional terms in the medical field, but due to usage habits, usage occasions, etc., there are often many different ways of writing the same diagnosis, surgery, medicine, examination, test, symptom, etc. In order to reduce the impact of this situation on the prediction accuracy of the model, the present invention further standardizes the terms by multi-strategy recall screening + positive and negative sample training after structuring the data in step S200, and maps the non-standard terms to standard terms. A large number of standard terms can be obtained through the medical knowledge base SNOMED CT, and the similarity between the original diagnostic terms of the current query and the standard terms is calculated, and a series of candidate terms with the highest similarity are returned. There are many ways to calculate similarity. The present invention uses three methods: minimum edit distance, TF-IDF, and Jaccard coefficient, which are defined as follows:

[0034] Edit distance refers to the minimum number of operations required to convert one string into another, including addition, deletion, and modification. The smaller the edit distance, the closer the two texts are.

[0035] TF-IDF is an efficient algorithm for calculating feature weights, which can be used to solve the problem of similarity of short texts. The present invention uses it as an algorithm for comparing the relevance of original terms and standard terms in the standard term library. This algorithm matches the original terms to some of the most relevant standard terms. First, the weights of the feature words in the standard term library are calculated, and each standard term in the standard term library C is segmented. For each segmented word w j , the weight calculation formula is formula (2):

[0036]

[0037] Where N(w j ) is the participle w j In standard terminology C i The number of occurrences, m is the total number of terms in the standard term base, M(w j ) is the standard terminology set containing the segmentation w j Next, calculate the relevance score between the original term and each concept in the concept library, segment the original term s to generate a word list v, and calculate s and C i The correlation score is as follows:

[0038]

[0039] The Jaccard correlation coefficient mainly calculates the similarity between individuals of symbolic metrics. For the original term s and the concept C i To calculate their Jaccard correlation coefficient, we can first transform s and C i They are divided into character set A and character set B respectively, then s and Ci The Jaccard coefficient is the ratio of the intersection and union of sets A and B, as shown in formula (4):

[0040]

[0041] The present invention uses multi-strategy recall, that is, the above three similarity calculation indicators are used to screen standard words. The sorting part uses semantic relevance training, and in addition to the marked positive samples, negative samples and standard words that do not correspond to the original diagnostic words are also generated. While reducing the distance between the original word and the positive sample, the distance between the original word and the negative sample is increased, thereby better distinguishing positive samples from negative samples. The present invention uses cosine similarity to measure the distance angle between two words, and the specific calculation is as shown in formula (5):

[0042]

[0043] Where a and b are the vector representations of the two words in the word vector space, and the vector angle is calculated using the ratio of the vector inner product and the length product to obtain the cosine similarity. The trained semantic network sorts the recalled data according to the semantics of the original word and the standard word, and selects the closest standardized term.

[0044] S400: After the data is structured and standardized, a large amount of redundant and noisy information is removed, and subsequent modeling and prediction can be performed. The TableBert model suitable for text table data is used to extract features and predict risks of structured information. TableBert is similar to Bert in structure, but in addition to word segmentation and position coding information, the model input also incorporates header data. Model pre-training uses mask prediction to randomly replace part of the table data with special characters [MASK]. The training goal is to predict the replaced table values. After the training is completed, the model can perform feature extraction on the table data. The present invention uses the maximum pooling layer to perform feature fusion on the output sequence to obtain the feature vector of the table.

[0045] S500: The fused features are connected to the classifier for training. The classifier is MLP (Multi-Layer Perceptron). The output of the classifier is converted into probability through the sigmoid function. The sigmoid function is as shown in formula (6):

[0046]

[0047] Use the data with labeled risks to train the classifier, input the risk to be predicted as a prompt word together with the table data into the model, and adjust the classifier parameters to improve the accuracy of risk prediction. After the model training is completed, the user can input text information and the risk to be predicted to get the risk probability.

[0048] Among them, the data in the present invention comes from a large number of electronic medical records, physical examination reports, and consultation records, including public datasets such as CBLUE and CMeEE, and unlabeled data crawled from public websites such as HaoDaiFu. The data is initially cleaned.

[0049] Embodiment 2

[0050] S100: First, the data is initially cleaned. After filtering sensitive content, the data processing part begins. A tokenizer is constructed using the BPE model to split common words into smaller sub-words, enhancing the ability to recognize out-of-vocabulary words and rare words. In this example, when inputting a medical text sentence, we first perform character segmentation, and then through a series of operations such as calculating frequencies, merging the characters with the highest frequencies, updating the vocabulary and frequencies, we obtain the final tokenization result. At the same time, in order to enhance the recognition ability of professional medical vocabulary, the present invention collects a large amount of materials to build a medical professional vocabulary library by itself and integrates it into the BPE tokenizer, greatly improving the recognition ability of professional and rare words in the medical field.

[0051] After tokenization, the tokenization results are merged into the tokenization of the pre-trained model, and then the Bert is fine-tuned for secondary pre-training to obtain a language model MedBert suitable for the medical field. An unsupervised training method of continuous masked prediction is adopted, replacing continuous segments in the original text with special characters [MASK], and the training objective of the model is to restore the replaced characters.

[0052] S200: The structured processing of the text is completed through entity and relationship extraction. There are a large number of entity overlaps and nestings in medical text. For example, in "due to the involvement of the respiratory center", both "respiratory center" and "involvement of the respiratory center" are entities that need to be extracted, which are part and symptom respectively. However, due to entity nesting, traditional sequence labeling is difficult to achieve an ideal extraction effect, and errors in entity extraction will accumulate to subsequent relationship and attribute extraction. The present invention constructs an N*N (N is the sequence length) segment matrix. As Figure 2 shown in the entity data annotation example, 0 represents an entity segment, the segment {呼}{枢}=1, representing that "respiratory center" is a part entity; the segment {呼}{累}=2, representing that "involvement of the respiratory center" is a symptom entity, using different serial numbers to distinguish entity types. Similar relationship classification annotations are obtained by constructing an N*C matrix (where N is the sequence length and C is the total number of relationship categories). The MedBert model is fine-tuned in a supervised manner, and at the same time, the losses of entity recognition and relationship classification are minimized. The fine-tuned model can identify key information such as symptoms and body parts in texts such as electronic medical records and physical examination reports, and then convert unstructured text data into structured templates. Table 1 is an example of text structuring.

[0053]

[0054] Table 1 Example of structured data

[0055] S300: After the data is structured, the terms are further standardized by multi-strategy recall screening + positive and negative sample training, and the non-standard terms are mapped to standard terms. A large number of standard terms can be obtained through the medical knowledge base SNOMED CT, and the similarity between the original diagnostic term of the current query and the standard term is calculated, and a series of candidate terms with the highest similarity are returned. The present invention uses the minimum edit distance and TF-IDF to calculate the similarity and screen the standard terms. The screened terms are sorted using semantic relevance. The semantic relevance calculation uses contrast training. In addition to the marked positive samples, negative samples are also generated during the process, that is, standard terms that do not correspond to the original diagnostic terms. This example uses triplet loss to reduce the distance between the original term and the positive sample while increasing the distance between the original term and the negative sample, so as to better distinguish between positive and negative samples. The distance of the word uses the cosine distance of formula (5). The trained network sorts the recalled data according to the semantics and selects the closest standardized terms. Table 2 is an example of medical term standardization.

[0056] Original word Standard word The right lung nodule metastasis may be large Pulmonary space-occupying lesions Hospitalization for right lung nodule Pulmonary space-occupying lesions Left upper lung subpleural nodule to be investigated Pleural mass

[0057] Table 2 Example diagram of term standardization

[0058] S400: After the data is structured and standardized, a large amount of redundant and noisy information is removed, and subsequent modeling and prediction can be performed. The TableBert model applicable to text table data is used to extract features and predict risks from structured information.

[0059] TableBert is similar to Bert in structure. The TableBert used in this example has 12 layers of Transformers, and the dimension of each layer is 768. When inputting data into the model, the embedded encoding of the table header text is used to replace the position encoding in Bert. The model training uses mask prediction to randomly replace part of the table data with special characters [MASK]. The training goal is to predict the replaced table values. After training, the model can extract features from the table data. In this example, the maximum pooling layer is used to fuse the output sequence to obtain the feature vector of the table.

[0060] S500: The fused features are connected to the classifier for training, and the output value is the risk probability. In this example, an MLP with 1 hidden layer and 512 hidden layer dimensions is used as the classifier, and the output result of the classifier is converted into a probability through the sigmoid function of formula (6). Use data labeled with risks to train the classifier model. The risk to be predicted is input into the model as a prompt word together with the table data, and the classifier model parameters are adjusted to improve the accuracy of risk prediction. After the model training is completed, the user can obtain the risk probability by inputting text information and the risk to be predicted. The model in this example can predict the risks of diseases including cardiovascular and cerebrovascular diseases, diabetes, etc. through text information such as electronic medical records and physical examination reports. At the same time, based on the powerful feature extraction capabilities of the MedBert and TableBert models, for rare risk types with prediction needs, only a small amount of labeled data is needed to fine-tune the classifier parameters to greatly improve the prediction accuracy. The overall prediction process is as follows Figure 3 shown.

[0061] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. The present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A medical risk prediction system based on NLP technology, including a data preprocessing module, a structured information extraction module, a terminology standardization processing module, a pre-trained model fine-tuning module and a risk prediction module, characterized in that: It consists of 5 steps: S100: The data preprocessing module is used for data cleaning and model pre-training. The collected medical text data is used to fine-tune the Bert model to obtain the language model MedBert suitable for the medical field. Bert refers to BidirectionalEncoder Representations from Transformers, which is a bidirectional encoding representation based on the Transformers architecture. S200: The structured information extraction module is used to structure the text data, extract entity, relationship and attribute information through the model, obtain structured text information, and filter irrelevant information; S300: The term standardization processing module is used for medical term standardization, mapping a given medical entity to a medical standard dictionary to reduce ambiguity; S400: The pre-trained model fine-tuning module uses the processed data to fine-tune the pre-trained Bert model and extract hidden features from the processed data; S500: The risk prediction module predicts the risk probability based on the prompt words, and can fine-tune new risk types using a small amount of data.

2. The medical field risk prediction system based on NLP technology according to claim 1 is characterized in that During data cleaning in S100, the data is cleaned and filtered to remove duplicate data, sensitive words, etc., and then a word segmentation model suitable for the medical field is constructed using the BPE method, so that the model learns the terminology and proper nouns in the medical field, and the training results are integrated with the word segmentation model of the Bert model to generate a word vector, wherein BPE refers to Byte Pair Encoding, an unsupervised word segmentation algorithm based on statistics.

3. The medical field risk prediction system based on NLP technology according to claim 1 is characterized in that The extraction in S200 is performed after fine-tuning the pre-trained MedBert model using data with entity and relationship annotations, and then jointly extracting entities and relationships by using the fine-tuned model.

4. The medical field risk prediction system based on NLP technology according to claim 3 is characterized in that The terminology standardization in S300 is to standardize the extracted information through the medical knowledge base SNOMED CT, screen relevant standard terms through multi-strategy recall + negative sample training, perform semantic relevance training, sort the recalled data according to semantics, and select the closest standardized term.

5. The medical field risk prediction system based on NLP technology according to claim 4 is characterized in that In S400, the TableBert model suitable for text tables is used to extract features from the text data that has been structured and standardized in S200 and S300, and the maximum pooling layer is used to perform feature fusion on the output sequence to obtain a feature vector of the table. When the structured data is input, the header text is also encoded and input into the TableBert model instead of the position encoding of the Bert model, and the model is pre-trained using mask prediction.

6. The medical field risk prediction system based on NLP technology according to claim 5 is characterized in that The S500 outputs the feature vector obtained by the S400, connects it to the classifier for training, and outputs the risk probability. The model is fine-tuned using the data annotated with the risks. The risks to be predicted are input into the model together with the table data as prompt words. The TableBert model parameters are adjusted to improve the accuracy of risk prediction. After the model training is completed, the user can obtain the risk probability by inputting text information and risk prompt words.

7. The medical field risk prediction system based on NLP technology according to claim 1 is characterized in that The data include electronic medical records, physical examination forms, consultation records and other data. Some of the data come from annotated public data sets such as CBLUE, CMeEE, etc., and there are also unlabeled data crawled from public websites such as Haodaifu.

Images