A medical text classification method and device based on prompt learning

Abstract

The application provides a medical text classification method and device based on prompt learning. The method comprises the following steps: obtaining prompt information for primary classification from original medical text based on event prior information and knowledge prior information, wherein the primary classification comprises a department category; filtering the original medical text, and inputting the filtered text and the prompt information into a large language generation model after integration; calculating the similarity between the result sequence output by the large language generation model and each secondary classification label representing a disease category under the primary classification, and taking the label category corresponding to the maximum value of the similarity as the secondary category output by the large language generation model. The application can realize primary classification based on the department category, and can also realize secondary classification based on the disease category under the primary classification, which is more consistent with the general cognition in the medical field, so that the classification result is more standardized; meanwhile, the classification label can be unfixed, so that multi-level text classification in an open domain can be effectively realized.

Description

Technical Field

[0001] This invention belongs to the field of text processing technology, specifically relating to a medical text classification method and apparatus based on cue learning. Background Technology

[0002] Classifying medical texts with specific information and domain knowledge has always been challenging. Clauses may exhibit ambiguity and conflict in text category due to technical terms, medical experience, contrast, and other narrative techniques, making it difficult for models to classify the text as a whole. For a long time, research institutions both domestically and internationally have proposed various deep learning models for text classification. However, limited by model analysis capabilities and hardware processing speed, small models cannot provide more accurate results, while large models face significant deployment difficulties. In recent years, with the rapid development of large language models and the emergence of new paradigms in natural language processing such as knowledge transfer, fine-tuning, and cue-based learning, small models can easily leverage the massive parameters of large language models to provide convenient and efficient text classification products and services.

[0003] Existing medical text classification methods based on template-based prompting typically involve concatenating the initial input text and prompt content, feeding this concatenation into a template generator, and then updating the prompt content within the same encoder model along with the initial input text. The updated template parameters are then extracted, concatenated with the unupdated initial input text, and fed into a large pre-trained language model. This leverages the massive parameters and training instances of the pre-trained model to improve text classification accuracy and other performance aspects. However, because this method updates template parameters using a multi-head self-attention network encoder, and the data source is a massive general-purpose text corpus, while it performs well for general text processing, it suffers from a lack of medical domain-specific knowledge. This leads to significant biases and deficiencies in understanding domain-specific knowledge, thus limiting the understanding of medical text context and semantics. Furthermore, this method does not consider important fact-related medical text information (such as existing medical entities, important indicator values, and relationships between entities), which hinders the classification of text content in specific contexts. While the ChatGPT model for generating classification labels can achieve open-domain text classification, providing a variety of non-given labels, this approach is insufficient. However, due to a lack of enhanced and standardized medical knowledge, the output categories are often uncontrollable, and usually only a first-level classification label can be obtained, which is not accurate enough for a deeper judgment on a certain disease. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, the present invention provides a medical text classification method and apparatus based on cue learning.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] In a first aspect, the present invention provides a medical text classification method based on cue learning, comprising the following steps:

[0007] Based on prior event information and prior knowledge information, prompts for primary classification are obtained from the original medical text, and the primary classification includes department categories;

[0008] The original medical text is filtered, and the filtered text is integrated with the prompt information and then input into the large language generation model.

[0009] Calculate the similarity between the output sequence of the large language generation model and each secondary classification label representing a disease category under the primary classification, and take the label category corresponding to the maximum similarity as the secondary category output by the large language generation model.

[0010] Furthermore, the method for obtaining the prompt information includes:

[0011] The original medical text is input into the medical event feature extraction model to obtain structured event prompt information A;

[0012] The original medical text is input into the medical knowledge feature extraction model to obtain structured knowledge prompt information B;

[0013] After integrating A and B, the results are output to the prompt template generation module to obtain prompt information for the primary category.

[0014] Furthermore, the medical event feature extraction model is a pre-trained RoBERTa model, fine-tuned using a dataset composed of publicly available medical information.

[0015] Furthermore, the medical knowledge feature extraction model is a pre-trained BioBERT model, which is fine-tuned using a dataset composed of publicly available medical knowledge graph data.

[0016] Furthermore, the prompt template generation module consists of four identical network layers, with each layer employing a Transformer decoder to remove the second attention layer structure.

[0017] Furthermore, the original medical text is filtered by inputting a text filter; the text filter is a binary classifier using a ConvNeXt Tiny network, which trains the text filter by manually annotating each sentence as a valid or invalid sentence through sentence-level annotation of the medical text.

[0018] Furthermore, before the prompt information is integrated with the filtered text, it is first mapped by the spatial mapping module to the feature space that the large language generation model can understand.

[0019] Furthermore, the primary classification includes at least cardiology, orthopedics, and neurology; the secondary classification under vascular medicine includes at least coronary heart disease, ischemic heart disease, stroke, aortic aneurysm, and renal artery stenosis; the secondary classification under orthopedics includes at least patellar fracture, ulnar nerve injury, congenital coxa vara, and infectious costochondritis; and the secondary classification under neurology includes at least cerebral hemorrhage, cerebral embolism, headache, and myelitis.

[0020] Furthermore, the method for calculating the similarity includes:

[0021] The pre-trained word vector model Word2Vec is used to embed the labels to be classified, denoted as (x1, x2, ..., x...). n );

[0022] Let the output sequence of the large language generation model be denoted as (y1, y2, ..., y). n );

[0023] Calculate (x1, x2, ..., x) using the following formula n ) and (y1,y2,…,y n Cosine similarity of )

[0024]

[0025] In the formula, K cos Let be the cosine similarity.

[0026] Secondly, the present invention provides a medical text classification device based on cue learning, comprising:

[0027] The prompt information acquisition module is used to acquire prompt information for primary classification from the original medical text based on prior event information and prior knowledge information, wherein the primary classification includes department category;

[0028] The information integration module is used to filter the original medical text and integrate the filtered text with the prompt information before inputting it into the large language generation model;

[0029] The secondary classification module is used to calculate the similarity between the result sequence output by the large language generation model and each secondary classification label representing a disease category under the primary classification, and the label category corresponding to the maximum similarity is taken as the secondary category output by the large language generation model.

[0030] Compared with the prior art, the present invention has the following beneficial effects.

[0031] This invention extracts prompts from original medical text based on prior event and knowledge information for primary classification, including departmental categories. The original medical text is then filtered, and the filtered text is integrated with the prompts before being input into a large language generation model. The similarity between the output sequence of the large language generation model and each secondary classification label representing a disease category under the primary classification is calculated. The label category corresponding to the maximum similarity is taken as the secondary category output by the large language generation model, thus achieving prompt-based medical text classification. This invention not only achieves primary classification based on departmental categories but also secondary classification based on disease categories, which is more consistent with common understanding in the medical field and makes the classification results more standardized. Furthermore, the classification labels can be flexible, effectively achieving multi-level text classification in open domains. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating a medical text classification method based on cue learning, as described in an embodiment of the present invention.

[0033] Figure 2 This is a schematic diagram of two stages in another embodiment of the present invention.

[0034] Figure 3 This is a structural diagram of the template generation module.

[0035] Figure 4 This is a block diagram of a medical text classification device based on cue learning, according to an embodiment of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer and more understandable, the invention will be further described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0037] Figure 1 This is a flowchart of a medical text classification method based on cue learning according to an embodiment of the present invention, including the following steps:

[0038] Step 101: Based on prior event information and prior knowledge information, obtain prompt information for primary classification from the original medical text, wherein the primary classification includes department category;

[0039] Step 102: Filter the original medical text, and then integrate the filtered text with the prompt information and input it into the large language generation model;

[0040] Step 103: Calculate the similarity between the result sequence output by the large language generation model and each secondary classification label representing a disease category under the primary classification, and take the label category corresponding to the maximum similarity as the secondary category output by the large language generation model.

[0041] In this embodiment, step 101 is mainly used to obtain cue information for primary classification. Cue learning utilizes templates to stimulate factual knowledge in a pre-trained language model, and then uses the obtained factual knowledge as the answer space, mapping it to the target space to complete the required downstream tasks. Cue learning can effectively help fine-tune downstream tasks using large language models, and achieves efficient parameter transfer learning with minimal fine-tuning computational power. Existing text classification techniques based on cue learning have two problems: one is using hard templates (discrete cue templates, which are actual text strings) for cue learning. Hard template construction is highly subjective, has limited content expression, and the model cannot fully utilize the information in the text, resulting in low prediction accuracy; the other is introducing soft cue templates (continuous cue templates, which are directly described in the embedding space of the underlying language model, usually in vector representation), which lack prior knowledge and rely solely on data from downstream tasks for optimization, making it difficult to achieve better performance and resulting in relatively high training costs. Therefore, this embodiment obtains a soft cue template based on knowledge extraction information and event extraction results. This template can effectively obtain the main medical events and medical-related knowledge in the text, better optimizing the classification effect of the large language generation model. Simultaneously, based on this soft prompt information, the text can be classified into primary categories, and its primary category labels can be initially obtained. In this embodiment, the primary categories mainly include departmental categories, such as cardiology and orthopedics.

[0042] In this embodiment, step 102 is mainly used to integrate the original medical text with the prompt information. The original medical text generally uses unstructured free text description, often containing a large amount of invalid information and fields, such as irrelevant unit information and addresses, watermarks, data interfaces, or special fields in HTML files. This results in a low density of effective information in the text, and directly obtaining all text features would introduce a lot of uncertain noise, weakening the core content of the text. Therefore, this embodiment first filters the original medical text, integrating the filtered text (after removing invalid sentences) with the prompt information. This integration involves concatenating the two feature vectors together.

[0043] For ease of understanding, the following is an excerpt from the original medical text:

[0044] After chemotherapy for right lung cancer, a CT scan from November 29, 2016, compared to the previous scan, shows: the lesion in the right upper lung has slightly decreased in size, and the surrounding inflammation has slightly decreased. Scattered small nodules remain in both lungs, roughly the same as before. Multiple lymph nodes in the left subclavian region and mediastinum, considered to be metastases, have slightly decreased in size compared to the previous scan. Liver cyst. Small cyst in the left kidney. After chemotherapy for right lung cancer, a CT scan from November 29, 2016, compared to the previous scan, shows: irregular nodular and patchy lesions in the right upper lobe, with indistinct borders, the largest measuring approximately 12mm × 8mm, with lobulated edges, uneven enhancement on contrast-enhanced scan, closely adhering to the oblique fissure pleura, with some areas slightly smaller than the previous scan. A few patchy, slightly high-density shadows are seen in the right upper lung, with indistinct borders, significantly reduced in size compared to the previous scan. No abnormalities were found in the right kidney and both subrenal glands. No enlarged lymph nodes were seen behind the diaphragm or beside the abdominal aorta. No signs of bone destruction were observed on the scan.

[0045] In this embodiment, step 103 is mainly used to output the secondary category. This embodiment inputs the integrated information obtained in step 102 into the large language generation model, calculates the similarity between the output sequence of the large language generation model and the secondary category label, and uses the label category corresponding to the maximum similarity as the output secondary category. The large language generation model refers to a language model containing massive (hundreds of billions or more) parameters, which needs to be trained on massive amounts of text data, such as the GPT-3 model. GPT-3 is obtained by stacking multiple layers of Transformer Decoder layers (with the second attention structure removed). GPT-3 is very powerful in text generation and can achieve amazing results in very complex NLP tasks. It represents a huge development for solutions in areas such as creating creative novels, stories, resumes, narratives, chatbots, and text summarization. Completing these tasks does not require supervised learning for model fine-tuning; for a new task, GPT-3 only needs very little data to understand the requirements of the task and achieve results close to or exceeding the current best methods. In this embodiment, the secondary classification is a disease category under the primary classification (department category), such as coronary heart disease, ischemic heart disease, etc. To achieve open-domain classification, the classification labels can be flexible. When obtaining learnable embeddings for the classification labels, not only are the secondary classification labels represented as vectors, but the combined labels of the corresponding primary and secondary classification labels are also represented and their similarity is calculated. Therefore, this embodiment can achieve not only primary classification but also secondary open-domain classification under the primary classification.

[0046] As an optional embodiment, the method for obtaining the prompt information includes:

[0047] The original medical text is input into the medical event feature extraction model to obtain structured event prompt information A;

[0048] The original medical text is input into the medical knowledge feature extraction model to obtain structured knowledge prompt information B;

[0049] After integrating A and B, the results are output to the prompt template generation module to obtain prompt information for the primary category.

[0050] This embodiment provides a technical solution for obtaining the prompt information. For example... Figure 2 As shown, in this embodiment, the original medical text is first input into the medical event feature extraction model and the medical knowledge feature extraction model, respectively, to extract event prompts and knowledge prompts. For example, inputting the sentence "unclear boundaries, maximum layer size approximately 12mm × 8mm" into the medical event feature extraction model outputs "primary lesion size: 12mm × 8mm"; inputting the triple <malignant lung tumor, disease-corresponding drug, gefitinib> into the medical knowledge feature extraction model outputs "oncology". Then, the extracted event prompts and knowledge prompts are integrated (concatenated) and sent to the template generation module to obtain prompts for primary classification.

[0051] As an optional embodiment, the medical event feature extraction model is a pre-trained RoBERTa model, which is fine-tuned using a dataset composed of publicly available medical information.

[0052] This embodiment presents a network structure and training method for a medical event feature extraction model. The medical event feature extraction model in this embodiment is a pre-trained RoBERTa model. BERT is a classic network in deep learning, and BERT-based network models perform well in natural language processing, especially in text information representation and feature encoding. RoBERT is an enhanced version of BERT, a more finely tuned version of the BERT model. RoBERT uses more parameters, more training data, and introduces dynamic masks and a larger vocabulary for training. Extensive literature and experiments show that the RoBERTa model performs excellently in information extraction tasks. Therefore, this embodiment selects RoBERT as the medical event feature extraction model. This embodiment uses publicly available medical information to construct a training dataset, and uses this dataset to fine-tune the pre-trained RoBERTa model. After training, it can be used to extract structured event information from unstructured text.

[0053] As an optional embodiment, the medical knowledge feature extraction model is a pre-trained BioBERT model, which is fine-tuned using a dataset composed of publicly available medical knowledge graph data.

[0054] This embodiment presents a network structure and training method for a medical knowledge feature extraction model. The medical knowledge feature extraction model in this embodiment is a pre-trained BioBERT model. The BioBERT model is also a more refined and optimized version of the BERT model. BioBERT is a pre-trained biomedical language representation model used for biomedical text mining. By enhancing the infusion of medical knowledge, the model contains a large amount of medical text information, resulting in excellent performance on biomedical text mining tasks. Therefore, this embodiment selects BioBERT as the medical knowledge feature extraction model. This embodiment utilizes publicly available large-scale medical knowledge graph data to fine-tune the BioBERT pre-trained model, employing a multi-label classification task based on knowledge departments. After training, it can be used to extract structured knowledge information from unstructured text.

[0055] As an optional embodiment, the prompt template generation module consists of four identical network layers, with each network layer employing a Transformer decoder to remove the second attention layer structure.

[0056] This embodiment presents a network structure for a template generation module. The template generation module in this embodiment consists of four identical stacked networks, with each layer employing a Transformer decoder to remove the second attention layer. The specific structure is as follows: Figure 3 As shown.

[0057] As an optional embodiment, the original medical text is filtered by inputting a text filter; the text filter is a binary classifier using a ConvNeXt Tiny network, which trains the text filter by manually annotating each sentence as a valid or invalid sentence through sentence-level annotation of the medical text.

[0058] This embodiment presents a technical solution for filtering raw medical text. This embodiment filters the raw medical text by setting a text filter. The text filter is a binary classifier, outputting two categories: valid sentences and invalid sentences. This embodiment performs sentence-level manual data annotation on the medical text, performing binary classification annotation (valid sentence and invalid sentence) on each sentence, and uses a ConvNeXt Tiny network for binary classification task training. The trained model is the text filter. After a piece of text passes through the text filter, it is judged sentence by sentence, retaining valid sentences and removing invalid sentences. For example, the following raw medical text is input into the text filter:

[0059] Right lower lobe anterior basal segment nodule: highly likely peripheral lung cancer. Left lower lobe anterior medial basal segment small nodule: metastasis? Follow-up after treatment is recommended. Enlarged mediastinal lymph nodes. Bronchitis. Multiple renal cysts. Reviewing Physician: Liu Haipeng; Reporting Physician: Liu Haipeng; Issuing Physician: Qiu Bin. Report Date: June 14, 2019, 11:22:03. This report is for clinical reference only and is invalid without the issuing physician's (electronic) signature. XIMC-06-ZY-YL-0142018-A1.

[0060] The text filter outputs the following text:

[0061] Right lower lobe anterior basal segment nodule: highly likely peripheral lung cancer. Left lower lobe anterior medial basal segment small nodule: metastasis? Follow-up after treatment is recommended. Enlarged mediastinal lymph nodes. Bronchitis. Multiple renal cysts. Report date: June 14, 2019, 11:22:03.

[0062] As can be seen, the filtered text removed the invalid sentences "Reviewing Physician: Liu Haipeng; Reporting Physician: Liu Haipeng; Issuing Physician: Qiu Bin" and "This report is for clinical reference only. It is invalid without the issuing physician's (electronic) signature. XIMC-06-ZY-YL-0142018-A1".

[0063] As an optional embodiment, before the prompt information is integrated with the filtered text, it is first mapped by the spatial mapping module to the feature space in which it is located to the feature space that the large language generation model can understand.

[0064] In this embodiment, a spatial mapping module is set before the large language generation model to map the feature space where the prompt information is located to the feature space that the large language generation model can understand. The spatial mapping module includes a Transformer encoding layer and a fully connected layer.

[0065] As an optional embodiment, the primary classification includes at least cardiology, orthopedics, and neurology; the secondary classification under vascular medicine includes at least coronary heart disease, ischemic heart disease, stroke, aortic aneurysm, and renal artery stenosis; the secondary classification under orthopedics includes at least patellar fracture, ulnar nerve injury, congenital coxa vara, and infected costochondritis; and the secondary classification under neurology includes at least cerebral hemorrhage, cerebral embolism, headache, and myelitis.

[0066] This embodiment provides several specific primary and secondary classification categories. It is worth noting that this embodiment is only a preferred implementation method and does not exclude or negate other feasible implementation methods, such as those for different departments or different disease categories.

[0067] As an optional embodiment, the method for calculating the similarity includes:

[0068] The pre-trained word vector model Word2Vec is used to embed the labels to be classified, denoted as (x1, x2, ..., x...). n );

[0069] Let the output sequence of the large language generation model be denoted as (y1, y2, ..., y). n );

[0070] Calculate (x1, x2, ..., x) using the following formula n ) and (y1,y2,…,y n Cosine similarity of )

[0071]

[0072] In the formula, K cos Let be the cosine similarity.

[0073] This embodiment presents a technical solution for calculating the similarity between the output sequence of a large language generation model and the secondary classification labels. This embodiment first uses Word2Vec to embed the labels to be classified, and then calculates the cosine similarity between the output sequence and the embedded representation. The calculation formula is as shown above.

[0074] Figure 4 This is a schematic diagram illustrating the composition of a medical text classification device for prompting learning according to an embodiment of the present invention. The device includes:

[0075] The prompt information acquisition module 11 is used to acquire prompt information for primary classification from the original medical text based on prior event information and prior knowledge information, wherein the primary classification includes department category;

[0076] The information integration module 12 is used to filter the original medical text and integrate the filtered text with the prompt information before inputting it into the large language generation model.

[0077] The secondary classification module 13 is used to calculate the similarity between the result sequence output by the large language generation model and each secondary classification label representing a disease category under the primary classification, and to take the label category corresponding to the maximum value of the similarity as the secondary category output by the large language generation model.

[0078] The apparatus of this embodiment can be used to perform Figure 1 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0079] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A medical text classification method based on cue learning, characterized in that, Includes the following steps: Based on prior event information and prior knowledge information, prompts for primary classification are obtained from the original medical text, and the primary classification includes department categories; The original medical text is filtered, and the filtered text is integrated with the prompt information and then input into the large language generation model. Calculate the similarity between the output sequence of the large language generation model and each secondary classification label representing a disease category under the primary classification, and take the label category corresponding to the maximum similarity as the secondary category output by the large language generation model; The process of obtaining prompts for primary classification includes: The original medical text is input into the medical event feature extraction model to obtain structured event prompt information A; The original medical text is input into the medical knowledge feature extraction model to obtain structured knowledge prompt information B; After integrating A and B, the results are output to the prompt template generation module to obtain prompt information for the primary category.

2. The medical text classification method based on cue learning according to claim 1, characterized in that, The medical event feature extraction model is a pre-trained RoBERTa model, which is fine-tuned using a dataset composed of publicly available medical information.

3. The medical text classification method based on cue learning according to claim 1, characterized in that, The medical knowledge feature extraction model is a pre-trained BioBERT model, which is fine-tuned using a dataset composed of publicly available medical knowledge graph data.

4. The medical text classification method based on cue learning according to claim 1, characterized in that, The prompt template generation module consists of four identical network layers, with each layer employing a Transformer decoder to remove the second attention layer structure.

5. The medical text classification method based on cue learning according to claim 1, characterized in that, The original medical text is filtered by inputting a text filter; the text filter is a binary classifier using a ConvNeXt Tiny network. By manually annotating the medical text at the sentence level, each sentence is labeled as a valid sentence or an invalid sentence, thereby training the text filter.

6. The medical text classification method based on cue learning according to claim 1, characterized in that, Before being integrated with the filtered text, the prompt information is first mapped from its feature space to a feature space that the large language generation model can understand, via a spatial mapping module.

7. The medical text classification method based on cue learning according to claim 1, characterized in that, The primary classification includes at least cardiology, orthopedics, and neurology; the secondary classification under vascular medicine includes at least coronary heart disease, ischemic heart disease, stroke, aortic aneurysm, and renal artery stenosis; the secondary classification under orthopedics includes at least patellar fracture, ulnar nerve injury, congenital coxa vara, and infectious costochondritis; and the secondary classification under neurology includes at least cerebral hemorrhage, cerebral embolism, headache, and myelitis.

8. The medical text classification method based on cue learning according to claim 1, characterized in that, The method for calculating the similarity includes: The pre-trained word vector model Word2Vec is used to embed the labels to be classified, denoted as (x1, x2, ..., x...). n ); Let the output sequence of the large language generation model be denoted as (y1, y2, ..., y). n ); Calculate (x1, x2, ..., x) using the following formula n ) and (y1,y2,…,y n Cosine similarity of ) In the formula, K cos Let be the cosine similarity.

9. A medical text classification device based on cue learning, characterized in that, include: The prompt information acquisition module is used to acquire prompt information for primary classification from the original medical text based on prior event information and prior knowledge information, wherein the primary classification includes department category; The information integration module is used to filter the original medical text and integrate the filtered text with the prompt information before inputting it into the large language generation model; The secondary classification module is used to calculate the similarity between the result sequence output by the big language generation model and each secondary classification label representing a disease category under the primary classification, and the label category corresponding to the maximum similarity is taken as the secondary category output by the big language generation model. The process of obtaining prompts for primary classification includes: The original medical text is input into the medical event feature extraction model to obtain structured event prompt information A; The original medical text is input into the medical knowledge feature extraction model to obtain structured knowledge prompt information B; After integrating A and B, the results are output to the prompt template generation module to obtain prompt information for the primary category.

Images