A fetal congenital heart disease diagnosis and treatment text generation method and system based on a pre-trained language model

Abstract

The application discloses a kind of based on pre-training language model's fetal congenital heart disease diagnosis and treatment text generation method and system, it is related to fetal congenital heart disease diagnosis and treatment technical field.The method is by selecting biomedical field pre-training model, and further fine-tuning is carried out in fetal congenital heart disease field, while constructing the knowledge graph of fetal congenital heart disease, and using search enhancement generation technology real-time into generation process with professional knowledge.The direction of generation is guided by designing structured prompt word, and an expert verification and feedback mechanism is introduced for iterative optimization, and finally a high-quality, personalized simulation diagnosis and treatment text that meets clinical practice is generated.The application effectively solves the problem of professional, accuracy and logical consistency in medical text generation, and the generated text can be used for decision support, clinical teaching and research data enhancement, with high fidelity, personalization and real-time knowledge advantages.

Description

Technical Field

[0001] This invention relates to the field of fetal congenital heart disease diagnosis and treatment technology, and in particular to a method for generating text for fetal congenital heart disease diagnosis and treatment based on a pre-trained language model. Background Technology

[0002] Congenital heart disease (CHD) is a structural malformation of the heart caused by abnormal development of the heart and blood vessels during fetal development. It is the most common birth defect in my country and one of the leading causes of neonatal death. Diagnostic and treatment documents for CHD, such as ultrasound reports, surgical protocol records, and postoperative follow-up summaries, are important knowledge carriers for clinical teaching, physician training, and research analysis. However, the current generation of such high-quality, structured case texts heavily relies on manual writing by experienced physicians, facing bottlenecks of low efficiency, limited scale, and susceptibility to subjective experience. In recent years, although natural language generation technology based on pre-trained large language models (such as the GPT series) has achieved significant success in general fields, its direct application in the highly specialized and precise medical scenario of fetal CHD faces the following key problems that urgently need to be addressed.

[0003] 1. Lack of Domain Knowledge and Inaccurate Terminology. While the training corpora of general pre-trained language models cover a wide range, they lack in-depth, structured medical expertise. When directly used to generate congenital heart disease cases, the models are prone to "illusion" problems such as incorrect descriptions of anatomical locations, confusion of diagnostic criteria, and inappropriate use of intervention terminology. For example, they may confuse the pathological features of "ventricular septal defect" with "atrial septal defect." Although models such as BioBERT and ClinicalBERT, which are further trained on biomedical literature, provide a starting point for solutions, they still lack deep adaptation to the specific subfield of "fetal congenital heart disease," and their knowledge is broad rather than precise.

[0004] 2. Lagging Knowledge Updates and Static Nature: Medical knowledge, especially intervention guidelines and techniques, is constantly and rapidly updated. Language models trained on static data have knowledge up to the point in time of the training data and cannot automatically integrate the latest clinical research evidence and guideline changes. This inherent static nature means that the content generated by the model may contain outdated or even refuted treatment plans, failing to meet the stringent requirements of timely knowledge in clinical practice. Existing methods lack a mechanism to dynamically introduce and integrate the latest authoritative knowledge.

[0005] 3. Uncontrollable and Logically Disjointed Generation Process. Using simple open-ended prompts (such as "Write a case about congenital heart disease in a fetus"), the logical consistency between the generated content and the disease progression, examination, intervention, and prognosis is difficult to guarantee. Mismatches may occur between intervention methods and disease classifications, or prognostic descriptions may contradict the severity of the condition. Current technology lacks an effective method to strongly constrain the generation process with clinical pathways and medical logic, resulting in low clinical credibility of the generated text.

[0006] 4. Lack of professional evaluation and absence of an optimization loop. In the medical field, the quality of generated text cannot be judged solely by general indicators such as fluency and grammar; clinical accuracy is paramount. Most existing solutions rely on one-time generation or use automated metrics (such as BLEU and ROUGE) for evaluation, lacking a systematic verification and feedback process deeply involved by domain experts (such as pediatric cardiologists). Without this loop, the model cannot learn from errors, hindering continuous performance improvement and clinical alignment, and posing potential application risks.

[0007] In summary, existing technologies suffer from a core contradiction: the conflict between the capabilities of general generative models and the extremely high requirements for accuracy, timeliness, logical consistency, and verifiability in the medical vertical field. Simply applying or fine-tuning existing models cannot fundamentally solve these problems. Therefore, there is an urgent need for a systematic and innovative approach that organically integrates domain knowledge modeling, real-time knowledge retrieval, structured guided generation, and expert feedback loops to achieve the automatic generation of high-quality, highly reliable, and iteratively optimizable diagnostic texts for congenital heart disease in fetuses. Summary of the Invention

[0008] To obtain fetal congenital heart disease (CHD) diagnosis and treatment texts more efficiently and reliably, this invention provides a method and system for generating CHD diagnosis and treatment texts based on a pre-trained language model. This addresses bottlenecks in clinical teaching and research caused by the scarcity of high-quality clinical case data, low efficiency of manual writing, and insufficient professionalism in content generated by general models. The core of the method provided by this invention lies in the systematic integration of a pre-trained language model, dynamic knowledge graph, retrieval enhancement, and expert feedback mechanisms to achieve the automatic generation of high-quality, personalized CHD diagnosis and treatment texts that conform to clinical practice standards. This is suitable for medical education, research data enhancement, and decision support. This invention is specifically implemented through the following technologies.

[0009] This invention provides a method for generating text for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model, comprising the following steps:

[0010] A pre-trained language model in the biomedical field was selected as the base model, and LoRA fine-tuning was performed on the base model in the field of fetal congenital heart disease.

[0011] Collect medical data related to congenital heart disease in fetuses and construct a corresponding knowledge graph;

[0012] Design a structured prompt word template, which contains required key fields for guiding the generation of simulated diagnosis and treatment text, and dynamically fills them based on the content of the knowledge graph;

[0013] Upon receiving a request to generate simulated medical text, a real-time retrieval is performed in the knowledge graph, and the retrieved knowledge is added to the input of the generation model to guide the model to generate simulated medical text that conforms to medical facts.

[0014] The simulated diagnosis and treatment text generated by the model is mixed with the real diagnosis and treatment text, and then manually verified to obtain feedback. Based on the feedback, the model parameters and prompt word templates are iteratively optimized to obtain the final model.

[0015] Input prompts and output corresponding simulated medical text.

[0016] Furthermore, the underlying model is based on the Transformer architecture and has been pre-trained on a biomedical corpus.

[0017] Furthermore, the biomedical corpus consists of MIMIC-III clinical notes or PubMed biomedical literature.

[0018] Furthermore, the underlying model includes, but is not limited to, BioGPT or BioBERT.

[0019] Furthermore, the method for LoRA fine-tuning of the basic model in the field of fetal congenital heart disease is as follows:

[0020] An initial dataset was constructed based on professional literature and desensitized case texts related to congenital heart disease in fetuses.

[0021] Freeze the original parameters of the base model, and in the self-attention mechanism of each Transformer module of the base model, inject trainable LoRA adapters into the query projection matrix and the value projection matrix in parallel.

[0022] The original weight matrix in the basic model The forward propagation process is modified to

[0023] ;

[0024] in, and The low-rank matrix introduced for LoRA, where r is the rank, and x is the input activation value; during fine-tuning, the original weight matrix W is frozen, and only the low-rank matrices B and A are updated;

[0025] Using the initial dataset as training corpus, and with masked language modeling and next sentence prediction as training objectives, the hyperparameters of the LoRA adapter are fine-tuned.

[0026] After fine-tuning, the trained LoRA adapter weights B and A are merged with the frozen base model weights W; or, the model is dynamically enhanced during inference by a lightweight additional computation BAx.

[0027] Furthermore, the hyperparameters of the LoRA adapter are fine-tuned as follows: LoRA rank r=8, LoRA scaling parameter α=16, learning rate is set to 5e-4, batch size is 16, training epochs are 3-5, and the Adam optimizer is used to avoid overfitting.

[0028] Furthermore, the loss function for mask language modeling is:

[0029] ;

[0030] Where M is the set of mask positions, x i For the masked word, x \M For context;

[0031] Furthermore, the loss function for the next prediction is:

[0032] ;

[0033] IsNext indicates whether the two sentences are consecutive.

[0034] Furthermore, the method for collecting medical data related to congenital heart disease in fetuses and constructing a corresponding knowledge graph is as follows:

[0035] Extract entities and relationships related to congenital heart disease in fetuses from authoritative structured databases;

[0036] The extracted entities and relationships are organized into a graph structure to form the knowledge graph in the field of congenital heart disease in fetuses; where nodes represent entities and edges represent relationships between entities.

[0037] The knowledge graph is stored and updated in real time.

[0038] Furthermore, the authoritative sources include, but are not limited to, authoritative medical textbooks, UpToDate, clinical guidelines, academic papers, and structured databases of hospital electronic medical record systems.

[0039] Furthermore, the entities include, but are not limited to, disease types, anatomical structures, diagnostic methods, interventions, drugs, and prognostic indicators.

[0040] The relationships mentioned include, but are not limited to, anatomical relationships, disease-related relationships, diagnostic relationships, treatment relationships, and prognostic relationships.

[0041] Specifically, the relationship includes, but is not limited to:

[0042] (1) Anatomical relationships: such as location (e.g., "the heart is located in the thoracic cavity"), branches (e.g., "the branches of the aorta are the ascending aorta and the aortic arch"), and adjacency (e.g., "the left ventricle is adjacent to the interventricular septum").

[0043] (2) Disease-related relationships: such as a type (e.g., "Tetralogy of Fallot is a type of cyanotic congenital heart disease"), clinical manifestations (e.g., "Clinical manifestations of ventricular septal defect include heart murmurs"), complications (e.g., "VSD complications include pulmonary hypertension"), etiology (e.g., "Chromosome 22q11.2 microdeletion is the cause of conus trunk malformation").

[0044] (3) Diagnostic relationships: such as diagnostic methods (e.g., "echocardiography is used to diagnose ventricular septal defects"), diagnostic indicators (e.g., "the diameter of the defect is a diagnostic indicator of the severity of VSD").

[0045] (4) Treatment relationship: such as treatment used (e.g., "interventional closure is used for perimembranous ventricular septal defect"), medication used (e.g., "indomethacin is used to promote ductus arteriosus closure"), surgical procedure (e.g., "establishing vascular access is a surgical procedure of interventional closure"), contraindications (e.g., "severe pulmonary hypertension is a contraindication of interventional closure").

[0046] (5) Prognostic relationship: such as affecting prognosis (e.g., "the size of the defect affects the prognosis of VSD"), outcome (e.g., "small VSD may be naturally closed").

[0047] Furthermore, the methods for updating the knowledge graph include, but are not limited to, periodically crawling the latest knowledge from authoritative sources and combining it with feedback from manual verification.

[0048] Furthermore, the required key fields include, but are not limited to: case type, fetal gestational age, specific type of congenital heart disease, main clinical manifestations, proposed intervention method, and expected text length.

[0049] Furthermore, the method for dynamically filling in the knowledge graph content is as follows: when the user inputs or the system specifies a value for a certain required key field, the system automatically retrieves typical values ​​or reasonable options for other fields most relevant to that value from the knowledge graph and fills in the value.

[0050] For example, when a user specifies the disease type as "ventricular septal defect", the system automatically retrieves and populates the common intervention methods from the knowledge graph as "interventional closure" or "surgical repair".

[0051] Furthermore, natural language sentence structures are used to combine the various required key fields.

[0052] Furthermore, upon receiving the request to generate the simulated medical text, a real-time retrieval is performed within the knowledge graph, and the retrieved knowledge is incorporated into the input of the generation model to guide the model in generating simulated medical text that conforms to medical facts.

[0053] Upon receiving a request to generate simulated medical text, the key information in the request is parsed and used as query conditions.

[0054] Based on the query conditions, the knowledge graph is searched in real time to obtain several knowledge subgraphs or entity relationship pairs that are most relevant to the current generation request.

[0055] The knowledge subgraph or entity relationship pairs are converted into natural language descriptions to form an enhanced context.

[0056] The enhanced context and the prompt word template are concatenated and used together as input to the basic model to guide the model in generating simulated diagnosis and treatment text that conforms to medical facts.

[0057] The criterion for determining "most relevant" is to select the top K items based on their similarity from highest to lowest.

[0058] Furthermore, the real-time retrieval is performed using a vector retrieval method.

[0059] Optionally, the real-time retrieval method is as follows: embed the entities and relationships in the knowledge graph into vectors, and return the most relevant Top-K knowledge fragments by calculating the similarity between the query vector and the knowledge vector;

[0060] Optionally, the similarity is calculated using the cosine similarity formula.

[0061] Furthermore, the simulated medical text generated by the model is manually verified, and the model parameters and prompt word templates are iteratively optimized based on the feedback obtained from the manual verification.

[0062] The simulated medical texts are mixed with real medical texts and then given to medical experts in the relevant medical fields for blind review and evaluation.

[0063] A feedback optimization loss function is constructed based on the feedback data from the blind review evaluation, and the model parameters are fine-tuned using the gradient descent algorithm.

[0064] Analyze the common problems related to the prompt word template in the feedback data, and optimize and update the required key fields of the prompt word template.

[0065] Furthermore, the scoring dimensions of the blind review assessment include clinical accuracy, logical consistency, and linguistic fluency.

[0066] Furthermore, it is recommended that the optimization updates be performed at least once a month.

[0067] Furthermore, the feedback optimization loss function is composed of the generation loss L gen And utility loss L utility The weighted summation is calculated using the following formula:

[0068] ;

[0069] Where α is a hyperparameter.

[0070] This invention also provides a fetal congenital heart disease diagnosis and treatment text generation system based on a pre-trained language model, including a knowledge graph construction module, a retrieval enhancement module, a prompt word design module, and a verification optimization module;

[0071] The knowledge graph construction module is used to collect medical data related to congenital heart disease in fetuses and construct the corresponding knowledge graph.

[0072] The retrieval enhancement module is used to perform real-time retrieval in the knowledge graph when a request to generate simulated medical text is received, and to enhance the retrieved knowledge into the input of the generation model, guiding the model to generate simulated medical text that conforms to medical facts.

[0073] The prompt word design module is used to design prompt word templates, which contain required key fields for generating the simulated diagnosis and treatment text and are dynamically filled based on the content of the knowledge graph.

[0074] The verification and optimization module is used to manually verify the simulated diagnosis and treatment text generated by the model, and to iteratively optimize the model parameters and prompt word templates based on the feedback obtained from the manual verification.

[0075] Compared with the prior art, the advantages of the present invention are:

[0076] 1. High fidelity. The simulated medical texts generated using the method of this invention conform to clinical standards, with accurate terminology and logical coherence.

[0077] 2. High degree of personalization. The method of this invention supports the generation of customized cases based on specific parameters (such as gestational age and disease type).

[0078] 3. Ensuring the timeliness of knowledge. This invention integrates the latest medical knowledge through a retrieval enhancement mechanism.

[0079] 4. Scalable. The modular design makes it easy to adapt to other medical text generation tasks. Attached Figure Description

[0080] Figure 1 This is a schematic diagram illustrating the steps of the method for generating simulated diagnosis and treatment text for congenital heart disease in fetuses based on a pre-trained language model, as provided by the present invention.

[0081] Figure 2 This is a schematic diagram of the knowledge graph formed in the embodiment. Detailed Implementation

[0082] The technical solution of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] Example 1: A method for generating text for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model

[0084] This embodiment provides a method for generating text for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model, such as... Figure 1 As shown, it includes the following steps.

[0085] 1. Basic Model Selection

[0086] This step is fundamental to the methodology and aims to ensure that the model possesses medical domain knowledge:

[0087] When selecting a model, it is necessary to consider the balance between model size and computing resources. Lightweight models with 100 million to 300 million parameters are recommended.

[0088] For example, one could choose a Transformer-based model pre-trained on biomedical corpora (such as PubMed or MIMIC-III), such as BioGPT or BioBERT (variants pre-trained on MIMIC-III or PubMed corpora). These models can effectively capture the semantic relationships of terms related to congenital heart disease.

[0089] 2. LoRA fine-tuning of the basic model in the field of fetal congenital heart disease.

[0090] Efficient fine-tuning of the base model's parameters can be achieved using Low-Rank Adaptation (LoRA) technology. The core principle of LoRA fine-tuning is to use low-rank matrix factorization to approximate weight updates by training only newly added small-scale low-rank matrices (A and B) while freezing the original parameters of the pre-trained model, thus achieving efficient and low-cost task adaptation. Specifically, it includes the following steps:

[0091] (1) Based on professional knowledge related to congenital heart disease in fetuses, an initial dataset was constructed; Before performing adaptive training on the base model in relevant medical fields, an initial dataset needs to be prepared for iterative training of the model. This dataset is the foundation for the model to learn knowledge in the field of fetal congenital heart disease and mainly consists of knowledge from sources such as professional literature texts, anonymized case texts, and hospital diagnosis and treatment reports.

[0092] Specifically, the initial dataset comes from two main parts:

[0093] ① Professional literature texts: Chapters or abstracts from publicly available unstructured texts such as academic papers, clinical guidelines, and textbooks related to congenital heart disease in fetuses, after being de-anonymized.

[0094] ② De-identified medical record texts: Clinical texts related to congenital heart disease in fetuses obtained from the information systems of partner hospitals, which have undergone strict de-identification processing, such as ultrasound report summaries and consultation record excerpts. All data has had personally identifiable information removed to ensure compliance with privacy protection regulations.

[0095] (2) Freeze the original parameters of the base model, and inject trainable LoRA adapters into the query projection matrix and value projection matrix in parallel in the self-attention mechanism of each Transformer module of the base model;

[0096] For the original weight matrix Its forward propagation process is modified as follows:

[0097] ;

[0098] in, and The low-rank matrix introduced for LoRA; r is the rank, and x is the input activation value; during fine-tuning, the original weight matrix W is frozen, and only the low-rank matrices B and A are updated;

[0099] (3) Using the initial dataset as training corpus, and with masked language modeling and next sentence prediction as training objectives, the hyperparameters of the LoRA adapter are fine-tuned;

[0100] The hyperparameters of the LoRA adapter are fine-tuned as follows: LoRA rank r=8, LoRA scaling parameter α=16, learning rate is set to 5e-4, batch size is 16, training epochs are 3-5, and the Adam optimizer is used to avoid overfitting.

[0101] Specifically, the loss function for mask language modeling is:

[0102] ;

[0103] Where M is the set of mask positions, x i For the masked word, x \M For context;

[0104] Specifically, the loss function for the next prediction is:

[0105] ;

[0106] Here, IsNext indicates whether the two sentences are in a continuous relationship, and this probability is predicted by the model.

[0107] (4) Model deployment and inference

[0108] After fine-tuning, the trained LoRA adapter weights B and A are merged with the frozen base model weights W to obtain the complete fine-tuned model for inference deployment.

[0109] Alternatively, the original model can be dynamically enhanced during inference by using lightweight additional computation BAx without changing the original model parameters, thus achieving efficient multi-task adaptation.

[0110] The LoRA fine-tuning method described above significantly reduces the number of trainable parameters (only 0.01-1% of the original model parameters need to be updated), while maintaining performance similar to full parameter fine-tuning. It greatly reduces computational overhead and storage requirements, improves training efficiency, and facilitates the maintenance and management of multiple lightweight LoRA adapters for different fetal congenital heart disease text generation tasks.

[0111] 3. Knowledge Graph Construction

[0112] Knowledge graphs provide structured knowledge support for this method, and the construction process includes:

[0113] (1) Knowledge extraction

[0114] Extract entities (such as disease types, anatomical structures, diagnostic methods, interventions, drugs, prognostic indicators, etc.) and relationships (such as anatomical relationships, disease-related relationships, diagnostic relationships, treatment relationships, prognostic relationships, etc.) from authoritative sources (such as UpToDate, clinical guidelines, academic papers).

[0115] Specifically, the relationship includes, but is not limited to:

[0116] ① Anatomical relationships: such as location (e.g., "the heart is located in the thoracic cavity"), branches (e.g., "the branches of the aorta are the ascending aorta and the aortic arch"), and adjacency (e.g., "the left ventricle is adjacent to the interventricular septum").

[0117] ② Disease-related relationships: such as a type (e.g., "Tetralogy of Fallot is a type of cyanotic congenital heart disease"), clinical manifestations (e.g., "Clinical manifestations of ventricular septal defect include heart murmurs"), complications (e.g., "VSD complications include pulmonary hypertension"), etiology (e.g., "Chromosome 22q11.2 microdeletion is the cause of conus medullaris").

[0118] ③ Diagnostic relationships: such as diagnostic methods (e.g., "echocardiography is used to diagnose ventricular septal defects") and diagnostic indicators (e.g., "the diameter of the defect is a diagnostic indicator of the severity of VSD").

[0119] ④ Treatment relationship: such as treatment used (e.g., "interventional closure is used for perimembranous ventricular septal defects"), medication used (e.g., "indomethacin is used to promote ductus arteriosus closure"), surgical procedure (e.g., "establishing vascular access is a surgical procedure of interventional closure"), contraindications (e.g., "severe pulmonary hypertension is a contraindication of interventional closure").

[0120] ⑤ Prognostic relationship: such as affecting prognosis (e.g., "the size of the defect affects the prognosis of VSD"), outcome (e.g., "small VSD may result in natural closure").

[0121] BERT-based Named Entity Recognition (NER) and relation extraction models (e.g., BERT-based extraction models) are employed to ensure data accuracy and integrity. The entity recognition process is represented as follows:

[0122] ;

[0123] ;

[0124] Where E is the set of entities and R is the set of relations.

[0125] At the same time, data is directly mapped from the structured database of the hospital's electronic medical record system.

[0126] (2) Atlas organization

[0127] ① Use a graph database (such as Neo4j) to store knowledge, where nodes represent entities and edges represent relationships, such as... Figure 2 As shown.

[0128] For example, a congenital heart disease knowledge graph can contain nodes such as "ventricular septal defect" and "interventional treatment", with edges representing "treatment options".

[0129] ② The atlas needs to be updated regularly (e.g., quarterly) to reflect medical progress. The update mechanism includes regularly crawling the latest knowledge from authoritative sources and updating it in conjunction with expert review.

[0130] 4. Structured prompt word template design, dynamically filled based on knowledge graph.

[0131] (1) Prompt word template design

[0132] The above format can be flexibly controlled by adjusting fields such as [Case Type] in the text generation request prompt to meet the needs of different scenarios. The design principles of the text generation request prompt are as follows:

[0133] ① Prompt word template structure

[0134] The prompt template includes required key fields, such as: case type, fetal gestational age, specific type of congenital heart disease, main clinical manifestations, proposed intervention method, and expected text length.

[0135] The example template is in natural language sentence format. For example: "Generate a detailed case of [case type], for a fetus at [gestational week], diagnosed with [congenital heart disease type], exhibiting [clinical manifestations], receiving [intervention method] treatment, approximately [number of words]."

[0136] ② Dynamic filling

[0137] Template fields are dynamically populated based on knowledge graph content to ensure personalized and accurate prompts. Specifically, when a user inputs or the system specifies a value for a key field, the system automatically retrieves typical values ​​or reasonable options from other fields most relevant to that value from the knowledge graph and populates the prompts.

[0138] For example, if the search results show that a certain disease is suitable for a specific treatment, the intervention method field in the suggestion words will be automatically adjusted. For example, when the user specifies the disease type as "ventricular septal defect", the system will automatically retrieve and populate the common intervention methods from the knowledge graph as "interventional closure" or "surgical repair".

[0139] Then, natural language sentences are used to combine the required key fields.

[0140] (2) Reference for the design format of simulated diagnosis and treatment text

[0141] The simulated medical records should cover various medical document formats throughout the entire process of fetal congenital heart disease diagnosis and treatment, including but not limited to:

[0142] ① Prenatal ultrasound diagnostic report: focuses on an objective description of cardiac anatomy and hemodynamics;

[0143] ② Multidisciplinary consultation records: focusing on the comprehensive assessment opinions of obstetrics, pediatrics, and cardiac surgery experts;

[0144] ③ Doctor-patient communication records: These records focus on providing informed consent by explaining the patient's condition, prognosis risks, and intervention costs in plain language.

[0145] ④ Perinatal management plan: covering the process planning of intrauterine monitoring and immediate transfer after birth;

[0146] ⑤ Surgical plan: A detailed plan of the surgical procedures to be developed after birth.

[0147] 5. Upon receiving a request to generate simulated medical text, the system searches the knowledge graph in real time and incorporates the retrieved knowledge into the input of the generation model, guiding the model to generate simulated medical text that conforms to medical facts.

[0148] During the generation of simulated medical text, knowledge graphs are retrieved in real time to dynamically introduce external knowledge and improve text quality. The specific methods are as follows.

[0149] (1) Search enhancement mechanism

[0150] ① When a prompt for a text generation request is received, the key information (such as disease type) in the prompt is parsed and used as a query condition.

[0151] ② Vectorize the entities and relationships in the knowledge graph (e.g., using embedding models such as Sentence-BERT) and perform vector similarity retrieval in the knowledge graph.

[0152] Specifically, vector similarity retrieval uses the FAISS (Facebook AI Similarity Search) tool, embedding knowledge entities and relationships as vectors. Similarity is calculated using the following cosine similarity formula:

[0153] ;

[0154] Where q is the query vector and k is the knowledge vector.

[0155] The Top-K (K=5~10) related knowledge points are returned by calculating cosine similarity.

[0156] (2) Knowledge integration to generate simulated diagnosis and treatment text

[0157] ① Convert search results into natural language descriptions to create enhanced context. For example, "ventricular septal defects are commonly treated with interventional therapy."

[0158] ② The enhanced context and prompt words are concatenated and used together as input to the basic model.

[0159] ③ The model uses an attention mechanism to weight and fuse this information to generate a simulated diagnosis and treatment text that ultimately conforms to medical facts.

[0160] The fusion weights can be adjusted through training to balance internal knowledge and external retrieval. The attention weights are calculated using the following formula:

[0161] ;

[0162] Among them, h 模型 h represents the hidden state of the model. 知识 Let d be the knowledge vector, and d be the dimension.

[0163] In the domain-adaptive training phase, the selected base model is trained using a predefined training set. The objective functions are Masked Language Modeling (MLM) and Next Sentence Prediction Loss (NSP) to enhance the model's understanding of relevant domain terminology and clinical context.

[0164] 6. Verification and Optimization

[0165] This step involves iteratively optimizing the model and prompts based on expert feedback to ensure the clinical accuracy of the generated text and achieve continuous improvement.

[0166] (1) Expert verification

[0167] An expert verification interface was established. Simulated clinical texts generated by the model were randomly mixed with real clinical texts, and then pediatric cardiology experts were invited to conduct blind reviews of the generated texts. The scoring dimensions included clinical accuracy (0-10 points), logical consistency, and language fluency. The evaluation results (feedback data) were recorded as structured data.

[0168] (2) Iterative optimization

[0169] ① Based on expert feedback, calculate utility loss (such as rating discrepancy) and weight it with generation loss (such as cross-entropy) to form the total loss function:

[0170] ;

[0171] Among them, L gen To generate loss (such as cross-entropy loss); L utility α represents the utility loss (related to expert ratings, such as mean squared error); α is a hyperparameter (recommended value 0.5-1.0).

[0172] ② Update the model parameters using gradient descent, and simultaneously analyze common problems related to the prompt word template in the feedback data to optimize the required key fields of the prompt word template. For example, add required key fields or modify the descriptions of required key fields.

[0173] Generally, common problems include:

[0174] Specific clinical details are missing or vague: The generated text does not provide sufficient detail on key prognostic factors, contraindications, or indications for personalized treatment.

[0175] For example, it repeatedly generates general statements such as "good long-term prognosis", but lacks differentiated assessments for children of different weights and the risk of specific complications (such as pulmonary hypertension).

[0176] The text structure does not conform to clinical standards: the generated documents of different types (such as "preoperative assessment" and "postoperative follow-up" reports) have similar structures and fail to reflect their respective focuses.

[0177] For example, the preoperative assessment lacks a specific section on "surgical indications," while the postoperative follow-up provides an excessively detailed description of the surgical procedures.

[0178] Inappropriate or abstract use of medical terminology: Too many technical terms were used in scenarios requiring colloquial explanation (such as "doctor-patient communication records"); while overly colloquial expressions were used in clinical reports requiring rigor (such as "multidisciplinary consultation records"), failing to automatically adjust the language style according to the case type.

[0179] A broken logical chain: There is a weak logical connection between diagnostic findings and intervention plans.

[0180] For example, the prompt clearly states "aortic stenosis," but the generated text does not fully reflect the key decision-making basis of "the need to assess the development of the left ventricle" when formulating the intervention plan.

[0181] To address the common issues mentioned above, the method for optimizing the prompt word template is as follows:

[0182] Add required key fields: Add structured fields to the template, mandating that the model must include the corresponding content. For example: Add a [Prognostic Assessment] field to the template and set it as required. The example is modified to: "...must include short-term (within 1 year) and long-term (more than 5 years) prognostic assessments, with special explanation of the impact of [weight factors] and [common complications] on the prognosis."

[0183] Modify the field description guidance: Optimize the descriptive text of the fields themselves to better guide the model to focus on key points. Change the description of the intervention method in the prompt words from "receive [intervention method] treatment" to "based on [diagnostic findings] and [clinical manifestations], [intervention method] is proposed as the preferred option, and its logical connection with the condition is explained."

[0184] Introduce negative example constraints: explicitly prohibit certain vague or inappropriate expressions in the prompts. Add instructions at the end of the template, such as: "Note: If the case type is 'patient-doctor communication record,' avoid using professional abbreviations such as 'VSD' and 'interventional closure,' and use more colloquial terms such as 'ventricular septal defect' and 'minimally invasive interventional surgery.'"

[0185] The optimization cycle is recommended to be once a month to ensure model adaptability.

[0186] 7. Iterate model training and deploy it in practical applications.

[0187] The model is continuously improved through training iterations. In practical applications, the user inputs prompt words, and the model generates and outputs simulated diagnosis and treatment text that conforms to medical facts.

[0188] Example 2

[0189] This embodiment provides a system for generating text for the diagnosis and treatment of congenital heart disease in fetuses in Embodiment 1. The system includes a knowledge graph construction module, a retrieval enhancement module, a prompt word design module, and a verification optimization module.

[0190] The knowledge graph construction module is used to collect medical data related to congenital heart disease in fetuses and construct the corresponding knowledge graph.

[0191] The retrieval enhancement module is used to perform real-time retrieval in the knowledge graph when a request to generate simulated medical text is received, and to enhance the input of the generation model with the retrieved knowledge, thereby guiding the model to generate simulated medical text that conforms to medical facts.

[0192] The prompt word design module is used to design prompt word templates, which contain required key fields for generating the simulated diagnosis and treatment text, and are dynamically filled based on the content of the knowledge graph.

[0193] The verification and optimization module is used by medical experts to manually verify the simulated diagnosis and treatment text generated by the model, and to iteratively optimize the model parameters and prompt word templates based on the feedback from the medical experts.

[0194] Application examples

[0195] This application example uses the generation of a "ventricular septal defect fetal interventional treatment case" as an example, and uses the method of the above embodiment to obtain a personalized simulated diagnosis and treatment text that meets clinical standards.

[0196] 1. User inputs prompt words

[0197] For example, the user provides the following text generation request prompt: "Please generate a professional clinical case report of congenital heart disease in a fetus. Describe a case of a fetus at 24 weeks of gestation diagnosed by echocardiography with a 'perimembranous ventricular septal defect (3 mm in diameter)', assessed as an isolated, restrictive defect with minimal intrauterine impact. The proposed treatment plan is to perform 'percutaneous catheter closure' at 3-6 months of age and a birth weight of over 5 kg. The report should be complete, including diagnostic details, multidisciplinary consultation, preoperative examinations, surgical plan, postoperative follow-up plan, and prognostic assessment, with a total length of approximately 400 words."

[0198] 2. Processing flow

[0199] (1) Text request parsing and knowledge retrieval

[0200] ① After receiving the text request input by the user, the system first performs in-depth analysis of the prompt words based on predefined grammatical rules in order to accurately extract structured key information.

[0201] "Predefined syntax rules" specifically refer to:

[0202] Rule-based pattern matching: For highly structured keywords, regular expressions are used for precise matching. For example, the pattern "pregnancy(\d+) weeks" is defined to extract the "pregnancy week" field, and the pattern "diagnosis" is defined as ['"]([^'"]+)['"] to extract the "disease type" enclosed in quotation marks.

[0203] Medical field dictionary matching: Establish a professional terminology dictionary in the field of fetal congenital heart disease (such as a disease classification dictionary and an intervention method dictionary), and identify professional terms appearing in prompt words through exact string matching or fuzzy matching.

[0204] Dependency parsing: This involves performing syntactic analysis on a sentence to identify the dependency relationships between components such as subject, predicate, and object, thereby accurately associating attributes with entities. For example, in the sentence "diagnosed as a perimembranous ventricular septal defect (3 mm in diameter)," analysis can establish that "perimembranous ventricular septal defect" is the object of "diagnosis," while "3 mm in diameter" is its attribute modifier.

[0205] By combining the above rules, the system can reliably extract key target fields from the diverse natural language expressions of users. For example, it can accurately extract from user input: target case type (“clinical case report”), fetal gestational age (“24 weeks”), congenital heart disease classification (“perimembranous ventricular septal defect”), key clinical manifestations (“diameter 3 mm”, “isolated, restrictive defect”), and intervention method (“percutaneous catheter closure”).

[0206] ②Then, the system will convert the extracted key information (especially disease classification and intervention methods) into query vectors.

[0207] The query vector is generated using the Sentence-BERT embedding model to ensure its semantic space consistency with entity vectors in the knowledge graph.

[0208] The system uses the FAISS vector retrieval tool to calculate the cosine similarity between the query vector and all entity vectors in the pre-built knowledge graph of congenital heart disease in fetuses.

[0209] This search was set to K=8, returning the top 8 knowledge fragments most relevant to "perimembranous ventricular septal defect" and "interventional closure". For example: "Indications for interventional treatment of perimembranous VSD: defect diameter 3-10 mm", "The optimal intervention window for isolated VSD is 3-6 months after birth", "Common complications of interventional closure: residual shunt, occluder displacement", etc.

[0210] (2) Enhancing input construction for retrieval

[0211] The retrieved knowledge fragments are converted into coherent natural language descriptions, forming an enhanced context.

[0212] For example: "According to the guidelines, if the diameter of a perimembranous ventricular septal defect is within the range of 3-10 mm and it is isolated, it is suitable to perform percutaneous catheter-based closure 3-6 months after birth and after the target weight is achieved. Attention should be paid to the risk of complications such as residual shunt after the operation."

[0213] Subsequently, the enhanced context and the original user prompts are concatenated using structured tags with explicit semantic boundary functions to form the final input of the pre-trained language model.

[0214] Specifically, this method uses XML-like tags (e.g., <knowledge> Enhanced context< / knowledge> and <query> Original prompt words for users< / query> This clearly defines the different modules, thereby explicitly informing the model of the roles of each part.

[0215] This step ensures that the generation process is not only based on the model's implicit knowledge, but is also guided and constrained in real time by external authority and structured knowledge.

[0216] (3) Simulated diagnosis and treatment text generation and preliminary quality inspection

[0217] The concatenated enhanced prompts are fed into the domain-adaptive BioGPT model. This model, based on an attention mechanism, integrates internal parameter knowledge with external retrieval knowledge to autoregressively generate complete simulated medical text.

[0218] After generating the simulated medical text, a quick scan is performed on the simulated medical text to check whether its basic attributes meet the requirements in the text generation request entered by the user.

[0219] For example: Is the text length close to 400 words? Does it contain user-specified key components such as "diagnostic details" or "multidisciplinary consultation"?

[0220] After passing this check, the text enters the expert verification stage.

[0221] (4) Expert verification and structured feedback generation

[0222] ① The generated simulated diagnosis and treatment text is mixed with several real desensitized case reports through an expert verification interface and presented to at least two pediatric cardiology experts for evaluation in a blind review.

[0223] ② Based on established standards, experts scored the students on dimensions such as clinical accuracy (score: 9 / 10), logical consistency (score: 8 / 10), and language fluency.

[0224] Meanwhile, pediatric cardiology experts provided specific feedback in the expert verification interface. For example, "The prognostic assessment section is too general; it is recommended to supplement it with specific risks for children of different weights."

[0225] ③ The system automatically collects all expert ratings and textual opinions and transforms them into structured feedback data.

[0226] For example, the opinion that "the prognostic assessment section is too general" can be mapped to a negative label for the "prognostic assessment" paragraph in the text and the specific reasons for it.

[0227] (5) Model Iteration Optimization and Prompt Word Update

[0228] ① The system performs iterative optimization based on the collected structured feedback data.

[0229] First, calculate the utility loss L. utility This refers to the difference between the expert scores and the perfect score for the model-generated text (measured by mean squared error). Subsequently, the utility loss is compared with the standard generation loss L. gen According to the formula (In this application example, α=0.7) We perform weighted summation to obtain the total loss function L.

[0230] ②Use the gradient descent algorithm and this total loss function to fine-tune the model parameters so that the model can better meet the expert's evaluation criteria in the next generation.

[0231] Meanwhile, the system analyzed common patterns in the feedback data and found that multiple experts pointed out that the "prognostic assessment" section needed improvement. Therefore, the system automatically optimized the prompt template, explicitly adding the "[Prognostic Assessment]" field to the template and requiring the description to "include the risk differences among children of different weights."

[0232] ③ The optimized prompt word template will be used in subsequent generation tasks.

[0233] 3. Generate and output personalized simulated diagnosis and treatment texts that meet clinical standards.

[0234] The following is an example of the output simulated diagnosis and treatment text.

[0235] [Basic Information]: The fetus is 24 weeks pregnant. During a systematic prenatal ultrasound screening, abnormalities were found in the four-chamber view and outflow tract view of the heart, indicating a perimembranous ventricular septal defect.

[0236] [Diagnostic Findings]: Detailed echocardiography revealed a ventricular septal defect (VSD) approximately 3 mm in diameter, with a left-to-right shunt and a percutaneous blood flow velocity of approximately 2.5 m / s. The sizes of the heart's chambers and ventricles were essentially normal, the foramen ovale valve opened and closed well, and no pericardial effusion or other complex malformations were observed. The overall assessment concluded an isolated, restrictive VSD with minimal intrauterine hemodynamic impact.

[0237] [Treatment Plan]: After multidisciplinary team (MDT) consultation, the following management pathway will be established: regular monitoring during the neonatal period. If there are no complications such as pulmonary hypertension, it is recommended to perform "percutaneous catheter closure" at 3-6 months of age and when the baby weighs more than 5 kg. Preoperative evaluation requires electrocardiogram, chest X-ray, and transthoracic echocardiography.

[0238] [Follow-up and Prognosis]: Follow-up visits are scheduled at 1, 3, 6, and 12 months post-surgery for echocardiography to assess occluder position and cardiac function. Long-term prognosis is excellent; the vast majority of children experience a quality of life comparable to that of healthy children, with a projected long-term survival rate exceeding 95%.

[0239] The above detailed embodiments describe the implementation of the present invention; however, the present invention is not limited to the specific details described in the above embodiments. Within the scope of the claims and technical concept of the present invention, various simple modifications and changes can be made to the technical solution of the present invention, and these simple modifications all fall within the protection scope of the present invention.

Claims

1. A method for generating text for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model, characterized in that, Includes the following steps: A pre-trained language model in the biomedical field was selected as the base model, and LoRA fine-tuning was performed on the base model in the field of fetal congenital heart disease. Collect medical data related to congenital heart disease in fetuses and construct a corresponding knowledge graph; Design a structured prompt word template, which contains required key fields for guiding the generation of simulated diagnosis and treatment text, and dynamically fills them based on the content of the knowledge graph; Upon receiving a request to generate simulated medical text, a real-time retrieval is performed in the knowledge graph, and the retrieved knowledge is added to the input of the generation model to guide the model to generate simulated medical text that conforms to medical facts. The simulated diagnosis and treatment text generated by the model is mixed with the real diagnosis and treatment text, and then manually verified to obtain feedback. Based on the feedback, the model parameters and prompt word templates are iteratively optimized to obtain the final model. Input prompts and output corresponding simulated medical text.

2. The method for generating text for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model according to claim 1, characterized in that, The underlying model is based on the Transformer architecture and has been pre-trained on a biomedical corpus. Furthermore, the biomedical corpus consists of MIMIC-III clinical notes or PubMed biomedical literature; Furthermore, the underlying model includes, but is not limited to, BioGPT or BioBERT.

3. The method for generating text for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model according to claim 1, characterized in that, The method for LoRA fine-tuning of the basic model in the field of fetal congenital heart disease is as follows: An initial dataset was constructed based on professional literature and desensitized case texts related to congenital heart disease in fetuses. Freeze the original parameters of the base model, and in the self-attention mechanism of each Transformer module of the base model, inject trainable LoRA adapters into the query projection matrix and the value projection matrix in parallel. The original weight matrix in the basic model The forward propagation process is modified to ； in, and The low-rank matrix introduced for LoRA, where r is the rank, and x is the input activation value; during fine-tuning, the original weight matrix W is frozen, and only the low-rank matrices B and A are updated; Using the initial dataset as training corpus, and with masked language modeling and next sentence prediction as training objectives, the hyperparameters of the LoRA adapter are fine-tuned. After fine-tuning, the trained LoRA adapter weights B and A are merged with the frozen base model weights W; or, the model is dynamically enhanced during inference by a lightweight additional computation BAx. Furthermore, the hyperparameters of the LoRA adapter are fine-tuned as follows: LoRA rank r=8, LoRA scaling parameter α=16, learning rate is set to 5e-4, batch size is 16, training epochs are 3-5, and the Adam optimizer is used to avoid overfitting. Furthermore, the loss function for mask language modeling is: ； Where M is the set of mask positions, x i For the masked word, x \M For context; Furthermore, the loss function for the next prediction is: ； IsNext indicates whether the two sentences are consecutive.

4. The method for generating text for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model according to claim 1, characterized in that, The method for collecting medical data related to congenital heart disease in fetuses and constructing a corresponding knowledge graph is as follows: Extract entities and relationships related to congenital heart disease in fetuses from authoritative structured databases; The extracted entities and relationships are organized into a graph structure to form the knowledge graph in the field of congenital heart disease in fetuses; Nodes represent entities, and edges represent relationships between entities; The knowledge graph is stored and updated in real time. Furthermore, the authoritative sources include, but are not limited to, authoritative medical textbooks, UpToDate, clinical guidelines, academic papers, and structured databases of hospital electronic medical record systems; Furthermore, the entities include, but are not limited to, disease types, anatomical structures, diagnostic methods, interventions, drugs, and prognostic indicators; the relationships include, but are not limited to, anatomical relationships, disease-related relationships, diagnostic relationships, treatment relationships, and prognostic relationships. Furthermore, the methods for updating the knowledge graph include, but are not limited to, periodically crawling the latest knowledge from authoritative sources and updating it in combination with feedback from manual verification.

5. The method for generating text for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model according to claim 1, characterized in that, The required key fields include, but are not limited to: case type, fetal gestational age, specific type of congenital heart disease, main clinical manifestations, proposed intervention method, and expected text length; The method for dynamically filling in the knowledge graph content is as follows: when the user inputs or the system specifies the value of a certain required key field, the system automatically retrieves typical values ​​or reasonable options of other fields most related to that value from the knowledge graph and fills in the value. Furthermore, natural language sentence structures are used to combine the various required key fields.

6. The method for generating text for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model according to claim 1, characterized in that, Upon receiving the request to generate the simulated medical text, the key information in the generation request is parsed and used as query conditions; Based on the query conditions, the knowledge graph is searched in real time to obtain several knowledge subgraphs or entity relationship pairs that are most relevant to the current generation request. The knowledge subgraph or entity relationship pairs are converted into natural language descriptions to form an enhanced context. The enhanced context and the prompt word template are concatenated and used together as input to the basic model to guide the model in generating simulated diagnosis and treatment text that conforms to medical facts.

7. The method for generating text for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model according to claim 6, characterized in that, The real-time retrieval is performed using a vector retrieval method. Furthermore, the real-time retrieval method is as follows: the entities and relationships in the knowledge graph are vectorized and embedded, and the most relevant Top-K knowledge fragments are returned by calculating the similarity between the query vector and the knowledge vector; Furthermore, the similarity is calculated using the cosine similarity formula.

8. The method for generating text for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model according to claim 1, characterized in that, The method for manually validating the simulated medical text generated by the model, and iteratively optimizing the model parameters and prompt word templates based on the feedback obtained from the manual validation, is as follows: The simulated medical texts are mixed with real medical texts and then given to medical experts in the relevant medical fields for blind review and evaluation. A feedback optimization loss function is constructed based on the feedback data from the blind review evaluation, and the model parameters are fine-tuned using the gradient descent algorithm. Analyze the common problems related to the prompt word template in the feedback data, and optimize and update the required key fields of the prompt word template; Furthermore, the scoring dimensions of the blind review assessment include clinical accuracy, logical consistency, and linguistic fluency; Furthermore, it is recommended that the optimization update be performed at least once a month.

9. The method for generating text for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model according to claim 8, characterized in that, The feedback optimization loss function is composed of the generation loss L. gen And utility loss L utility The weighted summation is calculated using the following formula: ； Where α is a hyperparameter.

10. A text generation system for the diagnosis and treatment of congenital heart disease in fetuses based on a pre-trained language model, characterized in that, It includes a knowledge graph construction module, a search enhancement module, a prompt word design module, and a verification optimization module; The knowledge graph construction module is used to collect medical data related to congenital heart disease in fetuses and construct the corresponding knowledge graph. The retrieval enhancement module is used to perform real-time retrieval in the knowledge graph when a request to generate simulated medical text is received, and to enhance the retrieved knowledge into the input of the generation model, guiding the model to generate simulated medical text that conforms to medical facts. The prompt word design module is used to design prompt word templates, which contain required key fields for generating the simulated diagnosis and treatment text and are dynamically filled based on the content of the knowledge graph. The verification and optimization module is used to manually verify the simulated diagnosis and treatment text generated by the model, and to iteratively optimize the model parameters and prompt word templates based on the feedback obtained from the manual verification.

Images