LLM-based children immunodeficiency retrieval question and answer model evaluation method and system

Abstract

The invention discloses a child immunodeficiency retrieval question and answer model evaluation method and system based on LLM, and relates to the field of question and answer models. The method specifically comprises the following steps: constructing a vector database of children immunodeficiency knowledge; retrieving the vector database according to the user question, and acquiring multiple groups of question and answer data pairs; constructing a children immunodeficiency retrieval question-answer model through the multiple groups of question-answer data pairs; the multiple clinicians evaluate the multiple groups of question and answer data pairs; according to an application scene and clinician features, cue words of each group of optimized question and answer data pairs are selected through a cue word project; and sequentially inputting each group of optimized question and answer data pairs and cue words into a children immunodeficiency retrieval question and answer model for training. According to the method, the ability level of the question and answer model in the children immunodeficiency specialized field can be improved by evaluating the question and answer data of the problem immunodeficiency more comprehensively.

Description

Technical Field

[0001] The present invention relates to the field of question - answering models, and particularly to a method and system for evaluating a retrieval question - answering model for childhood immunodeficiency based on LLM. Background Art

[0002] General large - language models, such as GPT - 4, ERNIE Bot, Zhipu Qingyan, etc., are trained with general knowledge. Although they perform powerfully in general language processing tasks, their training data comes from a wide range of non - specialized datasets. This means that they may not have in - depth understanding of some highly specialized fields. In particular, the medical field usually involves very professional terms, concepts, and background knowledge, and general large models usually cannot accurately understand or generate detailed information in these fields. In addition, general large models rely on a large amount of pre - trained data, and these datasets are usually fixed at a certain point in time. For the medical field, the knowledge update speed is relatively fast. Since general models cannot dynamically update their knowledge bases, they cannot ensure the mastery of these new knowledge. In specialized fields, especially in highly complex and sensitive medical fields such as Primary Immunodeficiency (PID), there is currently no unified standard for evaluating question - answering models. The existing evaluation methods and standards for general LLM are insufficient in such specialized fields, and there is an urgent need to develop multi - dimensional, domain - specific evaluation methods to ensure the efficient performance of models in multiple aspects such as knowledge accuracy and clinical relevance. In summary, general large - language models (such as GPT - 4, etc.) face problems such as lack of professional knowledge, lag in knowledge update, and inability to handle complex terms in specialized fields (childhood immunodeficiency). In view of the problem that there is no unified evaluation standard for large - language question - answering models in the current specialized field, especially in the face of childhood immunodeficiency, therefore, there is currently a need for a method and system for evaluating a retrieval question - answering model for childhood immunodeficiency based on LLM, which can improve the ability level of question - answering models in the specialized field of childhood immunodeficiency. Summary of the Invention

[0003] The technical problem to be solved by the present invention is the problem that there is no unified evaluation standard for large - language question - answering models in the current specialized field. The purpose is to provide a method and system for evaluating a retrieval question - answering model for childhood immunodeficiency based on LLM, which improves the ability level of question - answering models in the specialized field of childhood immunodeficiency by more comprehensively evaluating the question - answering data of immunodeficiency, and solves the above problems.

[0004] The present invention is achieved by the following technical solutions:

[0005] A method for evaluating a retrieval question - answering model for childhood immunodeficiency based on LLM includes the following steps:

[0006] Build a vector database of children's immunodeficiency knowledge; retrieve the above vector database according to the user's question, and collect multiple groups of Q&A data pairs;

[0007] Build a children's immunodeficiency retrieval Q&A model through multiple groups of Q&A data pairs;

[0008] Multiple clinicians evaluate multiple groups of the above Q&A data pairs;

[0009] According to the application scenario and clinician characteristics, select the prompts for each group of the above optimized Q&A data pairs through prompt engineering;

[0010] Input each group of the above optimized Q&A data pairs and prompts into the above children's immunodeficiency retrieval Q&A model for training in sequence;

[0011] When each group of the above optimized Q&A data pairs and prompts is input into the above children's immunodeficiency retrieval Q&A model for training each time, evaluate the above children's immunodeficiency retrieval Q&A model according to basic knowledge ability, professional scenario analysis ability, and diagnosis and treatment case analysis ability to select the final above children's immunodeficiency retrieval Q&A model.

[0012] Evaluate the above children's immunodeficiency retrieval Q&A model according to the above basic knowledge ability, including: inputting multiple questions of multiple question types into the above children's immunodeficiency retrieval Q&A model; the above question types include any one or more of clinical characteristics and diagnosis, genetics and family risk, etiology and pathophysiology, treatment and management, and research and new progress; each question covers any one of the above question types; evaluate the model output results.

[0013] Evaluate the above children's immunodeficiency retrieval Q&A model according to the above professional scenario analysis ability, including: inputting multiple questions of multiple question types into the above children's immunodeficiency retrieval Q&A model; the above question types include any one or more of clinical characteristics and diagnosis, genetics and family risk, etiology and pathophysiology, treatment and management, and research and new progress; each question covers one or more of the above question types; evaluate the model output results.

[0014] The above clinical features and diagnosis include any one or more of the main symptoms, main pathological features, diagnostic specific indicators, gold standard for diagnosis, differential diagnosis, main disease phenotypes, main genotypes, and main imaging features; the above genetics and family risks include any one or more of the main symptoms, main pathological features, diagnostic specific indicators, gold standard for diagnosis, differential diagnosis, main disease phenotypes, main genotypes, and main imaging features; the above etiology and pathophysiology include any one or more of the molecular mechanism of disease occurrence, pathological mechanism of disease occurrence, molecular typing of the disease, pathological typing of the disease, common risk factors, main mutated genes and mutation sites; the above treatment and management include any one or more of the main treatment strategies, common auxiliary examination items, main preventive measures, main therapeutic drugs, and main complications; the above research and new progress include any one or more of the latest research results, latest pathogenesis, and latest drugs currently in clinical trials.

[0015] The above-mentioned children's immunodeficiency retrieval Q&A model is evaluated according to the above-mentioned medical record analysis ability, including: the patient information of multiple patients is respectively obtained through the above-mentioned children's immunodeficiency retrieval Q&A model to obtain medical diagnosis results, and the above-mentioned medical diagnosis results of multiple patients are evaluated; the above-mentioned patient information includes any one or more of the patient's chief complaint, current medical history, auxiliary examinations, genetic tests, and past medical history; the output results of the model are evaluated.

[0016] The above method for evaluating the children's immunodeficiency retrieval Q&A model based on LLM further includes: multiple clinicians answer questions about multiple application scenarios and evaluate each other, and collect multiple groups of Q&A pairs of clinical data; use multiple groups of the above-mentioned Q&A pairs of clinical data to construct a children's immunodeficiency clinical Q&A model; adopt retrieval-enhanced generation technology to combine the above-mentioned children's immunodeficiency retrieval Q&A model and the above-mentioned children's immunodeficiency clinical Q&A model to obtain a children's immunodeficiency Q&A large model.

[0017] The above application scenarios include any one or more of infection management and prevention, disease knowledge and education, clinical symptoms and diagnosis, quality of life and long-term management, genetics and family-related issues, vaccination and preventive measures, treatment recommendations and plans, and treatment progress and research.

[0018] The evaluation system for the children's immunodeficiency retrieval Q&A model based on LLM includes:

[0019] Data acquisition module: construct a vector database of children's immunodeficiency knowledge; retrieve the above vector database according to the user's question and collect multiple groups of Q&A data pairs;

[0020] Model construction module: construct a children's immunodeficiency retrieval Q&A model through multiple groups of Q&A data pairs;

[0021] Data optimization module: Multiple clinicians evaluate multiple groups of the above Q&A data pairs.

[0022] Prompt extraction module: According to the application scenario and clinician characteristics, select the prompts for each group of the optimized above Q&A data pairs through prompt engineering.

[0023] Model training module: Input each group of the optimized above Q&A data pairs and prompts into the above-mentioned pediatric immunodeficiency retrieval Q&A model for training in sequence.

[0024] Model optimization module: When each group of the optimized above Q&A data pairs and prompts are input into the above-mentioned pediatric immunodeficiency retrieval Q&A model for training each time, evaluate the above-mentioned pediatric immunodeficiency retrieval Q&A model according to the basic knowledge ability, professional scenario analysis ability and diagnosis and treatment case analysis ability, so as to select the final above-mentioned pediatric immunodeficiency retrieval Q&A model.

[0025] An electronic device includes a memory, a processor, and a computer program running on the above processor, and is characterized in that: when the above processor executes the above computer program, the steps of the above method for evaluating a pediatric immunodeficiency retrieval Q&A model based on LLM are implemented.

[0026] A computer-readable storage medium stores a computer program, and is characterized in that: when the above computer program is executed by a processor, the steps of the above method for evaluating a pediatric immunodeficiency retrieval Q&A model based on LLM are implemented.

[0027] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0028] The present invention aims at the problem that there is no unified evaluation standard for large language Q&A models in the current pediatric immunodeficiency specialty field, and designs a method capable of comprehensively evaluating the ability level of Q&A models in the pediatric immunodeficiency specialty field. Based on the pre-trained pediatric immunodeficiency retrieval Q&A model, the present invention designs a specialty Q&A model in the pediatric immunodeficiency field by combining retrieval and doctor models; the present invention proposes a method for evaluating the Q&A ability of a large model by evaluating the model through basic knowledge ability, professional scenario analysis ability and diagnosis and treatment case analysis ability, and combining the doctor's ability level, providing a reference for other similar specialty Q&A models. Description of the Drawings

[0029] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings required for the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts. In the drawings:

[0030] Figure 1 It is a schematic diagram of the evaluation method for the retrieval and question-answering model of children's immunodeficiency based on LLM in the embodiments of this application. Specific embodiments

[0031] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in combination with the embodiments and the accompanying drawings. The illustrative embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention.

[0032] Embodiment

[0033] As Figure 1 shown, the embodiments of this application provide an evaluation method for the retrieval and question-answering model of children's immunodeficiency based on LLM, including the following steps:

[0034] Construct a vector database of children's immunodeficiency knowledge; retrieve the above vector database according to the user's question, and collect multiple groups of question-and-answer data pairs;

[0035] Construct a retrieval and question-answering model for children's immunodeficiency through multiple groups of question-and-answer data pairs;

[0036] Multiple clinicians evaluate multiple groups of the above question-and-answer data pairs;

[0037] According to the application scenario and clinician characteristics, select the prompt words for each group of optimized above question-and-answer data pairs through prompt engineering;

[0038] Input each group of optimized above question-and-answer data pairs and prompt words into the above retrieval and question-answering model for children's immunodeficiency in sequence for training;

[0039] When each group of optimized above question-and-answer data pairs and prompt words are input into the above retrieval and question-answering model for children's immunodeficiency for training each time, evaluate the above retrieval and question-answering model for children's immunodeficiency according to the basic knowledge ability, professional scenario analysis ability, and diagnosis and treatment case analysis ability to select the final above retrieval and question-answering model for children's immunodeficiency.

[0040] Evaluate the above-mentioned retrieval Q&A model for childhood immunodeficiency according to the above basic knowledge ability, including: inputting multiple questions of multiple question types into the above-mentioned retrieval Q&A model for childhood immunodeficiency; the above-mentioned question types include any one or more of clinical features and diagnosis, genetics and family risk, etiology and pathophysiology, treatment and management, and research and new progress; each question covers any one of the above-mentioned question types; evaluate the output results of the model.

[0041] Evaluate the above-mentioned retrieval Q&A model for childhood immunodeficiency according to the above professional scenario analysis ability, including: inputting multiple questions of multiple question types into the above-mentioned retrieval Q&A model for childhood immunodeficiency; the above-mentioned question types include any one or more of clinical features and diagnosis, genetics and family risk, etiology and pathophysiology, treatment and management, and research and new progress; each question covers one or more of the above-mentioned question types; evaluate the output results of the model.

[0042] The above-mentioned clinical features and diagnosis include any one or more of main symptoms, main pathological features, diagnostic specific indicators, gold standard for diagnosis, differential diagnosis, main disease phenotypes, main genotypes, and main imaging features; the above-mentioned genetics and family risk include any one or more of main symptoms, main pathological features, diagnostic specific indicators, gold standard for diagnosis, differential diagnosis, main disease phenotypes, main genotypes, and main imaging features; the above-mentioned etiology and pathophysiology include any one or more of the molecular mechanism of disease occurrence, pathological mechanism of disease occurrence, molecular typing of disease, pathological typing of disease, common high-risk factors, main mutant genes and mutation sites; the above-mentioned treatment and management include any one or more of main treatment strategies, common auxiliary examination items, main preventive measures, main treatment drugs, and main complications; the above-mentioned research and new progress include any one or more of recent research results, latest pathogenesis, and latest drugs currently in clinical trials.

[0043] Evaluate the above-mentioned retrieval Q&A model for childhood immunodeficiency according to the above medical record analysis ability, including: obtaining medical diagnosis results for multiple patients' patient information through the above-mentioned retrieval Q&A model for childhood immunodeficiency respectively, and evaluating the above-mentioned medical diagnosis results for multiple patients; the above-mentioned patient information includes any one or more of patient complaints, current medical history, auxiliary examinations, genetic tests, and past medical history; evaluate the output results of the model.

[0044] The above-mentioned method for evaluating the retrieval and question-answering model of childhood immunodeficiency based on LLM further includes: multiple clinicians answer questions in multiple application scenarios and evaluate each other, and collect multiple groups of question-and-answer pairs of clinical data; use multiple groups of the above-mentioned question-and-answer pairs of clinical data to construct a clinical question-and-answer model for childhood immunodeficiency; adopt retrieval-augmented generation technology to combine the above-mentioned retrieval and question-answering model of childhood immunodeficiency and the above-mentioned clinical question-and-answer model of childhood immunodeficiency to obtain a large question-and-answer model for childhood immunodeficiency.

[0045] Retrieval-Augmented Generation (RAG) is a natural language processing method that combines information retrieval and text generation technologies, aiming to improve the accuracy and flexibility of large language models in knowledge acquisition, question answering, and generation tasks. The RAG technology combines a pre-trained retrieval and question-answering model for childhood immunodeficiency with an information retrieval system, enabling the model to not only rely on its own parameters to generate answers, but also retrieve relevant information from an external database or knowledge base and generate more accurate answers based on the retrieval results.

[0046] The RAG technology effectively solves these problems by combining the retrieval and question-answering model for childhood immunodeficiency and the clinical question-and-answer model for childhood immunodeficiency. In terms of improving professionalism: RAG provides accurate domain knowledge by retrieving external professional knowledge bases, enhancing the model's performance in the field of childhood immunodeficiency; in terms of data update, RAG can help the model obtain information from the latest literature and research to avoid outdated knowledge; in terms of interpretability, RAG provides literature or data support to ensure that the answers are well-founded and improve the reliability of the answers. Therefore, RAG can make up for the deficiencies of general large models in the field of childhood immunodeficiency specialty and improve the question-and-answer ability of large models in the field of childhood immunodeficiency.

[0047] Optionally, when each clinician evaluates the answers of others and evaluates the above-mentioned retrieval and question-answering model of childhood immunodeficiency, the scoring is based on accuracy, relevance, logic, and conciseness.

[0048] The above-mentioned application scenarios include any one or more of infection management and prevention, disease knowledge and education, clinical symptoms and diagnosis, quality of life and long-term management, genetics and family-related issues, vaccination and preventive measures, treatment recommendations and plans, and treatment progress and research.

[0049] Large Language Model (LLM) refers to a model trained through deep learning technology that can process and generate natural language text. Large language models (such as GPT-3, GPT-4, BERT, etc.) have extensive applications in the field of natural language processing (NLP). They can perform various language tasks, such as dialogue generation, text translation, question-answering systems, and text classification, etc.

[0050] When constructing the vector database for the field of childhood immunodeficiency knowledge, first, through methods such as Pubmed retrieval, an original text database is obtained, including more than 50,000 scientific research literatures related to childhood immunodeficiency, 104 case literatures, and 10 textbooks / clinical guidelines. These knowledge are used to construct the vector database. Specifically, the document formats include PDF and TXT. The documents in a unified format are obtained through a document loader, and then a text splitter and text embedding are used in sequence, and finally the vector database is constructed.

[0051] When the user asks a question, the user's question will be embedded and converted into a question vector. Knowledge related to the question is retrieved from the vector database constructed in the previous step, and then the retrieved knowledge is sorted and filtered according to the semantic similarity with the question, and the knowledge most relevant to the question is retained to obtain multiple groups of question-answer data pairs.

[0052] When constructing the childhood immunodeficiency retrieval question-answer model with multiple groups of question-answer data pairs, first select a pre-trained large text model. The open-source pre-trained text model Qwen2-72B-instruct can be selected. When multiple clinicians optimize the question-answer data pairs, each clinician obtains the incorrect and duplicate data according to the evaluation results and modifies and deletes them manually. According to the application scenario and the working characteristics of the clinicians who optimize the question-answer data pairs, the prompt words for the question-answer data pairs are obtained.

[0053] The childhood immunodeficiency retrieval question-answer model is retrained with the optimized question-answer data pairs and prompt words. Optionally, the input question is automatically recognized to obtain the corresponding application scenario. Clinician characteristics can include doctor level, ability, area of expertise, etc., so as to adapt to the popular science capabilities of different types of clinicians and obtain the prompt words for the corresponding question-answer data pairs.

[0054] When evaluating the above-mentioned retrieval and question-answering model for childhood immunodeficiency according to basic knowledge ability, professional scenario analysis ability, and diagnosis and treatment case analysis ability, and optimizing the above-mentioned retrieval and question-answering model for childhood immunodeficiency, it is determined whether to select the retrieval and question-answering model for childhood immunodeficiency trained with the question-and-answer data of each group each time according to the evaluation results, so as to further screen the optimized training data. Optionally, the retrieval and question-answering model for childhood immunodeficiency is horizontally evaluated by comparing it with other question-answering models. In the embodiments of the present application, Wenxin Yiyan, Spark Model, Zhipu Qingyan, Huatuo GPT, and GPT4 are selected as the objects for horizontal comparison. In order to comprehensively evaluate the advantages and disadvantages of the question-answering model of the present invention and other question-answering models, three evaluation stages of basic knowledge ability evaluation, professional scenario analysis ability evaluation, and diagnosis and treatment case analysis ability evaluation are designed, and scores are given based on a six-level scoring standard (0 points - 5 points) in four evaluation dimensions of accuracy, relevance, logic, and conciseness.

[0055] When evaluating the basic knowledge ability, a basic knowledge ability evaluation set containing 100 - 200 questions can be established for testing. Each question covers one of the following question types, and all questions should cover all question types; at the same time, try to cover as many specific diseases of childhood immunodeficiency as possible.

[0056] The following are examples of questions of different question types:

[0057] (1) Etiology and pathophysiology:

[0058] 1. What is the molecular mechanism of Wiskott-Aldrich syndrome?

[0059] 2. What are the molecular mechanisms of hyper-IgE syndrome?

[0060] 3. What is the specific molecular mechanism of chronic granulomatous disease?

[0061] 4. What is the main molecular mechanism of congenital agammaglobulinemia?

[0062] 5. What is the molecular mechanism of familial Mediterranean fever?

[0063] 6. How to diagnose and select IgA deficiency?

[0064] (2) Clinical features and diagnosis:

[0065] 1. What are the diagnostic criteria for X-linked agammaglobulinemia (XLA)?

[0066] 2. What are the clinical symptoms of selective IgA deficiency?

[0067] 3. How to diagnose selective IgA deficiency?

[0068] 4. What are the common symptoms of agranulocytosis?

[0069] 5. What are the main clinical manifestations of chronic granulomatous disease?

[0070] (3)Treatment and management:

[0071] 1. What are the treatment strategies for X-linked agammaglobulinemia (XLA)?

[0072] 2. What is the treatment method for severe combined immunodeficiency (SCID)?

[0073] 3. What is the treatment strategy for hyper IgE syndrome?

[0074] 4. What is the main treatment strategy for familial Mediterranean fever?

[0075] 5. What are the suggestions for infection prevention and management in patients with chronic granulomatous disease?

[0076] (4)Genetics and family risk:

[0077] 1. How to evaluate the family genetic risk of primary immunodeficiency diseases?

[0078] 2. What is the genetic mechanism of Wiskott-Aldrich syndrome?

[0079] 3. What is the genetic mechanism of congenital agammaglobulinemia?

[0080] (5)Research and new progress:

[0081] 1. What are the new treatment breakthroughs in the research of immunodeficiency?

[0082] 2. What is CAR-T cell therapy? What is its application in the treatment of immunodeficiency?

[0083] 3. What is the progress of gene therapy for SCID?

[0084] When conducting a professional scenario analysis ability assessment, a professional scenario analysis ability assessment set containing 100 - 200 questions can be established for testing. Each question can contain multiple interrogative sentences and cover one or more question types, and all questions should cover all question types; try to cover as many specific diseases as possible.

[0085] The following are multiple question examples for professional scenario analysis ability assessment:

[0086] Q1: What are the clinical manifestations of X-linked severe combined immunodeficiency disease? What is the diagnostic basis? What are the current main treatment suggestions? What are the main preventive measures?

[0087] Question Q1 covers 3 question types: clinical features and diagnosis including main symptoms, main pathological features, diagnostic specific indicators and differential diagnosis, treatment and management including main treatment measures and common auxiliary examination items, research and new progress including recent research results.

[0088] Q2: What primary immunodeficiency diseases can STAT1 gene mutations cause? What are the pathogenic mechanisms and inheritance patterns respectively? What are the differences in the corresponding clinical manifestations?

[0089] Question Q2 covers 3 question types: clinical features and diagnosis including main disease phenotypes, genetics and family risk including genetic characteristics and genetic risks, treatment and management including main preventive measures, main treatment drugs and main complications.

[0090] Q3: What are the diagnostic criteria for CVID? What are the pathogenic genes?

[0091] Question Q3 covers 3 question types: clinical features and diagnosis including main genotypes, etiology and pathophysiology including molecular mechanisms of disease occurrence and pathological mechanisms of disease occurrence, research and new progress including the latest pathogenesis and the latest drugs entering clinical trials.

[0092] Q4: How many types is leukocyte adhesion molecule deficiency mainly divided into currently? What are the pathogenic genes respectively? What is the main treatment method?

[0093] Question Q4 covers 1 question type: etiology and pathophysiology including common high-risk factors, main mutant genes and mutant sites.

[0094] Q5: What are the pathogenic genes of congenital neutropenia? What are the differences in the clinical manifestations of the diseases caused by different pathogenic genes? What is the main treatment method?

[0095] Question Q5 covers 1 question type: etiology and pathophysiology including common high-risk factors, main mutant genes and mutant sites.

[0096] Q6: What imaging manifestations can the pulmonary lesions caused by COPA syndrome have? How are they different from the imaging manifestations of pulmonary infections?

[0097] Question Q6 covers 1 question type: clinical features and diagnosis including main imaging features.

[0098] When evaluating the ability of diagnosis and treatment case analysis, a professional scenario analysis ability evaluation set containing about 50 question-and-answer pairs can be established for testing. According to the patient information of the patient's one chief complaint and five histories provided, the hospital diagnosis result is obtained as follows for example:

[0099] Chief complaint: Repeated infections and pneumonia for more than 9 years since birth.

[0100] Present illness history: The child has had repeated infections after birth, suffered from pneumonia several times, was hospitalized 1 - 3 times a year on average, accompanied by recurrent and refractory oral ulcers, and red rashes can be seen on the palms. Recently, the child complained of joint pain, mainly involving the left hip joint and the right ilium.

[0101] Complete auxiliary examinations: Immunoglobulin levels are normal, TBNK: The percentage of CD3⁺ T cells decreased (57.13% - 62.61%, reference value 64.62% - 77.08%). Autoantibody tests showed that anti - SM, anti - SSB, anti - Ro - 52, and anti - Scl - 70 antibodies were transiently positive. Anti - neutrophil cytoplasmic antibody (ANCA) was negative. Serum cytokine tests: MIP - 1α: 138.4 pg / ml, MIP - 1β: 722.4 pg / ml, IL - 1β: 44 pg / ml, IL - 6 149.4 pg / ml, IL - 8 221.2 pg / ml;

[0102] Genetic testing suggested: Maternal ELF4 gene mutation. The child has no fever, no cough, no pain or limp in both lower extremities. For a clear diagnosis and further treatment, the child came to our hospital outpatient clinic and was admitted to the hospital with the diagnosis of "primary immunodeficiency disease". After the illness, the spirit is okay, the appetite is okay, urination is normal, suffering from intractable constipation, and the weight gain is average.

[0103] Past history: Suffered from severe pneumonia at 2 months old and was discharged after improvement with mechanical ventilation treatment in the PICU. The child has been vaccinated with vaccines including Bacillus Calmette - Guérin (BCG), hepatitis B, poliomyelitis, diphtheria - pertussis - tetanus (DPT), without abnormal reactions.

[0104] Please give a preliminary and rigorous admission diagnosis from the perspective of a professional doctor based on the above information of the patient.

[0105] Examples of the six - level scoring criteria for the four optional evaluation dimensions are as follows:

[0106] Accuracy: The degree to which the answer content conforms to the actual principle and the real situation.

[0107] 0 - 1 point: 50% of the content in the answer is incorrect;

[0108] 2 - 3 points: The answer content is basically consistent with the actual principle, with a few inconsistent places;

[0109] 4 - 5 points: The answer content is completely consistent with the actual principle.

[0110] Relevance: On the premise that there are no obvious errors in the answer content, the answer content must specifically answer the questions in the question.

[0111] 0 - 1 point: The answer content can hardly answer the questions in the question;

[0112] 2 - 3 points: The answer can address 50% of the questions in the query.

[0113] 4 - 5 points: The answer can address 100% of the questions in the query.

[0114] Logic: On the premise that there are no obvious errors in the answer content, the answer content is well - structured.

[0115] 0 - 1 point: The answer is incorrect and the content is logically chaotic.

[0116] 2 - 3 points: There are no obvious errors in the answer, and there are a few cases of logical chaos in the answer content.

[0117] 4 - 5 points: There are no obvious errors in the answer, and the answer content is well - structured.

[0118] Conciseness: On the premise that there are no obvious errors in the answer content, the answer content is concise and refined, with little irrelevant content.

[0119] 0 - 1 point: The answer is incorrect and the content is verbose.

[0120] 2 - 3 points: There are no obvious errors in the answer, and 50% of the answer content is irrelevant.

[0121] 4 - 5 points: There are no obvious errors in the answer, the answer is refined, and there is almost no irrelevant content in the answer content.

[0122] In terms of basic knowledge ability, the Q&A model of the present invention performs better than other large models in terms of accuracy, relevance, and conciseness. In terms of the logical dimension, all large models perform well; in terms of professional scenario analysis ability, the Q&A model of the present invention performs better than other large models in terms of accuracy, relevance, logic, and conciseness; in terms of diagnosis and treatment case analysis ability, the Q&A model of the present invention performs better than other large models in terms of accuracy, relevance, and conciseness; in terms of the logical dimension, all large models perform well.

[0123] In this embodiment, 2 primary clinicians, 2 intermediate clinicians, and 2 senior clinicians, a total of 6 clinicians, are selected to answer questions in multiple application scenarios and evaluate each other. The above - mentioned evaluation dimensions and scoring criteria are the same as before, that is, four evaluation dimensions (accuracy, relevance, logic, conciseness) and a six - level scoring standard (0 points - 5 points).

[0124] The following are examples of questions in multiple application scenarios:

[0125] Infection management and prevention:

[0126] Why are immunocompromised patients prone to infection?

[0127] How to manage long-term infections in patients with T cell deficiencies?

[0128] Disease knowledge and education:

[0129] What is secondary immunodeficiency disease?

[0130] What is bone marrow transplantation and how is it used to treat primary immunodeficiency?

[0131] Clinical symptoms and diagnosis:

[0132] My child has been diagnosed with Behcet's disease and has been taking thalidomide. Recently, she has been complaining of pain in her chest, waist, and thighs. Do we need to do any tests?

[0133] What are the main characteristics of autoimmune lymphoproliferative syndrome (ALPS)?

[0134] Quality of life and long-term management:

[0135] A female patient with high IgE syndrome and STAT3 gene deficiency is already an adult. Currently, her physical condition is normal. Can she get married and have children in the future? If so, what precautions need to be taken?

[0136] Can patients with primary immunodeficiency have children?

[0137] Genetics and family-related issues

[0138] How to evaluate the family genetic risk of primary immunodeficiency disease?

[0139] Does primary immunodeficiency affect the growth and development of children?

[0140] Vaccination and preventive measures:

[0141] My child has severe combined immunodeficiency and has been transplanted for more than 5 years and has been off medicine for 3 years. Can he receive attenuated vaccines in this situation?

[0142] My child has agammaglobulinemia and was vaccinated with BCG by the school without my knowledge. I want to know what to do now. Do I need to go to the hospital for a check-up?

[0143] Treatment recommendations and plans:

[0144] My son was diagnosed with DGs syndrome in 2015. Are there any new treatment methods now? Currently, the child has atelectasis in the left lung, has been infected with COVID-19 4 times, and has had a long-term EBV infection. His condition is not optimistic. Is cell therapy and other treatments feasible?

[0145] What are the treatment options for CVID?

[0146] Treatment progress and research:

[0147] What are the new treatment breakthroughs in the study of immunodeficiency?

[0148] Can primary immunodeficiency be cured?

[0149] In summary, the embodiments of the present application provide a method and system for evaluating a retrieval question-answering model for children's immunodeficiency based on LLM:

[0150] In response to the problem that there is no unified evaluation standard for the large language question-answering model in the current children's immunodeficiency specialty field, the present invention designs a method capable of comprehensively evaluating the ability level of the question-answering model in the children's immunodeficiency specialty field. Based on the pre-trained retrieval question-answering model for children's immunodeficiency, the present invention designs a specialty question-answering model in the field of children's immunodeficiency by combining retrieval and doctor models; the present invention proposes a method for evaluating the question-answering ability of the model through basic knowledge ability, professional scenario analysis ability, and diagnosis and treatment case analysis ability, and combining the evaluation of the doctor's ability level, providing a reference for other similar specialty question-answering models.

[0151] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. Evaluation method for a question-answering model for retrieving information on childhood immunodeficiency based on large language models (LLMs), characterized in that, It includes the following steps: Construct a vector database of knowledge about childhood immunodeficiency; retrieve the vector database according to the user's question, and collect multiple groups of Q&A data pairs; Construct a retrieval Q&A model for childhood immunodeficiency through multiple groups of Q&A data pairs; Multiple clinicians evaluate each group of the Q&A data pairs to optimize each group of the Q&A data pairs; According to the application scenario and clinician characteristics, select the prompting words for each group of optimized Q&A data pairs through prompting word engineering; Input each group of optimized Q&A data pairs and prompting words into the retrieval Q&A model for childhood immunodeficiency in sequence for training; When each time each group of optimized Q&A data pairs and prompting words are input into the retrieval Q&A model for childhood immunodeficiency for training, evaluate the retrieval Q&A model for childhood immunodeficiency according to the basic knowledge ability, professional scenario analysis ability, and diagnosis and treatment case analysis ability, so as to select the final retrieval Q&A model for childhood immunodeficiency.

2. The method for evaluating a question-answering model for retrieving children's immunodeficiency based on LLM according to claim 1, wherein, Evaluating the retrieval Q&A model for childhood immunodeficiency according to the basic knowledge ability includes: inputting multiple questions of multiple question types into the retrieval Q&A model for childhood immunodeficiency; the question types include any one or more of clinical features and diagnosis, genetics and family risk, etiology and pathophysiology, treatment and management, and research and new progress; each question covers any one of the question types; evaluate the model output results.

3. The method for evaluating a question-answering model for retrieving children's immunodeficiency based on an LLM according to claim 1, wherein Evaluating the retrieval Q&A model for childhood immunodeficiency according to the professional scenario analysis ability includes: inputting multiple questions of multiple question types into the retrieval Q&A model for childhood immunodeficiency; the question types include any one or more of clinical features and diagnosis, genetics and family risk, etiology and pathophysiology, treatment and management, and research and new progress; each question covers one or more of the question types; evaluate the model output results.

4. The method for evaluating the question answering model for retrieving children's immunodeficiency based on LLM according to claim 2 or 3, wherein The clinical features and diagnosis include any one or more of the following contents: main symptoms, main pathological features, diagnostic specific indicators, gold standard for diagnosis, differential diagnosis, main disease phenotypes, main genotypes, main imaging features; the genetics and family risk include any one or more of the following contents: main symptoms, main pathological features, diagnostic specific indicators, gold standard for diagnosis, differential diagnosis, main disease phenotypes, main genotypes, main imaging features; the etiology and pathophysiology include any one or more of the following contents: molecular mechanism of disease occurrence, pathological mechanism of disease occurrence, molecular typing of disease, pathological typing of disease, common high-risk factors, main mutated genes and mutation sites; the treatment and management include any one or more of the following contents: main treatment strategies, commonly used auxiliary examination items, main preventive measures, main treatment drugs, main complications; the research and new progress include any one or more of the following contents: latest research results, latest pathogenesis, latest drugs currently in clinical trials.

5. The method for evaluating a question-answering model for retrieving children's immunodeficiency based on an LLM according to claim 1, wherein, Evaluating the pediatric immunodeficiency retrieval question - answering model according to the medical record analysis ability, including: obtaining medical diagnosis results for the patient information of multiple patients through the pediatric immunodeficiency retrieval question - answering model respectively, and evaluating the medical diagnosis results for multiple patients; the patient information includes any one or more of patient's chief complaint, current medical history, auxiliary examinations, genetic testing, and past medical history; evaluating the model output results.

6. The method for evaluating a retrieval question-answering model for children's immunodeficiency based on LLM according to claim 1, wherein, It also includes: Multiple clinicians answer questions for multiple application scenarios and evaluate each other, collecting multiple groups of question - answer pairs of clinical data; Using multiple groups of the question - answer pairs of clinical data to construct a pediatric immunodeficiency clinical question - answering model; adopting retrieval - enhanced generation technology to combine the pediatric immunodeficiency retrieval question - answering model and the pediatric immunodeficiency clinical question - answering model to obtain a pediatric immunodeficiency question - answering large model.

7. The method for evaluating a question answering model for retrieving children's immunodeficiency based on LLM according to claim 1 or 6, characterized in that, The application scenarios include any one or more of infection management and prevention, disease knowledge and education, clinical symptoms and diagnosis, quality of life and long - term management, genetics and family - related issues, vaccination and preventive measures, treatment suggestions and plans, treatment progress and research.

8. The evaluation system for the question-answering model of retrieving children's immunodeficiency based on LLM, characterized in that, It includes: Data collection module: constructing a vector database of pediatric immunodeficiency knowledge; Retrieving the vector database according to the user's question and collecting multiple groups of question - answer data pairs; Model construction module: constructing a pediatric immunodeficiency retrieval question - answering model through multiple groups of question - answer data pairs; Data optimization module: multiple clinicians evaluate multiple groups of the question - answer data pairs; Prompt extraction module: selecting prompts for each group of optimized question - answer data pairs through prompt engineering according to the application scenario and clinician characteristics; Model training module: sequentially inputting each group of optimized question - answer data pairs and prompts into the pediatric immunodeficiency retrieval question - answering model for training; Model optimization module: when each group of optimized question - answer data pairs and prompts are input into the pediatric immunodeficiency retrieval question - answering model for training each time, evaluating the pediatric immunodeficiency retrieval question - answering model according to basic knowledge ability, professional scenario analysis ability, and medical record analysis ability to select the final pediatric immunodeficiency retrieval question - answering model.

9. An electronic device, comprising a memory, a processor, and a computer program running on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the method for evaluating the LLM - based pediatric immunodeficiency retrieval question - answering model according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the method for evaluating the LLM - based pediatric immunodeficiency retrieval question - answering model according to any one of claims 1 to 7.

Images