Large language model clinical diagnosis capability fine-grained evaluation method and system

Abstract

The present application relates to the technical field of natural language processing, and discloses a large language model clinical diagnosis capability fine-grained evaluation method and system. The method comprises: obtaining a patient medical record data set; synchronously performing the following tasks: scoring a question-standard answer-model answer triple based on a predetermined scoring dimension, calculating the medical knowledge score of the evaluated model; the evaluated model performs a recall task and a differential task, confirms or excludes a differential diagnosis list, and obtains a final main diagnosis; linking the differential diagnosis list and the final main diagnosis output by the evaluated model in the patient medical record; and analyzing factors affecting the clinical diagnosis capability of the evaluated model based on the linking result. The present application aims to deeply and synchronously evaluate the medical knowledge and reasoning capability of a large language model based on the same clinical diagnosis situation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a large language model clinical diagnosis capability fine-grained evaluation method and system. BACKGROUND

[0002] In recent years, large language models (LLMs) have shown significant progress in clinical diagnosis tasks, mainly due to their extensive medical knowledge and reasoning ability. Specifically, medical knowledge refers to the information that large language models learn from medical literature and clinical data, which forms the basis for their understanding of the medical field and clinical context. Reasoning ability refers to the cognitive process of integrating and using medical knowledge as evidence to arrive at medical conclusions. Although large language models have made progress in these areas, their accuracy in clinical diagnosis tasks has not yet reached the level of medical experts. Therefore, it is necessary to further focus and in-depth explore the weak links that limit the effectiveness of large language models in this process. To this end, it is necessary to establish a fine-grained and systematic evaluation method to comprehensively measure the performance of large language models in clinical diagnosis tasks.

[0003] Many existing evaluation methods have shortcomings and are difficult to comprehensively measure the clinical diagnosis ability of large language models. First, traditional evaluation methods rely heavily on medical textbooks or examination questions, and these datasets lack real clinical data support, which is quite different from actual medical scenarios. To better meet the needs of clinical diagnosis, recent evaluation methods have incorporated patient medical record information. However, such methods mostly use static single-step evaluation paradigms, which cannot reflect the dynamic and iterative nature of real clinical diagnosis. In addition, some evaluation methods use multi-agent collaboration frameworks to simulate doctor-patient interactions or communication between doctors to enhance the dynamic nature of the diagnosis process. However, such methods focus on simulating clinical scenarios and fail to deeply analyze the medical knowledge reserves and reasoning ability of large language models in the diagnosis process, which is complementary to the focus of the present application. Recent research attempts to use dynamic evaluation frameworks and has made valuable insights. However, these methods treat medical knowledge evaluation and diagnosis reasoning as independent tasks and fail to examine both simultaneously in the same diagnostic context, making it difficult to accurately determine whether incorrect diagnoses are due to a lack of medical knowledge or insufficient reasoning ability. Therefore, it is necessary to establish a method that follows a standard dynamic diagnosis paradigm and can evaluate large language models in the same diagnostic task. SUMMARY

[0004] To solve the above technical problems, the present application provides a large language model clinical diagnosis capability fine-grained evaluation method and system.

[0005] To solve the above technical problems, the present application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a method for evaluating the clinical diagnosis capability of a large language model in a fine-grained manner, comprising:

[0007] obtaining a patient medical record dataset containing a list of differential diagnoses and a final primary diagnosis;

[0008] synchronously performing the following tasks in the same clinical diagnosis scenario:

[0009] medical knowledge evaluation task: the large language model generates questions based on the final primary diagnosis of the patient medical record and obtains standard answers, obtains the answers of the evaluated model to the questions, forms a question-standard answer-model answer triple, scores the triple based on predetermined scoring dimensions, and calculates the medical knowledge score of the evaluated model;

[0010] reasoning ability evaluation task: the evaluated model performs a recall task based on the patient medical record, generates a list of differential diagnoses, and performs a differential task to confirm or exclude the list of differential diagnoses to obtain a unique final primary diagnosis; based on the international disease classification standard, link the list of differential diagnoses and the final primary diagnosis in the patient medical record and the corresponding list of differential diagnoses and the final primary diagnosis of the evaluated model;

[0011] Based on the linking results, evaluate the hit rate of the evaluated model in the recall task, the differential accuracy in the differential task, and the clinical diagnosis accuracy; calculate the correlation coefficients between the medical knowledge score, the hit rate, the differential accuracy, and the clinical diagnosis accuracy, respectively, to analyze the factors affecting the clinical diagnosis capability of the evaluated model.

[0012] In one embodiment, the large language model generates questions based on the final primary diagnosis of the patient medical record and obtains standard answers, obtains the answers of the evaluated model to the questions, forms a question-standard answer-model answer triple, scores the triple based on predetermined scoring dimensions, and calculates the medical knowledge score of the evaluated model, specifically comprising:

[0013] For each medical record, the large language model uses the final primary diagnosis field in the patient medical record based on doctor annotation to generate questions, and forms question-standard answer pairs with the corresponding answers, and further forms question-standard answer-model answer triples with the answers of the evaluated model to the questions;

[0014] The large language model scores triples corresponding to each medical record based on the degree of fit with the questions, the consistency with actual clinical manifestations, the degree of conformity with medical facts, and the rigor of the reasoning process, adds up the scores corresponding to all medical records and takes the average to obtain the medical knowledge score of the evaluated model.

[0015] In one of the embodiments, the evaluated model performs a recall task based on the patient medical record, generates a differential diagnosis list, and performs a differential task to confirm or exclude the differential diagnosis list to obtain a unique final main diagnosis, specifically including:

[0016] The patient medical record includes patient basic information, chief complaint, medical history, physical examination results, and auxiliary examination results;

[0017] Recall task: the evaluated model recalls a differential diagnosis list based on patient basic information, chief complaint, medical history, and physical examination results, including multiple differential diagnoses;

[0018] Differential task: the evaluated model combines auxiliary examination results, uses clinical reasoning to confirm or exclude each differential diagnosis and outputs the corresponding differential reason until a unique final main diagnosis is obtained.

[0019] In one of the embodiments, it further includes a process of quality evaluation of the differential reason, specifically including:

[0020] Through a large language model, extract atomic medical evidence with negative implications and clear support for differential diagnosis from the differential reason output by the evaluated model and the differential reason corresponding to the differential diagnosis of the patient medical record;

[0021] Based on the extracted evidence, evaluate the reasonableness and completeness of the differential reason output by the evaluated model;

[0022] Reasonableness: define a binary variable , if the large language model considers that the jth evidence corresponding to the differential reason output by the evaluated model is reasonable in medicine, , otherwise ; the reasonableness of the differential reason output by the evaluated model is

[0023] ;

[0024] is the total number of evidence corresponding to the differential reason output by the evaluated model;

[0025] Completeness: define a binary variable , if the large language model considers that the jth evidence corresponding to the differential reason output by the evaluated model is mentioned or covered by the differential reason in the patient medical record annotated by the doctor, , otherwise ; the completeness of the differential reason output by the evaluated model is

[0026] ;​​

[0027] wherein, represents the total number of evidence extracted by the large language model from the differential reasons in the patient medical record.

[0028] In one embodiment, the differential diagnosis list and the final primary diagnosis in the patient medical record and the differential diagnosis list and the final primary diagnosis corresponding to the evaluated model are linked based on the international disease classification standard, specifically comprising:

[0029] The differential diagnosis list and the final primary diagnosis in the patient medical record and the differential diagnosis list and the final primary diagnosis corresponding to the evaluated model are standardized into terms conforming to the international disease classification standard using a large language model; the standardized terms are vectorized with the items of the international disease classification standard using a medical encoder SapBERT to obtain a term vector and an international disease classification standard vector database, the cosine similarity between the term vector and all item vectors in the international disease classification standard vector database is calculated, and the disease name corresponding to the item vector with the highest cosine similarity is selected as the linking result.

[0030] In a second aspect, the present application provides a large language model clinical diagnosis ability fine-grained evaluation system, comprising:

[0031] A medical record acquisition module: acquires a patient medical record data set containing a differential diagnosis list and a final primary diagnosis;

[0032] An evaluation module: synchronously performs the following tasks under the same clinical diagnosis scenario:

[0033] A medical knowledge evaluation task: the large language model generates questions based on the final primary diagnosis of the patient medical record and obtains standard answers, obtains the answers of the evaluated model to the questions to form a question-standard answer-model answer triple; score the triple based on predetermined scoring dimensions to calculate the medical knowledge score of the evaluated model;

[0034] A reasoning ability evaluation task: the evaluated model performs a recall task based on the patient medical record, generates a differential diagnosis list, and performs a differential task to confirm or exclude the differential diagnosis list to obtain a unique final primary diagnosis; the differential diagnosis list and the final primary diagnosis in the patient medical record and the differential diagnosis list and the final primary diagnosis corresponding to the evaluated model are linked based on the international disease classification standard;

[0035] An analysis module: based on the linking result, the hit rate of the evaluated model in the recall task, the differential accuracy of the evaluated model in the differential task, and the clinical diagnosis accuracy are evaluated; the correlation coefficients between the medical knowledge score, the hit rate, the differential accuracy, and the clinical diagnosis accuracy are calculated to analyze the factors affecting the clinical diagnosis ability of the evaluated model.

[0036] In one of the embodiments, the large language model generates questions based on the final principal diagnosis of the patient medical record and obtains standard answers, obtains the answers of the evaluated model to the questions, forms question-standard answer-model answer triples; scores the triples based on predetermined scoring dimensions, calculates the medical knowledge score of the evaluated model, specifically including:

[0037] For each medical record, the large language model generates questions based on the final principal diagnosis field in the patient medical record annotated by doctors, and forms question-standard answer pairs with the corresponding answers, and further forms question-standard answer-model answer triples with the answers of the evaluated model to the questions;

[0038] The large language model scores the triples corresponding to each medical record based on the degree of fit with the questions, the consistency with the actual clinical manifestations, the degree of conformity with medical facts, and the rigor of the reasoning process, adds and averages the scores corresponding to all medical records to obtain the medical knowledge score of the evaluated model.

[0039] In one of the embodiments, the evaluated model performs a recall task based on the patient medical record, generates a list of differential diagnoses, and performs a differential task to confirm or exclude the list of differential diagnoses to obtain a unique final principal diagnosis, specifically including:

[0040] The patient medical record includes patient basic information, chief complaint, medical history, physical examination results, and auxiliary examination results;

[0041] Recall task: the evaluated model recalls a list of differential diagnoses based on patient basic information, chief complaint, medical history, and physical examination results;

[0042] Differential task: the evaluated model combines auxiliary examination results, uses clinical reasoning to confirm or exclude each differential diagnosis and outputs the corresponding differential reasons until a unique final principal diagnosis is obtained.

[0043] In one of the embodiments, it further includes a quality evaluation process of the differential reasons, specifically including:

[0044] Through the large language model, atomic medical evidence with negative implications and clear support for differential diagnosis is extracted from the differential reasons output by the evaluated model and the differential reasons corresponding to the differential diagnoses of the patient medical record;

[0045] Based on the extracted evidence, the reasonableness and completeness of the differential reasons output by the evaluated model are evaluated;

[0046] Reasonableness: define a binary variable , if the large language model considers that the jth evidence corresponding to the differential reasoning output by the evaluated model is medically reasonable, , otherwise ; the reasonableness of the differential reasoning output by the evaluated model is

[0047] ;

[0048] is the total number of evidence corresponding to the differential reasoning output by the evaluated model;

[0049] Completeness: define the completeness binary variable , if the large language model considers that the jth evidence corresponding to the differential reasoning output by the evaluated model is mentioned or covered by the differential reasoning in the patient's medical record labeled by the doctor, , otherwise ; the completeness of the differential reasoning output by the evaluated model is

[0050] ;

[0051] wherein, denotes the total number of evidence extracted by the large language model from the differential reasoning of the patient's medical record.

[0052] In one of the embodiments, the differential diagnosis list and the final principal diagnosis in the patient's medical record and the corresponding differential diagnosis list and the final principal diagnosis of the evaluated model are linked based on the international disease classification standard, specifically comprising:

[0053] The differential diagnosis list and the final principal diagnosis in the patient's medical record and the corresponding differential diagnosis list and the final principal diagnosis of the evaluated model are standardized to terms conforming to the international disease classification standard using the large language model; the standardized terms are vectorized with the items of the international disease classification standard using the medical encoder SapBERT, obtaining a term vector and an international disease classification standard vector database, calculating the cosine similarity between the term vector and all item vectors in the international disease classification standard vector database, and selecting the disease name corresponding to the item vector with the highest cosine similarity as the linking result.

[0054] Compared with the prior art, the beneficial technical effects of the present application are:

[0055] ​​Based on a deep understanding of clinical diagnostic methods and the need for systematic analysis and attribution of clinical diagnostic performance, this invention proposes a fine-grained unified assessment method. This method aims to deeply and synchronously evaluate the medical knowledge and reasoning ability of a large language model within the same clinical diagnostic context. Through a systematic and comprehensive assessment framework, this invention precisely explores the key factors influencing the clinical diagnostic performance of large language models from multiple perspectives, thereby enabling a more rigorous and comprehensive evaluation of their clinical diagnostic performance. The construction of this framework presents significant technical challenges and is of great importance in promoting the application of large language models in the field of clinical diagnosis. Attached Figure Description

[0056] Figure 1 This is the core evaluation framework in the embodiments of the present invention.

[0057] Figure 2 This describes the implementation process of the link module in this embodiment of the invention.

[0058] Figure 3 This is a flowchart illustrating the data cleaning process in an embodiment of the present invention.

[0059] Figure 4 This is a statistical chart showing the proportion of each department in the assessment dataset in this embodiment of the invention.

[0060] Figure 5 This is a line graph showing the comprehensive clinical diagnostic performance of the large language model in this embodiment of the invention.

[0061] Figure 6 This is a bar chart showing the correlation coefficients between the three key capabilities in this invention and the accuracy of clinical diagnosis.

[0062] Figure 7 This is a statistical chart showing the classification of medical records in the validation dataset during the evaluation of different models in this embodiment of the invention.

[0063] Figure 8 This is a scatter plot showing the relationship between medical knowledge and clinical diagnostic accuracy in an embodiment of the present invention.

[0064] Figure 9 This is a scatter plot showing the relationship between model recall capability and discrimination capability in an embodiment of the present invention.

[0065] Figure 10 This is a line graph illustrating the rationality and completeness of the identification reasons in the embodiments of the present invention. Detailed Implementation

[0066] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0067] The core evaluation framework of this invention, the implementation process of the linking modules in the evaluation framework, the data cleaning process, and the high-quality evaluation dataset constructed accordingly will be introduced in turn below.

[0068] 1. Core assessment framework.

[0069] The input to this invention is a real patient medical record dataset containing a list of differential diagnoses and a final primary diagnosis field. Optionally, the patient medical record also contains the reasons for the differential diagnoses and the reasons for the final primary diagnosis. For each patient medical record, this invention performs a medical knowledge assessment task and a reasoning ability assessment task to ensure the same clinical diagnostic scenario.

[0070] This invention defines a medical knowledge assessment task as a task that predicts related symptoms given a final primary diagnosis. The task flow is described below. Figure 1 (a) of the task requires the evaluated model to perform single-step knowledge retrieval rather than multi-step reasoning, thereby decoupling knowledge evaluation from reasoning ability evaluation.

[0071] Specifically, for each medical record, this invention first uses the GPT-4o Large Language Model to generate 10 questions around the final primary diagnosis annotated by the doctor (i.e., the final primary diagnosis in the patient's medical record), and simultaneously extracts the corresponding standard answers to form question-standard answer pairs (i.e., QG pairs). These questions do not broadly inquire about general information about the disease, but rather focus on the typical symptoms of the final primary diagnosis, including its causes, risk factors, abnormal findings in physical examination and auxiliary examinations, and particularly emphasize key details such as numerical ranges and thresholds. Subsequently, the model being evaluated needs to answer these questions and match them with the corresponding QG pairs to obtain question-standard answer-model answer triples (i.e., QGA triples). The purpose of this is to unify the correspondence among the three for overall consideration, ensuring a more rigorous and objective evaluation. Finally, the GPT-4o Large Language Model scores the answers and standard answers in these QGA triples from the following four aspects: 1) the degree of fit with the question, 2) the consistency with actual clinical manifestations, 3) the degree of conformity with medical facts, and 4) the rigor of the reasoning process. These carefully selected evaluation angles are sufficient to comprehensively measure the medical knowledge level of the evaluated model, avoiding overestimation or underestimation of the model's capabilities based solely on surface accuracy.

[0072] This invention uses a five-point Likert scale for scoring, suggesting that the GPT-4o large language model assign an ordered positive integer score (i.e., an ordered discrete variable) to each patient's medical record. , Indicates the sequence number of the patient's medical record. For the first The positive integer fractions of each patient's medical record at this stage, of which and is an integer, denotes the total number of patient records in the patient record dataset. It is worth mentioning that the higher the score, the higher the level of medical knowledge. Based on this, the medical knowledge score of the evaluated model is defined as:

[0073] ; (1)

[0074] In addition, when drawing the result graph, the medical knowledge score will be divided by 5 and converted into a percentage to ensure consistency.

[0075] It is worth mentioning that in the medical knowledge evaluation task, GPT-4o large language model is not only widely used in auxiliary functions such as QG pair construction that involve semantics and are irrelevant to evaluation, but also often acts as a referee to undertake the work of evaluation or scoring. This is mainly due to its excellent performance in reasoning, alignment and multilingual benchmarks, as well as its strong robustness and cross-task generalization ability; at the same time, the experimental results based on GPT-4o large language model have good reproducibility, making it a widely recognized third-party high-level model in academia and industry. Therefore, it will still be applied to similar scenarios in subsequent evaluations, especially in the evaluation or scoring link. It needs to be emphasized that GPT-4o large language model as a referee participating in evaluation does not conflict with its own being included in evaluation.

[0076] It is not easy to evaluate the medical reasoning ability in detail, because different evaluated models show diversified reasoning patterns. In order to compare these models fairly, the invention adopts the widely recognized hypothesis-deductive reasoning paradigm in clinical practice, further refines the reasoning into recall task and discrimination task, and designs the reasoning ability evaluation task flowchart as shown in (b) of Figure 1 . Specifically, the model first recalls a list of hypothetical diagnoses (i.e. differential diagnosis list) based on the patient's basic information (i.e. age, gender, occupation, etc. demographic information), medical history and physical examination results, to complete the recall task. The recall of each differential diagnosis follows three key clinical principles: 1) give priority to common diseases, 2) give priority to serious and potentially life-threatening diseases, and 3) follow the principle of diagnostic parsimony (try to explain all symptoms and signs with one disease). Subsequently, the evaluated model combines the auxiliary examination results and uses clinical reasoning to confirm or exclude each hypothesis until the final diagnosis is made, i.e. the discrimination process. At the same time, the evaluated model also needs to output the reasons for confirming or excluding each differential diagnosis. The invention carefully disassembles the high-difficulty overall evaluation task of reasoning ability into two independent and lower-difficulty sub-evaluation tasks, so as to clearly judge whether the wrong diagnosis is due to the inability to recall the differential diagnosis or the inability to effectively distinguish the differential diagnosis.

[0077] In the process of clinical diagnosis, physical examination and auxiliary examination are complementary examination methods. Physical examination focuses on the direct observation and sign finding of doctors, while auxiliary examination focuses on providing quantitative, imaging or molecular level objective evidence. Specifically, physical examination is usually completed by doctors directly at the bedside through basic methods such as visual examination, palpation, percussion, auscultation, etc. The main content includes general condition (body temperature, pulse, respiration, blood pressure, body position, nutritional status, etc.), skin mucosa, superficial lymph nodes, head and neck, chest and lungs, heart, abdomen, limbs and joints, and nervous system, etc. aspects of systematic examination, which is used to find clinical signs and make preliminary judgments. Auxiliary examination is to obtain objective data by means of laboratory or medical imaging, which covers hematology, biochemistry, immunology, infection, urine and feces laboratory tests, X-ray, ultrasound, CT, MRI, endoscopy, electrocardiogram, lung function and pathology examination, etc.

[0078] As shown in Figure 1 , the medical knowledge evaluation task and the reasoning ability evaluation task of the present application use the same high-quality real medical record data as a link, embodying the unity of the medical scene.

[0079] It needs to be further emphasized that the existing technologies generally ignore an important clinical cognition, that is, patients usually go to the hospital because of their most urgent or most pressing health problem (i.e. the only final main diagnosis). However, these existing technologies often generate a wide differential diagnosis list covering secondary or irrelevant diseases when evaluating clinical diagnosis, but rarely emphasize identifying or prioritizing the real clinical concern. This leads to a conceptual disconnection between these benchmarks and real-world clinical diagnosis. The present application is particularly concerned about this point, ensuring that only the only final main diagnosis is selected and other differential diagnoses are excluded in the differential stage, which is a significant difference in concept and methodology between the present application and previous work.

[0080] For the only final main diagnosis and multiple differential diagnoses generated by the doctor's annotation and the evaluated model, the present application uses a linking module to convert them into standardized Chinese medical diagnosis names conforming to the International Classification of Diseases 11th Edition (ICD-11). This step realizes the alignment of medical semantics and reduces the misjudgment caused by non-standard model expression. After the linking is completed, the present application evaluates the recall ability, discrimination ability and clinical diagnosis accuracy.

[0081] The hit rate measures the recall ability of the evaluated model, which indicates whether the final main diagnosis annotated by the doctor is included in the differential diagnosis list generated by the evaluated model. Specifically, the present application defines the hit rate binary variable , indicates that the final main diagnosis is included, indicates that the final main diagnosis is not included, wherein denotes the sequence number of the patient medical record, wherein and is an integer, denotes the total number of patient medical records in the patient medical record data set. Based on this, the hit rate The calculation formula is:

[0082] ; (2)

[0083] In addition, the present application also uses the score to evaluate the performance of differential diagnosis, which is defined as:

[0084] ; (3)

[0085] wherein, denotes the number of differential diagnoses that coincide in the list of differential diagnoses marked by the doctor and the list of differential diagnoses generated by the evaluated model, denotes the total number of differential diagnoses in the list of differential diagnoses marked by the doctor.

[0086] In the evaluation of differential ability, the present application is based on following the framework shown in (b) in the Figure 1 , the input of the differential stage is replaced by the list of differential diagnoses marked by the doctor to avoid unfair comparison due to the difference in recall stage of each model. The present application uses differential accuracy as an evaluation index. Specifically, the differential accuracy binary variable is defined as: when the final primary diagnosis output by the evaluated model is consistent with the final primary diagnosis marked by the doctor, otherwise wherein, denotes the sequence number of the patient medical record, wherein and is an integer, denotes the total number of patient medical records in the patient medical record data set. Based on this, the calculation formula is:

[0087] (4)

[0088] The application further evaluates the quality of the differential reason, and designs an evidence extraction-based quality evaluation scheme. Specifically, the application strictly follows the carefully designed extraction rules, and uses the GPT-4o large language model to extract medical evidence with negative meaning and clear support for identification from the differential reason generated by the evaluated model and the doctor's labeled differential reason. The extraction rules require: 1) each piece of evidence must only contain an independent, clear, specific and negative meaning medical fact, 2) there should be direct contradiction or conflict with the diagnosis result in each piece of evidence, 3) background information unrelated to the diagnosis, simple repetition of objective examination results, and conclusion judgments or neutral descriptions without specific facts should be excluded in the extraction process. In addition, the application also focuses on semantic merging and restoration when extracting evidence. For semantic merging, the GPT-4o large language model needs to merge evidence items with repetitive expressions or similar semantics into one to ensure clear expression; for semantic restoration, the GPT-4o large language model needs to trace back each medical fact referred to when it detects a referential negative summary such as "no above symptoms" or "no relevant performance" and restore it to negative atomic evidence instead of just extracting neutral words or symptom names. Through this scheme, the application converts long text that is difficult to evaluate into key items that are easy to evaluate.

[0089] The application evaluates the quality of the differential reason from the perspectives of rationality and completeness. For rationality, the rationality binary variable is defined, where if the GPT-4o large language model considers the jth evidence to be medically reasonable, then , otherwise , where j is the sequence number of the evidence extracted by the GPT-4o large language model from the differential reason output by the evaluated model, j is an integer, and j is an integer,

[0090] On this basis, the rationality of the differential reason is:

[0091] (5)

[0092] For completeness, the completeness binary variable is defined, where if the GPT-4o large language model considers the jth evidence to be mentioned or covered by the doctor's labeled differential reason, then , otherwise , where the definition of j is the same as that in the rationality, so it is not repeated here.

[0093] On this basis, the completeness of the differential reason is:

[0094] (6)

[0095] where denotes the total number of evidence extracted from the physician-annotated differential reasons by the GPT-4o large language model. It is noted that the completeness focuses on “whether covered by the standard answer”, which is the average of the standard answer, so it is divided by instead of .

[0096] Clinical diagnosis accuracy The evaluation follows the procedure shown in (b) in Figure 1 . Define the diagnosis accuracy binary variable when the final primary diagnosis output by the evaluated model is consistent with the physician-annotated final primary diagnosis, otherwise , and The definitions remain the same as before.

[0097] On this basis, the calculation formula is:

[0098] (7)

[0099] Based on these indicators, the present application uses the correlation coefficient to evaluate the influence of each factor on the clinical diagnosis accuracy, and then determines the key factors affecting the clinical diagnosis ability of the evaluated model.

[0100] For the mathematical definition of the correlation coefficient, let and be two binary variables, then the Pearson correlation coefficient between them is:

[0101] (8)

[0102] where is the total number of samples, is the sample number, and is an integer, and represent the mean of and , and are the corresponding th sample of and When it comes to the relationship between binary variables with values of 0 or 1 and ordinal variables, it is more appropriate to use the point biserial correlation coefficient.

[0103] Specifically, let the ordered variable be... A binary variable that takes the value 0 or 1 is The point-to-bicol correlation coefficient between the two is... for:

[0104] (9)

[0105] in and They are respectively according to The two groups are divided by whether the value is 1 or 0. The mean, for standard deviation and for according to The total number of samples in each of the two groups defined by whether the value is 1 or 0. .

[0106] In assessing the impact between recall capability and clinical diagnostic accuracy, this invention calculates the hit rate. With clinical diagnostic accuracy Pearson correlation coefficient between these two indicators ,in The corresponding hit rate binary variable That is , The corresponding bivariate variable of diagnostic accuracy That is In assessing the impact between discriminative ability and clinical diagnostic accuracy, this invention calculates... and Pearson correlation coefficient between these two indicators ,in The corresponding binary variable of identification accuracy That is , The corresponding bivariate variable of diagnostic accuracy That is In assessing the impact of medical knowledge scores on clinical diagnostic accuracy, this invention calculates... (correspond )and (correspond The point-to-bivariate correlation coefficient between these two indicators ,in, Corresponding positive integer fractions That is , The corresponding bivariate variable of diagnostic accuracy i.e. .

[0107] It is worth noting that when a variable is a binary variable, the point two column correlation coefficient is mathematically equivalent to the Pearson correlation coefficient. In the present invention, the calculation of all correlation coefficients, is and the index corresponds to a binary variable . Therefore, in the evaluation process of the present invention, the two correlation coefficients can be compared. In addition, although the two correlation coefficients are themselves existing statistical methods, overcoming their strict limit conditions and introducing them into the evaluation of the clinical diagnostic ability of large language models is the innovation of the present invention. This innovation benefits from the unified medical scenario constructed by the present invention: each medical record sample covers four dimensions of medical knowledge, recall ability, discrimination ability and clinical diagnostic accuracy, rather than being measured in isolation as in previous work. This design ensures that the influence of each factor can be explicitly quantified and reliably compared, thereby rigorously revealing the key driving factors of the clinical performance of the large language model being evaluated.

[0108] 2. Linking module implementation process.

[0109] Measuring the consistency between the output of the evaluated model and the medical physician annotation requires careful and precise processing. On the one hand, the output of the evaluated model often differs due to subtle changes in input format; on the other hand, the annotation results are influenced by factors such as personal experience, expression habits, and patient individual differences. Even if they are the same in medical sense, the difference in wording may lead to inconsistent judgments. Such biases can significantly distort the calculation results of the evaluation indicators and may lead to misjudgment of the model performance. Therefore, before evaluation, the linking module must standardize each diagnostic name to eliminate the influence of wording differences as much as possible, and then ensure the fairness and accuracy of the results.

[0110] The implementation process of the linking module designed by the present invention is shown in Figure 2 . The linking module, under the premise of ensuring correct alignment of medical semantics, realizes the standardization of diagnostic terms, and is an important module to ensure that the present invention can correctly evaluate the recall and discrimination tasks.

[0111] For each medical record, the input of the linking module includes the differential diagnosis list and the final principal diagnosis output by the evaluated model and labeled by doctors. The module processes each differential diagnosis in turn and outputs the corresponding standardized Chinese disease entry conforming to the International Classification of Diseases, 11th Revision (ICD-11). Specifically, the linking module first uses a stable large language model to standardize each diagnosis name into an ICD-11-compliant term. The purpose of this is to fully leverage the advantages of large language models in medical knowledge integration and semantic understanding, minimize potential semantic deviations or information loss in the conversion process, and thus lay a solid foundation for subsequent standardized mapping. Next, the module uses the excellent Chinese medical encoder SapBERT to perform text vectorization processing on the standardized terms and all valid entries in the ICD-11 file, obtaining a term vector and an ICD-11 vector database. Although SapBERT is an existing method, the design of combining it with ICD-11 valid entries is a unique solution carefully selected through extensive practice. In contrast, other encoders lack adaptability or performance in the Chinese medical context, and commonly used ICD-10 files are not fully covered, while methods using ICD-11 often ignore validity screening. The combination of the present invention ensures high-precision representation of Chinese medical semantics and relies on the integrity and authority of ICD-11 to achieve high coverage and accurate matching. Finally, the cosine similarity between the term vector and all entry vectors in the ICD-11 vector database is calculated, and the disease name corresponding to the entry vector with the highest cosine similarity is selected as the final linking result.

[0112] 3. Data cleaning process.

[0113] Evaluation methods require the support of data sets, but the data used by traditional methods are mostly selected, judged, or filled in the form of medical textbooks or examination questions. These data lack the support of real clinical data and have a large gap with actual medical scenarios. Although the latest methods incorporate patient real medical record information, the quality is not high and there are still problems such as unclear differentiation and difficulty in fully supporting the final principal diagnosis with medical record information. To overcome these defects and ensure a consistent medical context, the present invention designs a systematic and detailed data cleaning process, as shown in the flowchart Figure 3 The input is a non-standard medical record text file containing the differential diagnosis field, and the output is a standardized high-quality medical record text file. The process will be described in detail below.

[0114] First, data anonymization was performed using regular expressions to remove sensitive information such as institution names and addresses, ensuring data standardization and security and preventing information leakage. Second, medical records containing inappropriate medical content were removed, including: 1) diagnoses consisting of physical examination or auxiliary examination results, normal findings, or other non-disease information; 2) overlap between the final primary diagnosis and differential diagnosis; 3) overlap between differential diagnoses; and 4) diagnoses containing combined symptoms, accompanying symptoms, or additional explanations, violating the principle of singleness of symptoms. This approach comprehensively considered several key medical points that were often overlooked in previous datasets, ensuring the effectiveness and reliability of subsequent evaluation indicator calculations, improving data quality and the efficiency of subsequent screening processes, and addressing shortcomings in previous research in this area. Third, the GPT-4o large language model was used to assign each medical record to a corresponding medical department, with the assignment guidelines strictly following the standard clinical department classification compiled in the "Directory of Medical Institution Diagnosis and Treatment Subjects." The purpose of this step was both to provide a technical foundation for the deduplication process in subsequent structural screening and to support in-depth and detailed analysis of the performance of the evaluated model across different departments. Fourth, adjust medical record fields based on rules and the GPT-4o large language model, such as refining medical history (further refining past medical history into past medical history, family history, and allergy history), re-extracting differential diagnoses (if the field is not explicitly defined), and matching differential diagnoses with corresponding reasons. Fifth, structural-level screening. This screening step focuses on the external form of medical records, and the detailed procedures are as follows: 1) Remove medical records with missing or duplicate fields; 2) Remove highly similar medical records within the same department, i.e., if two medical records have the same chief complaint, present illness, and diagnosis, or if the cosine similarity after encoding into vectors is higher than 0.95, only one of them is retained; 3) Perform fine matching based on basic information (i.e., demographic information such as age, gender, and occupation) and medical history, and remove medical records containing duplicate patients; 4) Exclude records with incomplete information based on rules to ensure that all listed diagnoses in the medical records are supported by corresponding evidence. Finally, semantic-level screening. This screening process focuses on the internal meaning of medical records. The detailed procedures are as follows: 1) Using the GPT-4o large language model, medical records with low semantic relevance between differential diagnosis and final primary diagnosis are removed; 2) Using the GPT-4o large language model, medical records with semantically missing or unclear differential diagnoses are removed. Through this dual-layer structural and semantic screening, this invention transforms non-standard medical records into high-quality standard medical records. These standard medical records possess completeness, standardization, and clinical validity in both form and meaning, laying a solid foundation for subsequent evaluation on this dataset.

[0115] Following the above process, this invention constructs a unified and challenging evaluation dataset containing 410 rigorously screened, high-quality, standardized patient medical records covering eight major medical departments. The percentage of data across each department is shown in the following chart. Figure 4 As shown.

[0116] It is emphasized that the data source of the present application is the medical website in Chinese language and other open source data sets for obtaining medical record data. Through strict and meticulous screening, no identifiable personal health information is involved in the data set. Each standard medical record data constructed by the present application provides complete diagnostic information, including: basic information (age, gender, occupation, etc. population statistics), chief complaint, medical history (current history, past history, allergy history, family history, personal history), physical examination, auxiliary examination, differential diagnosis list, differential diagnosis reason list (optional), final main diagnosis, final main diagnosis reason (optional), and medical department.

[0117] The present application systematically evaluates nine different large language models, including the specialized models GPT-4o, O4-mini, Claude-3.7-Sonnet, Qwen2.5-Max and GPT-3.5-turbo, and the open source models Deepseek-V3, Deepseek-R1, Llama-3.3-70B and Qwen2.5-32B-Instruct.

[0118] The present application first evaluates the performance of large language models in clinical diagnosis, and the evaluation results are shown in Figure 5 The figure shows the comprehensive performance of the nine mainstream large language models evaluated by the present application in clinical diagnosis, with the horizontal coordinate representing the model name and the vertical coordinate representing the corresponding percentage value. Each model focuses on five indicators corresponding to four key dimensions, namely: medical knowledge score ), recall indicator , ), differential accuracy ), and clinical diagnosis accuracy .

[0119] It can be seen that the clinical diagnosis accuracy ) of none of the models exceeds 60%, which cannot meet the strict requirements of medical clinical environment. This shows that the mainstream large language models still have a lot of room for improvement in clinical diagnosis. In addition, the hit rate of all large language models on the recall task ) is less than 60%, and the highest score is less than 30%, which is extremely weak performance. This shows that even the most advanced large language models still perform poorly in differential diagnosis tasks and have a significant gap with medical experts. Figure 5 The results show the urgency of improving the clinical diagnosis capability of large language models.

[0120] On this basis, the present application further explores the main reasons for the poor performance of clinical diagnosis through correlation coefficient. As Figure 6As shown, all large language models exhibit the same pattern: a correlation between recall and clinical diagnostic accuracy. ) is significantly higher than the difference between differential diagnosis ability and medical knowledge and the accuracy of clinical diagnosis. and ). Figure 6 The horizontal axis represents the name of the large language model, and the vertical axis represents the correlation coefficient, with values ​​ranging from 0 to 1. The bar charts corresponding to each large language model, from left to right, show the correlation coefficient values ​​between the three capabilities of medical knowledge, recall, and discrimination and the clinical diagnostic accuracy.

[0121] Therefore, poor diagnostic performance primarily stems from the recall task—the first stage of reasoning—failing to recall an accurate list of differential diagnoses, rather than from the non-differentiation task—the second stage of reasoning—lacking in differential diagnostic ability or medical knowledge. To further verify this conclusion, such as... Figure 7 As shown, this invention divides each medical record in the evaluation dataset into three categories: recall failure, recall success but identification failure, and complete success.

[0122] The results show that recall failures occur far more frequently than discrimination failures across all models, further demonstrating that recall capability is the main bottleneck in the reasoning process. Furthermore, discrimination failures remain consistently low in both number and proportion, indicating that existing large language models perform well overall in terms of discrimination capability.

[0123] Following this, the present invention provides a more detailed evaluation of the differences in key capabilities among large language models. Figure 5 Further drawing based on Figure 8 and Figure 9 . Figure 8 This study focuses on the relationship between medical knowledge and diagnostic accuracy. Specifically, mainstream advanced large language models show relatively small differences in medical knowledge, such as Claude-3.7-Sonnet, O4-mini, and Deepseek-V3. It is worth noting that a higher level of medical knowledge does not necessarily lead to higher diagnostic accuracy. For example, while Claude-3.7-Sonnet ranks first in medical knowledge score, its diagnostic accuracy is only at a moderate level; conversely, some large language models with slightly lower medical knowledge scores (such as Deepseek-V3, GPT-4o, and Qwen2.5-Max) actually outperform in clinical diagnosis. Figure 9Further focusing on the relationship between recall ability and discrimination ability in the reasoning process, the results show that the performance gap of each large language model in recall ability is much larger than that in discrimination ability. Among them, the recall ability presents a clear gradient distribution: Qwen2.5-Max and Deepseek-V3 perform best, followed by GPT-4, Claude-3.7-Sonnet and Deepseek-R1, and the remaining models perform relatively weakly. In contrast, the difference between each model in discrimination ability is small, and is concentrated in Figure 9 the upper area. Among them, the reasoning model O4-mini and Deepseek-R1 perform best. These results further confirm that recall ability is the key factor driving reasoning performance.

[0124] Figure 8 The abscissa of is , and the ordinate is Figure 9 . .

[0125] The present application deeply excavates the knowledge mastery level of large language models in various medical departments. Tables 1 to 8 show in detail the medical knowledge, recall ability and exclusion ability of each large language model in different departments. In general, mainstream large language models perform well in common departments such as internal medicine and surgery, but have obvious shortcomings in professional departments such as pediatrics, obstetrics and gynecology, and oncology, especially in pediatrics. In addition, almost all large language models have the weakest recall ability in each department, which is significantly lower than other abilities, further verifying that recall ability is the main bottleneck of diagnostic performance. It is worth noting that whether the same large language model in different departments or different large language models in the same department, the recall ability shows a large fluctuation of more than 10%, even 20%, indicating that its stability in recall tasks needs to be improved. In contrast, the exclusion ability of each large language model in the same department has a small difference, and only in the oncology and pediatrics departments does it show a significant decrease, and the rest of the departments remain relatively stable. In terms of knowledge mastery, Deepseek-V3 and Claude-3Sonnet perform outstandingly; O4-mini has a knowledge difference of only 5.8% between departments, which is much smaller than other large language models, showing excellent cross-department consistency.

[0126] Table 1, ability performance of each evaluated model in the infectious department:

[0127]

[0128] Table 2, ability performance of each evaluated model in the pediatric department:

[0129] ​

[0130] Table 3, the ability performance of each evaluated model in obstetrics and gynecology:

[0131]

[0132] Table 4, the ability performance of each evaluated model in psychiatry:

[0133]

[0134] Table 5, the ability performance of each evaluated model in internal medicine:

[0135]

[0136] Table 6, the ability performance of each evaluated model in surgery:

[0137]

[0138] Table 7, the ability performance of each evaluated model in pediatric surgery:

[0139]

[0140] Table 8, the ability performance of each evaluated model in oncology:

[0141]

[0142] Finally, the present application evaluates the rationality and completeness of the differential reasons, as shown in Table 9. The results show that the rationality scores of mainstream large language models are all above 77%, indicating that they can generally generate medically reasonable explanations. Among them, the reasoning model performs best, with Deepseek-R1 scoring the highest and O4-mini second. This shows that the reasoning model has a significant advantage in providing medically reasonable differential reasons. However, the completeness scores of mainstream large language models are relatively low, even if GPT-4o, which is the most complete, only reaches 42.4%. This shows that although existing models can often generate medically correct explanations, the points covered are still far from the comprehensive points involved in expert reasoning in clinical practice. Figure 10 Table 9, the rationality and completeness of the differential reasons of each evaluated model. Figure 10 Among them, the horizontal coordinate is the name of the large language model, and the vertical coordinate is the score of the index.

[0143] All models used in the present application are obtained by API calls, and the values of the two key parameters of temperature parameter and random seed are set to 0.0 to ensure the reproducibility of the results, and the remaining parameters are default values. For the mixed inference model Claude-3.7-Sonnet, the present application sets it to non-inference mode. In addition, the number of retrieved differential diagnoses is fixed to be consistent with the number of diagnoses labeled in each medical record, and does not need to be manually set, and will be automatically set when reading data. All experiments of the present application are implemented based on Python3.8.

[0144] Before use, the key of the API needs to be set to the key registered by the user personally. When using, the process of Figure 1 needs to be followed to set the name of the large language model to be evaluated and the data set path, and the present application can complete the index calculation and evaluation link of medical knowledge, recall task, identification task and clinical diagnosis accuracy.

[0145] The terms used herein are merely intended to describe specific embodiments and are not intended to limit the present application. The terms "include", "contain" and the like used herein indicate the presence of the described features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0146] Based on the description of the above method embodiments, the present application also provides a system. The system can be a software (application), module, component, server, client, etc. using the method described in the embodiments of the present application, combined with the necessary implementation hardware. Based on the same innovative concept, the system in one or more embodiments provided by the present application is described in the following embodiments. Since the implementation scheme of the system to solve the problem is similar to the method, the specific system implementation of the present application can be referred to the implementation of the foregoing method, and the repeated parts will not be described again. The term "module" or "module" used below is a combination of software and / or hardware that can implement a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware implementation is also possible and is conceived.

[0147] A large language model clinical diagnosis capability fine-grained evaluation system, comprising:

[0148] Medical record acquisition module: acquire patient medical record data set containing differential diagnosis list and final main diagnosis;

[0149] Evaluation module: synchronously execute the following tasks in the same clinical diagnosis scenario:

[0150] The medical knowledge evaluation task is as follows: the large language model generates a question based on the final main diagnosis of the patient medical record, obtains a standard answer, obtains the answer of the evaluated model to the question, and forms a question-standard answer-model answer triple; the triple is scored based on a predetermined scoring dimension, and the medical knowledge score of the evaluated model is calculated;

[0151] The reasoning ability evaluation task is as follows: the evaluated model performs a recall task based on the patient medical record, generates a differential diagnosis list, and performs a differential task to confirm or exclude the differential diagnosis list to obtain a unique final main diagnosis; based on the international disease classification standard, the differential diagnosis list and the final main diagnosis in the patient medical record and the differential diagnosis list and the final main diagnosis corresponding to the evaluated model are linked;

[0152] The analysis module is as follows: based on the linking result, the hit rate of the evaluated model in the recall task, the differential accuracy rate in the differential task, and the clinical diagnosis accuracy rate are evaluated; the correlation coefficients between the medical knowledge score, the hit rate, the differential accuracy rate, and the clinical diagnosis accuracy rate are calculated to analyze the factors affecting the clinical diagnosis ability of the evaluated model.

[0153] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present disclosure.

[0154] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered as exemplary and non-limiting, and the scope of the present application is defined by the appended claims, not the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application, and any reference signs in the claims should not be considered as limiting the claims.

[0155] In addition, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be combined appropriately to form other embodiments that those skilled in the art can understand.

Claims

1. A fine-grained assessment method for the clinical diagnostic capability of a large language model, characterized in that, include: Obtain a patient medical record dataset containing a list of differential diagnoses and the final primary diagnosis; Perform the following tasks simultaneously within the same clinical diagnostic setting: Medical knowledge assessment task: The large language model generates questions based on the final primary diagnosis of the patient's medical records and obtains the standard answer. The model being evaluated then obtains its answer to the questions, forming a triplet of question-standard answer-model answer. The triplet is scored based on a predetermined scoring dimension, and the medical knowledge score of the model being evaluated is calculated. Reasoning Ability Assessment Task: The model being assessed performs a recall task based on the patient's medical records to generate a list of differential diagnoses, and then performs an identification task to confirm or exclude the differential diagnoses to arrive at a unique final primary diagnosis. Specifically, the patient's medical records include basic patient information, chief complaint, medical history, physical examination results, and auxiliary examination results. Recall Task: Based on the patient's basic information, chief complaint, medical history, and physical examination results, the model being assessed recalls a list of differential diagnoses, including multiple differential diagnoses. Identification Task: Combining the auxiliary examination results, the model being assessed uses clinical reasoning to confirm or exclude each differential diagnosis and outputs the corresponding identification reason until a unique final primary diagnosis is obtained. Based on the International Classification of Diseases, the differential diagnosis list and final primary diagnosis in the patient's medical record are linked with the differential diagnosis list and final primary diagnosis corresponding to the evaluated model; The hit rate of the evaluated model in the recall task, the identification accuracy in the identification task, and the clinical diagnosis accuracy were evaluated based on the linking results. The correlation coefficients between medical knowledge score, hit rate, identification accuracy and clinical diagnosis accuracy were calculated to analyze the factors affecting the clinical diagnosis ability of the evaluated model. It also includes a process for quality assessment of the differential diagnoses: using a large language model, atomic medical evidence with negative meaning that clearly supports the differential diagnosis is extracted from the differential diagnoses output by the model being evaluated and the differential diagnoses in the patient's medical records; the reasonableness and completeness of the differential diagnoses output by the model being evaluated are assessed based on the extracted evidence; reasonableness: defining a binary variable of reasonableness. If the large language model deems the j-th piece of evidence corresponding to the reason for identification output by the evaluated model to be medically reasonable, then ,otherwise The reasonableness of the identification reasons output by the evaluated model. ; The total number of pieces of evidence corresponding to the reasons for identification output by the model being evaluated; Completeness: Define a completeness binary variable. If the large language model considers the j-th piece of evidence corresponding to the identification reason output by the evaluated model to be mentioned or covered by the identification reason in the patient's medical record annotated by the doctor, then ,otherwise The completeness of the identification reasons output by the evaluated model. ; This represents the total amount of evidence extracted by the large language model from the patient's medical records for the reasons for identification.

2. The method for fine-grained evaluation of the clinical diagnostic capability of a large language model according to claim 1, characterized in that, The large language model generates a question based on the final primary diagnosis of the patient's medical record, obtains the standard answer, and obtains the model's response to the question, forming a question-standard answer-model response triplet. The triples are scored based on predetermined scoring dimensions, and the medical knowledge score of the evaluated model is calculated, specifically including: For each medical record, the large language model generates a final primary diagnosis field based on the doctor-annotated patient medical record. The questions are presented, along with their corresponding answers. A question-and-answer pair, which, together with the evaluated model's response to the question, constitutes... One question - standard answer - model answer triplet; The large language model assigns each medical record a specific language level based on its relevance to the problem, consistency with actual clinical manifestations, alignment with medical facts, and the rigor of its reasoning process. Each triple is scored, and the scores corresponding to all medical records are added together and averaged to obtain the medical knowledge score of the evaluated model.

3. The method for fine-grained evaluation of the clinical diagnostic capability of a large language model according to claim 1, characterized in that, Based on the International Classification of Diseases (ICD), the differential diagnosis list and final primary diagnosis in the patient's medical record are linked to the differential diagnosis list and final primary diagnosis corresponding to the evaluated model. Specifically, this includes: The differential diagnosis list and final primary diagnosis in patient medical records, as well as the differential diagnosis list and final primary diagnosis corresponding to the evaluated model, were normalized into terms conforming to the International Classification of Diseases (ICD) standard using a large language model. The normalized terms and ICD entries were vectorized using the medical encoder SapBERT to obtain term vectors and an ICD standard vector database. The cosine similarity between the term vectors and all entry vectors in the ICD standard vector database was calculated, and the disease name corresponding to the entry vector with the highest cosine similarity was selected as the link result.

4. A fine-grained assessment system for the clinical diagnostic capabilities of a large language model, characterized in that, include: Medical record acquisition module: Acquires patient medical record datasets containing a list of differential diagnoses and the final primary diagnosis; Assessment module: Simultaneously executes the following tasks within the same clinical diagnostic scenario: Medical knowledge assessment task: The large language model generates questions based on the final primary diagnosis of the patient's medical records and obtains the standard answer. The model being evaluated then obtains its answer to the questions, forming a triplet of question-standard answer-model answer. The triplet is scored based on a predetermined scoring dimension, and the medical knowledge score of the model being evaluated is calculated. Reasoning Ability Assessment Task: The model being assessed performs a recall task based on the patient's medical record to generate a list of differential diagnoses, and then performs an identification task to confirm or exclude the differential diagnoses to arrive at a unique final primary diagnosis. Specifically, the patient's medical record includes basic patient information, chief complaint, medical history, physical examination results, and auxiliary examination results. Recall Task: Based on the patient's basic information, chief complaint, medical history, and physical examination results, the model being assessed recalls a list of differential diagnoses, including multiple diagnoses. Identification Task: Combining the auxiliary examination results, the model being assessed uses clinical reasoning to confirm or exclude each differential diagnosis and outputs the corresponding identification reason until a unique final primary diagnosis is reached. Based on the International Classification of Diseases (ICD), the differential diagnosis list and final primary diagnosis in the patient's medical record, as well as the differential diagnosis list and final primary diagnosis of the model being assessed, are linked. Analysis module: Based on the link results, evaluate the hit rate of the evaluated model in the recall task, the identification accuracy in the identification task, and the clinical diagnosis accuracy; calculate the correlation coefficients between medical knowledge score, hit rate, identification accuracy and clinical diagnosis accuracy, respectively, to analyze the factors affecting the clinical diagnosis ability of the evaluated model. It also includes a quality assessment process for the reasons for identification, specifically including: Using a large language model, atomic medical evidence with negative meaning that can clearly support the differential diagnosis is extracted from the differential reasoning output by the evaluated model and the differential diagnosis corresponding to the differential diagnosis in the patient's medical records. The reasonableness and completeness of the identification reasons output by the evaluated model are assessed based on the extracted evidence; Reasonableness: Define a binary variable for reasonableness. If the large language model deems the j-th piece of evidence corresponding to the reason for identification output by the evaluated model to be medically reasonable, then ,otherwise The reasonableness of the identification reasons output by the evaluated model. for: ; The total number of pieces of evidence corresponding to the reasons for identification output by the model being evaluated; Completeness: Define a completeness binary variable. If the large language model considers the j-th piece of evidence corresponding to the identification reason output by the evaluated model to be mentioned or covered by the identification reason in the patient's medical record annotated by the doctor, then ,otherwise The completeness of the identification reasons output by the evaluated model. for: ; in, This represents the total amount of evidence extracted by the large language model from the patient's medical records for the reasons for identification.

5. The fine-grained assessment system for the clinical diagnostic capability of a large language model according to claim 4, characterized in that, The large language model generates a question based on the final primary diagnosis of the patient's medical record, obtains the standard answer, and obtains the model's response to the question, forming a question-standard answer-model response triplet. The triples are scored based on predetermined scoring dimensions, and the medical knowledge score of the evaluated model is calculated, specifically including: For each medical record, the large language model generates a final primary diagnosis field based on the doctor-annotated patient medical record. The questions are presented, along with their corresponding answers. A question-and-answer pair, which, together with the evaluated model's response to the question, constitutes... One question - standard answer - model answer triplet; The large language model assigns each medical record a specific language level based on its relevance to the problem, consistency with actual clinical manifestations, alignment with medical facts, and the rigor of its reasoning process. Each triple is scored, and the scores corresponding to all medical records are added together and averaged to obtain the medical knowledge score of the evaluated model.

6. The fine-grained assessment system for the clinical diagnostic capability of a large language model according to claim 4, characterized in that, Based on the International Classification of Diseases (ICD), the differential diagnosis list and final primary diagnosis in the patient's medical record are linked to the differential diagnosis list and final primary diagnosis corresponding to the evaluated model. Specifically, this includes: The differential diagnosis list and final primary diagnosis in patient medical records, as well as the differential diagnosis list and final primary diagnosis corresponding to the evaluated model, were normalized into terms conforming to the International Classification of Diseases (ICD) standard using a large language model. The normalized terms and ICD entries were vectorized using the medical encoder SapBERT to obtain term vectors and an ICD standard vector database. The cosine similarity between the term vectors and all entry vectors in the ICD standard vector database was calculated, and the disease name corresponding to the entry vector with the highest cosine similarity was selected as the link result.

Images