Large language model medical numerical reasoning capability process evaluation method and system

Abstract

The application relates to the technical field of medical artificial intelligence, and discloses a large language model medical numerical reasoning ability process evaluation method and system. The method comprises the following steps: obtaining a structured abstract of medical research literature; inputting the structured abstract into a large language model to be evaluated, performing binary judgment on each candidate conclusion, and outputting a correct candidate conclusion set supported by numerical evidence and context information and a reasoning text; determining the correctness of the large language model at the answer level; aligning the reasoning text with a preset structured standard explanation item by item to evaluate the reasoning process of the large language model; and comprehensively evaluating the correctness at the answer level and the evaluation result of the reasoning process to output a comprehensive ability evaluation of the large language model in a medical numerical reasoning task. The application can systematically, finely and interpretively evaluate the comprehensive numerical reasoning ability of the large language model in a real medical research context.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, specifically to a process-oriented evaluation method and system for medical numerical reasoning ability of large language models. Background Technology

[0002] In recent years, large language models have demonstrated strong capabilities in tasks such as medical question answering, medical literature processing, electronic medical record analysis, and clinical decision support, showing their potential for application in assisting information processing and decision analysis in the medical field. However, medical research and clinical practice widely involve complex information such as numerical data, statistical indicators, time points, dosage changes, inter-group differences, and significance tests. Relying solely on medical knowledge memorization or general textual reasoning is insufficient to fully measure a model's ability to understand and reason about medical numerical evidence.

[0003] Existing evaluation methods for large language models in the medical field largely focus on medical knowledge question answering, clinical text understanding, medical record analysis, or qualitative reasoning tasks, with relatively insufficient assessment of numerical reasoning capabilities. While some existing methods involve medical computation, numerical extraction, or quantification problem solving, they typically rely on pre-defined formulas, fixed rules, or simple numerical comparisons. For example, they might perform arithmetic operations based on explicitly given calculation instructions or directly extract single numerical results from text. These evaluation methods struggle to cover the complex numerical relationships present in real-world medical research scenarios. Actual medical literature and clinical research results often contain information such as multiple control groups, multiple time-point observations, multiple clinical indicators, dose differences, confidence intervals, p-values, and statistical significance. Models need to comprehensively understand the context, determine the direction of numerical changes and statistical significance, and further deduce research conclusions in the absence of predefined rules. Therefore, existing evaluation methods are insufficient to adequately measure the comprehensive numerical reasoning capabilities of large language models in the medical field.

[0004] Furthermore, existing evaluation methods mostly focus on whether the final answer is correct, lacking fine-grained analysis of the model's reasoning process. When the model gives an incorrect conclusion, it is difficult to determine whether the error stems from incomplete extraction of numerical evidence, incorrect medical or statistical numerical judgments, or a problem with the logical chain of reasoning between the evidence and the candidate conclusion. Therefore, there is an urgent need for a process-oriented evaluation method and system for comprehensive numerical reasoning capabilities in the medical field, enabling systematic, fine-grained, and interpretable evaluation of the answer results and reasoning processes of large language models in real medical research contexts. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a process-oriented evaluation method and system for the medical numerical reasoning capabilities of large language models. This method comprehensively measures the overall capabilities of large language models in numerical evidence understanding, statistical judgment, and logical reasoning formation within a real medical research context. The purpose of this invention is to provide a scientific basis for the capability verification, performance analysis, and optimization of medical artificial intelligence models, thereby promoting the reliability and interpretability of large language models in medical research and clinical applications.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: In a first aspect, the present invention provides a process-oriented evaluation method for the medical numerical reasoning ability of a large language model, comprising: Obtain structured summaries of medical research literature, which include contextual information, experimental results containing numerical evidence, and a set of candidate conclusions consisting of correct and incorrect conclusions; The structured summary is input into the large language model to be evaluated, which then performs binary judgments on each candidate conclusion and outputs a set of correct candidate conclusions supported by numerical evidence and contextual information, as well as the reasoning text. The correct candidate conclusion set is compared with the preset standard answer set to determine the correctness of the large language model at the answer level. The reasoning text is aligned with the pre-defined structured standard interpretation item by item, and the reasoning process of the large language model is evaluated from three process evaluation dimensions: numerical evidence coverage, numerical judgment consistency, and reasoning logic sufficiency. By combining the correctness of the answer with the evaluation results of the reasoning process, a comprehensive assessment of the large language model's ability in medical numerical reasoning tasks is output.

[0007] In one embodiment, the structured standard interpretation includes three parts: a standard numerical evidence extraction part, a standard numerical judgment part, and a standard reasoning basis part, wherein: The standard numerical evidence extraction section records the numerical evidence required to support the correct candidate conclusion; The standard numerical judgment section records numerical judgments based on numerical evidence; The standard reasoning basis section records the standard reasoning basis that connects numerical evidence, numerical judgments, and correct candidate conclusions.

[0008] In one embodiment, the numerical evidence includes: measurements, time points, doses, percentages, inter-group differences, significance probabilities, and confidence intervals.

[0009] In one embodiment, the evaluation of the reasoning process of the large language model from three dimensions—numerical evidence coverage, consistency of numerical judgment, and sufficiency of reasoning logic—specifically includes: The string matching method is used to determine whether the reasoning text covers each piece of numerical evidence in the standard numerical evidence extraction section, and the evidence coverage ratio is calculated. The evaluation model is used to determine whether the reasoning text meets the standard numerical judgment criteria in the standard numerical judgment section, and the judgment consistency ratio is calculated. The evaluation model is used to compare the consistency of the reasoning chain between the reasoning text and the standard reasoning basis, and the reasoning quality score is output.

[0010] In one embodiment, the evaluation results of the combined answer level and the reasoning process output a comprehensive assessment of the large language model's ability in medical numerical reasoning tasks, specifically including: Pearson correlation coefficients were calculated between the evaluation dimensions of each process and the answer accuracy of the large language model to analyze the degree of influence of different reasoning stages on the final answer performance of the large language model.

[0011] Secondly, this invention provides a process-oriented evaluation system for medical numerical reasoning ability using a large language model, comprising: The input construction module is used to obtain structured summaries of medical research literature, which include contextual information, experimental results containing numerical evidence, and a set of candidate conclusions consisting of correct and incorrect conclusions. The model response module is used to input structured summaries into the large language model to be evaluated, so that the large language model can make binary judgments on each candidate conclusion and output a set of correct candidate conclusions supported by numerical evidence and contextual information, as well as the reasoning text. The answer analysis module is used to compare the set of correct candidate conclusions with the preset set of standard answers to determine the correctness of the large language model at the answer level. The standard interpretation alignment module is used to align the reasoning text with the preset structured standard interpretation item by item, and evaluate the reasoning process of the large language model from three process evaluation dimensions: numerical evidence coverage, numerical judgment consistency and reasoning logic sufficiency. The evaluation results output module integrates the correctness of the answer with the evaluation results of the reasoning process, and outputs a comprehensive evaluation of the large language model's ability in medical numerical reasoning tasks.

[0012] In one embodiment, the structured standard interpretation includes three parts: a standard numerical evidence extraction part, a standard numerical judgment part, and a standard reasoning basis part, wherein: The standard numerical evidence extraction section records the numerical evidence required to support the correct candidate conclusion; The standard numerical judgment section records numerical judgments based on numerical evidence; The standard reasoning basis section records the standard reasoning basis that connects numerical evidence, numerical judgments, and correct candidate conclusions.

[0013] In one embodiment, the numerical evidence includes: measurements, time points, doses, percentages, inter-group differences, significance probabilities, and confidence intervals.

[0014] In one embodiment, the evaluation of the reasoning process of the large language model from three dimensions—numerical evidence coverage, consistency of numerical judgment, and sufficiency of reasoning logic—specifically includes: The string matching method is used to determine whether the reasoning text covers each piece of numerical evidence in the standard numerical evidence extraction section, and the evidence coverage ratio is calculated. The evaluation model is used to determine whether the reasoning text meets the standard numerical judgment criteria in the standard numerical judgment section, and the judgment consistency ratio is calculated. The evaluation model is used to compare the consistency of the reasoning chain between the reasoning text and the standard reasoning basis, and the reasoning quality score is output.

[0015] In one embodiment, the evaluation results of the combined answer level and the reasoning process output a comprehensive assessment of the large language model's ability in medical numerical reasoning tasks, specifically including: Pearson correlation coefficients were calculated between the evaluation dimensions of each process and the answer accuracy of the large language model to analyze the degree of influence of different reasoning stages on the final answer performance of the large language model.

[0016] The system and method in this invention correspond to each other; the specific technical solutions applicable to the method are also applicable to the system.

[0017] Compared with the prior art, the beneficial technical effects of the present invention are: This invention enables a systematic, fine-grained, and interpretable evaluation of the comprehensive numerical reasoning capabilities of large language models within a real-world medical research context. Through a multi-answer, multi-select task, this invention reduces subjectivity in open-ended generative evaluation while retaining the challenge of complex medical numerical reasoning tasks. Through structured standard interpretation, this invention can pinpoint specific weaknesses in the model's reasoning process from three dimensions: data entity extraction, numerical judgment, and reasoning basis. Through Pearson correlation analysis and error type analysis, this invention can further reveal which capability aspects most directly affect the model's final answer performance. Therefore, this invention can not only be used for performance comparisons between different large language models but also for model error diagnosis, prompt strategy optimization, training data improvement, and reliability verification before medical application deployment, demonstrating strong technical practical value and positive effects. Attached Figure Description

[0018] Figure 1 This is the core evaluation framework diagram of the present invention.

[0019] Figure 2 This is a schematic diagram of the multi-answer, multi-select medical numerical reasoning task of the present invention.

[0020] Figure 3 A flowchart for constructing the medical numerical reasoning sample dataset of this invention.

[0021] Figure 4 This is an example diagram of the medical numerical reasoning sample dataset of the present invention.

[0022] Figure 5 This is an example diagram illustrating a structured standard interpretation of an instance of the medical numerical reasoning sample dataset of the present invention.

[0023] Figure 6 This is a distribution diagram of the medical numerical reasoning sample dataset of the present invention across various medical specialties.

[0024] Figure 7 This is a distribution diagram of the medical numerical reasoning sample dataset of the present invention across four types of numerical reasoning tasks.

[0025] Figure 8 This is a flowchart of the method of the present invention. Detailed Implementation

[0026] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0027] like Figure 8 As shown, this invention provides a procedural evaluation method for the medical numerical reasoning ability of large language models, including the following steps: S1, Obtain a structured summary of medical research literature, the structured summary including contextual information, experimental results containing numerical evidence, and a set of candidate conclusions consisting of correct and incorrect conclusions; S2, input the structured summary into the large language model to be evaluated, so that the large language model can make binary judgments on each candidate conclusion and output the set of correct candidate conclusions supported by numerical evidence and contextual information and the reasoning text. S3, compare the set of correct candidate conclusions with the preset set of standard answers to determine the correctness of the large language model at the answer level; S4 aligns the reasoning text with the pre-defined structured standard interpretation item by item, and evaluates the reasoning process of the large language model from three process evaluation dimensions: numerical evidence coverage, numerical judgment consistency, and reasoning logic sufficiency. S5, combining the correctness of the answer and the evaluation results of the reasoning process, outputs a comprehensive evaluation of the large language model's ability in medical numerical reasoning tasks.

[0028] This invention, based on numerical evidence and research conclusions from medical research literature, constructs a multi-answer, multi-select medical numerical reasoning evaluation task. It combines structured standard interpretation to simultaneously evaluate the answer results and reasoning process of a large language model. The core evaluation framework, process-oriented evaluation index system, and the process for constructing the medical numerical reasoning sample dataset of this invention will be introduced below.

[0029] 1. Core assessment framework.

[0030] The core evaluation framework of this invention uses structured abstracts from medical research literature as input to conduct a procedural and fine-grained evaluation of the comprehensive numerical reasoning ability of a large language model in the medical field. Each evaluation sample includes the research background and purpose, research methods, experimental results containing numerical evidence, and a set of candidate conclusions. The evaluated model needs to determine whether each candidate conclusion is supported by the research results based on the medical numerical evidence and contextual information (including research background and purpose, research methods) in the abstract, and output its selected set of conclusions and the corresponding reasoning process. Through this framework, this invention can simultaneously examine the model's final answer performance and intermediate reasoning process within a unified medical research context. The core evaluation framework is as follows: Figure 1 As shown.

[0031] This invention does not employ traditional single-answer selection, simple numerical calculation, or open-ended conclusion generation methods in its task design. Instead, it designs the comprehensive numerical reasoning ability assessment in the medical field as a multi-answer, multi-select conclusion judgment task based on medical research abstracts, such as... Figure 2 As shown. The core concept of this design is to preserve the complexity of numerical reasoning in real medical research scenarios while ensuring that the evaluation results can be standardized and automatically calculated. Conclusions in medical research abstracts are usually not derived directly from a single numerical value, but require judgment based on a combination of information, including background, experimental methods, results from multiple groups of subjects, observations at different time points, changes in clinical indicators, statistical significance, and differences between groups. Therefore, relying solely on single-choice questions or fixed formulas is insufficient to fully assess the model's comprehensive reasoning ability; while open-ended conclusion generation is easily influenced by differences in expression, language style, and subjective interpretation, leading to unstable evaluations. This invention employs a multi-answer, multi-select format, requiring the model to separately determine whether each candidate conclusion is supported by numerical evidence, thus simultaneously considering task complexity, evaluation objectivity, and result reproducibility.

[0032] Specifically, for the first A sample of medical research abstracts is used to define a set of candidate conclusions. ,in, Indicates the first All candidate conclusions in a sample of medical research abstracts Indicates the first The first medical research abstract sample There are n candidate conclusions, where n represents the total number of candidate conclusions in the medical research abstract sample. This invention denotes the set of standard answers for the medical research abstract sample as... , representing the set of candidate conclusions All correct conclusions that can be supported by medical numerical evidence, therefore yes A subset of can be represented as The modeling task can be formalized as processing each candidate conclusion in the set. Functions for performing binary judgments : ; in, The set of numerical evidence representing the experimental results. This provides contextual information about the research background and methodology.

[0033] Accordingly, the prediction set output by the large language model is defined as follows: ; This task requires the set of correct candidate conclusions predicted by a large language model. With the set of standard answers Completely consistent. This task covers multiple capability requirements in medical numerical reasoning. The large language model first needs to locate numerical evidence from the experimental results, then determine numerical differences, directions of change, statistical significance, and the meaning of clinical indicators, and finally, map the numerical evidence to candidate conclusions in conjunction with the research background and experimental methods. Since candidate conclusions often have high semantic similarity, incorrect options may only differ slightly from correct options in terms of time point, indicator direction, statistical significance, intervention group, clinical indicator subjects, or the scope of application of the conclusion. Therefore, the large language model must complete comprehensive reasoning under conditions lacking predefined calculation formulas or fixed reasoning rules. Through the above task design, this invention can more realistically evaluate the ability of the large language model to derive research conclusions from numerical evidence in medical research scenarios.

[0034] After defining the task, this invention further designed a method for aligning the large language model's response process with the inference text to comprehensively record the behavior of the large language model in medical numerical reasoning tasks. For each sample of medical research abstracts, the large language model first receives complete input, including the research background and purpose, research methods, experimental results, and a set of candidate conclusions. The experimental results section provides numerical evidence, while the background and methods section provides additional context to help the model understand the experimental design, intervention groups, control settings, and outcome indicators. Based on this understanding, the large language model needs to determine whether each candidate conclusion is supported by evidence, while simultaneously forming intermediate inference text.

[0035] Large language models can employ various prompting methods for answering questions, including direct answering, stepwise reasoning, or structured stepwise reasoning. In direct answering mode, the large language model directly outputs its set of correctly predicted candidate conclusions. In the stepwise reasoning mode, the model generates natural language reasoning steps before outputting the answer. In the structured stepwise reasoning mode, the large language model outputs intermediate reasoning in a preset format, including question focus, key numerical evidence, and logical reasoning chains that map the evidence to candidate conclusions. The large language model's response not only yields the final answer but also generates structured information that can be used for process-based evaluation. Through the above settings, this invention can compare the impact of different prompting methods on the large language model's answer results and reasoning text under the same evaluation task, and provide analyzable large language model outputs for subsequent process-based evaluation.

[0036] It is important to emphasize that this invention does not presuppose that the reasoning process of a large language model is inherently composed of several fixed parts. Instead, it uses manually annotated structured standard interpretations as a reference to perform a process-oriented comparison of the reasoning text generated by the large language model. For each correct candidate conclusion in a sample, the manually annotated structured standard interpretation includes three parts: standard numerical evidence extraction (DataEntity), standard numerical judgment (NumericalJudgement), and standard reasoning basis (ReasoningBasis).

[0037] Based on the aforementioned structured standard interpretation, this invention performs a point-by-point alignment and comparison between the reasoning text generated by the large language model and the manually annotated structured standard interpretation. Let the manually annotated structured standard interpretation of the i-th sample be... ,in Extraction of standard numerical evidence Standard numerical judgment section and standard reasoning basis part Composition. Let the reasoning text generated by the large language model be... This invention does not require the output text of the large language model to be in the same form as... Completely consistent, but judging the reasoning text of the large language model. Is it covered? Does the numerical evidence in the data meet the requirements? The necessary numerical judgments in the process, and whether a relationship has been formed with Consistent or equivalent evidence leads to logical conclusions. Therefore, process-oriented assessment can perform fine-grained analysis of large language model reasoning texts, starting from structured standard interpretations, at the evidence, judgment, and reasoning levels.

[0038] This invention combines final answer evaluation with reasoning process evaluation through this process. If the large language model's final answer is correct, but its reasoning text fails to contain numerical evidence or correctly interprets statistical meaning, it indicates that the large language model may have issues with accidental correct answers or insufficient evidence. If the large language model's final answer is incorrect, further analysis can be conducted to determine whether the error stems from missing key evidence, incorrect numerical direction or significance judgment, or insufficient reasoning logic between the evidence and the candidate conclusion. Therefore, this invention can not only determine whether the large language model has selected the correct conclusion, but also further reveal the specific capability bottlenecks of the large language model in the medical numerical reasoning process, providing a basis for large language model optimization, capability diagnosis, and reliability assessment in medical scenarios.

[0039] From a system implementation perspective, the core evaluation framework of this invention can be organized into a series of stages, including input construction, large language model response, answer parsing, standard interpretation alignment, and evaluation result output. The input construction stage organizes the background and purpose, research methods, experimental results, and candidate conclusions from the medical research abstract into a unified format. The large language model response stage calls the large language model being evaluated and obtains its output through direct response, stepwise reasoning, or structured stepwise reasoning. The answer parsing stage extracts the set of predicted candidate conclusions from the large language model's response. The standard interpretation alignment stage compares the reasoning text generated by the large language model with manually annotated structured standard interpretations. The evaluation result output stage summarizes the performance of the large language model in aspects such as answer selection, numerical evidence coverage, numerical judgment, and reasoning basis. Through the above steps, this invention forms a complete evaluation closed loop from medical research abstract input, large language model conclusion judgment, inference text collection to process analysis output. It can not only fairly compare the final performance of different large language models on the same medical numerical reasoning task, but also reveal the differences in the capabilities of large language models in specific reasoning stages, thereby supporting the large language model error diagnosis, prompt strategy optimization, training data improvement, and reliability verification before medical application deployment.

[0040] 2. Process-oriented evaluation indicator system.

[0041] This invention, based on the core evaluation framework, further constructs an evaluation index system that combines answer-level and process-level metrics. Answer-level metrics measure whether the large language model correctly completes the multiple-answer, multiple-choice conclusion determination task, while process-level metrics measure the consistency between the large language model's reasoning text and the manually annotated structured standard interpretation. Specifically, this invention first rigorously matches the large language model's predicted answer set with the standard answer set to obtain answer accuracy. Then, based on the manually annotated structured standard interpretation, it performs a fine-grained evaluation of the reasoning text generated by the large language model from three dimensions: data entity extraction, numerical judgment, and reasoning basis. Finally, it further calculates the correlation between each process-level metric and the final answer accuracy to analyze the impact of different reasoning stages on the final performance of the large language model.

[0042] In the answer-level evaluation, for the i-th medical research abstract sample, the set of correctly predicted candidate conclusions by the large language model is denoted as . The set of standard answers is denoted as This invention uses strict set consistency as the criterion for judging the correctness of the answer; that is, the large language model is considered to have answered correctly on the sample only when all candidate conclusions selected by the large language model are completely consistent with the standard answer set. Define binary variables. Let represent the correctness of the answer for the i-th sample, then: ; in, This includes situations where the large language model misses any correct conclusion or incorrectly selects any candidate conclusion that is not supported by evidence. Assuming there are N samples in the evaluation set, the overall accuracy (Acc) of the large language model at the answer level is defined as: .

[0043] This metric reflects the overall answer selection ability of large language models in multi-answer, multi-select medical numerical reasoning tasks. Unlike the lenient partial matching approach, this metric emphasizes the importance of selecting all supported conclusions, thus avoiding overestimation when large language models only identify partially correct conclusions.

[0044] In the process-level evaluation, this invention uses manually annotated structured standard interpretations as the gold standard to perform fine-grained comparisons of the reasoning texts generated by the large language model. For each medical research abstract sample, the manually annotated structured standard interpretation is denoted as... , It consists of three parts: Standard Numerical Evidence Extraction (DataEntity), Standard Numerical Judgment (NumericalJudgement), and Standard Reasoning Basis (Reasoning Basis), denoted as follows: , and ,Right now: ; The standard numerical evidence extraction (DataEntity) part This represents the set of numerical evidence required to support the correct candidate conclusion in the i-th sample. This section, derived from the results section of a medical research abstract, records the raw numerical expressions necessary to reach a conclusion, including measurements, time points, doses, percentages, inter-group differences, p-values, confidence intervals, and other statistical results that directly support the conclusion. It can be represented as: .

[0045] in, Let represent the k-th numerical evidence in the i-th sample. The purpose of this set is to clarify which raw numerical information the large language model should identify and utilize during the reasoning process, thereby avoiding the large language model making judgments based solely on surface semantics or generalized medical knowledge.

[0046] Standard numerical judgment section Indicates to The direct interpretation and judgment of numerical evidence. This section does not simply restate the original values, but further explains the directional relationships between the values, the magnitude of the differences, the meaning of the clinical indicators, and statistical significance. It can be formally expressed as: .

[0047] in, This represents the numerical judgment based on the k-th numerical evidence, where g represents the judgment made by the human annotator based on the numerical evidence. and context information The process of making a judgment This section provides contextual information. Through this part, the structured standard interpretation not only indicates "which values ​​are important," but also further indicates "what these values ​​illustrate."

[0048] Standard Reasoning Basis This section outlines the standard reasoning that connects numerical evidence, numerical judgments, and correct candidate conclusions. It explains why the corresponding numerical evidence and judgments support a given correct candidate conclusion, and, in conjunction with relevant medical knowledge, demonstrates the logical chain from medical numerical evidence to research conclusions. It can be represented as: .

[0049] in, Let j represent the j-th correct candidate conclusion in the i-th sample. Let f represent the set of standard answers for the i-th sample, and let f represent the process by which the human annotator synthesizes numerical evidence and numerical judgment, and combines it with medical-related knowledge to conduct logical reasoning to obtain the correct candidate conclusion, thus reflecting the logical reasoning process.

[0050] With standardized interpretation, this invention will transform the reasoning text generated by the large language model. With structured standard interpretation Layer-by-layer alignment and procedural evaluation are performed. For the numerical evidence extraction (DataEntity) dimension, this invention evaluates large language model inference text. Whether it covers the key original numerical expressions required to complete the reasoning. Specifically, this invention uses a string matching method to match the structured standard interpretation. Each piece of partially labeled numerical evidence is used sequentially to infer the text using the large language model. Matching is performed within this framework. If a certain numerical evidence can be found... If a corresponding expression is accurately found or expressed in an equivalent numerical form, the evidence is considered successfully covered by the large language model; if no corresponding expression is found, the evidence is considered not covered. Finally, the score on this dimension is... Defined as the percentage of numerical evidence that successfully matches a large language model The proportion of the total amount of numerical evidence.

[0051] Regarding the numerical judgment dimension, this invention assesses whether the large language model correctly understands the direct numerical relationships and meanings expressed by the standard numerical evidence. Numerical judgment mainly includes judgments on numerical differences, effect direction, clinical indicator meaning, and statistical significance. Specifically, this invention addresses the structured standard interpretation... Each numerical judgment in the part can be evaluated and scored using an evaluation model (such as the GPT-5-Mini model) to judge the inference text generated by the large language model. Whether a direct numerical or statistical conclusion identical to the standard numerical judgment is explicitly reached. A binary scoring method is used: if the model's inference text can achieve the same judgment, the judgment is recorded as 1; if the large language model omits the judgment, is too vague, gives an incorrect effect direction, uses the wrong comparison object, misunderstands statistical significance, or contradicts the gold standard numerical judgment, the judgment is recorded as 0. Finally, the score of the i-th sample on the Numerical Judgment dimension is calculated. The number of standard numerical judgments defined as having a score of 1 accounts for the majority. The proportion of the total quantity is determined by the standard value.

[0052] Regarding the reasoning basis dimension, this invention evaluates whether the reasoning text generated by the large language model can connect numerical evidence, contextual information, and relevant medical knowledge to conduct complete logical reasoning and arrive at the correct candidate conclusion. Specifically, this invention uses GPT-5-Mini as the evaluation model and tests the reasoning text generated by the tested large language model. With structured standard interpretation The comparison is performed to determine whether the large language model forms an inference chain consistent with or equivalent to the gold standard around the correct candidate conclusion, and the explanation process generated by the large language model is scored on an integer scale from 0 to 10. The assessment focuses on whether the large language model's reasoning text possesses logical consistency, whether it fully integrates numerical evidence and judgments, and whether it clearly explains how this evidence supports the corresponding candidate conclusions. Higher scores indicate a more complete reasoning chain, more comprehensive evidence integration, and clearer support for the selected candidate conclusions; lower scores indicate logical inconsistencies, insufficient evidence integration, or an inability to effectively support the selected candidate conclusions.

[0053] The three process indicators mentioned above characterize the reasoning ability of large language models from different levels. Data Entity primarily focuses on whether the large language model has found the correct key original numerical evidence; Numerical Judgment focuses on whether the large language model has correctly understood this numerical evidence; and Reasoning Basis further focuses on whether the large language model can transform numerical interpretations into a complete reasoning chain supporting candidate conclusions. These three indicators are progressively related, and this indicator system can reveal the specific shortcomings of large language models in the medical numerical reasoning process in a more detailed way than a single answer accuracy rate.

[0054] To further analyze the relationship between process-oriented indicators and final answer performance, this invention uses the Pearson correlation coefficient to conduct correlation analysis on different process evaluation dimensions. Assume there are T large language models or combinations of large language models and prompting methods under the same evaluation conditions, and let the final answer accuracy of the t-th evaluation subject be denoted as... Its overall score on a certain process indicator is denoted as ,in They can be represented as follows: , or .set up The average value is , The average value is The Pearson correlation coefficient r between this process-oriented indicator and the answer accuracy rate is defined as: ; Where t represents the t-th evaluated object, and T represents the total number of evaluated objects participating in the correlation analysis. The value of r ranges from -1 to 1. When r is close to 1, it indicates a strong positive correlation between the process indicator and the final answer accuracy; when r is close to 0, it indicates a weak linear correlation; and when r is less than 0, it indicates a negative correlation between the process indicator and the final answer. By calculating the Pearson correlation coefficient between each process evaluation dimension and the answer accuracy, we can analyze which reasoning stages have a stronger impact on the final answer performance of the large language model.

[0055] To ensure the reliability of the automated process evaluation, this invention can also introduce a manual review mechanism to verify the consistency of the automated scoring results. Specifically, a certain number of samples can be randomly selected from the evaluation sample, and human annotators with medical backgrounds can review and score the reasoning text of the large language model according to the same scoring criteria. The manual scoring results are then compared with the automated evaluation results for consistency analysis. When the manual scoring and automated evaluation results show a high degree of consistency, it indicates that the process evaluation system can stably reflect the true reasoning quality of the large language model. By combining answer accuracy, three types of process indicators, Pearson correlation analysis, and manual consistency verification, this invention forms a complete evaluation indicator system covering the final result, reasoning process, and ability attribution.

[0056] 3. Medical numerical inference sample dataset construction process.

[0057] To support the aforementioned core evaluation framework and process-oriented evaluation indicator system, this invention constructs a medical numerical reasoning sample dataset for evaluating comprehensive medical numerical reasoning capabilities. This dataset is built upon structured summaries of randomized controlled trials (RCTs). RCTs typically contain clearly defined research objectives, experimental methods, experimental results, and research conclusions. Their results often include numerical information such as multiple controls, time points, clinical indicators, and statistical significance, making them naturally suitable as a data source for evaluating the medical numerical reasoning capabilities of large language models. By selecting such literature, this invention ensures that the evaluation samples possess strong medical authenticity and clinical research relevance. The construction process of the medical numerical reasoning sample dataset is as follows: Figure 3 As shown.

[0058] During the data collection phase, this invention screens randomized controlled trials containing structured abstracts from the PubMed database, prioritizing samples where research conclusions need to be derived from numerical information in the experimental results. For each structured abstract, this invention uses the results section as the primary basis for reasoning, sub-conclusions in the conclusion section as sources of candidate correct conclusions, and other sections such as background and purpose, and research methods as supplementary contextual information. Since the original results section may contain judgmental expressions such as "increase," "decrease," and "significantly improve," this invention further uses a large language model to rewrite such judgmental expressions into neutral statements. While preserving the original numerical information and statistical meaning, it minimizes linguistic cues in the text that directly imply the conclusions, thereby improving the numerical reasoning ability required for the assessment task.

[0059] In the multiple-answer multiple-choice question construction phase, this invention adopts a uniform question format for each sample, requiring the large language model to select all options supported by research findings from candidate conclusions based on numerical evidence in the abstract. Specifically, this invention first breaks down the original conclusion into several sub-conclusions supported by outcome evidence, and selects one or more of these sub-conclusions as correct options. Subsequently, this invention generates several deceptive incorrect options, which, together with the correct options, form a candidate conclusion set. Incorrect options are designed to closely resemble the correct options in terms of language length, expression style, and medical semantics, but lack supporting numerical evidence. Common construction methods include modifying time points, altering statistical significance, replacing clinical indicators, adjusting intervention groups, or changing the direction of the conclusion. In this way, each sample forms a multiple-answer multiple-choice question containing five candidate conclusions, and the large language model must determine whether each candidate conclusion is supported by numerical evidence.

[0060] In the structured standard interpretation generation stage, this invention constructs a corresponding structured standard interpretation for each correct candidate conclusion supported by evidence. This structured interpretation includes a standard numerical evidence extraction (DataEntity) part, a standard numerical judgment (NumericalJudgement) part, and a standard reasoning basis (ReasoningBasis) part. Specifically, the standard numerical evidence extraction (DataEntity) part records key numerical expressions related to the conclusion derivation in the original text, such as measurement values, time points, doses, percentages, p-values, confidence intervals, and other experimental results; the standard numerical judgment (NumericalJudgement) part records direct local judgments on these numerical expressions, such as numerical differences, direction of change, inter-group comparisons, and statistical significance; and the standard reasoning basis (ReasoningBasis) part records the reasoning basis connecting the numerical judgments, medical background knowledge, and candidate conclusions. By constructing structured interpretations for correct candidate conclusions, this invention not only obtains standard answers at the answer level but also standard references required for process-level evaluation.

[0061] To ensure dataset quality, this invention further incorporates a manual review and quality control process. First, candidate randomized controlled trial abstracts undergo preliminary screening, checking whether the results and conclusions sections are sufficiently detailed and mutually verifiable, removing approximately 25% of samples whose conclusions lack supporting experimental results. Subsequently, annotators with medical backgrounds manually verify the candidate samples, candidate options, and structured interpretations. For candidate options, annotators check whether correct options are indeed supported by the results data and whether incorrect options are misleading but not supported by numerical evidence. For structured interpretations, annotators focus on verifying the completeness and accuracy of key values ​​in the standard numerical evidence extraction (DataEntity) section, whether there are omissions or errors in the numerical judgments in the standard numerical judgment (NumericalJudgement) section, and whether the reasoning logic from evidence to conclusion in the standard reasoning basis (ReasoningBasis) section is sufficient, clear, and consistent. For any issues discovered, annotators delete, supplement, correct, or rewrite the data to improve sample quality and the reliability of the standard interpretations.

[0062] Ultimately, this invention resulted in a dataset containing 970 medical numerical reasoning instances. Each instance includes the research background and purpose, research methods, experimental results containing numerical data, five candidate conclusion options, and a manually reviewed structured standard interpretation, such as... Figure 4 and Figure 5 As shown in the figure. This dataset covers more than 20 medical specialties and further encompasses various numerical reasoning tasks such as statistical interpretation, clinical indicator understanding, time reasoning, and comparative and trend analysis. The statistical results are as follows. Figure 6 and Figure 7 As shown. Through the above construction process, this invention obtains a data foundation that has both real medical research sources and standardized answers and procedural explanations, providing reliable support for the systematic evaluation of the comprehensive numerical reasoning capabilities of large language models in medicine.

[0063] This invention systematically evaluates various types of large language models, including general closed-source models GPT-4o, O4-Mini, GPT-5.1, Qwen3.6-Plus, Claude-Opus-4.5, and Doubao-Seed-2-0-Pro; general open-source models DeepSeek-R1, DeepSeek-V3.2, Qwen2.5-Max, and Qwen3-Max; and medical-specific models Baichuan-M2, Baichuan-M3, and HuatuoGPT-o1-8B. By uniformly evaluating these models on a medical numerical reasoning dataset, this invention can systematically compare the final answer performance and procedural reasoning capabilities of different types of large language models in comprehensive medical numerical reasoning tasks.

[0064] This invention first evaluated the accuracy of various language models in a multi-answer, multi-select medical numerical reasoning task, and the evaluation results are shown in Table 1. It can be seen that the performance differences among different large language models are quite significant. Claude-Opus-4.5 achieved the highest average accuracy of 72.50%; Qwen3.6-Plus also performed well with an average accuracy of 72.19%; in contrast, HuatuoGPT-o1-8B had the lowest average accuracy of only 37.19%. Furthermore, even the best-performing large language model only achieved a maximum single-item accuracy of 77.50%, which is still insufficient to meet the high reliability requirements of numerical evidence understanding and conclusion derivation in rigorous medical research scenarios. This indicates that while current advanced large language models possess certain medical numerical reasoning capabilities, there is still significant room for improvement in complex tasks requiring the simultaneous processing of multiple sets of experimental results, statistical significance, clinical indicator meanings, and candidate conclusion support relationships. It is noteworthy that medical-specific models generally performed weaker than most general-purpose models, indicating that while existing medical-specific models may have advantages in medical knowledge, they still suffer from insufficient training in medical numerical reasoning.

[0065] Table 1. Performance of various language models in conclusion reasoning tasks in medical numerical reasoning:

[0066] Regarding prompting strategies, this invention compares the impact of standard prompts, ordinary thought chain prompts, and structured thought chain prompts on the performance of large language models. Table 1 shows that ordinary thought chain prompts do not always improve the accuracy of the final answer of large language models. For some large language models, free-form reasoning processes may introduce irrelevant values, incorrect comparison objects, or unsupported information, leading to the propagation of errors in subsequent answer generation. In contrast, structured thought chain prompts, by requiring the large language model to organize around the question focus, key evidence, option judgment, reasoning bridging, and final answer, can constrain the reasoning path to a certain extent, reducing irrelevant expansion and the introduction of incorrect evidence. Therefore, this invention can not only evaluate the medical numerical reasoning ability of large language models themselves, but also analyze the impact of different prompting strategies on the answer results and reasoning quality of large language models, providing a basis for the optimization of prompts for large language models in medical scenarios. The values ​​in Table 1 are all accuracy rates (percentages) under strict set consistency, meaning that a model is only considered correct when the set of candidate conclusions selected by the large language model is completely consistent with the set of standard answers. "" indicates the best performance of each large language model under the three prompting strategies, bold indicates the large language model with the highest average accuracy, and underline indicates the best result under a single setting.

[0067] Based on the final answer evaluation, this invention further conducted a fine-grained evaluation of the reasoning process generated by the large language model, and the evaluation results are shown in Table 2. Overall, structured thought chain prompts improve the performance of procedural metrics on most large language models, especially in the data entity extraction dimension. For example, GPT-4o scored 26.05% in data entity extraction with ordinary thought chain prompts, but improved to 77.01% with structured thought chain prompts, indicating that structured prompts significantly help large language models locate numerical evidence. The numerical judgment and reasoning basis dimensions also showed improvement on most large language models, indicating that structured reasoning enhances the ability of large language models to organize numerical differences, statistical significance, and evidence-to-conclusion relationships.

[0068] Table 2. Procedural evaluation results of various language models in medical numerical reasoning tasks:

[0069] Table 2 shows the reasoning performance of the large language model under two settings: ordinary thought chain prompts and structured thought chain prompts. DE represents the accuracy rate of numerical evidence extraction, and NJ represents the accuracy rate of numerical judgment, both expressed as percentages; RB represents the quality score of the reasoning basis from evidence to candidate conclusion, ranging from 0 to 10. Pearson r represents the Pearson correlation coefficient between each process indicator and the final answer accuracy rate, used to analyze the impact of different reasoning stages on answer performance. Bold and underlined lines represent the best and second-best results in each dimension, respectively. "" represents the maximum Pearson coefficient under a certain prompting strategy.

[0070] In terms of performance across process-oriented language models, GPT-5.1 excels in multiple process metrics. With structured thought chain prompts, it achieves a DataEntity score of 78.75%, a NumericalJudgement score of 81.15%, and a ReasoningBasis score of 9.10. Claude-Opus-4.5 performs best in ReasoningBasis, reaching 9.14 with structured thought chain prompts, while its NumericalJudgement score reaches 83.23%, indicating strong numerical judgment and reasoning chain organization capabilities. Other large language models such as Qwen3-Max, Qwen3.6-Plus, and Doubao-Seed-2-0-Pro also demonstrate strong process-oriented reasoning capabilities with structured thought chain prompts. In contrast, HuatuoGPT-o1-8B performed poorly in extracting numerical evidence (DataEntity), achieving only 16.96% even with structured thought chain prompts, indicating that the medical-specific large language model still has significant shortcomings in the localization of numerical evidence.

[0071] This invention further analyzed the relationship between process-oriented indicators and the accuracy of the final answer using Pearson correlation coefficients. Table 2 shows that, under ordinary thought chain prompts, the Pearson correlation coefficients between the three evaluation dimensions—DataEntity, NumericalJudgement, and ReasoningBasis—and the accuracy of the final answer are 0.34, 0.57, and 0.59, respectively; under structured thought chain prompts, the Pearson correlation coefficients are 0.44, 0.81, and 0.77, respectively. It can be seen that all process-oriented indicators are positively correlated with the accuracy of the final answer, while the correlation between NumericalJudgement and ReasoningBasis is significantly higher than that between DataEntity and NumericalJudgement. This indicates that medical numerical reasoning tasks not only require large language models to extract raw numerical values, but more importantly, to correctly interpret numerical differences, effect directions, and statistical significance, and to coherently map these judgments to candidate conclusions. These results further demonstrate that the process-oriented evaluation indicator system proposed in this invention can effectively reveal the impact of different reasoning stages on the performance of the final answer.

[0072] This invention further analyzes the error types of large language models in medical numerical reasoning. Specifically, this invention randomly selected 200 samples and manually classified and statistically analyzed the errors in the reasoning results of the large language models, as shown in Table 3. This invention categorizes large language model reasoning errors into three types: the first type is numerical evidence extraction errors, i.e., omissions, mismatches, or illusions when the large language model extracts numerical evidence from the context; the second type is numerical understanding and judgment errors, i.e., errors when the large language model interprets medical numerical values, judges directions, or understands statistical significance; and the third type is logical chain errors, i.e., insufficient logic or connection errors in the reasoning process from evidence to candidate conclusions. Statistical results show that numerical understanding and judgment errors account for the highest proportion, averaging 0.39; followed by numerical evidence extraction errors, averaging 0.32; and logical chain errors, averaging 0.18. This indicates that the main bottleneck of current large language models in medical numerical reasoning is not simply the ability to organize language logic, but rather focuses more on medical numerical understanding, statistical significance judgment, and numerical evidence localization.

[0073] Table 3. Distribution of error types of major language models in medical numerical reasoning tasks:

[0074] Table 3 shows the percentage of different types of inference errors in the large language model obtained through manual analysis of 200 randomly sampled samples. Type A represents errors in numerical evidence extraction, including: data omission (A1), where the large language model fails to extract the numerical evidence required for inference; data mismatch (A2), where the large language model uses incorrect or mismatched numerical evidence for inference; and data illusion (A3), where the large language model generates numerical evidence that does not exist in the original text. Type B represents errors in numerical understanding and judgment, including: errors in understanding medical indicators (B1), where the large language model misunderstands the meaning of medical indicators or clinical numerical values; errors in direction judgment (B2), where the large language model incorrectly judges directional relationships such as increase, decrease, better than, and worse than; and errors in judging statistical significance (B3), where the large language model misunderstands p-values, confidence intervals, or significance conclusions. Type C represents errors in logical chains, including: errors in reasoning chains (C1), where the large language model has problems such as insufficient logic, incorrect evidence connection, or invalid conclusion support relationships when deriving candidate conclusions from numerical evidence and numerical judgments. Error Rate represents the overall error rate of the corresponding large language model in the manually analyzed sample. In Table 3, the underlined type represents the most widespread major error type, and the bold type represents the second most widespread major error type.

[0075] From the perspective of error subtypes, data omission is the most common problem in numerical evidence extraction, with an average proportion of 0.15%, higher than data mismatch and numerical illusion. This indicates that large language models are more likely to ignore numerical evidence than to simply extract erroneous or fabricated values. Among numerical understanding and judgment errors, errors in understanding medical indicators have the highest proportion, averaging 0.20, while errors in direction judgment and statistical significance judgment are also relatively common. This further illustrates that when faced with medical research abstracts, large language models often need not only to identify the numerical values ​​themselves but also to understand their specific meaning within particular medical indicators, experimental groups, and statistical tests.

[0076] To verify the reliability of the process-based automated evaluation, this invention further compared the automated scoring results with the human scoring results for evaluating large language models, as shown in Table 4. Under both ordinary and structured thought chain prompts, the automated and human scoring maintained a high degree of consistency overall. Although the automated scoring was slightly higher than the human scoring overall, the trends of the two were basically consistent, indicating that the automated process evaluation method used in this invention can stably reflect the reasoning quality of large language models. This result proves that the process-based evaluation system of this invention not only has fine-grained analysis capabilities but also possesses good operability and evaluation reliability.

[0077] Table 4. Consistency verification between process-automated evaluation results and manual scoring results:

[0078] In Table 4, Numerical Judgement represents the evaluation of the numerical judgment part; Reasoning Basis represents the evaluation of the reasoning basis part.

[0079] Figure 1 The overall evaluation process of this invention is demonstrated. The system takes a medical research abstract and a set of candidate conclusions as input, calls the large language model being evaluated to generate a set of predicted answers and inference text, and combines the standard answer set and manually annotated structured standard interpretations to comprehensively evaluate the final answer result and intermediate inference process of the large language model. Finally, it outputs the answer accuracy, process indicators, correlation analysis and error type analysis results.

[0080] Figure 2 This paper demonstrates a multi-answer, multiple-choice medical numerical reasoning task. The large language model relies on the results section of the medical research abstract as the primary numerical evidence, and combines this with contextual information provided by the research background, objectives, and methods to evaluate the evidence support for each candidate conclusion option, ultimately selecting all the correct options supported by the research results.

[0081] Figure 3 The main construction process of the medical numerical reasoning dataset is demonstrated.

[0082] Figure 4 The data structure of an instance of a medical numerical reasoning dataset. Figure 5 It displays a structured explanation of the correct options that have been manually reviewed.

[0083] Figure 6 This demonstrates the distribution of the medical numerical reasoning dataset across various medical specialties. The dataset covers more than 20 specialty areas, including cardiology, dermatology, nephrology, and urology.

[0084] Figure 7 This paper presents the distribution of the medical numerical reasoning dataset across four types of numerical reasoning tasks. We categorize numerical reasoning tasks into four types: statistical interpretation, clinical indicator understanding, temporal reasoning, and comparison and trend analysis. The medical numerical reasoning dataset covers all four types of tasks.

[0085] Example: 1. Key settings: All large language models used in this invention can be obtained via API calls. To ensure the stability and reproducibility of the experimental results, the temperature parameter is set to 0.0 when calling the large language model, while other parameters can use default settings. The system operating environment can be implemented based on Python, and the API key, large language model name, dataset path, and prompting method of the large language model to be evaluated are pre-configured. For evaluation settings that require reasoning text, the system can choose between ordinary thought chain prompts or structured thought chain prompts; for settings that only evaluate the final answer, the system can directly obtain the candidate conclusion set selected by the large language model using the standard prompting method.

[0086] 2. Usage process: When using this invention, the API key is first set to the user's personally registered key, and the name of the large language model to be evaluated, the path to the medical numerical reasoning dataset, and the evaluation prompt method are specified. Then, the system automatically reads the research background and purpose, research methods, experimental results, and candidate conclusion set from each sample according to the core evaluation framework, and calls the large language model to be evaluated to complete the multi-answer, multi-select medical numerical reasoning task. After the large language model outputs, the system automatically parses the predicted answer set and extracts the large language model's reasoning text when procedural evaluation is required, comparing it with the standard answer set and manually annotated structured standard explanations. Finally, the system automatically outputs scores for procedural indicators such as answer accuracy, data entity extraction, numerical judgment, and reasoning basis, and can further calculate Pearson correlation coefficients and error type analysis results.

[0087] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0088] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0089] Based on the description of the above method embodiments, the present invention also provides a system. The system may be a system that uses software (applications), modules, components, servers, clients, etc., using the methods described in the embodiments of this specification, combined with necessary implementation hardware. Since the implementation schemes and methods for solving the problem are similar, the specific system implementations in the embodiments of this specification can be found in the implementations of the foregoing methods, and repeated details will not be described again. Although the system described in the embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0090] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0092] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A procedural evaluation method for medical numerical reasoning ability using a large language model, characterized in that, include: Obtain structured summaries of medical research literature, which include contextual information, experimental results containing numerical evidence, and a set of candidate conclusions consisting of correct and incorrect conclusions; The structured summary is input into the large language model to be evaluated, which then performs binary judgments on each candidate conclusion and outputs a set of correct candidate conclusions supported by numerical evidence and contextual information, as well as the reasoning text. The correct candidate conclusion set is compared with the preset standard answer set to determine the correctness of the large language model at the answer level. The reasoning text is aligned with the pre-defined structured standard interpretation item by item, and the reasoning process of the large language model is evaluated from three process evaluation dimensions: numerical evidence coverage, numerical judgment consistency, and reasoning logic sufficiency. By combining the correctness of the answer with the evaluation results of the reasoning process, a comprehensive assessment of the large language model's ability in medical numerical reasoning tasks is output.

2. The method for procedural evaluation of medical numerical reasoning ability using a large language model according to claim 1, characterized in that, The structured standard interpretation comprises three parts: standard numerical evidence extraction, standard numerical judgment, and standard reasoning basis. The standard numerical evidence extraction section records the numerical evidence required to support the correct candidate conclusion; The standard numerical judgment section records numerical judgments based on numerical evidence; The standard reasoning basis section records the standard reasoning basis that connects numerical evidence, numerical judgments, and correct candidate conclusions.

3. The method for procedural evaluation of medical numerical reasoning ability using a large language model according to claim 2, characterized in that, The numerical evidence includes: measurements, time points, doses, percentages, differences between groups, significance probabilities, and confidence intervals.

4. The method for procedural evaluation of medical numerical reasoning ability using a large language model according to claim 2, characterized in that, The evaluation of the reasoning process of the large language model is carried out from three dimensions: coverage of numerical evidence, consistency of numerical judgment, and sufficiency of reasoning logic. Specifically, it includes: The string matching method is used to determine whether the reasoning text covers each piece of numerical evidence in the standard numerical evidence extraction section, and the evidence coverage ratio is calculated. The evaluation model is used to determine whether the reasoning text meets the standard numerical judgment criteria in the standard numerical judgment section, and the judgment consistency ratio is calculated. The evaluation model is used to compare the consistency of the reasoning chain between the reasoning text and the standard reasoning basis, and the reasoning quality score is output.

5. The method for procedural evaluation of medical numerical reasoning ability using a large language model according to claim 1, characterized in that, The evaluation results of the correctness and reasoning process at the comprehensive answer level output a comprehensive assessment of the large language model's ability in medical numerical reasoning tasks, specifically including: Pearson correlation coefficients were calculated between the evaluation dimensions of each process and the answer accuracy of the large language model to analyze the degree of influence of different reasoning stages on the final answer performance of the large language model.

6. A process-oriented evaluation system for medical numerical reasoning ability based on large language models, characterized in that, include: The input construction module is used to obtain structured summaries of medical research literature, which include contextual information, experimental results containing numerical evidence, and a set of candidate conclusions consisting of correct and incorrect conclusions. The model response module is used to input structured summaries into the large language model to be evaluated, so that the large language model can make binary judgments on each candidate conclusion and output a set of correct candidate conclusions supported by numerical evidence and contextual information, as well as the reasoning text. The answer analysis module is used to compare the set of correct candidate conclusions with the preset set of standard answers to determine the correctness of the large language model at the answer level. The standard interpretation alignment module is used to align the reasoning text with the preset structured standard interpretation item by item, and evaluate the reasoning process of the large language model from three process evaluation dimensions: numerical evidence coverage, numerical judgment consistency and reasoning logic sufficiency. The evaluation results output module integrates the correctness of the answer with the evaluation results of the reasoning process, and outputs a comprehensive evaluation of the large language model's ability in medical numerical reasoning tasks.

7. The process-oriented evaluation system for medical numerical reasoning ability based on a large language model according to claim 6, characterized in that, The structured standard interpretation comprises three parts: standard numerical evidence extraction, standard numerical judgment, and standard reasoning basis. The standard numerical evidence extraction section records the numerical evidence required to support the correct candidate conclusion; The standard numerical judgment section records numerical judgments based on numerical evidence; The standard reasoning basis section records the standard reasoning basis that connects numerical evidence, numerical judgments, and correct candidate conclusions.

8. The process-oriented evaluation system for medical numerical reasoning ability based on a large language model according to claim 7, characterized in that, The numerical evidence includes: measurements, time points, doses, percentages, differences between groups, significance probabilities, and confidence intervals.

9. The process-oriented evaluation system for medical numerical reasoning ability based on a large language model according to claim 7, characterized in that, The evaluation of the reasoning process of the large language model is carried out from three dimensions: coverage of numerical evidence, consistency of numerical judgment, and sufficiency of reasoning logic. Specifically, it includes: The string matching method is used to determine whether the reasoning text covers each piece of numerical evidence in the standard numerical evidence extraction section, and the evidence coverage ratio is calculated. The evaluation model is used to determine whether the reasoning text meets the standard numerical judgment criteria in the standard numerical judgment section, and the judgment consistency ratio is calculated. The evaluation model is used to compare the consistency of the reasoning chain between the reasoning text and the standard reasoning basis, and the reasoning quality score is output.

10. A process-oriented evaluation system for medical numerical reasoning ability based on a large language model according to claim 6, characterized in that, The evaluation results of the correctness and reasoning process at the comprehensive answer level output a comprehensive assessment of the large language model's ability in medical numerical reasoning tasks, specifically including: Pearson correlation coefficients were calculated between the evaluation dimensions of each process and the answer accuracy of the large language model to analyze the degree of influence of different reasoning stages on the final answer performance of the large language model.

Images