Clinical thinking examination question generation and step-by-step analysis construction method and system based on large language model

Abstract

The invention provides a clinical thinking examination question generation and step-by-step analysis construction method and system based on a large language model, and relates to the technical field of large language models. The method comprises the steps of analyzing a teaching outline and real questions over the years, constructing a multi-level clinical knowledge graph, intercepting a knowledge sub-graph according to target difficulty, and generating case question stems, candidate options and a preliminary reasoning path covering the sub-graph; the reasoning path consistency is verified through a logic engine, a simulation answer sample is constructed, item reaction modeling is executed, and the question difficulty is estimated; if the difficulty does not accord with the target difficulty, parameters are automatically fine-tuned and re-generated, and iteration is carried out until the standard is reached; performing error selection rate driven optimization on the interference options to form a final question version; and collecting student answering data feedback atlas weight and difficulty, and writing the content into the versioned question bank. According to the invention, the accuracy of examination question generation, the teaching suitability and the continuous updating ability are improved, and the intelligent evaluation of clinical thinking ability is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a method and system for generating and step-by-step parsing clinical thinking assessment questions based on a large language model. Background Art

[0002] In recent years, large-scale language models based on deep learning have been gradually applied to medical teaching and assessment. Some universities and online medical education platforms have begun experimenting with using these models to automatically generate case questions, providing reference answers and simple explanations at a level close to that of resident physicians. This demonstrates that the basic conditions for automatically generating clinical assessment questions are already in place, both in terms of algorithms and computing power.

[0003] The focus of medical licensing exams and university assessments is shifting from simply verifying the correctness of conclusions to ensuring the rationality and transparency of the entire diagnostic reasoning process. Many medical schools plan to incorporate visualization of "clinical thought chains" as a core indicator to more accurately assess students' logical rigor, ability to utilize evidence, and areas prone to error, laying a data foundation for subsequent educational interventions.

[0004] However, existing tools still have three shortcomings: ① Insufficient knowledge alignment - the question stems are often misaligned with the fine-grained objectives of the syllabus, making it difficult to map the assessment results to classroom interventions; ② Unreliable reasoning trajectory - LLM may generate steps in the COT that are unreachable or inconsistent with clinical guidelines; ③ Lack of a closed loop - the question bank update and difficulty control are separated from student behavior data, making it difficult for the model to evolve continuously, and teachers still need to conduct a lot of manual screening and rewriting. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a method and system for generating and step-by-step parsing of clinical thinking assessment questions based on a large-scale language model, which improves the generation quality and teaching adaptability of clinical thinking assessment questions and realizes the efficient, accurate and sustainable construction of an intelligent question bank.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A method for generating and parsing clinical thinking assessment questions based on a large-scale language model, including:

[0008] Parse the teaching syllabus and past exam questions to generate a multi-level clinical knowledge graph. Within the clinical knowledge graph, extract the corresponding knowledge subgraph based on the target difficulty level. Use a large language model to output the case stem, candidate options, and preliminary reasoning paths corresponding to the content of the case stem that cover the knowledge subgraph. Each knowledge node in the clinical knowledge graph records the topic label, weight, and sequential dependency relationship.

[0009] Using a symbolic logic engine to verify node dependencies and guideline consistency of the preliminary reasoning path, eliminating questions with logical breakpoints or clinical conflicts, and obtaining questions that have passed the verification;

[0010] Constructing simulated answer samples, performing item response modeling on the questions that have passed the verification, calculating the answer accuracy and discrimination, and estimating the actual difficulty level of the questions that have passed the verification based on the answer accuracy and discrimination;

[0011] If there is a deviation between the actual difficulty level and the preset target difficulty, the adjustable parameters in the verified question are automatically fine-tuned, and the case stem and candidate options are regenerated until the actual difficulty level falls within the target difficulty, thereby obtaining the current question version;

[0012] For the initial version of the distractor options in the current question version, a generative-discriminative adversarial mechanism is used for iterative optimization, and the false positive rate of the simulated test subjects is statistically simulated. The distractor options with a false positive rate that reaches a set threshold are retained and written back into the candidate option set to form the final question version;

[0013] Collect students' real answer data, and use the real answer data as feedback signals to update the node weights and difficulty labels in the clinical knowledge graph, and synchronously write the question stem, options and analysis into the versioned question bank for release.

[0014] Preferably, after obtaining the current title version, the method further includes:

[0015] The retrieval-enhanced large language model is called for the current question version to generate a step-by-step analysis, and is stored in the "knowledge point - reasoning logic - authoritative document evidence" triple format.

[0016] Preferably, the adjustable parameters include patient age, disease duration, laboratory values ​​and interference item combinations.

[0017] Preferably, the teaching syllabus and past exam questions are parsed to generate a multi-level clinical knowledge graph. In the clinical knowledge graph, corresponding knowledge subgraphs are intercepted according to the target difficulty level, and a large language model is called to output case stems, candidate options, and preliminary reasoning paths corresponding to the content of the case stems covering the knowledge subgraphs, including:

[0018] Sentence segmentation and clinical entity recognition are performed on the teaching syllabus text and the previous years' exam questions to extract knowledge elements of disease diagnosis, symptoms and signs, laboratory tests, treatment plans and anatomical structures;

[0019] Divide the knowledge elements into a basic layer, an advanced layer, and a comprehensive layer according to the target difficulty level of the teaching, forming a corresponding multi-level knowledge node set;

[0020] Based on the explicit order of the textbook, the co-occurrence frequency of the real questions, and the recommended level of the guide, the forward and backward dependency relationship between the knowledge nodes in the knowledge node set is determined, and a weight coefficient is assigned to each dependency edge;

[0021] Writing the knowledge nodes and the preceding and following dependencies into a graph database to generate a multi-level clinical knowledge graph, and recording the topic label, weight, and preceding and following dependency information for each knowledge node;

[0022] receiving a target difficulty parameter, and employing a depth-limited search in the knowledge graph to extract a knowledge subgraph covering the difficulty level;

[0023] Converting the knowledge subgraph into prompt text containing basic patient information, a list of key knowledge nodes, and generation instructions, and inputting the prompt text into a large language model;

[0024] Receive the case stem, candidate options and preliminary reasoning path corresponding to the case stem output by the large language model, and establish a node-content mapping relationship to confirm that the generated content covers the knowledge subgraph.

[0025] Preferably, the calculation formula of the weight coefficient is:

[0026]

[0027] Among them, w ij is the dependency weight coefficient of knowledge node i pointing to knowledge node j; S ij is the textbook sequence similarity, defined as S ij =1 / (1+d ij ), where d ij is the normalized page distance between knowledge node i and knowledge node j in the textbook chapter; C ij G is the co-occurrence probability of past exam questions, which is equal to the ratio of the number of times knowledge node i and knowledge node j appear at the same time to the total number of times the two appear; ij G is the product of the strength of the recommendation of the guideline. ij =g i ×g j , where g k is the recommendation grade score of knowledge node k in the latest evidence-based guidelines; Z ij is the fusion score; exp is the exponential function exp(x) = e x , e is a natural constant.

[0028] Preferably, constructing simulated answer samples, performing item response modeling on the questions that have passed the verification, calculating the answer accuracy and discrimination, and estimating the actual difficulty level of the questions that have passed the verification based on the answer accuracy and discrimination, including:

[0029] According to the actual test scores of previous years, the normal distribution N(μ,σ 2 ) Extract the ability values ​​θ of N virtual subjects k (k=1…N), where μ and σ are the empirical statistical mean and standard deviation;

[0030] For each question q that passes the verification, a two-parameter logic model is used Calculate the probability that subject k answers correctly; where a q is the initial value of the topic discrimination, b q is the initial value of the difficulty of the question;

[0031] For each P kq Perform Bernoulli random sampling to obtain the virtual answer matrix R∈{0,1} N×Q ; Where Q is the number of questions, 1 means the answer is correct;

[0032] Use the maximum likelihood method to inversely calculate the discrimination of the virtual answer matrix R and difficulty And calculate the correct rate of the questions

[0033] Will and Substitute into the formula Get the actual difficulty level of the question

[0034] like If the question falls within the preset target difficulty range, it will be retained. Otherwise, the adjustable parameters of the case question will be adjusted and the step "calculating the probability that subject k answered correctly" will be returned to re-estimate until the requirements are met.

[0035] Preferably, if there is a deviation between the actual difficulty level and the preset target difficulty, the adjustable parameters of the verified question are automatically fine-tuned, and the case stem and candidate options are regenerated until the actual difficulty level falls within the target difficulty, thereby obtaining the current question version, including:

[0036] Computational difficulty bias And determine the deviation sign sgn(Δd q );d min and d max are the endpoint values ​​of the target difficulty interval respectively;

[0037] Find the linear sensitivity m of each adjustable parameter p∈{patient age, disease duration, test value} to the difficulty of the question in the pre-established sensitivity matrix M pq ;

[0038] Select|m pq |maximum and sgn(mpq )=sgn(Δd q ) parameter p * As the object of this round of fine-tuning;

[0039] According to the formula Calculate the parameter step size Δp and quantify Δp to a medically reasonable value; the medically reasonable value includes an integer year, day, or the smallest unit within the normal range of the test indicator;

[0040] The updated parameter p * Embed the original prompt text, call the large language model to regenerate the case stem and candidate options, and re-estimate

[0041] If the new Fall into [d min ,d max ], then output the current version of the question; otherwise, return to sub-step 2, and the number of iterations does not exceed the preset upper limit K.

[0042] Preferably, for the initial version of the distractor options in the current question version, a generative-discriminative adversarial mechanism is used for iterative optimization, the false positive rate of the test subject group is statistically simulated, and the distractor options with the false positive rate reaching a set threshold are retained and written back into the candidate option set to form the final question version, including:

[0043] Using the conditional generation model G, we generate a set of candidate interference options D based on the question text, correct answer, and knowledge subgraph. (0) ={d1,d2,…d m};

[0044] Call the discriminator D med To D (0) Each option in the test is tested for consistency with medical logic and guidelines, and options that contradict known facts or contain potential misleading information are eliminated to obtain the preliminary review set D. (1) ;

[0045] According to the ability distribution of the simulated test group, for each d∈D (1) Calculate the error rate where R k is the answer result of subject k, 1(·) is the indicator function;

[0046] All satisfy r d The option set <τ is fed back to the generative model G for adversarial training, the generated weights are updated and a new round of candidate set D is output (t+1) ; Wherein, τ is the set threshold;

[0047] If all candidate interference items satisfy r d≥τ or the number of iterations reaches the upper limit M, the distractor items that currently meet the threshold are written back into the candidate option set and together with the correct answer constitute the final question version.

[0048] Preferably, students' real answer data is collected and used as feedback signals to update the node weights and difficulty labels in the clinical knowledge graph, and the question stem, options and solutions are synchronously written into the versioned question bank for publication, including:

[0049] After the students finish answering the questions, the unique identifier of each question, the options selected by the students, the answering time and the submission time are recorded, and the recorded data is stored in the answer log database to obtain the original answer log;

[0050] The original answer log is anonymized and aggregated by question identifier to form a structured answer record for each question; the structured answer record includes statistical information such as the number of students answering the question, the number of times each option was selected, the accuracy rate, and the average time taken;

[0051] Associate the structured answer record corresponding to each question back to the dependent knowledge node, and establish a mapping relationship between the question and the knowledge node;

[0052] For questions with incorrect answer rates or incorrect answer concentrations greater than the preset weight adjustment threshold, the weight annotations of the knowledge nodes associated with them in the knowledge graph are increased;

[0053] Based on the accuracy rate, average time and distribution of incorrect answers, the difficulty level of the questions is updated, and the difficulty labels of the associated nodes in the clinical knowledge graph are adjusted simultaneously;

[0054] Writing the updated knowledge node weights and difficulty level information back to the clinical knowledge graph to form a new round of graph snapshots;

[0055] Based on the updated question performance and the new round of graph snapshots, new version numbers are generated for the question stem, options, and analysis, and are simultaneously written into the versioned question bank for use in teaching and the next round of model training.

[0056] A system for generating and analyzing clinical thinking assessment questions based on a large-scale language model, including:

[0057] A knowledge graph-driven content generation unit is used to parse the teaching syllabus and past exam questions to generate a multi-level clinical knowledge graph. Within the clinical knowledge graph, corresponding knowledge subgraphs are extracted based on the target difficulty level, and a large language model is used to output case stems, candidate options, and preliminary reasoning paths corresponding to the content of the case stems that cover the knowledge subgraphs. Each knowledge node in the clinical knowledge graph records topic labels, weights, and sequential dependencies.

[0058] A reasoning path consistency verification unit is used to use a symbolic logic engine to verify the node dependencies and guideline consistency of the preliminary reasoning path, eliminate questions with logical breakpoints or clinical conflicts, and obtain questions that have passed the verification;

[0059] a virtual question answering modeling and difficulty estimation unit, configured to construct simulated question answering samples, perform item response modeling on the questions that have passed the verification, calculate the correct answer rate and discrimination, and estimate the actual difficulty level of the questions that have passed the verification based on the correct answer rate and discrimination;

[0060] A question difficulty adaptive control unit is configured to automatically fine-tune the adjustable parameters of the verified question if there is a deviation between the actual difficulty level and the preset target difficulty level, and regenerate the case stem and candidate options until the actual difficulty level falls within the target difficulty level, thereby obtaining the current question version;

[0061] a distractor option adversarial optimization unit, configured to iteratively optimize the initial distractor options in the current question version using a generative-discriminative adversarial mechanism, statistically simulate the error rate of the test subject group, retain the distractor options whose error rate reaches a set threshold, and write them back into the candidate option set to form the final question version;

[0062] The answering behavior feedback-driven update unit is used to collect students' real answer data, and use the real answer data as feedback signals to update the node weights and difficulty labels in the clinical knowledge graph, and synchronously write the question stem, options and analysis into the versioned question bank for release.

[0063] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0064] By incorporating clinical knowledge graphs, symbolic logic verification, item response modeling, adversarial optimization, and behavioral feedback mechanisms, this method achieves a closed-loop question generation system that is tailored to teaching objectives, features clear difficulty control, verifiable reasoning logic, identifiable interference options, and can be dynamically updated with learning behavior. This method automatically generates clinical assessment questions that meet teaching requirements, possess appropriate difficulty, have clear reasoning paths, and complete analysis. This significantly improves the efficiency of question bank construction and teaching adaptability, helping to promote systematic training and accurate assessment of students' clinical thinking skills. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of the system structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0069] The purpose of the present invention is to provide a method and system for generating and step-by-step parsing clinical thinking assessment questions based on a large-scale language model, thereby improving the generation quality and teaching adaptability of clinical thinking assessment questions and realizing efficient, accurate and sustainable intelligent question bank construction.

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] Figure 1 A flow chart of the method provided in the embodiment of the present invention is shown in FIG. Figure 1 As shown, the present invention provides a method for generating and parsing clinical thinking assessment questions based on a large language model, including:

[0072] Step 100: Parse the teaching syllabus and past exam questions to generate a multi-level clinical knowledge graph. Within the clinical knowledge graph, extract the corresponding knowledge subgraph based on the target difficulty level. Use a large language model to output the case stem, candidate options, and preliminary reasoning paths corresponding to the content of the case stem that cover the knowledge subgraph. Each knowledge node in the clinical knowledge graph records the topic label, weight, and sequential dependency relationship.

[0073] Step 200: Using a symbolic logic engine to check node dependencies and guideline consistency on the preliminary reasoning path, eliminating questions with logical breakpoints or clinical conflicts, and obtaining questions that have passed the verification;

[0074] Step 300: Constructing simulated answer samples, performing item response modeling on the questions that have passed the verification, calculating the accuracy and discrimination of the answers, and estimating the actual difficulty level of the questions that have passed the verification based on the accuracy and discrimination;

[0075] Step 400: If the actual difficulty level deviates from the preset target difficulty level, the adjustable parameters of the verified question are automatically fine-tuned, and the case stem and candidate options are regenerated until the actual difficulty level falls within the target difficulty level, thereby obtaining the current question version;

[0076] Step 500: For the initial version of distractor options in the current question version, a generative-discriminative adversarial mechanism is used for iterative optimization. The false positive rate of the simulated test subjects is statistically analyzed. The distractor options with a false positive rate that reaches a set threshold are retained and written back into the candidate option set to form the final question version.

[0077] Step 600: Collect students' actual answer data, and use the actual answer data as feedback signals to update the node weights and difficulty labels in the clinical knowledge graph, and synchronously write the question stem, options and solutions into the versioned question bank for publication.

[0078] Preferably, after obtaining the current title version, the method further includes:

[0079] The retrieval-enhanced large language model is called for the current question version to generate a step-by-step analysis, and is stored in the "knowledge point - reasoning logic - authoritative document evidence" triple format.

[0080] Specifically, in this embodiment, after the current question version is generated, the question is further parsed step by step through a large language model that integrates literature retrieval capabilities. First, based on the core knowledge points involved in the current question, the corresponding keyword phrases are extracted, and this is used as a retrieval condition to obtain relevant guideline content or literature abstracts from public databases in the medical field. Then, the retrieved information is compared with the diagnostic path in the question stem, and combined with the standard reasoning template written by experts, reasoning steps including problem location, key information identification, preliminary judgment, elimination process and final conclusion are gradually generated. Each step of reasoning is marked with the knowledge point and reference source on which it is based, and is archived according to the structure of "knowledge point-reasoning logic-literature source" to form a structured step-by-step analysis content. This structured analysis is not only convenient for subsequent teaching use, but also can be used as a basis for question bank version management and model updates.

[0081] Preferably, the adjustable parameters include patient age, disease duration, laboratory values ​​and interference item combinations.

[0082] Preferably, the teaching syllabus and past exam questions are parsed to generate a multi-level clinical knowledge graph. In the clinical knowledge graph, corresponding knowledge subgraphs are intercepted according to the target difficulty level, and a large language model is called to output case stems, candidate options, and preliminary reasoning paths corresponding to the content of the case stems covering the knowledge subgraphs, including:

[0083] Sentence segmentation and clinical entity recognition are performed on the teaching syllabus text and the previous years' exam questions to extract knowledge elements of disease diagnosis, symptoms and signs, laboratory tests, treatment plans and anatomical structures;

[0084] Divide the knowledge elements into a basic layer, an advanced layer, and a comprehensive layer according to the difficulty level of the teaching target, forming a corresponding multi-level knowledge node set;

[0085] Based on the explicit order of the textbook, the co-occurrence frequency of the real questions, and the recommended level of the guide, the forward and backward dependency relationship between the knowledge nodes in the knowledge node set is determined, and a weight coefficient is assigned to each dependency edge;

[0086] Writing the knowledge nodes and the preceding and following dependencies into a graph database to generate a multi-level clinical knowledge graph, and recording the topic label, weight, and preceding and following dependency information for each knowledge node;

[0087] receiving a target difficulty parameter, and employing a depth-limited search in the knowledge graph to extract a knowledge subgraph covering the difficulty level;

[0088] Converting the knowledge subgraph into prompt text containing basic patient information, a list of key knowledge nodes, and generation instructions, and inputting the prompt text into a large language model;

[0089] Receive the case stem, candidate options and preliminary reasoning path corresponding to the case stem output by the large language model, and establish a node-content mapping relationship to confirm that the generated content covers the knowledge subgraph.

[0090] Specifically, in this embodiment, after completing the extraction and multi-level classification of clinical knowledge elements, the sequential dependency relationship between each knowledge node is further quantitatively modeled through the order of textbook content, co-occurrence statistics of real questions, and the recommendation level of authoritative guidelines. Among them, the textbook order is calculated by comparing the chapter page positions; the co-occurrence frequency is counted by the frequency of the combination of knowledge points in the question stem and options; the guideline recommendation level is converted and scored based on the recommended level of diagnosis, examination, and treatment measures in the public medical guidelines. After combining the above three types of indicators, the directed dependency strength between knowledge nodes is calculated according to a unified scoring strategy, and it is used as the edge weight assignment for subsequent knowledge graph edge construction.

[0091] In this embodiment, after receiving the externally set target difficulty parameter, this embodiment performs a depth-restricted search within the established knowledge graph, prioritizing knowledge units whose node weights fall within the target range to construct a knowledge subgraph that meets the requirements for question generation. Based on this subgraph, the system extracts core node information and typical combinations, combines it with a preset patient information template (including gender, age, duration of the main complaint, etc.), and assembles it into a natural language prompt text. This prompt text also embeds generation instruction keywords to control the domain, style, and format of the output content of the large language model.

[0092] In this embodiment, after obtaining the case stem, candidate options, and preliminary reasoning paths returned by the large-scale language model, the system automatically performs semantic parsing on the output content, identifying the medical concepts and operational processes involved, and mapping them back to the original knowledge subgraph node set, calculating coverage and deviation metrics. If the output content fails to effectively cover the items marked as "core nodes" in the knowledge subgraph, or if content tends to be inconsistent with the target difficulty level, the system will trigger a regeneration process to ensure that the generated results are highly consistent with the expectations in terms of knowledge content and difficulty control.

[0093] Preferably, the calculation formula of the weight coefficient is:

[0094]

[0095] Among them, w ij is the dependency weight coefficient of knowledge node i pointing to knowledge node j; S ij is the textbook sequence similarity, defined as S ij =1 / (1+d ij ), where d ij is the normalized page distance between knowledge node i and knowledge node j in the textbook chapter; C ij G is the co-occurrence probability of past exam questions, which is equal to the ratio of the number of times knowledge node i and knowledge node j appear at the same time to the total number of times the two appear; ij G is the product of the strength of the recommendation of the guideline. ij =g i ×g j , where g k is the recommendation grade score of knowledge node k in the latest evidence-based guidelines; Z ij is the fusion score; exp is the exponential function exp(x) = e x , e is a natural constant.

[0096] Specifically, in this embodiment, in order to achieve quantitative modeling of the dependency relationship between knowledge nodes in the clinical knowledge graph, after extracting the textbook sequence, the co-occurrence frequency of real questions and the guideline recommendation level, the three types of indicators are normalized respectively and uniformly converted into dimensionless scores. In the calculation process, the textbook sequence similarity is calculated by the page spacing of the knowledge points in the textbook structure, the co-occurrence probability of real questions in previous years is obtained by counting the frequency of two knowledge points appearing at the same time in past questions, and the guideline recommendation level is assigned a fixed score according to the recommendation level with reference to the latest version of the diagnosis and treatment guidelines. The above three indicators are averaged after logarithmic transformation and input into the exponential mapping function as a fusion score to obtain the final dependency weight coefficient. This coefficient can be directly used as the weight of the directed edge in the knowledge graph, and is used for subsequent modules such as subgraph extraction and path verification. Through this method, the adaptive fusion of three-source information can be achieved without relying on manually set weight factors, effectively enhancing the rationality and structural expression ability of the knowledge chain in the graph.

[0097] Preferably, constructing simulated answer samples, performing item response modeling on the questions that have passed the verification, calculating the answer accuracy and discrimination, and estimating the actual difficulty level of the questions that have passed the verification based on the answer accuracy and discrimination, including:

[0098] According to the actual test scores of previous years, the normal distribution N(μ,σ 2 ) Extract the ability values ​​θ of N virtual subjects k (k=1…N), where μ and σ are the empirical statistical mean and standard deviation;

[0099] For each question q that passes the verification, a two-parameter logic model is used Calculate the probability that subject k answers correctly; where a q is the initial value of the topic discrimination, b q is the initial value of the difficulty of the question;

[0100] For each P kq Perform Bernoulli random sampling to obtain the virtual answer matrix R∈{0,1} N×Q ; Where Q is the number of questions, 1 means the answer is correct;

[0101] Use the maximum likelihood method to inversely calculate the discrimination of the virtual answer matrix R and difficulty And calculate the correct rate of the questions

[0102] Will and Substitute into the formula Get the actual difficulty level of the question

[0103] like If the question falls within the preset target difficulty range, it will be retained. Otherwise, the adjustable parameters of the case question will be adjusted and the step "calculating the probability that subject k answered correctly" will be returned to re-estimate until the requirements are met.

[0104] Specifically, in this embodiment, to simulate the response behavior of examinees of different ability levels on the verification questions, the system constructs a normal ability model based on the distribution of medical examination scores over the years and randomly extracts the ability values ​​of multiple virtual subjects from this model. Each subject's ability value is set as a static parameter and does not change in subsequent answering simulations. For each question, the system calculates the probability of each virtual subject answering the question correctly, combining the initial difficulty and discrimination parameters of the question. Based on this probability, random sampling is performed to generate "correct / incorrect" response results, forming a virtual answer matrix for parameter estimation.

[0105] After completing the construction of the virtual answer matrix, this embodiment uses the maximum likelihood estimation method to inversely solve the parameters of each question, obtain the actual discrimination and inferred difficulty index of the question, and calculate the simulated accuracy of each question. To improve the convergence stability of the model, constraints are introduced during the parameter update process to prevent overfitting of discrimination or difficulty drift. Subsequently, the system uses the accuracy and discrimination of the question as input, calculates the actual difficulty level of the question, and compares it with the target difficulty range preset by the current system as the basis for whether to enter the next step of optimization.

[0106] If the actual difficulty level of the question exceeds the preset range, this embodiment will locate the preset adjustable parameters in the question stem, including but not limited to factors affecting the complexity of judgment such as patient age, duration of illness, laboratory values, and make slight adjustments to the parameters with greater influence according to the pre-established sensitivity table. The updated parameters are rewritten into the prompt template and submitted to the large language model to generate a new question stem and option content. The newly generated question will enter the probability calculation and simulated answering process again until its actual difficulty level converges to the target range, and it will be identified as the current question version.

[0107] Preferably, if there is a deviation between the actual difficulty level and the preset target difficulty, the adjustable parameters of the verified question are automatically fine-tuned, and the case stem and candidate options are regenerated until the actual difficulty level falls within the target difficulty, thereby obtaining the current question version, including:

[0108] Computational difficulty bias And determine the deviation sign sgn(Δd q );d min and d max are the endpoint values ​​of the target difficulty interval respectively;

[0109] Find the linear sensitivity m of each adjustable parameter p∈{patient age, disease duration, test value} to the difficulty of the question in the pre-established sensitivity matrix M pq ;

[0110] Select|m pq |maximum and sgn(m pq )=sgn(Δd q ) parameter p * As the object of this round of fine-tuning;

[0111] According to the formula Calculate the parameter step size Δp and quantify Δp to a medically reasonable value; the medically reasonable value includes an integer year, day, or the smallest unit within the normal range of the test indicator;

[0112] The updated parameter p * Embed the original prompt text, call the large language model to regenerate the case stem and candidate options, and re-estimate

[0113] If the new Fall into [d min ,d max ], then output the current version of the question; otherwise, return to sub-step 2, and the number of iterations does not exceed the preset upper limit K.

[0114] Preferably, for the initial version of the distractor options in the current question version, a generative-discriminative adversarial mechanism is used for iterative optimization, the false positive rate of the test subject group is statistically simulated, and the distractor options with the false positive rate reaching a set threshold are retained and written back into the candidate option set to form the final question version, including:

[0115] Using the conditional generation model G, we generate a set of candidate interference options D based on the question text, correct answer, and knowledge subgraph. (0) ={d1,d2,…d m};

[0116] Call the discriminator D med To D (0) Each option in the test is tested for consistency with medical logic and guidelines, and options that contradict known facts or contain potential misleading information are eliminated to obtain the preliminary review set D. (1) ;

[0117] According to the ability distribution of the simulated test group, for each d∈D (1) Calculate the error rate where R k is the answer result of subject k, 1(·) is the indicator function;

[0118] All satisfying r dThe option set <τ is fed back to the generative model G for adversarial training, the generated weights are updated and a new round of candidate set D is output (t+1) ; Wherein, τ is the set threshold;

[0119] If all candidate interference items satisfy r d ≥τ or the number of iterations reaches the upper limit M, the distractor items that currently meet the threshold are written back into the candidate option set and together with the correct answer constitute the final question version.

[0120] Optionally, in this embodiment, after completing the actual difficulty estimate for a verified question, the system first calculates the deviation between that difficulty value and the median of a preset target difficulty range. The system then uses the sign to determine whether the question is more difficult or easier relative to the target requirement. Subsequently, the system uses a pre-established parameter-difficulty sensitivity matrix to determine the responsiveness of each adjustable parameter in the question stem to changes in difficulty, prioritizing the most sensitive parameters for subsequent intervention.

[0121] In this embodiment, after completing parameter sensitivity screening, the parameter with the largest absolute sensitivity in the current deviation direction is selected as the target for this round of adjustment. This embodiment calculates the recommended adjustment step size based on the inverse relationship between the parameter's sensitivity coefficient and the target deviation value, and quantifies it in conjunction with a medical knowledge base to ensure that the adjusted parameter values ​​conform to actual clinical logic. For example, age should be an integer number of years, disease duration should be an integer number of days, and test indicators should be within an acceptable accuracy within the normal or warning range to avoid generating medically unreasonable or pedagogically inappropriate scenarios.

[0122] In this embodiment, after parameter adjustment is complete, it is embedded into the corresponding position in the original prompt template and a large language model is used to regenerate the case question stem and candidate options. The newly generated content will re-enter the difficulty estimation process. If its actual difficulty value falls within the set range, it will be determined as the current question version; if there is still a deviation, it will return to the sensitivity search step to continue a new round of fine-tuning. To ensure generation efficiency and content stability, this embodiment sets a maximum number of iterations. If this value is exceeded, a prompt will be triggered or a failure status will be recorded for subsequent manual review or annotation.

[0123] Preferably, students' real answer data is collected and used as feedback signals to update the node weights and difficulty labels in the clinical knowledge graph, and the question stem, options and solutions are synchronously written into the versioned question bank for publication, including:

[0124] After the students finish answering the questions, the unique identifier of each question, the options selected by the students, the answering time and the submission time are recorded, and the recorded data is stored in the answer log database to obtain the original answer log;

[0125] The original answer log is anonymized and aggregated by question identifier to form a structured answer record for each question; the structured answer record includes statistical information such as the number of students answering the question, the number of times each option was selected, the accuracy rate, and the average time taken;

[0126] Associate the structured answer record corresponding to each question back to the dependent knowledge node, and establish a mapping relationship between the question and the knowledge node;

[0127] For questions with incorrect answer rates or incorrect answer concentrations greater than the preset weight adjustment threshold, the weight annotations of the knowledge nodes associated with them in the knowledge graph are increased;

[0128] Based on the accuracy rate, average time and distribution of incorrect answers, the difficulty level of the questions is updated, and the difficulty labels of the associated nodes in the clinical knowledge graph are adjusted simultaneously;

[0129] Writing the updated knowledge node weights and difficulty level information back to the clinical knowledge graph to form a new round of graph snapshots;

[0130] Based on the updated question performance and the new round of graph snapshots, new version numbers are generated for the question stem, options, and analysis, and are simultaneously written into the versioned question bank for use in teaching and the next round of model training.

[0131] Specifically, in this embodiment, after students complete answering a question, the question ID, answer options, time taken, and submission time are automatically recorded and summarized as the original answer log. To protect data privacy, this embodiment anonymizes the log data, removes student identity information, and aggregates and organizes it by question ID to form a structured answer record for each question. This record includes statistical indicators such as the number of respondents, the number of times each option was selected, the overall accuracy rate, and the average answer time, which serves as the basic data source for subsequent diagnostic performance analysis and knowledge node feedback.

[0132] In this embodiment, the structured answer record of each question is retroactively mapped to the set of knowledge nodes that it relied on when the question was generated, and a "question-node" mapping relationship is established. For nodes with a high error rate or a concentration of false positives exceeding a preset threshold, this embodiment increases their weight value in the clinical knowledge graph to enhance the significance of the node in subsequent generation tasks. At the same time, based on performance indicators such as the accuracy rate of answering questions, time and false positive distribution, the difficulty level of the question is re-evaluated, and the difficulty labels of its associated nodes are adjusted accordingly to achieve dynamic feedback optimization of the graph structure.

[0133] In this embodiment, the adjusted knowledge node weights and difficulty level information are written back to the original clinical knowledge graph, and a new graph snapshot is generated to mark the current knowledge status. Based on the graph snapshot, this embodiment assigns a new version number to the updated question stem, candidate options, and corresponding parsing content. The number includes the graph snapshot identifier and question generation batch information. Finally, the versioned content is written into a question bank document in a standard format for subsequent teaching use and model iterative calls, enabling continuous accumulation and traceability management of question bank content.

[0134] Corresponding to the above method, such as Figure 2 As shown, this embodiment also provides a system for generating and analyzing clinical thinking assessment questions based on a large language model, including:

[0135] A knowledge graph-driven content generation unit is used to parse the teaching syllabus and past exam questions to generate a multi-level clinical knowledge graph. Within the clinical knowledge graph, corresponding knowledge subgraphs are extracted based on the target difficulty level, and a large language model is used to output case stems, candidate options, and preliminary reasoning paths corresponding to the content of the case stems that cover the knowledge subgraphs. Each knowledge node in the clinical knowledge graph records topic labels, weights, and sequential dependencies.

[0136] A reasoning path consistency verification unit is used to use a symbolic logic engine to verify the node dependencies and guideline consistency of the preliminary reasoning path, eliminate questions with logical breakpoints or clinical conflicts, and obtain questions that have passed the verification;

[0137] a virtual question answering modeling and difficulty estimation unit, configured to construct simulated question answering samples, perform item response modeling on the questions that have passed the verification, calculate the correct answer rate and discrimination, and estimate the actual difficulty level of the questions that have passed the verification based on the correct answer rate and discrimination;

[0138] A question difficulty adaptive control unit is configured to automatically fine-tune the adjustable parameters of the verified question if there is a deviation between the actual difficulty level and the preset target difficulty level, and regenerate the case stem and candidate options until the actual difficulty level falls within the target difficulty level, thereby obtaining the current question version;

[0139] a distractor option adversarial optimization unit, configured to iteratively optimize the initial distractor options in the current question version using a generative-discriminative adversarial mechanism, statistically simulate the error rate of the test subject group, retain the distractor options whose error rate reaches a set threshold, and write them back into the candidate option set to form the final question version;

[0140] The answering behavior feedback-driven update unit is used to collect students' real answer data, and use the real answer data as feedback signals to update the node weights and difficulty labels in the clinical knowledge graph, and synchronously write the question stem, options and analysis into the versioned question bank for release.

[0141] The beneficial effects of the present invention are as follows:

[0142] (1) The present invention constructs a multi-level clinical knowledge graph covering the teaching syllabus and extracts knowledge subgraphs based on the target difficulty, ensuring that the generated examination questions are highly aligned with the teaching objectives in terms of content scope and examination depth, thereby significantly improving the adaptability and coverage of the question bank.

[0143] (2) The present invention introduces a symbolic logic verification and item response modeling mechanism, which can verify the dependency consistency of the reasoning path of the questions and estimate the actual difficulty based on the students' simulated answering performance, thereby constructing a set of questions with reasonable reasoning and controllable difficulty, thereby enhancing the discrimination and teaching explanation of the evaluation results.

[0144] (3) The present invention is equipped with an automatic fine-tuning mechanism for adjustable parameters and an optimization process for counteracting interference options. By iteratively adjusting the question parameters and screening interference items with high error rates, it ensures that each question has both real-world context and a reasonable level of thinking challenge, effectively improving the teaching value of the test questions.

[0145] (4) The present invention establishes a graph feedback update mechanism based on students' real answer data, realizes the dynamic adjustment of knowledge node weights and difficulty levels, and simultaneously writes into the versioned question bank, constructing a closed-loop system of "generation-evaluation-feedback-regeneration", and improving the intelligence and sustainable evolution capabilities of the question bank system.

[0146] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0147] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A method for generating and analyzing clinical thinking assessment questions based on a large language model, characterized by: include: Parse the teaching syllabus and past exam questions to generate a multi-level clinical knowledge graph. Within the clinical knowledge graph, extract the corresponding knowledge subgraph based on the target difficulty level. Use a large language model to output the case stem, candidate options, and preliminary reasoning paths corresponding to the content of the case stem that cover the knowledge subgraph. Each knowledge node in the clinical knowledge graph records the topic label, weight, and sequential dependency relationship. Using a symbolic logic engine to verify node dependencies and guideline consistency of the preliminary reasoning path, eliminating questions with logical breakpoints or clinical conflicts, and obtaining questions that have passed the verification; Constructing simulated answer samples, performing item response modeling on the questions that have passed the verification, calculating the answer accuracy and discrimination, and estimating the actual difficulty level of the questions that have passed the verification based on the answer accuracy and discrimination; If there is a deviation between the actual difficulty level and the preset target difficulty, the adjustable parameters in the verified question are automatically fine-tuned, and the case stem and candidate options are regenerated until the actual difficulty level falls within the target difficulty, thereby obtaining the current question version; For the initial version of the distractor options in the current question version, a generative-discriminative adversarial mechanism is used for iterative optimization, and the false positive rate of the simulated test subjects is statistically simulated. The distractor options with a false positive rate that reaches a set threshold are retained and written back into the candidate option set to form the final question version; Collect students' real answer data, and use the real answer data as feedback signals to update the node weights and difficulty labels in the clinical knowledge graph, and synchronously write the question stem, options and analysis into the versioned question bank for release.

2. The method for generating and analyzing clinical thinking assessment questions based on a large language model according to claim 1 is characterized in that: After getting the current title version, it also includes: The retrieval-enhanced large language model is called for the current question version to generate a step-by-step analysis, which is stored in the "knowledge point - reasoning logic - authoritative document evidence" triple format.

3. The method for generating and analyzing clinical thinking assessment questions based on a large language model according to claim 1 is characterized in that: The adjustable parameters include patient age, disease duration, laboratory values ​​and interference item combinations.

4. The method for generating and analyzing clinical thinking assessment questions based on a large language model according to claim 1 is characterized in that: Parse the teaching syllabus and past exam questions to generate a multi-level clinical knowledge graph. Within the clinical knowledge graph, extract the corresponding knowledge subgraph based on the target difficulty level. Then use a large language model to output the case stem, candidate options, and preliminary reasoning paths corresponding to the content of the case stem that cover the knowledge subgraph, including: Sentence segmentation and clinical entity recognition are performed on the teaching syllabus text and the previous years' exam questions to extract knowledge elements of disease diagnosis, symptoms and signs, laboratory tests, treatment plans and anatomical structures; Divide the knowledge elements into a basic layer, an advanced layer, and a comprehensive layer according to the target difficulty level of the teaching, forming a corresponding multi-level knowledge node set; Based on the explicit order of the textbook, the co-occurrence frequency of the real questions, and the recommended level of the guide, the forward and backward dependency relationship between the knowledge nodes in the knowledge node set is determined, and a weight coefficient is assigned to each dependency edge; Writing the knowledge nodes and the preceding and following dependencies into a graph database to generate a multi-level clinical knowledge graph, and recording the topic label, weight, and preceding and following dependency information for each knowledge node; receiving a target difficulty parameter, and employing a depth-limited search in the knowledge graph to extract a knowledge subgraph covering the difficulty level; Converting the knowledge subgraph into prompt text containing basic patient information, a list of key knowledge nodes, and generation instructions, and inputting the prompt text into a large language model; Receive the case stem, candidate options and preliminary reasoning path corresponding to the case stem output by the large language model, and establish a node-content mapping relationship to confirm that the generated content covers the knowledge subgraph.

5. The method for generating and analyzing clinical thinking assessment questions based on a large language model according to claim 4 is characterized in that: The calculation formula of the weight coefficient is: Among them, w ij is the dependency weight coefficient of knowledge node i pointing to knowledge node j; S ij is the textbook sequence similarity, defined as S ij =1 / (1+d ij ), where d ij is the normalized page distance between knowledge node i and knowledge node j in the textbook chapter; C ij G is the co-occurrence probability of past exam questions, which is equal to the ratio of the number of times knowledge node i and knowledge node j appear at the same time to the total number of times the two appear; ij G is the product of the strength of the recommendation of the guideline. ij =g i ×g j , where g k is the recommendation grade score of knowledge node k in the latest evidence-based guidelines; Z ij is the fusion score; exp is the exponential function exp(x) = e x , e is a natural constant.

6. The method for generating and analyzing clinical thinking assessment questions based on a large language model according to claim 1 is characterized in that: Constructing simulated answer samples, performing item response modeling on the questions that have passed the verification, calculating the answer accuracy and discrimination, and estimating the actual difficulty level of the questions that have passed the verification based on the answer accuracy and discrimination, including: According to the actual test scores of previous years, the normal distribution N(μ,σ 2 ) Extract the ability values ​​θ of N virtual subjects k (k=1…N), where μ and σ are the empirical statistical mean and standard deviation; For each question q that passes the verification, a two-parameter logic model is used Calculate the probability that subject k answers correctly; where a q is the initial value of the topic discrimination, b q is the initial value of the difficulty of the question; For each P kq Perform Bernoulli random sampling to obtain the virtual answer matrix R∈{0,1} N×Q ; Where Q is the number of questions, 1 means the answer is correct; Use the maximum likelihood method to inversely calculate the discrimination of the virtual answer matrix R and difficulty And calculate the correct rate of the questions Will and Substitute into the formula Get the actual difficulty level of the question like If the answer falls within the preset target difficulty range, it will be retained. Otherwise, the adjustable parameters of the case question will be adjusted and the process will return to step "Calculate the probability that subject k answered correctly" to re-estimate until the requirements are met.

7. The method for generating and analyzing clinical thinking assessment questions based on a large language model according to claim 6 is characterized in that: If the actual difficulty level deviates from the preset target difficulty level, the adjustable parameters of the verified question are automatically fine-tuned, and the case stem and candidate options are regenerated until the actual difficulty level falls within the target difficulty level, and the current question version is obtained, including: Computational difficulty bias And determine the deviation sign sgn(Δd q );d min and d max are the endpoint values ​​of the target difficulty interval respectively; Find the linear sensitivity m of each adjustable parameter p∈{patient age, disease duration, test value} to the difficulty of the question in the pre-established sensitivity matrix M pq ; Select|m pq |maximum and sgn(m pq )=sgn(Δd q ) parameter p * As the object of this round of fine-tuning; According to the formula Calculate the parameter step size Δp and quantify Δp to a medically reasonable value; the medically reasonable value includes an integer year, day, or the smallest unit within the normal range of the test indicator; The updated parameter p * Embed the original prompt text, call the large language model to regenerate the case stem and candidate options, and re-estimate If the new Fall into [d min ,d max ], then output the current version of the question; otherwise, return to sub-step 2, and the number of iterations does not exceed the preset upper limit K.

8. The method for generating and analyzing clinical thinking assessment questions based on a large language model according to claim 1 is characterized in that: For the initial version of the distractor options in the current question version, a generative-discriminative adversarial mechanism is used for iterative optimization. The false selection rate of the test subject group is statistically simulated. The distractor options with a false selection rate that reaches a set threshold are retained and written back into the candidate option set to form the final question version, including: Using the conditional generation model G, we generate a set of candidate interference options D based on the question text, correct answer, and knowledge subgraph. (0) ={d1,d2,…d m }; Call the discriminator D med To D (0) Each option in the test is tested for consistency with medical logic and guidelines, and options that contradict known facts or contain potential misleading information are eliminated to obtain the preliminary review set D. (1) ; According to the ability distribution of the simulated test group, for each d∈D (1) Calculate the error rate where R k is the answer result of subject k, 1(·) is the indicator function; All satisfying r d The option set <τ is fed back to the generative model G for adversarial training, the generated weights are updated and a new round of candidate set D is output (t+1) ; Wherein, τ is the set threshold; If all candidate interference items satisfy r d ≥τ or the number of iterations reaches the upper limit M, the distractor items that currently meet the threshold are written back into the candidate option set and together with the correct answer constitute the final question version.

9. The method for generating and analyzing clinical thinking assessment questions based on a large language model according to claim 1 is characterized in that: Collect students' real answer data, and use the real answer data as feedback signals to update the node weights and difficulty labels in the clinical knowledge graph, and synchronously write the question stem, options and solutions into the versioned question bank for release, including: After the students finish answering the questions, the unique identifier of each question, the options selected by the students, the answering time and the submission time are recorded, and the recorded data is stored in the answer log database to obtain the original answer log; The original answer log is anonymized and aggregated by question identifier to form a structured answer record for each question; the structured answer record includes statistical information such as the number of students answering the question, the number of times each option was selected, the accuracy rate, and the average time taken; Associate the structured answer record corresponding to each question back to the dependent knowledge node, and establish a mapping relationship between the question and the knowledge node; For questions with incorrect answer rates or incorrect answer concentrations greater than the preset weight adjustment threshold, the weight annotations of the knowledge nodes associated with them in the knowledge graph are increased; Based on the accuracy rate, average time and distribution of incorrect answers, the difficulty level of the questions is updated, and the difficulty labels of the associated nodes in the clinical knowledge graph are adjusted simultaneously; Writing the updated knowledge node weights and difficulty level information back to the clinical knowledge graph to form a new round of graph snapshots; Based on the updated question performance and the new round of graph snapshots, new version numbers are generated for the question stem, options, and analysis, and are simultaneously written into the versioned question bank for use in teaching and the next round of model training.

10. A system for generating and analyzing clinical thinking assessment questions based on a large language model, characterized by: include: A knowledge graph-driven content generation unit is used to parse the teaching syllabus and past exam questions to generate a multi-level clinical knowledge graph. Within the clinical knowledge graph, corresponding knowledge subgraphs are extracted based on the target difficulty level, and a large language model is used to output case stems, candidate options, and preliminary reasoning paths corresponding to the content of the case stems that cover the knowledge subgraphs. Each knowledge node in the clinical knowledge graph records topic labels, weights, and sequential dependencies. A reasoning path consistency verification unit is used to use a symbolic logic engine to verify the node dependencies and guideline consistency of the preliminary reasoning path, eliminate questions with logical breakpoints or clinical conflicts, and obtain questions that have passed the verification; a virtual question answering modeling and difficulty estimation unit, configured to construct simulated question answering samples, perform item response modeling on the questions that have passed the verification, calculate the correct answer rate and discrimination, and estimate the actual difficulty level of the questions that have passed the verification based on the correct answer rate and discrimination; A question difficulty adaptive control unit is configured to automatically fine-tune the adjustable parameters of the verified question if there is a deviation between the actual difficulty level and the preset target difficulty level, and regenerate the case stem and candidate options until the actual difficulty level falls within the target difficulty level, thereby obtaining the current question version; a distractor option adversarial optimization unit, configured to iteratively optimize the initial distractor options in the current question version using a generative-discriminative adversarial mechanism, statistically simulate the error rate of the test subject group, retain the distractor options whose error rate reaches a set threshold, and write them back into the candidate option set to form the final question version; The answering behavior feedback-driven update unit is used to collect students' real answer data, and use the real answer data as feedback signals to update the node weights and difficulty labels in the clinical knowledge graph, and synchronously write the question stem, options and analysis into the versioned question bank for release.

Images