Closed-loop reinforcement learning diagnosis method and system based on high-fidelity virtual clinical environment

Abstract

The invention provides a closed-loop reinforcement learning diagnosis method and system based on a high-fidelity virtual clinical environment, and the method comprises the steps: 1, carrying out the structural processing of basic information and an examination record set of a patient, and obtaining a standardized examination and diagnosis data set; 2, training to obtain a high-fidelity clinical environment system; and step 3, based on the high-fidelity clinical environment system, obtaining a diagnosis agent through reinforcement learning training. A simulation examination result highly consistent with the state of a patient can be generated based on real electronic health record data, a diagnosis agent performs multiple rounds of interaction in a virtual environment, autonomously explores a diagnosis path, and performs strategy optimization through a double-reward mechanism, so that the crossing of a diagnosis model from static prediction to dynamic decision is realized, and the diagnosis efficiency is improved. And the diagnosis accuracy and the reasonability of examination recommendation are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning and diagnostic systems technology, specifically to a closed-loop reinforcement learning diagnostic method and system based on a high-fidelity virtual clinical environment. Background Technology

[0002] Currently, in the field of medical artificial intelligence, especially in the area of ​​clinical auxiliary diagnosis based on large language models (LLM), there is a significant demand for diagnostic systems with high accuracy and multi-turn interaction capabilities. Most existing medical large language models are still primarily trained on static data. While they can demonstrate some reasoning ability in single-turn question-and-answer or static case analysis scenarios, in real clinical environments, due to the lack of interactive, high-fidelity virtual clinical environments, these models cannot continuously optimize diagnostic strategies through reinforcement learning in a secure closed loop. Consequently, when dealing with complex conditions, their diagnostic accuracy and the rationality of examination recommendations both suffer from significant bottlenecks.

[0003] For example, the Baichuan-M1 model developed by Baichuan Intelligence (reference: Baichuan-M1: Pushing the Medical Capability of Large Language Models) has achieved good performance in medical reasoning ability by statically fine-tuning medical data. However, this method does not introduce a dynamic interactive training mechanism and cannot dynamically adjust the order of subsequent examinations based on real-time patient feedback, which can easily lead to insufficient information acquisition in multi-round consultation tasks.

[0004] For example, Google's MedGemma model, released in 2025 (see MedGemma Technical Report), is based on the Gemma3 framework and supports multimodal fusion processing of medical record text and medical image information, capable of processing both structured and unstructured clinical information simultaneously. However, its training process is still based on static historical data and lacks a closed-loop training mechanism for multi-round decision-making. Therefore, in actual diagnostic reasoning, it cannot dynamically adjust diagnostic paths and examination strategies based on intermediate examination results.

[0005] Furthermore, the multi-agent collaborative diagnostic systems described in the 2023 papers *MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning* and *MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making*, which simulate multidisciplinary consultations through role-playing, can improve the comprehensiveness of diagnosis to some extent. However, the performance of such systems is highly dependent on the large models they are based on, and in real multi-round consultations, they are prone to problems such as premature termination and omission of key examinations, thus affecting the accuracy of the final diagnosis.

[0006] Patent application CN113239972A describes an AI-assisted diagnostic model construction system for medical images, comprising a data center module, a data anonymization module, a data preprocessing module, and a model building module connected in sequence. The data center module receives and stores image data and its corresponding tags. The data anonymization module performs anonymization processing on each image data, removing privacy information. The data preprocessing module converts all privacy-removed image data into image format and inputs it into the corresponding preprocessing pipeline for data processing. The model building module divides the preprocessed image data into training and validation samples, imports them into a pre-trained model based on federated machine learning for training and validation, and obtains the final AI-assisted diagnostic model. However, this patent cannot completely solve the existing technical problems, nor can it meet the needs of this invention.

[0007] In summary, the common drawback of the aforementioned existing technologies is that they all lack a high-fidelity, interactive virtual clinical environment, and cannot continuously optimize diagnostic strategies through reinforcement learning under safe and controllable closed-loop conditions, resulting in difficulty in further improving diagnostic accuracy and examination recommendation quality in complex clinical scenarios. Summary of the Invention

[0008] In view of the shortcomings of the prior art, the purpose of this invention is to provide a closed-loop reinforcement learning diagnostic method and system based on a high-fidelity virtual clinical environment.

[0009] The closed-loop reinforcement learning diagnostic method based on a high-fidelity virtual clinical environment provided by the present invention includes: Step 1: Structure the patient's basic information and examination record set, construct a time-series examination chain and standardize it, construct multi-round diagnostic trajectory data and standardize it; uniformly map the names of laboratory and microbiology examination items, extract and complete the missing examination names in the imaging examinations from the reports through string matching, sort all examination records by timestamp, and retain only the earliest record for repeated examinations of the same item, thus obtaining a standardized examination and diagnosis dataset. Step 2: Based on the standardized examination and diagnosis dataset, design a simulator training task, defining the training task as an autoregressive text generation problem, predicting the next examination result given the patient's basic information and existing examination records; perform model training and implementation, using a language model as the basic model for initialization, and train it through distributed optimization to obtain a high-fidelity clinical environment system; Step 3: Based on the high-fidelity clinical environment system, a diagnostic agent is obtained through reinforcement learning training.

[0010] Preferably, in step 2, the process of predicting the result of the next inspection is expressed as follows:

[0011] in, For the predicted first The results of the inspection, For generating functions of high-fidelity clinical environment systems, For patient background information, This is a collection of completed inspection records. For the first The name of the item to be checked; In autoregressive generative modeling, Consider it as a sequence consisting of L smallest units of text information. The generated conditional probability P is decomposed as follows:

[0012] in, Indicates that in generating the first The historical sequence of the smallest unit of text information has already been generated; during prediction, firstly... , and Encode it as a context vector, and then generate each one sequentially according to the conditional probabilities described above. until a terminator is generated or the maximum length is reached. Finally, the smallest unit sequence of the generated text information is concatenated to form the predicted inspection result. During training, the loss function is:

[0013] in, This indicates the total number of examinations performed on the patient. Indicates the first The smallest unit of text information in the inspection results. Indicates the first The first item of the inspection results The smallest unit of real text information This indicates that the first item in the inspection results The smallest unit of all real text information preceding the smallest unit of text information. This represents the loss function value.

[0014] Preferably, step 3 includes: Step 3.1: In the cold start phase, the diagnostic agent is fine-tuned using multiple rounds of diagnostic trajectory samples through a loss function; Step 3.2: Reinforcement learning phase, the diagnostic agent after cold start is trained in real time through interactive training with the clinical environment system.

[0015] Preferably, when using autoregressive generative modeling, the conditional probability P is decomposed as follows:

[0016] in, The smallest unit of complete dialogue text information sequence. The trainable parameters of the diagnostic agent are represented. This represents the historical context when generating the i-th smallest unit of text information; during cold start fine-tuning, the loss is calculated and backpropagated only for the smallest unit of text information corresponding to the assistant role, and a gradient mask is defined. for:

[0017] The loss function is expressed as:

[0018] Where N is the number of the smallest units of text information.

[0019] Preferably, in each training iteration, the agent receives initial consultation information as its state, generates a preliminary diagnosis and suggestions for the next examination, inputs the results into the clinical environment system as feedback, forms a new state, and repeats the above process until a final diagnosis is generated or the maximum number of interaction rounds is reached. During training, the GRPO strategy optimization algorithm is used, and the reward function consists of three parts: a reward for the accuracy of the final diagnosis, a reward for the relevance of the examination recommendations, and a reward for the number of rounds. The reward for the accuracy of the final diagnosis is 1 when the predicted diagnosis is semantically consistent with the reference diagnosis, and 0 otherwise. The reward for the relevance of the examination recommendations is the F1 score of the reference examination set and the predicted examination set. The reward for the number of rounds is 0.1 when the number of dialogue rounds does not exceed the maximum number of rounds, and 0 otherwise. Let the policy function of the diagnostic agent be... In a given patient state At that time, the strategy outputs an action. The probability distribution is The GRPO algorithm updates its policy based on relative reward advantage, and its optimization objective function is... Defined as expected cumulative reward:

[0020] in, For mathematical expectation symbol, This represents a complete diagnostic trajectory. As a discount factor, For the trajectory length, This is the immediate reward for step t; Total reward function Defined as:

[0021] in, For weight hyperparameters, Rewards are given for the accuracy of the final diagnosis. To check the recommendation relevance reward, The reward is a round-based penalty; the specific definitions of the three sub-rewards are as follows:

[0022]

[0023] in, To predict the check set, For reference check set;

[0024] in, This is the preset maximum number of dialogue turns; The GRPO algorithm samples multiple trajectories in each training batch, calculates the cumulative reward for each trajectory, and then applies the relative advantage function. To adjust the policy gradient, where This is the average reward for all trajectories within the batch; The update formula for the policy parameters in the k-th iteration is:

[0025] in, For the first The parameters for the next iteration For the updated parameters, For learning rate, Indicates the parameter The gradient.

[0026] The closed-loop reinforcement learning diagnostic system based on a high-fidelity virtual clinical environment provided by the present invention includes: Module M1: Structures the patient's basic information and examination record set, constructs and standardizes the time-series examination chain, constructs and standardizes multi-round diagnostic trajectory data; unifies the mapping of laboratory and microbiology examination item names, extracts and completes missing examination names in imaging examinations from the report through string matching, sorts all examination records by timestamp, and retains only the earliest record for repeated examinations of the same item, thus obtaining a standardized examination and diagnosis dataset. Module M2: Based on the standardized examination and diagnosis dataset, a simulator training task is designed, which is defined as an autoregressive text generation problem. Given the patient's basic information and existing examination records, the task is to predict the result of the next examination. The model is trained and implemented, using a language model as the basic model for initialization, and trained through distributed optimization to obtain a high-fidelity clinical environment system. Module M3: Based on the high-fidelity clinical environment system, a diagnostic agent is obtained through reinforcement learning training.

[0027] Preferably, in module M2, the process of predicting the result of the next inspection is represented as follows:

[0028] in, For the predicted first The results of the inspection, For generating functions of high-fidelity clinical environment systems, For patient background information, This is a collection of completed inspection records. For the first The name of the item to be checked; In autoregressive generative modeling, Consider it as a sequence consisting of L smallest units of text information. The generated conditional probability P is decomposed as follows:

[0029] in, Indicates that in generating the first The historical sequence of the smallest unit of text information has already been generated; during prediction, firstly... , and Encode it as a context vector, and then generate each one sequentially according to the conditional probabilities described above. until a terminator is generated or the maximum length is reached. Finally, the smallest unit sequence of the generated text information is concatenated to form the predicted inspection result. During training, the loss function is:

[0030] in, This indicates the total number of examinations performed on the patient. Indicates the first The smallest unit of text information in the inspection results. Indicates the first The first item of the inspection results The smallest unit of real text information This indicates that the first item in the inspection results The smallest unit of all real text information preceding the smallest unit of text information. This represents the loss function value.

[0031] Preferably, the module M3 includes: Module M3.1: In the cold start phase, the diagnostic agent is fine-tuned using multiple rounds of diagnostic trajectory samples through a loss function; Module M3.2: Reinforcement learning phase, which involves real-time interactive training between the diagnostic agent after cold start and the clinical environment system.

[0032] Preferably, when using autoregressive generative modeling, the conditional probability P is decomposed as follows:

[0033] in, The smallest unit of complete dialogue text information sequence. The trainable parameters of the diagnostic agent are represented. This represents the historical context when generating the i-th smallest unit of text information; during cold start fine-tuning, the loss is calculated and backpropagated only for the smallest unit of text information corresponding to the assistant role, and a gradient mask is defined. for:

[0034] The loss function is expressed as:

[0035] Where N is the number of the smallest units of text information.

[0036] Preferably, in each training iteration, the agent receives initial consultation information as its state, generates a preliminary diagnosis and suggestions for the next examination, inputs the results into the clinical environment system as feedback, forms a new state, and repeats the above process until a final diagnosis is generated or the maximum number of interaction rounds is reached. During training, the GRPO strategy optimization algorithm is used, and the reward function consists of three parts: a reward for the accuracy of the final diagnosis, a reward for the relevance of the examination recommendations, and a reward for the number of rounds. The reward for the accuracy of the final diagnosis is 1 when the predicted diagnosis is semantically consistent with the reference diagnosis, and 0 otherwise. The reward for the relevance of the examination recommendations is the F1 score of the reference examination set and the predicted examination set. The reward for the number of rounds is 0.1 when the number of dialogue rounds does not exceed the maximum number of rounds, and 0 otherwise. Let the policy function of the diagnostic agent be... In a given patient state At that time, the strategy outputs an action. The probability distribution is The GRPO algorithm updates its policy based on relative reward advantage, and its optimization objective function is... Defined as expected cumulative reward:

[0037] in, For mathematical expectation symbol, This represents a complete diagnostic trajectory. As a discount factor, For the trajectory length, This is the immediate reward for step t; Total reward function Defined as:

[0038] in, For weight hyperparameters, Rewards are given for the accuracy of the final diagnosis. To check the recommendation relevance reward, The reward is a round-based penalty; the specific definitions of the three sub-rewards are as follows:

[0039]

[0040] in, To predict the check set, For reference check set;

[0041] in, This is the preset maximum number of dialogue turns; The GRPO algorithm samples multiple trajectories in each training batch, calculates the cumulative reward for each trajectory, and then applies the relative advantage function. To adjust the policy gradient, where This is the average reward for all trajectories within the batch; The update formula for the policy parameters in the k-th iteration is:

[0042] in, For the first The parameters for the next iteration For the updated parameters, For learning rate, Indicates the parameter The gradient.

[0043] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention adopts the structure of "high-fidelity virtual clinical environment + multi-round reinforcement learning diagnostic agent", which solves the problem that the existing medical big language model cannot optimize the diagnostic strategy in dynamic and multi-round interaction, overcomes the defect of insufficient reasoning ability of static instruction fine-tuning when facing incomplete information and atypical cases, and achieves a double improvement in diagnostic accuracy and examination recommendation efficiency. (2) The present invention can generate simulated examination results that are highly consistent with the patient's condition based on real electronic health record data. The diagnostic agent then interacts in multiple rounds in the virtual environment, autonomously explores the diagnostic path, and optimizes the strategy through a dual reward mechanism, thereby realizing the leap from static prediction to dynamic decision-making in the diagnostic model, and greatly improving the diagnostic accuracy and the rationality of examination recommendations. Attached Figure Description

[0044] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the overall process of the closed-loop reinforcement learning method of this invention. Figure 2 Prepare a workflow diagram for data. Detailed Implementation

[0045] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0046] Example 1 like Figure 1 This invention provides a closed-loop reinforcement learning diagnostic method based on a high-fidelity virtual clinical environment, comprising: Step 1: Construct a standardized examination and diagnosis dataset; Step 1.1: Structured processing of patient basic information and examination record set; like Figure 2 In this step, we first extract patient electronic medical records containing multimodal information such as discharge summaries, laboratory tests, microbiological tests, and imaging examinations from the MIMIC-IV large clinical database. Based on this, the raw data is structured and expanded in terms of disease coverage to ensure the dataset can support diagnostic reasoning training for various disease scenarios. For each patient's medical record data, it is reorganized into two parts: basic patient information and a set of examination records arranged chronologically. During the extraction of basic patient information, the discharge summary is parsed, and a heuristic string matching method is used to extract fields such as physical examination, chief complaint, present illness, past medical history, social history, family history, and discharge diagnosis, which are then converted into structured text format. In the data cleaning stage, cases without physical examination records are first removed; then, the DeepSeek-V3 language model is used, combined with specific prompt words, to perform a second screening of cases, removing those with a final diagnosis already stated in the past medical history or present illness history, thus avoiding the introduction of already diagnosed referral cases, which typically lack the key examination information required for diagnostic reasoning.

[0047] Step 1.2: Check the construction and standardization of the time sequence check chain; During the construction of the examination record set, the aforementioned physical examination texts were converted into a structured table conforming to JSON format using the DeepSeek-V3 model, and this table was used as the first examination record in the examination chain. Subsequently, relevant data were extracted from laboratory tests, microbiological tests, and imaging tests within the day prior to admission and appended to the examination chain. During data standardization, the names of laboratory and microbiological test items were uniformly mapped; for example, "red blood cell count" was merged into the "complete blood cell count" category. For missing test names in imaging tests, string matching was used to extract and complete them from the EXAMINATION paragraph of the imaging report. All examination records were strictly sorted according to timestamps. For repeated tests of the same item, only the earliest record was retained to ensure the temporal consistency of the examination chain. Finally, the reconstructed dataset contained 118,478 patient electronic medical records, covering 4,897 diseases, with an average of 29 tests per patient (26 laboratory tests, 2 microbiological tests, and 1 imaging test).

[0048] Step 1.3: Construction and standardization of multi-round diagnostic trajectory data; Furthermore, to train the diagnostic agent, the aforementioned structured electronic medical record data needs to be transformed into a multi-round diagnostic trajectory format. The specific method is as follows: First, using the DeepSeek-V3 model, combined with the original discharge summary and structured electronic medical record data, initial consultation information is generated for each case. This information includes the chief complaint, present illness, past medical history, family history, etc., but does not include the final diagnosis. Then, a reference multi-round diagnostic trajectory is constructed based on a time-series examination chain. Each round includes the current preliminary diagnosis, the next recommended examination and its rationale, and the actual result of that examination. The preliminary diagnosis and rationale are generated by a large language model, while the examination order and results are directly derived from the actual electronic medical record. Finally, the final diagnosis is refined into a clear conclusion for a single major disease, ensuring consistency with the examination and reasoning process in the aforementioned trajectory. Regarding data quality control, a two-stage filtering strategy is adopted. Specific prompt words are used to detect and remove samples that reveal the final diagnosis in the initial consultation or intermediate reasoning, and samples with logical errors or non-standard formats are also removed.

[0049] Step 2: Based on the standardized examination and diagnosis dataset, a high-fidelity clinical environment system is trained; Step 2.1: Simulator training task design; In this step, the previously constructed EHR dataset is input into a high-fidelity clinical environment system for training, enabling it to simulate patient performance and examination results under different examination items in a real clinical environment. The training task is defined as an autoregressive text generation problem, that is, given basic patient information and existing examination records, predicting the result of the next examination. This result can be numerical (e.g., blood indicators) or textual (e.g., imaging descriptions), and the model uniformly treats it as a free text sequence for generation. Specifically, let the patient's examination sequence be... Each examination consists of an item name and a result. During training, the model is input with basic patient information and completed examination records, and outputs a sequence of results for the target examination. The loss function uses the standard token-level negative log-likelihood form.

[0050] Let the patient's background information and the set of completed examination records be... ,in Indicates the first The name of the item to be checked, This indicates the result of the examination. The generation function of a high-fidelity clinical environment system is denoted as... The process of predicting the result of the next test can be expressed as:

[0051] in, For patient background information, This is a collection of completed inspection records. For the first The name of the item to be checked, For the predicted first The results of this check will be used in autoregressive generative modeling. Consider it as a sequence of L tokens The conditional probability generated can be decomposed into:

[0052] in, Indicates that in generating the first The historical sequence for each token has already been generated. During prediction, first... , and Encode it as a context vector, and then generate each one sequentially according to the conditional probabilities described above. until a terminator is generated or the maximum length is reached. Finally, the generated token sequences are concatenated to form the prediction check result. During training, the loss function is:

[0053] in, This indicates the total number of examinations performed on the patient. Indicates the first The number of tokens in the item check results Indicates the first The first item of the inspection results A real token, This indicates that the first item in the inspection results All real tokens prior to this token This represents the loss function value.

[0054] Step 2.2: Model Training and Implementation Details; In the implementation process, Qwen2.5-Instruct-7B was used as the base model for initialization, with a maximum input length of 8192 tokens and a learning rate of 100%. The training run consisted of 15 epochs and was distributed across eight NVIDIA A100 GPUs using DeepSpeed ​​ZeRO Stage 2. During training, the model treated all examination results (both numerical and textual) as free text for generation, enabling it to simultaneously learn the statistical correlations and semantic patterns between numerical and descriptive results. Specifically, the AdamW optimizer was chosen to improve convergence speed and generalization performance. During training, each mini-batch's input consisted of the patient's background information, completed examination records, and the target examination item; the output was the actual text of that examination item. The training objective was to minimize the token-level negative log-likelihood loss function defined in step 2.1. The convergence criterion is that the loss values ​​on the training and validation sets continuously decrease and tend to stabilize, while multiple indicators on the independent test set meet the expected requirements. For example, in the task of generating simulated examinations of real cases, the instance-level clinical consistency reaches 96.91%, the 1-Wasserstein distance of numerical results is 0.128, and the Fréchet Inception Distance (FID) of textual results reaches 0.747, all of which are significantly better than existing baseline models.

[0055] After training convergence, the clinical environment system can generate high-fidelity examination results that conform to clinical logic based on different disease scenarios and previous examination results. It can even generate reasonable simulated results for examinations not observed in the training set, thus providing a stable, controllable, and realistic interactive environment for the diagnostic agent. This capability stems from the autoregressive modeling and generalization characteristics of the high-fidelity virtual clinical environment generation system. Traditional training methods based on replaying real medical records can only generate results for examinations actually performed by the patient, lacking corresponding records for examinations not performed, and thus failing to provide simulated feedback. In contrast, this method learns the statistical relationships and semantic patterns between patient background information, previous examination results, and potential examination items through a unified conditional generation framework during the training phase. Therefore, during inference, even if the target examination item is not directly associated with the patient in the training set, the system can still perform conditional inference based on existing information and generate results that conform to medical logic. This zero-shot generalization capability allows the diagnostic agent to explore more potentially valuable examination items during interaction, without being limited by the coverage of historical data, thereby significantly improving the flexibility and comprehensiveness of the virtual environment.

[0056] Step 3: Based on a high-fidelity clinical environment system, a diagnostic agent is obtained through reinforcement learning training; Step 3.1: Cold Start Phase; During the cold start phase, 1000 rigorously manually selected multi-turn diagnostic trajectories were chosen as training samples. These samples covered various disease types and exhibited no formatting errors, logical jumps, or diagnostic information leakage during diagnostic reasoning. These samples were organized into a standardized multi-turn dialogue format, with each turn including preliminary diagnoses and examination suggestions generated by the agent, as well as examination results returned by DiagGym. During training, the initial parameters of the diagnostic agent were set to Qwen2.5-Instruct-7B or other equivalent large-scale language models. Fine-tuning was performed using an autoregressive text generation method, with gradients calculated only for the assistant role's output. This allowed the model to learn the logical structure and format specifications of diagnostic reasoning, unaffected by environmental feedback text.

[0057] Specifically, each multi-turn diagnostic dialogue is represented as alternating user role messages and assistant role messages. The user role messages contain input information such as the patient's background information and examination results, while the assistant role messages contain the diagnostic agent's response (which may be examination suggestions or a final diagnosis). Let the complete dialogue token sequence be... Some tokens belong to the user role, and some belong to the assistant role. When using autoregressive generative modeling, the conditional probability P is decomposed as follows:

[0058] in, The trainable parameters of the diagnostic agent are represented. This represents the historical context when generating the i-th token. During cold start fine-tuning, we only calculate the loss and backpropagate it for the tokens corresponding to the assistant role, i.e., we define a gradient mask. for:

[0059] The loss function can then be expressed as:

[0060] This approach ensures that gradients come only from the output tokens of the assistant role, thus avoiding the model passively learning the input patterns or environmental feedback text of the user role during the cold start phase. Instead, it focuses on optimizing its own diagnostic reasoning output capabilities.

[0061] Training parameters include a maximum input length of 8192 tokens and a learning rate. The optimizer is AdamW, combined with DeepSpeed ​​ZeRO Stage 2 to achieve efficient distributed training. By monitoring the generation format accuracy and inference coherence on the validation set, the cold start phase is considered converged when the metrics show no significant improvement over multiple consecutive epochs. After this phase, the model can generate multi-turn diagnostic dialogues that conform to clinical logic without relying on reinforcement learning, laying the foundation for subsequent stages.

[0062] Step 3.2: Reinforcement Learning Phase; During the reinforcement learning phase, the diagnostic agent, after a cold start, is trained in real-time with the clinical environment system to optimize its diagnostic reasoning strategy and examination decision-making capabilities. In each training iteration, the agent receives initial consultation information as its state. The system generates a preliminary diagnosis and suggestions for further examinations. These are input into the clinical environment system to obtain examination results as feedback, forming a new state. This process is repeated until a final diagnosis is generated or the maximum number of interaction rounds is reached. The GRPO strategy optimization algorithm is used during training. The reward function consists of three parts: a reward for the accuracy of the final diagnosis, a reward for the relevance of the examination recommendations, and a reward for the number of rounds. The final diagnosis reward is 1 if the predicted diagnosis is semantically consistent with the reference diagnosis, and 0 otherwise. The examination recommendation reward is the F1 score of the reference examination set and the predicted examination set. The round penalty is 0.1 if the number of dialogue rounds does not exceed the maximum number of rounds, and 0 otherwise.

[0063] Specifically, let the policy function of the diagnostic agent be... ,in These are trainable parameters, given a patient state. When (including initial patient information and historical examination results), the strategy outputs an action. The probability distribution of (recommended examinations or diagnostic outputs) is as follows: The core of the GRPO (Group Relative Policy Optimization) algorithm is to update the policy based on relative reward advantage; its optimization objective function is... Defined as expected cumulative reward:

[0064] in, For mathematical expectation symbol, This represents a complete diagnostic trajectory. As a discount factor, For the trajectory length, Let be the immediate reward at step t. In this method, the total reward function is... Defined as:

[0065] in, These are the weight hyperparameters (set to 1.0, 0.5, 1.0 in the experiment). Rewards are given for accurate diagnosis. To check the recommendation relevance reward, The reward is a round-based penalty. The specific definitions of the three sub-rewards are as follows:

[0066]

[0067] in, To predict the check set, For reference check set.

[0068]

[0069] in, The maximum number of dialogue turns was preset (12 in the experiment). The GRPO algorithm samples multiple trajectories in each training batch, calculates the cumulative reward for each trajectory, and then applies a relative advantage function. To adjust the policy gradient, where This represents the average reward for all trajectories within the batch. The update formula for the policy parameters in the k-th iteration is:

[0070] in, For the first The parameters for the next iteration For the updated parameters, For learning rate, Indicates the parameter The gradient. Through this relative reward mechanism, GRPO can reduce reward variance, accelerate convergence, and enable the diagnostic agent to gradually learn to balance diagnostic accuracy, examination recommendation quality, and diagnostic efficiency in a virtual clinical environment, ultimately forming an optimized diagnostic strategy.

[0071] In implementation, DiagGym is deployed on two node servers, providing inference services through the vLLM framework; Qwen2.5-Instruct-72B is deployed as an independent evaluator on a separate node, used to calculate semantic consistency and check relevance metrics. Reinforcement learning training is performed in parallel on four node servers, each configured with 8 NVIDIA A100 GPUs, a batch size of 512, a maximum response length of 8192 tokens, and a learning rate of [missing information]. Each iteration executes 5 rollouts, converging after approximately 200 steps. After this training phase, the diagnostic agent can autonomously select the most informative examination items in various disease scenarios and complete accurate diagnosis with the fewest interaction rounds, thereby significantly improving the efficiency and accuracy of clinical diagnosis.

[0072] Example 2 The present invention also provides a closed-loop reinforcement learning diagnostic system based on a high-fidelity virtual clinical environment. The closed-loop reinforcement learning diagnostic system based on a high-fidelity virtual clinical environment can be implemented by executing the process steps of the closed-loop reinforcement learning diagnostic method based on a high-fidelity virtual clinical environment. That is, those skilled in the art can understand the closed-loop reinforcement learning diagnostic method based on a high-fidelity virtual clinical environment as a preferred embodiment of the closed-loop reinforcement learning diagnostic system based on a high-fidelity virtual clinical environment.

[0073] The system comprises: Module M1: Structures the patient's basic information and examination record set, constructs and standardizes a time-series examination chain, constructs and standardizes multi-round diagnostic trajectory data; unifies the mapping of laboratory and microbiology examination item names, extracts and completes missing examination names from reports through string matching in imaging examinations, sorts all examination records by timestamp, and retains only the earliest record for repeated examinations of the same item, resulting in a standardized examination and diagnosis dataset; Module M2: Based on the standardized examination and diagnosis dataset, designs a simulator training task, defining the training task as an autoregressive text generation problem, predicting the result of the next examination given the patient's basic information and existing examination records; performs model training and implementation, uses a language model as the basic model for initialization, and trains through distributed optimization to obtain a high-fidelity clinical environment system; Module M3: Based on the high-fidelity clinical environment system, trains a diagnostic agent through reinforcement learning.

[0074] In module M2, the process of predicting the result of the next check is represented as follows:

[0075] in, For the predicted first The results of the inspection, For generating functions of high-fidelity clinical environment systems, For patient background information, This is a collection of completed inspection records. For the first The name of the item to be checked; In autoregressive generative modeling, Consider it as a sequence consisting of L smallest units of text information. The generated conditional probability P is decomposed as follows:

[0076] in, Indicates that in generating the first The historical sequence of the smallest unit of text information has already been generated; during prediction, firstly... , and Encode it as a context vector, and then generate each one sequentially according to the conditional probabilities described above. until a terminator is generated or the maximum length is reached. Finally, the smallest unit sequence of the generated text information is concatenated to form the predicted inspection result. During training, the loss function is:

[0077] in, This indicates the total number of examinations performed on the patient. Indicates the first The smallest unit of text information in the inspection results. Indicates the first The first item of the inspection results The smallest unit of real text information This indicates that the first item in the inspection results The smallest unit of all real text information preceding the smallest unit of text information. This represents the loss function value.

[0078] The module M3 includes: Module M3.1: Cold start phase, which uses a loss function to fine-tune the diagnostic agent using multiple rounds of diagnostic trajectory samples; Module M3.2: Reinforcement learning phase, which performs real-time interactive training between the cold-started diagnostic agent and the clinical environment system.

[0079] When using autoregressive generative modeling, the conditional probability P is decomposed as follows:

[0080] in, The smallest unit of complete dialogue text information sequence. The trainable parameters of the diagnostic agent are represented. This represents the historical context when generating the i-th smallest unit of text information; during cold start fine-tuning, the loss is calculated and backpropagated only for the smallest unit of text information corresponding to the assistant role, and a gradient mask is defined. for:

[0081] The loss function is expressed as:

[0082] Where N is the number of the smallest units of text information.

[0083] In each training iteration, the agent receives initial consultation information as its state, generates a preliminary diagnosis and suggestions for the next examination, inputs the results of the examinations into the clinical environment system as feedback, forms a new state, and repeats the above process until a final diagnosis is generated or the maximum number of interaction rounds is reached. The GRPO strategy optimization algorithm is used during training. The reward function consists of three parts: a reward for the accuracy of the final diagnosis, a reward for the relevance of the examination recommendations, and a reward for the number of rounds. The reward for the accuracy of the final diagnosis is 1 when the predicted diagnosis is semantically consistent with the reference diagnosis, and 0 otherwise. The reward for the relevance of the examination recommendations is the F1 score of the reference examination set and the predicted examination set. The reward for the number of rounds is 0.1 when the number of dialogue rounds does not exceed the maximum number of rounds, and 0 otherwise. Let the policy function of the diagnostic agent be... In a given patient state At that time, the strategy outputs an action. The probability distribution is The GRPO algorithm updates its policy based on relative reward advantage, and its optimization objective function is... Defined as expected cumulative reward:

[0084] in, For mathematical expectation symbol, This represents a complete diagnostic trajectory. As a discount factor, For the trajectory length, This is the immediate reward for step t; Total reward function Defined as:

[0085] in, For weight hyperparameters, Rewards are given for the accuracy of the final diagnosis. To check the recommendation relevance reward, The reward is a round-based penalty; the specific definitions of the three sub-rewards are as follows:

[0086]

[0087] in, To predict the check set, For reference check set;

[0088] in, This is the preset maximum number of dialogue turns; The GRPO algorithm samples multiple trajectories in each training batch, calculates the cumulative reward for each trajectory, and then applies the relative advantage function. To adjust the policy gradient, where This is the average reward for all trajectories within the batch; The update formula for the policy parameters in the k-th iteration is:

[0089] in, For the first The parameters for the next iteration For the updated parameters, For learning rate, Indicates the parameter The gradient.

[0090] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0091] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A closed-loop reinforcement learning diagnostic method based on a high-fidelity virtual clinical environment, characterized in that, include: Step 1: Structure the patient's basic information and examination record set, construct a time-series examination chain and standardize it, construct multi-round diagnostic trajectory data and standardize it; uniformly map the names of laboratory and microbiology examination items, extract and complete the missing examination names in the imaging examinations from the reports through string matching, sort all examination records by timestamp, and retain only the earliest record for repeated examinations of the same item, thus obtaining a standardized examination and diagnosis dataset. Step 2: Based on the standardized examination and diagnosis dataset, design a simulator training task, defining the training task as an autoregressive text generation problem, predicting the next examination result given the patient's basic information and existing examination records; perform model training and implementation, using a language model as the basic model for initialization, and train it through distributed optimization to obtain a high-fidelity clinical environment system; Step 3: Based on the high-fidelity clinical environment system, a diagnostic agent is obtained through reinforcement learning training.

2. The closed-loop reinforcement learning diagnostic method based on a high-fidelity virtual clinical environment according to claim 1, characterized in that, In step 2, the process of predicting the result of the next inspection is expressed as follows: in, For the predicted first The results of the inspection, For generating functions of high-fidelity clinical environment systems, For patient background information, This is a collection of completed inspection records. For the first The name of the item to be checked; In autoregressive generative modeling, Consider it as a sequence consisting of L smallest units of text information. The generated conditional probability P is decomposed as follows: in, Indicates that in generating the first The historical sequence has been generated when the smallest unit of text information is obtained; during prediction, the first step is to... , and Encode it as a context vector, and then generate each one sequentially according to the conditional probabilities described above. until a terminator is generated or the maximum length is reached. Finally, the smallest unit sequence of the generated text information is concatenated to form the predicted inspection result. During training, the loss function is: in, This indicates the total number of examinations performed on the patient. Indicates the first The smallest unit of text information in the inspection results. Indicates the first The first item of the inspection results The smallest unit of real text information This indicates that the first item in the inspection results The smallest unit of all real text information preceding the smallest unit of text information. This represents the loss function value.

3. The closed-loop reinforcement learning diagnostic method based on a high-fidelity virtual clinical environment according to claim 2, characterized in that, Step 3 includes: Step 3.1: In the cold start phase, the diagnostic agent is fine-tuned using multiple rounds of diagnostic trajectory samples through a loss function; Step 3.2: Reinforcement learning phase, the diagnostic agent after cold start is trained in real time through interactive training with the clinical environment system.

4. The closed-loop reinforcement learning diagnostic method based on a high-fidelity virtual clinical environment according to claim 3, characterized in that, When using autoregressive generative modeling, the conditional probability P is decomposed as follows: in, The smallest unit sequence of complete dialogue text information. The trainable parameters of the diagnostic agent are represented. This represents the historical context when generating the i-th smallest unit of text information; during the cold start fine-tuning process, the loss is calculated and backpropagated only for the smallest unit of text information corresponding to the assistant role, and a gradient mask is defined. for: The loss function is expressed as: Where N is the number of the smallest units of text information.

5. The closed-loop reinforcement learning diagnostic method based on a high-fidelity virtual clinical environment according to claim 4, characterized in that, In each training iteration, the agent receives initial consultation information as its state, generates a preliminary diagnosis and suggestions for the next examination, inputs the results of the examinations into the clinical environment system as feedback, forms a new state, and repeats the above process until a final diagnosis is generated or the maximum number of interaction rounds is reached. The GRPO strategy optimization algorithm is used during training. The reward function consists of three parts: a reward for the accuracy of the final diagnosis, a reward for the relevance of the examination recommendations, and a reward for the number of rounds. The reward for the accuracy of the final diagnosis is 1 when the predicted diagnosis is semantically consistent with the reference diagnosis, and 0 otherwise. The reward for the relevance of the examination recommendations is the F1 score of the reference examination set and the predicted examination set. The reward for the number of rounds is 0.1 when the number of dialogue rounds does not exceed the maximum number of rounds, and 0 otherwise. Let the policy function of the diagnostic agent be... In a given patient state At that time, the strategy outputs an action. The probability distribution is The GRPO algorithm updates its policy based on relative reward advantage, and its optimization objective function is... Defined as expected cumulative reward: in, For mathematical expectation symbol, This represents a complete diagnostic trajectory. As a discount factor, For the trajectory length, This is the immediate reward for step t; Total reward function Defined as: in, For weight hyperparameters, Rewards are given for the accuracy of the final diagnosis. To check the recommendation relevance reward, The reward is based on the number of rounds; the specific definitions of the three sub-rewards are as follows: in, To predict the check set, For reference check set; in, This is the preset maximum number of dialogue turns; The GRPO algorithm samples multiple trajectories in each training batch, calculates the cumulative reward for each trajectory, and then applies the relative advantage function. To adjust the policy gradient, where This is the average reward for all trajectories within the batch; The update formula for the policy parameters in the k-th iteration is: in, For the first The parameters for the next iteration For the updated parameters, For learning rate, Indicates the parameter The gradient.

6. A closed-loop reinforcement learning diagnostic system based on a high-fidelity virtual clinical environment, characterized in that, include: Module M1: Structures the patient's basic information and examination record set, constructs and standardizes the time-series examination chain, constructs and standardizes multi-round diagnostic trajectory data; unifies the mapping of laboratory and microbiology examination item names, extracts and completes missing examination names in imaging examinations from the report through string matching, sorts all examination records by timestamp, and retains only the earliest record for repeated examinations of the same item, thus obtaining a standardized examination and diagnosis dataset. Module M2: Based on the standardized examination and diagnosis dataset, a simulator training task is designed, which is defined as an autoregressive text generation problem. Given the patient's basic information and existing examination records, the task is to predict the result of the next examination. The model is trained and implemented, using a language model as the basic model for initialization, and trained through distributed optimization to obtain a high-fidelity clinical environment system. Module M3: Based on the high-fidelity clinical environment system, a diagnostic agent is obtained through reinforcement learning training.

7. The closed-loop reinforcement learning diagnostic system based on a high-fidelity virtual clinical environment according to claim 6, characterized in that, In module M2, the process of predicting the result of the next check is represented as follows: in, For the predicted first The results of the inspection, For generating functions of high-fidelity clinical environment systems, For patient background information, This is a collection of completed inspection records. For the first The name of the item to be checked; In autoregressive generative modeling, Consider it as a sequence consisting of L smallest units of text information. The generated conditional probability P is decomposed as follows: in, Indicates that in generating the first The historical sequence has been generated when the smallest unit of text information is obtained; during prediction, the first step is to... , and Encode it as a context vector, and then generate each one sequentially according to the conditional probabilities described above. until a terminator is generated or the maximum length is reached. Finally, the smallest unit sequence of the generated text information is concatenated to form the predicted inspection result. During training, the loss function is: in, This indicates the total number of examinations performed on the patient. Indicates the first The smallest unit of text information in the inspection results. Indicates the first The first item of the inspection results The smallest unit of real text information This indicates that the first item in the inspection results The smallest unit of all real text information preceding the smallest unit of text information. This represents the loss function value.

8. The closed-loop reinforcement learning diagnostic system based on a high-fidelity virtual clinical environment according to claim 7, characterized in that, The module M3 includes: Module M3.1: In the cold start phase, the diagnostic agent is fine-tuned using multiple rounds of diagnostic trajectory samples through a loss function; Module M3.2: Reinforcement learning phase, which involves real-time interactive training between the diagnostic agent after cold start and the clinical environment system.

9. The closed-loop reinforcement learning diagnostic system based on a high-fidelity virtual clinical environment according to claim 8, characterized in that, When using autoregressive generative modeling, the conditional probability P is decomposed as follows: in, The smallest unit of complete dialogue text information sequence. The trainable parameters of the diagnostic agent are represented. This represents the historical context when generating the i-th smallest unit of text information; during the cold start fine-tuning process, the loss is calculated and backpropagated only for the smallest unit of text information corresponding to the assistant role, and a gradient mask is defined. for: The loss function is expressed as: Where N is the number of the smallest units of text information.

10. The closed-loop reinforcement learning diagnostic system based on a high-fidelity virtual clinical environment according to claim 9, characterized in that, In each training iteration, the agent receives initial consultation information as its state, generates a preliminary diagnosis and suggestions for the next examination, inputs the results of the examinations into the clinical environment system as feedback, forms a new state, and repeats the above process until a final diagnosis is generated or the maximum number of interaction rounds is reached. The GRPO strategy optimization algorithm is used during training. The reward function consists of three parts: a reward for the accuracy of the final diagnosis, a reward for the relevance of the examination recommendations, and a reward for the number of rounds. The reward for the accuracy of the final diagnosis is 1 when the predicted diagnosis is semantically consistent with the reference diagnosis, and 0 otherwise. The reward for the relevance of the examination recommendations is the F1 score of the reference examination set and the predicted examination set. The reward for the number of rounds is 0.1 when the number of dialogue rounds does not exceed the maximum number of rounds, and 0 otherwise. Let the policy function of the diagnostic agent be... In a given patient state At that time, the strategy outputs an action. The probability distribution is The GRPO algorithm updates its policy based on relative reward advantage, and its optimization objective function is... Defined as expected cumulative reward: in, For mathematical expectation symbol, This represents a complete diagnostic trajectory. As a discount factor, For the trajectory length, This is the immediate reward for step t; Total reward function Defined as: in, For weight hyperparameters, Rewards are given for the accuracy of the final diagnosis. To check the recommendation relevance reward, The reward is a round-based penalty; the specific definitions of the three sub-rewards are as follows: in, To predict the check set, For reference check set; in, This is the preset maximum number of dialogue turns; The GRPO algorithm samples multiple trajectories in each training batch, calculates the cumulative reward for each trajectory, and then applies the relative advantage function. To adjust the policy gradient, where This is the average reward for all trajectories within the batch; The update formula for the policy parameters in the k-th iteration is: in, For the first The parameters for the next iteration For the updated parameters, For learning rate, Indicates the parameter The gradient.

Images