Methods and devices for training large-scale medical models for digital doctors

By constructing a dynamic evaluation rule set and reinforcement learning driven by multi-dimensional reward signals, the shortcomings of existing large-scale medical models in terms of user experience and interaction strategies are addressed, thereby achieving continuous optimization of digital doctor service quality and improvement of user satisfaction.

CN123070185APending Publication Date: 2026-11-10ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611290729.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-24
Publication Date
2026-11-10

Smart Images

  • Figure CN123070185A_ABST
    Figure CN123070185A_ABST
Patent Text Reader

Abstract

This specification discloses a method and apparatus for training a large-scale medical model for digital doctors. The training method includes: first, acquiring a historical conversation sample set between a user and a digital doctor, the historical conversation sample set including high-scoring samples and low-scoring samples; then, processing the historical conversation sample set using a first large-scale model to generate an evaluation rule set for digital doctor services, the evaluation rule set including medical accuracy rules and interaction experience rules; next, using the large-scale medical model implementing the digital doctor, processing a first conversation context between a first user and the digital doctor to obtain corresponding candidate responses; then, determining a reward signal for the candidate responses based on the first conversation context, the candidate responses, and the evaluation rule set; and finally, training the large-scale medical model using a reinforcement learning algorithm based on the candidate responses and the reward signal.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One or more embodiments in this specification relate to the field of machine learning technology, and more particularly to a method and apparatus for training a large medical model for digital doctors, a computer-readable storage medium, and a computing device. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models (LMs) are increasingly being used in the healthcare field. Currently, the industry is using general-purpose large language models and massive amounts of medical literature, clinical guidelines, and anonymized doctor-patient dialogue data for pre-training or fine-tuning in vertical domains to build large medical models for healthcare scenarios.

[0003] Based on large-scale medical models, products such as digital doctors, intelligent health assistants, and voice consultation systems have emerged. These systems aim to provide users with services such as initial health consultations, triage suggestions, chronic disease management, and emotional support through natural language interaction. Especially in voice interaction scenarios, digital doctors can simulate real doctor-patient dialogues, lowering the barrier to entry for users and improving the accessibility of medical services.

[0004] However, in practical applications, it has been found that existing large-scale medical models are insufficient to meet the higher requirements for user experience and other aspects in real online scenarios. Summary of the Invention

[0005] This specification describes an embodiment of a medical large-scale model training method and apparatus for digital doctors, which can solve the above-mentioned technical problems.

[0006] Based on the first aspect, a method for training a large-scale medical model for digital doctors is provided. This training method includes:

[0007] A historical session sample set between the user and the digital doctor is obtained, including high-scoring and low-scoring samples. The historical session sample set is processed using a first large-scale model to generate an evaluation rule set for the digital doctor service, including rules related to medical accuracy and interactive experience. Using a large-scale medical model implementing the digital doctor, the first session context between the first user and the digital doctor is processed to obtain corresponding candidate responses. Based on the first session context, the candidate responses, and the evaluation rule set, a reward signal for the candidate responses is determined. Based on the candidate responses and the reward signal, a reinforcement learning algorithm is used to train the large-scale medical model.

[0008] In one embodiment, the historical conversation sample set is derived from historical voice calls, and the first conversation context includes transcribed text of the current round and historical multi-round voice dialogues. The interaction experience rules include one or more of the following: voice interruption repair rules, automatic speech recognition error-tolerant rules, and rules for naturalness in colloquial expression.

[0009] In one embodiment, obtaining a historical conversation sample set between a user and a digital doctor includes: obtaining a first historical conversation record and a corresponding first user evaluation feedback; if the first user evaluation feedback meets a preset high score condition, determining the first historical conversation record as a high-scoring sample; if the first user evaluation feedback meets a preset low score condition, determining the first historical conversation record as a low-scoring sample.

[0010] In one embodiment, obtaining a historical session sample set between a user and a digital doctor includes: obtaining a second historical session record and analyzing the second historical session record using a second large model; if the analysis results indicate that the second historical session record meets the high-score criteria, identifying it as a high-score sample; if the analysis results indicate that the second historical session record meets the low-score criteria, identifying it as a low-score sample.

[0011] In one embodiment, the historical session sample set is processed using a first large model to generate an evaluation rule set for digital doctor services, including: clustering the high-scoring samples and the low-scoring samples to obtain multiple sample clusters; inputting samples in the same sample cluster as a batch into the first large model to generate one or more evaluation rules corresponding to that cluster, and classifying them into the evaluation rule set.

[0012] In one embodiment, determining a reward signal for a candidate response based on the first session context, the candidate response, and the evaluation rule set includes: determining the user intent corresponding to the first session context; filtering an enabled subset of evaluation rules from the evaluation rule set based on the user intent; calculating a comprehensive score for the candidate response under the enabled subset of evaluation rules; and using the comprehensive score as a reward signal for the candidate response.

[0013] Further, in a specific embodiment, the evaluation rules in the evaluation rule set include pass criteria, failure criteria, and weights; wherein, calculating the comprehensive score of the candidate response under the enabled subset of evaluation rules includes:

[0014] By inputting the candidate response, the first session context, and the enabled subset of evaluation rules into the third model, it is determined whether the candidate response meets the pass or fail criteria of each evaluation rule. If it meets the pass criteria, a first score is assigned under the corresponding evaluation rule; if it meets the fail criteria, a second score is assigned under the corresponding evaluation rule; wherein the first score is higher than the second score. Using the weights corresponding to each evaluation rule in the enabled subset of evaluation rules, the scores under each evaluation rule are weighted and summed to determine the comprehensive score.

[0015] Furthermore, in one example, the weighted summation result is used as the initial comprehensive score, and the method further includes: detecting whether there is a key evaluation rule in the enabled evaluation rule subset, the key evaluation rule including medical accuracy-related rules or medical safety risk-related rules; if there is a key evaluation rule, and the score under the key evaluation rule is lower than a first preset threshold, then the smaller value between the initial comprehensive score and the preset low score is used as the final comprehensive score.

[0016] In one embodiment, the number of candidate responses is multiple; wherein, training the medical big model using a reinforcement learning algorithm based on the candidate responses and reward signals includes: constructing a training group based on multiple candidate responses and their corresponding reward signals; using a group relative strategy optimization algorithm to calculate the relative advantage of each candidate response within the training group relative to the average level within the group; and using the relative advantage as an optimization signal to update the parameters of the medical big model.

[0017] In one embodiment, the method further includes: obtaining a labeled reward signal for the candidate response as a baseline signal; determining that the reward signal generation is abnormal if the difference between the reward signal and the baseline signal is greater than a second preset threshold; processing the first session context, candidate response, evaluation rule set, reward signal and baseline signal using a fourth model to generate a cause analysis result for the abnormality in the reward signal generation; and adjusting the evaluation rule set based on the cause analysis result.

[0018] According to the second aspect, a medical large-scale model training device for digital doctors is provided. The training device includes: a conversation sample set acquisition module, configured to acquire a historical conversation sample set between a user and a digital doctor, the historical conversation sample set including high-scoring samples and low-scoring samples; an evaluation rule set generation module, configured to process the historical conversation sample set using a first large-scale model to generate an evaluation rule set for digital doctor services, the evaluation rule set including medical accuracy rules and interaction experience rules; a candidate response generation module, configured to process a first conversation context between a first user and the digital doctor using a medical large-scale model implementing the digital doctor to obtain corresponding candidate responses; a reward signal determination module, configured to determine a reward signal for the candidate responses based on the first conversation context, the candidate responses, and the evaluation rule set; and a medical large-scale model training module, configured to train the medical large-scale model using a reinforcement learning algorithm based on the candidate responses and the reward signal.

[0019] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method provided in the first aspect.

[0020] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method provided in the first aspect.

[0021] In summary, by employing the methods and apparatus disclosed in the embodiments of this specification, and by constructing a dynamic evaluation rule set based on historical conversation data mining, and by using multi-dimensional reward signals to drive reinforcement learning, the evaluation system is automated and dynamically updated. This enables digital doctors to quickly adapt to dynamically changing user needs, significantly improving service quality and user satisfaction in complex scenarios such as real voice calls. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the implementation architecture of the medical large-scale model training scheme for digital doctors disclosed in the embodiments of this specification;

[0024] Figure 2 This is a schematic diagram of the process steps of the medical large model training method for digital doctors disclosed in the embodiments of this specification;

[0025] Figure 3 This is a schematic diagram of the functional module structure of the medical large-scale model training device for digital doctors disclosed in the embodiments of this specification. Detailed Implementation

[0026] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0027] As mentioned earlier, existing large-scale medical models are insufficient to meet the higher requirements for user experience and other aspects in real-world online scenarios. Specifically, existing technologies mainly suffer from the following problems:

[0028] 1) Traditional medical large-scale model evaluation often focuses on the factual accuracy, format standardization, or medical knowledge coverage of single-round text question answering.

[0029] However, in real-world digital doctor service scenarios, user feedback depends not only on the accuracy of the medical content but also heavily on the model's interaction strategies. For example, can the model handle context to prevent users from repeating their symptoms, effectively soothe users' anxieties, and stop pushing the service after a user declines? In scenarios where the service is in the form of voice calls, it's also crucial to consider whether the model can identify and handle voice interruptions or Automatic Speech Recognition (ASR) errors.

[0030] 2) Evaluation rules built upon human experience are outdated and one-sided, making it difficult to consistently capture the preferences of real online users. Currently, the main method for optimizing model dialogue strategies is to manually summarize common problems and write prompts or fixed rules. This approach has significant limitations:

[0031] On the one hand, manual rules struggle to cover the massive and dynamically changing real user feedback, especially long-tail, non-standardized failure patterns (such as misunderstandings caused by specific dialect accents, complex multi-turn intent drift, etc.). On the other hand, manual rules are often static and cannot form an automated closed loop from "online user feedback" to "model training signals." When new negative review patterns emerge online, relying on manual discovery, analysis, and rule updates is time-consuming and costly, causing model optimization to lag behind changes in user needs.

[0032] 3) Traditional reward signal modeling methods are simplistic and struggle to balance the contradiction between medical rigor and interactive flexibility.

[0033] In reinforcement learning training, such as Reinforcement Learning from Human Feedback (RLHF), the quality of the reward signal directly determines the direction of model optimization. Current techniques typically use a single reward model to score responses. This single-dimensional scoring can easily mask key issues: for example, a response might be very friendly in tone (high score) but pose a slight hallucination or safety risk in medical advice (should be low score); or a response might be completely medically correct (high score) but its expression is too academic and verbose, unsuitable for voice broadcasting, resulting in a poor user experience (should be low score). A single total score signal cannot distinguish these dimensions, easily leading to "reward avoidance" behavior in the model during training—that is, sacrificing medical safety or ignoring the unique characteristics of voice interaction in pursuit of a high score.

[0034] In summary, how to construct a training method that can deeply integrate medical professionalism and interactive features, automatically mine preferences from real online user feedback, generate dynamic evaluation rules, and construct multi-dimensional reward signals to drive continuous closed-loop optimization of large models is a technical challenge that urgently needs to be solved in the field of digital doctor technology.

[0035] To address the above problems, this specification discloses an improved solution in its embodiments. For example... Figure 1 As shown, the implementation architecture of this scheme mainly includes four interrelated processing stages:

[0036] Phase 1 involves generating the evaluation rule set: using a large model to analyze the historical session sample set (including high-scoring and low-scoring samples) in the digital doctor service scenario, and automatically generating an evaluation rule set that includes medical accuracy and interactive experience categories.

[0037] Phase Two is candidate response generation: The first session context input is used to implement the digital doctor's large-scale medical model, which generates one or more corresponding candidate responses. It should be noted that the first session context refers to the interaction context data between the user to be processed and the digital doctor. Furthermore, the "first" in "first session context," as well as similar terms such as "second" and "third" elsewhere in the text, are all for the purpose of clearly distinguishing similar things and do not have any ranking or other limiting function.

[0038] Phase 3 is the reward signal calculation: The reward signal for the candidate response is calculated by the reward signal calculation module based on the context of the first session, the generated candidate response, and the evaluation rule set provided in the first phase.

[0039] Phase four is reinforcement learning training: based on the generated candidate responses and the calculated reward signals, reinforcement learning algorithms are used to update the parameters of the large medical model, thereby achieving closed-loop optimization of the model.

[0040] As can be seen from the above, the improved solution provided in the embodiments of this specification effectively solves many problems faced by existing large medical models in real online scenarios by constructing a complete technical architecture from historical session data mining, dynamic evaluation rule generation, multi-dimensional reward signal calculation to reinforcement learning closed-loop optimization.

[0041] Specifically, this solution utilizes a large model to automatically analyze high- and low-scoring samples and generate an evaluation rule set covering medical accuracy and interactive experience. This overcomes the shortcomings of manual rules, such as being outdated, one-sided, and unable to cover long-tail problems, thus achieving automation and dynamic updates to the evaluation system. Simultaneously, the reward signal calculated based on multi-dimensional evaluation rules can simultaneously consider distinguishing between medical rigor and interactive flexibility, avoiding reward avoidance behavior and ensuring that the model balances safety and user experience during optimization. Finally, by directly converting real online feedback into training signals to drive model parameter updates, an efficient data-driven closed-loop optimization mechanism is established, significantly reducing manual maintenance costs. This enables digital doctors to quickly adapt to dynamically changing user needs and continuously improve service quality and user satisfaction in complex scenarios such as real voice calls.

[0042] Next, the implementation process of the improvement scheme will be described in detail with reference to specific embodiments. Figure 2 The process of training a large medical model for digital doctors is shown.

[0043] It should be noted that the "digital doctor" in this manual may also be referred to as a "medical intelligent assistant" or a "virtual diagnostic agent." Its core interactive capabilities are driven by a large medical model, but are not limited to the model itself. In actual deployment, it can be combined with components such as Automatic Speech Recognition (ASR), Text-to-Speech (TTS), intent understanding modules, a knowledge base retrieval engine, and a dialogue state tracker to form a complete service system. This digital doctor supports multimodal interaction, including text chat and real-time voice calls, and can dynamically adapt to different communication formats in different scenarios.

[0044] Figure 2The steps shown correspond to the four stages described above as follows: steps S210 and S220 together correspond to the "evaluation rule set generation" stage; step S230 corresponds to the "candidate response generation" stage; step S240 corresponds to the "reward signal calculation" stage; and step S250 corresponds to the "reinforcement learning training" stage. These four steps are sequentially linked, forming a closed-loop iterative path from data mining to model optimization. Furthermore, the execution entity of this method can be any device, platform, server, or equipment cluster with computing and processing capabilities.

[0045] Figure 2 The process steps shown are as follows:

[0046] Step S210: Obtain a historical session sample set between the user and the digital doctor, the historical session sample set including high-scoring samples and low-scoring samples.

[0047] It should be understood that each historical session sample in the historical session sample set may contain one or more rounds of dialogue between the user and the digital doctor, as well as rich interaction log data.

[0048] Taking voice call scenarios as an example, historical conversation samples can be referred to as historical voice call samples. The real voice call data collected can include multi-turn user speech-to-text transcription, model response content, timestamps, interruptions, mute, interruptions, retries, and other call context information, as well as user ratings, positive or negative rating tags, reasons for low scores, and post-call evaluations. Simultaneously, it can also record online behavioral indicators such as model version, strategy version, user retention, interaction depth, call duration, whether the user continued to ask follow-up questions, and whether the user exited midway. It should be noted that, for ease of understanding, the following explanation mainly focuses on voice call scenarios; other scenarios such as text interaction can be inferred more easily.

[0049] For low-scoring samples, further low-scoring attribution results can be received or generated to distinguish whether the source of the problem is medical risk, hallucination, automatic speech recognition error, abnormal speech link, irrelevant answer, repeated follow-up questions, or forced diversion. After obtaining the above raw data, it is usually subjected to preprocessing operations such as de-identification, cleaning, and noise reduction to ensure data quality and compliance.

[0050] The definition of high-scoring and low-scoring samples can be derived from direct user feedback. In practical analysis, high-scoring and low-scoring samples are generally considered to have higher mining value, while the analytical significance of samples in the middle range is relatively limited. To reduce the burden of subsequent processing and focus on key issues, this step prioritizes selecting high-scoring and low-scoring samples to construct the sample set, with a particular emphasis on in-depth analysis of low-scoring samples.

[0051] In one embodiment, the specific method for obtaining the historical session sample set is as follows: obtain a first historical session record and its corresponding first user evaluation feedback; when the first user evaluation feedback meets the preset high score condition, the first historical session record is determined as a high-scoring sample; when the first user evaluation feedback meets the preset low score condition, it is determined as a low-scoring sample.

[0052] For example, the preset high score conditions include: the user's explicit rating is greater than or equal to 4 points (out of 5), or the user clicks positive tags such as "satisfied" or "helpful", or the user does not complain after the conversation ends and completes the subsequent recommended service process (such as successful registration or medication purchase); the preset low score conditions include: the user's explicit rating is less than or equal to 2 points, or the user clicks negative tags such as "dissatisfied", "useless", or "poor attitude", or the user exhibits behavioral characteristics such as hanging up midway through the conversation, exiting after a long period of silence, repeatedly asking the same question, or clearly expressing anger.

[0053] In another embodiment, sample selection can also be automated: a second historical session record is acquired and analyzed using a second large model. If the analysis results indicate that it meets the high-score criteria, it is identified as a high-score sample; if the analysis results indicate that it meets the low-score criteria, it is identified as a low-score sample. Based on this, the high-score and low-score labels can be used as service quality labels corresponding to the historical session samples, facilitating subsequent rule generation and reward signal calculation.

[0054] It's important to understand that the aforementioned high and low score criteria can be predefined and written into the prompts provided to the second-level model. These prompts guide the second-level model to comprehensively evaluate conversation quality from two dimensions: medical accuracy and interactive experience. For example, high score criteria specified in the prompts might include: correct medical logic in the response content, no risk of hallucinations, accurate identification of user intent, natural and fluent language with empathy, and effective handling of ASR errors in voice scenarios. Low score criteria might include: serious medical factual errors or safety risk suggestions, irrelevant answers or ignoring the user's core needs, a stiff and mechanical tone, lack of reassurance when the user expresses anxiety, and conversation interruptions or loops due to incomprehension of colloquial expressions. The second-level model scores or categorizes the second-level historical conversation records based on these criteria, thereby automatically filtering out high-quality and low-quality samples.

[0055] As described above, this step yields a session sample set with significant analytical value. It should be noted that the historical session sample set may consist only of high-scoring and low-scoring samples, or it may include other types of samples, such as medium-scoring samples or specialized samples from specific scenarios, to enrich the coverage dimensions of the generated evaluation rules.

[0056] Step S220: Process the historical session sample set using the first large model to generate an evaluation rule set for digital doctor services. The evaluation rule set includes medical accuracy rules and interactive experience rules.

[0057] It should be noted that the above-mentioned medical accuracy rules are mainly used to ensure the professionalism and safety of the digital doctor's response content. Their purpose is to provide clear medical compliance judgment criteria for the model's generated responses and prevent the output content from deviating from clinical norms or containing misleading information.

[0058] For example, medical accuracy rules can cover dimensions such as safety risk identification, fabrication and exaggeration detection, and factual consistency verification. Among them, safety risk identification is used to determine whether the model has omitted necessary medical risk warnings or given advice that may endanger the user's health; fabrication and exaggeration detection is used to determine whether the model has fabricated medical information or over-promised treatment effects without evidence; factual consistency verification is used to determine whether the model's response is consistent with the user's medical condition information confirmed in an authoritative medical knowledge base or dialogue, avoiding contradictions or deviations from common sense.

[0059] The aforementioned interactive experience rules are mainly used to optimize the fluency of communication and emotional comfort between users and digital doctors. Their purpose is to transform users' subjective interactive feelings into quantifiable and executable interactive behavior standards, enabling the model to naturally, empathetically, and efficiently advance the dialogue in real-time scenarios such as voice calls.

[0060] For example, interaction experience rules can cover the following core dimensions: contextual continuity and repetition avoidance, used to determine whether the model can remember the user's previously stated medical information and avoid requiring the user to repeatedly describe the same content; alignment with current needs, used to determine whether the model prioritizes responding to the user's most crucial question in the current round, rather than giving irrelevant answers or forcibly changing the topic; clarification when information is insufficient, used to determine whether the model will proactively ask for necessary details when the user's described symptoms lack key diagnostic elements, rather than blindly giving advice; and rejection signal recognition, used to determine whether the model will respond when the user clearly indicates that they do not want to continue answering or request a direct conclusion. The model should be able to switch to a concise and direct response strategy in a timely manner, rather than continuing to mechanically ask follow-up questions; ASR (Automatic Speech Recognition) tolerance should be used to determine whether the model can reasonably infer the user's intent based on the context when there are empty texts, typos, or semantic inconsistencies in the speech transcription, rather than directly reporting an error or ignoring it; naturalness of conversational expression should be used to determine whether the model's response conforms to everyday conversational habits, avoiding the use of overly formal, lengthy, or text-reading-appropriate but speech-reading-inappropriate expressions; and emotional continuity should be used to determine whether the model can recognize the user's anxiety, worry, and other emotions, and provide appropriate comfort and empathetic responses, rather than simply outputting cold medical information.

[0061] Regarding the specific structure of the evaluation rules, each rule can include fields such as rule name, triggering condition, pass criterion, failure criterion, scope of application marker, weight, risk level, and attribution label for failure. The triggering condition describes the conversation scenario in which the rule is activated, such as when a user interrupts their speech or expresses refusal. The pass and failure criteria define the expected and unexpected response characteristics, respectively. The scope of application marker limits the specific business scenario, conversation stage, or user group in which the rule takes effect, preventing the rule from being mistakenly triggered in irrelevant contexts. The weight indicates the relative importance of the rule within the evaluation rule set. The risk level can be used to identify the severity of the problem corresponding to the rule, for example, divided into high, medium, and low levels. The attribution label is used to mark the category of failure reason. Through these structured fields, the evaluation rule layer can transform abstract user preferences and complex failure patterns into standardized evaluation criteria that are judgmental, trainable, and traceable.

[0062] For example, an evaluation rule for voice interaction could be:

[0063] Name: "Voice Interruption Repair Rule";

[0064] Triggering condition: "An empty text is returned by the ASR or the user asks the same question repeatedly";

[0065] The standard states: "The model actively identifies content that was not heard clearly and provides a brief restatement option."

[0066] Failure criteria: "The model ignores the interruption and continues the original topic directly or outputs a long, irrelevant reply";

[0067] Weight: 0.8;

[0068] Attribution tag: "Interaction experience - interruption handling failure".

[0069] During the generation of evaluation rules, a structured prompt word template can be constructed to guide the output of the first large model, thereby enhancing the feasibility and standardization of the technical solution. This prompt word template instructs the large model to generate evaluation rules based on the input sample batches, according to a preset field format. It also includes explanations or examples of each field to ensure that the generated large-scale evaluation rules have a unified format and clear semantic boundaries, facilitating automated parsing and execution by the subsequent reward signal calculation module. For the specific definitions and functions of each field, please refer to the relevant descriptions in the aforementioned embodiments.

[0070] In one implementation, to improve the targeting and efficiency of rule generation, the high-scoring and low-scoring samples can be clustered first to obtain multiple sample clusters representing specific service modes or failure types. It should be understood that the clustering process can employ a hybrid clustering mode (merging high and low-scoring samples) or a separate clustering mode (processing high and low-scoring samples independently) to adapt to different rule mining granularity requirements.

[0071] Subsequently, samples from the same sample cluster are input into the first large model as a batch, generating one or more evaluation rules corresponding to that cluster and categorizing them into the evaluation rule set.

[0072] Clustering preprocessing enables massive heterogeneous session data to be structured and grouped according to semantic or behavioral patterns, allowing large models to focus on similar features when generating rules, avoiding interference between different patterns, thereby significantly improving the accuracy, coverage completeness and generation efficiency of rules.

[0073] In another implementation, the historical session sample set can be randomly divided into multiple different batches and then input into the first large model for processing without going through the clustering step, so as to simplify the process and ensure the diversity of samples.

[0074] As described above, this step enables the automatic generation of a dynamic evaluation rule set that combines medical expertise with interactive flexibility based on a historical session sample set, providing a refined and quantifiable evaluation basis for subsequent reward signal calculation.

[0075] Step S230: Using the medical big model that implements the digital doctor, process the first session context between the first user and the digital doctor to obtain the corresponding candidate response.

[0076] It should be understood that the first conversation context may include transcribed text of the current round and previous rounds of voice dialogue, as well as auxiliary data such as user profiles and past medical history tags, in order to fully restore the real context of the user's interaction with the digital doctor.

[0077] Furthermore, the number of candidate responses can be one or more; when multiple candidate responses are generated, they can be used to improve the sampling efficiency and optimization stability of reinforcement learning.

[0078] Step S240: Based on the first session context, the candidate response, and the evaluation rule set, determine the reward signal for the candidate response.

[0079] In one implementation, the process of determining the reward signal in this step can be further refined into three sub-steps: intent recognition, rule filtering, and comprehensive score calculation.

[0080] First, determine the user intent corresponding to the first session context. For example, the user intent could be a specific medical service scenario such as health consultation, medication guidance, appointment booking, interpretation of examination reports, or postoperative rehabilitation advice, or a non-medical scenario such as casual conversation. In actual execution, the first session context can be analyzed through a pre-trained intent classification model to accurately identify the user's core needs in the current round.

[0081] Secondly, based on the user intent, a subset of enabled evaluation rules is selected from the set of evaluation rules. In one embodiment, a set of rules associated with the intent can be found based on a pre-established "user intent-evaluation rule" mapping table. In another embodiment, a large model can be used to dynamically match the most relevant rules based on the user intent, thereby avoiding irrelevant rules from interfering with the scoring results.

[0082] Finally, the candidate response is calculated as a composite score under the enabled subset of evaluation rules, and this composite score is used as a reward signal for the candidate response.

[0083] In one embodiment, each evaluation rule in the evaluation rule set includes a pass criterion, a failure criterion, and a weight. For each candidate response, the candidate response, the first session context, and the enabled subset of evaluation rules are input into a third model. The third model determines whether the candidate response meets the pass or failure criteria of each evaluation rule: if it meets the pass criterion, a first score (e.g., 1 point) is assigned under the corresponding rule; if it meets the failure criterion, a second score (e.g., 0 points) is assigned, where the first score is higher than the second score. Subsequently, the scores under each rule are weighted and summed using the weights corresponding to each rule in the enabled subset of evaluation rules to obtain an initial comprehensive score.

[0084] Furthermore, to ensure medical safety, it is also possible to detect whether there are key evaluation rules (such as rules related to medical accuracy or medical safety risks, or rules with high risk levels) in the subset of enabled evaluation rules. If there are key evaluation rules and their corresponding scores are lower than the first preset threshold (such as 3 points), the smaller value between the initial comprehensive score (such as 7 points) and the preset low score (such as 0 points) will be used as the final comprehensive score, thereby achieving a veto for high-risk responses. If there are no violations of key evaluation rules, the initial comprehensive score obtained by weighted summation will be used as the final comprehensive score.

[0085] In another implementation, a lightweight approach that does not rely on large models can be used to calculate the reward signal: a subset of enabled evaluation rules is transformed into structured validation logic. Candidate responses are then automatically detected using regular expressions, keyword matching, or a rule engine to determine whether they meet the pass or fail criteria of each rule. Finally, a weighted sum is calculated according to preset weights to obtain a comprehensive score. This approach does not require calling large models, resulting in higher computational efficiency. It is suitable for deployment scenarios with stringent real-time requirements or limited computing resources.

[0086] As described above, this step enables the generation of reward signals that combine interactive experience orientation with medical safety constraints, based on multi-dimensional evaluation rules, multi-dimensional weighted scoring, and a safety fallback mechanism, providing reliable optimization targets for subsequent reinforcement learning training.

[0087] Step S250: Based on the candidate responses and reward signals, the medical large model is trained using a reinforcement learning algorithm.

[0088] In one embodiment, the number of candidate responses generated based on the first session context is multiple. Accordingly, this step may include: constructing a training group based on the multiple candidate responses and their corresponding reward signals; employing the Group Relative Policy Optimization (GRPO) algorithm to calculate the relative advantage of each candidate response within the training group relative to the group's average level; specifically, the GRPO algorithm does not rely on an independent value function network, but directly utilizes the reward signal distribution of multiple candidate responses generated under the same session context, and obtains the relative advantage value of each candidate response through normalization processing; subsequently, using the relative advantage as an optimization signal, combined with a pruning mechanism to limit the magnitude of policy updates, the parameters of the large-scale medical model are updated, thereby reducing training memory usage while improving the model's ability to learn preferences for high-quality responses.

[0089] As a result, refined reward signals generated based on dynamic evaluation rules can be used to drive targeted parameter iteration and optimization of large medical models. By adopting reinforcement learning algorithms such as GRPO, the model can be effectively guided to gradually learn generation strategies that take into account medical accuracy, safety and naturalness of interaction in complex medical dialogue scenarios, thereby achieving a closed-loop improvement in the quality of digital doctor services.

[0090] According to another embodiment, the improved solution also provides a monitoring and self-evolution mechanism for abnormal reward signal generation to address the difficulty of attributing low scores in existing technologies. Specifically, in existing digital doctor service systems, when users provide low scores online, it is often difficult to automatically and accurately pinpoint the root cause of the problem: is it a factual error due to a lack of medical knowledge in the model, a poor experience due to an interaction strategy failure, or a non-semantic problem caused by a voice link technology failure (such as ASR recognition error or silence interruption)? Due to the lack of structured attribution analysis methods, developers usually have to rely on manual sampling to troubleshoot problems, which is inefficient and has limited coverage.

[0091] Based on this, the proposed training method may further include: obtaining a labeled reward signal for the candidate response as a baseline signal; it should be noted that the labeled reward signal can be obtained through manual expert annotation, or it can be automatically annotated by calling a fifth model with performance superior to the model used to generate the reward signal, and the annotation content not only includes the comprehensive reward signal, but also covers the fine-grained scores under each scoring rule dimension. Subsequently, the difference between the reward signal and the baseline signal is calculated. If the difference is greater than a second preset threshold, or if the proportion of dimensions with excessively large differences in multiple scoring rule dimensions exceeds a preset proportion, it is determined that there is an anomaly in the current reward signal generation (also known as reward signal generation failure). The root cause may be the unreasonableness of the evaluation rule itself, or it may be the insufficient capability of the third model used to judge the rule.

[0092] For cases deemed abnormal, the fourth model is used to process the first session context, candidate responses, evaluation rule set, reward signal, benchmark signal, and actual and labeled scores for each dimension, generating a cause analysis result for the abnormality in the reward signal. Based on the cause analysis result, the system is adjusted or optimized accordingly: if the cause analysis result indicates that the problem mainly stems from overly broad trigger conditions of the evaluation rules, ambiguous definitions of pass / fail criteria, or unbalanced weight allocation, then the fourth model is used to generate rule modification suggestions, automatically updating the corresponding rule fields in the evaluation rule set to achieve self-correction of the evaluation system; if the cause analysis result indicates that the problem mainly stems from the third model's misunderstanding of specific complex contexts or logical reasoning errors, then the data containing the first session context, candidate responses, erroneous reward signal, benchmark signal, and cause analysis result is used to construct a negative sample error correction dataset, which is added to the historical session sample set or a dedicated fine-tuning dataset for incremental training or instruction fine-tuning of the third model or the medical big model, thereby improving the model's consistency in executing evaluation criteria.

[0093] Through the above mechanism, the system can automatically identify and fix "rule defects" or "model misjudgments" from online feedback, realizing the transformation from open-loop training to closed-loop self-evolution.

[0094] It should be noted that the medical big model, the first big model, the second big model, the third big model, the fourth big model, and the fifth big model mentioned in the text all refer to pre-trained language models with natural language processing capabilities, but they may differ in their specific functional positioning, parameter scale, and training objectives. Among them, the medical big model is the main model that ultimately provides services to users; the first big model is used to mine and generate evaluation rules from historical data, and usually needs strong inductive summarization and rule abstraction capabilities; the second big model is used to assist in screening high-scoring or low-scoring samples, and can have basic semantic understanding and classification capabilities; the third big model is used for online real-time evaluation of whether candidate responses meet the evaluation rules, requiring high reasoning accuracy and response speed; the fourth big model is used for in-depth analysis of the causes of abnormal reward signals, and usually needs stronger logical reasoning and causal analysis capabilities; the fifth big model serves as the annotation benchmark model, and its performance indicators (such as logical reasoning ability and medical knowledge reserve) should be superior to the third big model used to generate regular reward signals to ensure the authority and accuracy of the annotation results, and it is often used as a "teacher model" to guide the optimization of other models. In actual deployment, the above-mentioned major models can be different instances of the same general-purpose model, or they can be independent models that are specifically fine-tuned for different tasks. For example, in order to reduce computing power costs, the first and fourth major models can reuse the same high-performance base model, while the third major model can adopt a lightweight model to meet real-time requirements. The medical major model is usually deployed independently to ensure service stability and data security.

[0095] In summary, the improved solutions provided in the embodiments of this specification have significant beneficial effects:

[0096] First, by constructing a dynamic evaluation rule set based on historical conversation data mining and utilizing multi-dimensional reward signals to drive reinforcement learning, this approach effectively overcomes the shortcomings of traditional manual rules, such as being outdated, one-sided, and unable to cover long-tail problems. It achieves automated and dynamic updates to the evaluation system, enabling digital doctors to quickly adapt to dynamically changing user needs and significantly improving service quality and user satisfaction in complex scenarios such as real voice calls. In particular, addressing the unique noise interference, interruptions, and ASR errors in voice call scenarios, this solution significantly improves the robustness and service availability of digital doctors in non-ideal acoustic environments through specialized interactive experience rules and fault-tolerance mechanisms.

[0097] Secondly, by introducing a monitoring and self-evolution mechanism for abnormal reward signal generation, it can automatically locate the root cause of low-scoring samples as a lack of medical knowledge, interaction strategy errors, or technical malfunctions, and adjust the evaluation rules or optimize model parameters accordingly. This solves the problem of existing systems lacking effective attribution of low scores and rule self-evolution capabilities, establishes an efficient data-driven closed-loop optimization path, significantly reduces manual maintenance costs, and ensures the continuous iteration and safe control of digital doctor service capabilities.

[0098] Corresponding to the training method described above, this specification also discloses a training device. Figure 3 The medical large-scale model training device 300 for digital doctors includes the following functional modules:

[0099] The session sample set acquisition module 310 is configured to acquire a historical session sample set between the user and the digital doctor, the historical session sample set including high-scoring samples and low-scoring samples. The evaluation rule set generation module 320 is configured to process the historical session sample set using a first large-scale model to generate an evaluation rule set for the digital doctor service, the evaluation rule set including medical accuracy rules and interaction experience rules. The candidate response generation module 330 is configured to process the first session context between the first user and the digital doctor using a large-scale medical model implementing the digital doctor to obtain corresponding candidate responses. The reward signal determination module 340 is configured to determine the reward signal of the candidate response based on the first session context, the candidate response, and the evaluation rule set. The large-scale medical model training module 350 is configured to train the large-scale medical model using a reinforcement learning algorithm based on the candidate response and the reward signal.

[0100] In one embodiment, the historical conversation sample set is derived from historical voice calls, and the first conversation context includes transcribed text of the current round and historical multi-round voice dialogues. The interaction experience rules include one or more of the following: voice interruption repair rules, automatic speech recognition error-tolerant rules, and rules for naturalness in colloquial expression.

[0101] In one embodiment, the session sample set acquisition module 310 is specifically configured to: acquire a first historical session record and the corresponding first user evaluation feedback; if the first user evaluation feedback meets a preset high score condition, determine the first historical session record as a high-scoring sample; if the first user evaluation feedback meets a preset low score condition, determine the first historical session record as a low-scoring sample.

[0102] In one embodiment, the session sample set acquisition module 310 is specifically configured to: acquire a second historical session record and analyze the second historical session record using a second large model; if the analysis result indicates that the second historical session record meets the high score standard, determine it as a high-scoring sample; if the analysis result indicates that the second historical session record meets the low score standard, determine it as a low-scoring sample.

[0103] In one embodiment, the evaluation rule set generation module 320 is specifically configured to: perform clustering processing on the high-scoring samples and the low-scoring samples to obtain multiple sample clusters; input samples in the same sample cluster as a batch into the first large model to generate one or more evaluation rules corresponding to the cluster and classify them into the evaluation rule set.

[0104] In one embodiment, the reward signal determination module 340 is specifically configured to: determine the user intent corresponding to the first session context; based on the user intent, filter out an enabled subset of evaluation rules from the set of evaluation rules; calculate the comprehensive score of the candidate response under the enabled subset of evaluation rules, and use the comprehensive score as the reward signal of the candidate response.

[0105] Furthermore, in one specific embodiment, the evaluation rules in the evaluation rule set include pass criteria, failure criteria, and weights. The reward signal determination module 340 is configured to calculate a comprehensive score for the candidate response under the enabled subset of evaluation rules, specifically including:

[0106] By inputting the candidate response, the first session context, and the enabled subset of evaluation rules into the third model, it is determined whether the candidate response meets the pass or fail criteria of each evaluation rule. If it meets the pass criteria, a first score is assigned under the corresponding evaluation rule; if it meets the fail criteria, a second score is assigned under the corresponding evaluation rule; wherein the first score is higher than the second score. Using the weights corresponding to each evaluation rule in the enabled subset of evaluation rules, the scores under each evaluation rule are weighted and summed to determine the comprehensive score.

[0107] Furthermore, in one example, the weighted summation result is used as the initial comprehensive score, and the reward signal determination module 340 is further configured to: detect whether there is a key evaluation rule in the enabled evaluation rule subset, the key evaluation rule including medical accuracy-related rules or medical safety risk-related rules; if there is a key evaluation rule, and the score under the key evaluation rule is lower than a first preset threshold, then the smaller value between the initial comprehensive score and the preset low score is used as the final comprehensive score.

[0108] In one embodiment, the number of candidate responses is multiple; the medical large model training module 350 is specifically configured to: construct a training group based on multiple candidate responses and their corresponding reward signals; use a group relative strategy optimization algorithm to calculate the relative advantage of each candidate response in the training group relative to the average level of the group; and use the relative advantage as an optimization signal to update the parameters of the medical large model.

[0109] In one embodiment, the medical large model training device 300 further includes: a reward signal anomaly analysis module 360, configured to: acquire a labeled reward signal for a candidate response as a baseline signal; determine that the reward signal generation is abnormal when the difference between the reward signal and the baseline signal is greater than a second preset threshold; process the first session context, candidate response, evaluation rule set, reward signal, and baseline signal using a fourth large model to generate a cause analysis result for the reward signal generation anomaly; and an evaluation rule set adjustment module 370, configured to adjust the evaluation rule set based on the cause analysis result.

[0110] It should be noted that for a description of the above-mentioned functional modules or functional units, please refer to the relevant description of the process method in the foregoing embodiments.

[0111] In the embodiments of this specification, the Large Language Model (LLM) may also be simply referred to as the Large Model. A Large Language Model is a natural language processing model based on deep learning technology. It has a high number of parameters, typically ranging from billions to hundreds of billions or even higher, and possesses powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.). This architecture utilizes an attention mechanism to achieve global modeling of sequential data, efficiently handling long-distance dependencies and thus performing excellently in natural language tasks. Large Language Models can be used to perform various tasks such as text generation, language translation, text summarization, and question answering reasoning.

[0112] In some embodiments, large language models can also have the ability to understand and generate data from other modalities (such as visual, audio, etc.). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound.

[0113] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v18, published on March 8, 2026), and will not be repeated here.

[0114] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform... Figure 2 The method described.

[0115] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement... Figure 2 The method described.

[0116] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0117] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training a large-scale medical model for digital doctors, comprising: Obtain a historical session sample set between the user and the digital doctor, the historical session sample set including high-scoring samples and low-scoring samples; The historical session sample set is processed using the first major model to generate an evaluation rule set for digital doctor services, which includes medical accuracy rules and interactive experience rules. Using the medical big data model that implements the digital doctor, the first conversation context between the first user and the digital doctor is processed to obtain the corresponding candidate responses; Based on the first session context, the candidate responses, and the evaluation rule set, a reward signal for the candidate response is determined; Based on the candidate responses and reward signals, the large medical model is trained using a reinforcement learning algorithm.

2. The method according to claim 1, wherein, The interactive experience rules include one or more of the following: voice interruption repair rules, automatic speech recognition error tolerance rules, and naturalness rules for colloquial expressions.

3. The method according to claim 1, wherein, Obtain a sample set of historical conversations between users and digital doctors, including: Obtain the first historical session record and the corresponding first user review feedback; If the first user evaluation feedback meets the preset high score conditions, the first historical session record will be identified as a high score sample. If the first user evaluation feedback meets the preset low score condition, the first historical session record will be identified as a low score sample.

4. The method according to claim 1, wherein, Obtain a sample set of historical conversations between users and digital doctors, including: Obtain the second historical session record and analyze the second historical session record using the second major model; If the analysis results indicate that the second historical session record meets the high-score criteria, it is identified as a high-score sample; If the analysis results indicate that the second historical session record meets the low score criteria, it is identified as a low-scoring sample.

5. The method according to claim 1, wherein, The first major model is used to process the historical session sample set to generate an evaluation rule set for digital doctor services, including: Clustering is performed on the high-scoring samples and the low-scoring samples to obtain multiple sample clusters; Samples from the same sample cluster are input into the first large model as a batch, generating one or more evaluation rules corresponding to that cluster, which are then incorporated into the evaluation rule set.

6. The method according to claim 1, wherein, Based on the first session context, the candidate responses, and the evaluation rule set, the reward signal for the candidate response is determined, including: Determine the user intent corresponding to the first session context; Based on the user intent, a subset of enabled evaluation rules is selected from the set of evaluation rules; Calculate the comprehensive score of the candidate response under the enabled subset of evaluation rules, and use the comprehensive score as a reward signal for the candidate response.

7. The method according to claim 6, wherein, The evaluation rules in the set of evaluation rules include pass criteria, failure criteria, and weights; wherein, calculating the comprehensive score of the candidate response under the enabled subset of evaluation rules includes: By inputting the candidate response, the first session context, and the enabled subset of evaluation rules into the third model, it is determined whether the candidate response meets the pass or fail criteria of each evaluation rule. If the pass criteria are met, the first score will be assigned under the corresponding evaluation rules; if the failure criteria are met, the second score will be assigned under the corresponding evaluation rules. The scores under each evaluation rule are weighted and summed using the weights corresponding to each evaluation rule in the enabled evaluation rule subset to determine the comprehensive score.

8. The method according to claim 7, wherein, The result of the weighted summation is used as the initial comprehensive score, and the method further includes: The system detects whether there are key evaluation rules in the subset of enabled evaluation rules, including rules related to medical accuracy or rules related to medical safety risks. If a key evaluation rule exists, and the score under that key evaluation rule is lower than a first preset threshold, then the smaller of the initial comprehensive score and the preset low score will be used as the final comprehensive score.

9. The method according to claim 1, wherein, The number of candidate responses is multiple; wherein, based on the candidate responses and the reward signal, the large-scale medical model is trained using a reinforcement learning algorithm, including: A training group is constructed based on multiple candidate responses and their corresponding reward signals; A group-relative strategy optimization algorithm is used to calculate the relative advantage of each candidate response within the training group relative to the group average level. The parameters of the large medical model are updated using the relative advantage as an optimization signal.

10. The method according to any one of claims 1 to 9, wherein, The method further includes: Obtain the labeled reward signal for the candidate response as a reference signal; If the difference between the reward signal and the reference signal is greater than a second preset threshold, it is determined that the reward signal generation is abnormal. The fourth model is used to process the first session context, candidate responses, evaluation rule set, reward signal and benchmark signal to generate an analysis result of the cause of the anomaly in the reward signal; Based on the results of the cause analysis, the evaluation rule set is adjusted.

11. A large-scale medical model training device for digital doctors, comprising: The conversation sample set acquisition module is configured to acquire a historical conversation sample set between the user and the digital doctor, wherein the historical conversation sample set includes high-scoring samples and low-scoring samples. The evaluation rule set generation module is configured to process the historical session sample set using the first large model to generate an evaluation rule set for digital doctor services. The evaluation rule set includes medical accuracy rules and interactive experience rules. The candidate response generation module is configured to use the medical big model that implements the digital doctor to process the first conversation context between the first user and the digital doctor, and obtain the corresponding candidate responses. The reward signal determination module is configured to determine the reward signal of the candidate response based on the first session context, the candidate response, and the evaluation rule set. The medical large-scale model training module is configured to train the medical large-scale model using a reinforcement learning algorithm based on the candidate responses and reward signals.

12. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed in the computer, it causes the computer to perform the method according to any one of claims 1-10.

13. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-10.