An experience-based self-distillation medical agent learning training method and device

Abstract

The application discloses a medical intelligent agent learning training method and device based on experience self-distillation. The method forms a trajectory database through trajectory collection, and then distills trajectories in the trajectory database to generate strategy principles, thereby forming an experience database. When searching, the intelligent agent can directly search the strategy principles in the experience database to inject a reasoning process and realize online guidance of intelligent agent question and answer. According to a certain time period, the method periodically strengthens learning of the intelligent agent model according to historical trajectories in the experience database through a group relative strategy optimization method, thereby realizing autonomous evolution of the intelligent agent. On the other hand, the method periodically scores the experience data, and removes low-quality experience data, so that the experience data maintains dynamic high quality.

Description

Technical Field

[0001] This invention relates to artificial intelligence, and more particularly to reinforcement learning techniques using human feedback for large language models. Background Technology

[0002] Existing medical question-answering methods mainly include rule-based expert systems, direct generation using large language models, and traditional RAG enhancement methods. These methods have several shortcomings: rule systems are rigid and cannot flexibly handle complex and ever-changing clinical queries. While large language models possess strong generation capabilities (such as the GPT series), they are prone to generating illusions during use, potentially leading to misleading diagnoses or treatment recommendations in the medical field, posing high security risks. Traditional retrieval enhancement methods, although incorporating external data for observation, involve a passive and static retrieval process, lacking the system's ability to systematically learn and accumulate historical interaction data. This results in repeated errors for the same or similar questions over long-term operation, hindering dynamic optimization. Furthermore, medical scenarios involve a large amount of multimodal heterogeneous data (such as X-rays, pathology images, electronic medical records, PDF reports, etc.). Traditional systems struggle to effectively integrate this data and extract reusable experience. In environments with strict privacy requirements, continuous knowledge base updates also heavily rely on manual intervention, making it difficult to meet real-time evolution needs.

[0003] In the current integration of agent-related products with healthcare scenarios, such as clinical decision support tools, traditional technological approaches struggle to handle the real-time fusion and long-term evolution of multi-source heterogeneous data (EHR, images, literature). The fundamental limitation lies in the fact that existing systems generally treat each interaction as an isolated event, failing to autonomously extract reusable knowledge from past diagnostic trajectories. Specifically, they lack the ability to distill abstract guiding principles from the system's own successes and failures, failing to provide heuristic guidance for subsequent reasoning, resulting in inefficient decision-making paths and a tendency for the illusion problem to recur. Summary of the Invention

[0004] The problem this invention aims to solve is to achieve the evolution of medical intelligent agent models through reinforcement learning based on human feedback.

[0005] To solve the above problems, the present invention adopts the following solution: According to the present invention, a medical intelligent agent learning and training method based on empirical self-distillation includes the following steps: Step S1: When the user interacts with the agent, the evidence of the user's query and the agent's retrieval, the answer generated by the agent, and the user's feedback are collected to form trajectory data {(s(t),a(t))|t∈[1..T]}, and the trajectory data is stored in the trajectory database as historical trajectory data; where t is the time step number, s(t) represents the state at time step t, a(t) represents the action at time step t, and T is the total number of time steps; Step S2: Extract historical trajectory data from the trajectory database and perform empirical self-distillation to form empirical data, which is then stored in the empirical database; the empirical data includes strategy principles and a set of historical trajectory data. The strategy principles include principle type, text content, contextual information, indicator data, tag set, and medical entity set; the principle type includes at least guiding strategy principles and warning strategy principles; the guiding strategy principles indicate that the corresponding historical trajectory data is a successful case, and the warning strategy principles indicate that the corresponding historical trajectory data is a failed case; The empirical self-distillation includes the following steps: Step S21: Input the historical trajectory data into the large language model. The large language model analyzes the historical trajectory data as an expert reflector and extracts strategy principles. Step S22: Calculate the similarity between the extracted strategy principles and the strategy principles in the experience database, and find the experience data with the same principle type and the highest similarity as the extracted strategy principles as the most similar experience. Step S23: If the similarity between the strategy principle of the most similar experience and the extracted strategy principle is greater than a certain preset threshold, then the historical trajectory data is merged into the historical trajectory data set of the most similar experience and stored in the experience database; otherwise, new experience data is generated based on the extracted strategy principle and historical trajectory data and stored in the experience database. Step S3: Optimize the agent model using a group-relative strategy optimization method based on empirical data from the empirical database, including the following steps: Step S31: Calculate the overall reward for each historical trajectory in each empirical data set; Step S32: Cluster the strategy principles in each empirical data into K strategy groups; Step S33: Based on the historical trajectory corresponding to the strategy principle of each strategy group, standardize the overall reward of the historical trajectory into a relative advantage within the group. Step S34: Adjust the agent model parameters according to the relative advantages within the group.

[0006] The medical intelligent agent learning and training method based on experience self-distillation according to claim 1, characterized in that the method further includes: Step S4: When the agent reasones about the user query, it retrieves the experience database and finds the K most similar principle types that are guiding policy principles and injects them into its reasoning process.

[0007] Furthermore, according to the medical intelligent agent learning and training method based on experience self-distillation of the present invention, step S4 further includes recording the number of times the policy principle is retrieved, the number of times it is invoked and injected into its reasoning process, and the time of retrieval. The method also includes: Step S5: Calculate the experience score based on the number of times the recorded strategy principle is retrieved, the number of times it is invoked into its reasoning process, and the time it is retrieved. If the experience score is less than a pre-set threshold, delete the experience data corresponding to that strategy principle from the experience database. The score is calculated using the following formula: s(p)=(cj(p)+1)*exp(-ck*t(p)) / (cu(p)+2); where, p represents the strategy principle; s(p) represents the score of strategy principle p; cj(p) represents the number of times the strategy principle p is invoked into its reasoning process; cu(p) represents the number of times strategy principle p is retrieved; exp is an exponential function with respect to the natural constant e; ck is a preset parameter; t(p) is the time length from the time when strategy principle p was retrieved to the current time.

[0008] Furthermore, according to the medical agent learning and training method based on experience self-distillation of the present invention, step S31 further includes calculating the probability Pnew(s(t),a(t)) of the agent model generating action a(t) in the state s(t) of the historical trajectory, and calculating the probability ratio of the new and old strategies Pw(t)=Pnew(s(t),a(t)) / Pold(s(t),a(t)) with the probability Pold(s(t),a(t)) calculated in the previous training. When adjusting the agent model parameters based on the relative advantage within the group in step S34, calculate... loss=sum({gama(t)*min(Pw(t)*Av,clip(Pw(t),1-ex,1+ex)*Av))|t∈[1..T]}); Then, the agent's model is optimized based on the loss; where, `sum` represents the sum of the values ​​in the set within the parentheses; gama(t) represents the time decay factor; min represents the minimum value; Av indicates relative advantage within the group; clip represents the clipping function. If Pw(t) < 1 - ex, then the value is 1 - ex; if Pw(t) > 1 + ex, then the value is 1 + ex; otherwise, the value is Pw(t). ex is the preset clipping threshold.

[0009] According to the present invention, a medical intelligent agent learning and training device based on empirical self-distillation includes the following modules: Module M1: When the user interacts with the agent, it collects evidence from user queries and agent retrieval, answers generated by the agent, and user feedback to form trajectory data {(s(t),a(t))|t∈[1..T]}, and stores the trajectory data as historical trajectory data in the trajectory database; where t is the time step number, s(t) represents the state at time step t, a(t) represents the action at time step t, and T is the total number of time steps; Module M2: Extracts historical trajectory data from the trajectory database, performs empirical self-distillation, and stores the resulting empirical data in the empirical database; the empirical data includes strategy principles and a set of historical trajectory data. The strategy principles include principle type, text content, contextual information, indicator data, tag set, and medical entity set; the principle type includes at least guiding strategy principles and warning strategy principles; the guiding strategy principles indicate that the corresponding historical trajectory data is a successful case, and the warning strategy principles indicate that the corresponding historical trajectory data is a failed case; The empirical self-distillation includes the following modules: Module M21: Input historical trajectory data into the large language model. The large language model analyzes the historical trajectory data as an expert reflector and extracts strategy principles. Module M22: Calculate the similarity between the extracted strategy principles and the strategy principles in the experience database, and find the experience data with the same principle type and the highest similarity as the extracted strategy principles as the most similar experience. Module M23: If the similarity between the strategy principle of the most similar experience and the extracted strategy principle is greater than a certain preset threshold, then the historical trajectory data is merged into the historical trajectory data set of the most similar experience and stored in the experience database; otherwise, new experience data is generated based on the extracted strategy principle and historical trajectory data and stored in the experience database. Module M3: A device for optimizing the agent model using group-based relative strategies based on empirical data in an empirical database, including the following modules: Module M31 calculates the overall reward for each historical trajectory in each set of empirical data. Module M32 clusters the policy principles in each empirical data into K policy groups; Module M33 standardizes the overall reward of the historical trajectory into a relative advantage within the group based on the historical trajectory corresponding to the strategy principles of each strategy group. Module M34 adjusts the agent model parameters based on the relative advantages within the group.

[0010] The medical intelligent agent learning and training device based on experience self-distillation according to claim 1, characterized in that the device further comprises: Module M4: When the agent reasones about a user query, it retrieves the experience database and finds the K most similar policy principles that are guiding policy principles, and injects them into its reasoning process.

[0011] Furthermore, in the medical intelligent agent learning and training device based on experience self-distillation according to the present invention, the module M4 further includes recording the number of times the policy principle is retrieved, the number of times it is invoked and injected into its reasoning process, and the time of retrieval. The device also includes: Module M5: Calculates an experience score based on the number of times the recorded strategy principle is retrieved, the number of times it is invoked into its reasoning process, and the time it is retrieved. If the experience score is less than a pre-set threshold, the experience data corresponding to that strategy principle is deleted from the experience database. The score is calculated using the following formula: s(p)=(cj(p)+1)*exp(-ck*t(p)) / (cu(p)+2); where, p represents the strategy principle; s(p) represents the score of strategy principle p; cj(p) represents the number of times the strategy principle p is invoked into its reasoning process; cu(p) represents the number of times strategy principle p is retrieved; exp is an exponential function with respect to the natural constant e; ck is a preset parameter; t(p) is the time length from the time when strategy principle p was retrieved to the current time.

[0012] Furthermore, in the medical intelligent agent learning and training device based on experience self-distillation according to the present invention, module M31 further includes calculating the probability Pnew(s(t),a(t)) of the intelligent agent model generating action a(t) in the state s(t) of the historical trajectory, and calculating the probability ratio of the new and old strategies Pw(t)=Pnew(s(t),a(t)) / Pold(s(t),a(t)) with the probability Pold(s(t),a(t)) calculated during the previous training. When module M34 adjusts the agent model parameters based on the relative advantage within the group, it calculates... loss=sum({gama(t)*min(Pw(t)*Av,clip(Pw(t),1-ex,1+ex)*Av))|t∈[1..T]}); Then, the agent's model is optimized based on the loss; where, `sum` represents the sum of the values ​​in the set within the parentheses; gama(t) represents the time decay factor; min represents the minimum value; Av indicates relative advantage within the group; clip represents the clipping function. If Pw(t) < 1 - ex, then the value is 1 - ex; if Pw(t) > 1 + ex, then the value is 1 + ex; otherwise, the value is Pw(t). ex is the preset clipping threshold.

[0013] The technical effects of this invention are as follows: This invention achieves online guidance for intelligent agent question answering by collecting trajectory data, distilling experience data, and conducting dynamic quality assessment, and by using the experience database as the retrieval object. At the same time, it achieves autonomous optimization of the medical intelligent question answering system by evaluating historical trajectory data from the experience database. Attached Figure Description

[0014] Figure 1 This is a flowchart and data flow diagram of an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings.

[0017] Figure 2 An example of an electronic device is provided, which is a general-purpose computer device in the von Neumann architecture, comprising at least a processor 101 and a memory 102 connected together. The memory 102 is used to store computer program instruction sets and data. The processor 101 implements the experience-based self-distillation medical intelligent agent learning and training method described in this invention by loading and executing the computer program instruction sets stored in the memory 102. The memory 102 is typically a sustainable storage device, including but not limited to disks, magnetic tapes, solid-state drives, etc. It should be noted that the processor 101 here is a component capable of executing computer program instructions in a broad sense, and is not limited to general-purpose processors, or even a single component. For example, in this embodiment, the processor 101 is a combination of a CPU and a GPU, that is, the functions implemented by the processor 101 are jointly implemented by the CPU and the GPU.

[0018] Reference Figure 1The medical intelligent agent learning and training method based on experience self-distillation of the present invention includes a trajectory acquisition step, a trajectory distillation step, a strategy principle guidance step, a scoring and inferior elimination step, and a group relative strategy optimization step.

[0019] The trajectory acquisition step, namely step S1 mentioned above, involves collecting evidence from user queries and agent retrieval, answers generated by the agent, and user feedback to form trajectory data when the user interacts with the agent. This trajectory data is then stored in the trajectory database as historical trajectory data. In this embodiment, trajectory data is represented by a triplet of state, action, and reward, specifically (s(t), a(t), r(t)); where t is the time step number, s(t) represents the state at time step t, a(t) represents the action at time step t, and r(t) represents the reward for action a(t). When the total number of time steps for the entire trajectory is T, the trajectory data can be represented as {(s(t), a(t), r(t))|t∈[1..T]}.

[0020] The trajectory distillation step involves extracting historical trajectory data from the trajectory database and performing empirical self-distillation to generate empirical data, which is then stored in the empirical database. The empirical data includes policy principles and a set of historical trajectory data. Policy principles include principle types, text content, contextual information, indicator data, a set of labels, and a set of medical entities. Principle types include at least guiding policy principles and cautionary policy principles. Guiding policy principles indicate that the corresponding historical trajectory data represents successful cases, while cautionary policy principles indicate that the corresponding historical trajectory data represents unsuccessful cases.

[0021] Empirical autodistillation includes the following steps: Step S21: Input the historical trajectory data into the large language model. The large language model analyzes the historical trajectory data as an expert reflector and extracts strategy principles.

[0022] Step S22: Calculate the similarity between the extracted strategy principles and the strategy principles in the experience database, and find the experience data with the same principle type and the highest similarity as the extracted strategy principles as the most similar experience.

[0023] Step S23: If the similarity between the strategy principle of the most similar experience and the extracted strategy principle is greater than a certain preset threshold, then the historical trajectory data is merged into the historical trajectory data set of the most similar experience and stored in the experience database; otherwise, new experience data is generated based on the extracted strategy principle and historical trajectory data and stored in the experience database.

[0024] Step S22, similarity calculation, is obtained by analyzing text vectors. In this embodiment, the strategy principle is calculated according to the following formula: simP=a1*sim1+a2*sim2+a3*sim3+a4*sim4+a5*sim5; among them, simP represents the similarity between two policy principles; sim1, sim2, sim3, sim4, and sim5 represent the similarity of text content, contextual information, indicator data, label set, and medical entity set in the two policy principles, respectively. a1, a2, a3, a4, and a5 are pre-defined weighting coefficients.

[0025] In the trajectory distillation step, historical trajectory data is deleted from the trajectory database after self-distillation. Therefore, the trajectory database is cleared after each round of trajectory distillation. The trajectory distillation step is performed at pre-set times, such as at certain time intervals, when the amount of historical trajectory data in the trajectory database exceeds a certain limit, or at a time scheduled by the administrator.

[0026] The strategy principle guidance step, namely step S4 mentioned above, involves retrieving the experience database and identifying the K most similar principle types as guiding strategy principles, which are then injected into the reasoning process. In this embodiment, for ease of scoring, the number of times the strategy principle is retrieved, the number of times it is invoked into the reasoning process, and the time of retrieval are recorded during retrieval and optimization.

[0027] The scoring and de-optimization step, namely step S5 mentioned above, calculates an experience score based on the number of times the recorded strategy principle is retrieved, the number of times it is invoked into its reasoning process, and the time it is retrieved. If the experience score is less than a pre-set threshold, the experience data corresponding to that strategy principle is deleted from the experience database. The score is calculated using the following formula: s(p)=(cj(p)+1)*exp(-ck*t(p)) / (cu(p)+2); where, p represents the strategy principle; s(p) represents the score of strategy principle p; cj(p) represents the number of times the strategy principle p is invoked into its reasoning process; cu(p) represents the number of times strategy principle p is retrieved; exp is an exponential function with respect to the natural constant e; ck is a preset parameter; t(p) is the time length from the time when strategy principle p was retrieved to the current time.

[0028] The scoring and elimination process is carried out at a pre-set time, preferably at a certain time cycle.

[0029] The group-relative policy optimization step, also known as step S3 above, optimizes the agent model using the group-relative policy optimization method based on empirical data in the empirical database. This includes the following steps: Step S31: Calculate the overall reward for each historical trajectory in each empirical data set; Step S32: Cluster the strategy principles in each empirical data into K strategy groups; Step S33: Based on the historical trajectory corresponding to the strategy principle of each strategy group, standardize the overall reward of the historical trajectory into a relative advantage within the group. Step S34: Adjust the agent model parameters according to the relative advantages within the group.

[0030] In this embodiment, step S31 further includes calculating the probability Pnew(s(t), a(t)) of the agent model generating action a(t) in the state s(t) of the historical trajectory. Specifically, the probability Pnew(s(t), a(t)) is calculated by inputting the state s(t) into the agent model, sampling the original score vector output from the last layer of the agent model, performing a Softmax function regression operation on the original score vector, and then extracting the value corresponding to a(t). The overall reward of the historical trajectory in step S31 is calculated using the following formula: Rtotal(path) = w1 * Rformat(path) + w2 * Rprocess(path); where, path represents the historical trajectory; Rtotal(path) represents the total reward of the historical trajectory p; Rformat(path) represents the inference format integrity reward for the historical trajectory p; Rprocess(path) represents the process correctness reward for the historical trajectory p; w1 and w2 are weighting coefficients; where... Rformat(path)=sum({valid(qu(path,t),an(path,t))|t∈[1..T(path)]}) / T(path); Rprocess(path)=sum({clipupper(r(path,t))|t∈[1..T(path)]}) / sum({abs(r(path,t))|t∈[1..T(path)]}); where, `sum` represents the sum of the values ​​in the set within the parentheses; The `valid(qu(path,t),an(path,t))` method checks whether the inference labels appear in pairs. If they appear in pairs, the value is 1; otherwise, it is 0. The `clipupper(r(path,t))` function means that if `r(path,t))` is greater than 0, it takes the value `r(path,t)`; otherwise, it takes the value 0. abs(r(path,t)) means taking the absolute value of r(path,t); r(path,t) represents the action reward at the t-th time step of the historical trajectory path; T(path) represents the total time steps of the historical trajectory path.

[0031] In step S32, since each strategy principle of the experience data in this embodiment corresponds to a set of historical trajectories, in another optional implementation, the set of historical trajectories corresponding to each strategy principle is treated as a cluster group.

[0032] In step S33, the overall reward of the historical trajectory is standardized into relative advantage within the group, specifically using the following calculation formula: Av(path)=(Rtotal(path)-groupavg_Reword) / (sd_reword+emin); where, Av(path) represents the group relative advantage of the historical trajectory path; Rtotal(path) represents the total reward of the historical trajectory p; groupavg_Reword represents the average overall reward of each historical trajectory p within the group; sd_reword represents the standard deviation of the overall reward for each historical trajectory p within the group; emin is used to avoid small quantities with a denominator of 0.

[0033] Step S34 involves calculating the loss and then optimizing the agent model using an optimizer based on the loss.

[0034] In an alternative implementation, the loss is calculated using the following formula: loss=﹣avg({log(Pnew(s(t),a(t)))*Av|t∈[1..T]}); where, loss refers to the amount of money lost. t represents the time step; T represents the total number of time steps in the historical trajectory; Pnew(s(t),a(t)) represents the probability of generating action a(t) in state s(t); Av represents relative advantage within the group; log represents the logarithmic function.

[0035] In another optional implementation, in step S31, the probability Pnew(s(t),a(t)) is saved, and the probability ratio of the new and old strategies Pw(t) = Pnew(s(t),a(t)) / Pold(s(t),a(t)) is calculated with the probability Pold(s(t),a(t)) calculated during the previous training. The loss is calculated using the following formula: loss=sum({gama(t)*min(Pw(t)*Av,clip(Pw(t),1-ex,1+ex)*Av))|t∈[1..T]}); where, t represents the time step; T is the total number of time steps in the historical trajectory; `sum` represents the sum of the values ​​in the set within the parentheses; gama(t) represents the time decay factor; min represents the minimum value; Av indicates relative advantage within the group; clip represents the clipping function. If Pw(t) < 1 - ex, then the value is 1 - ex; if Pw(t) > 1 + ex, then the value is 1 + ex; otherwise, the value is Pw(t). ex is the preset clipping threshold.

[0036] The group-based strategy optimization steps are executed at pre-set times, usually on a fixed time period.

[0037] Furthermore, it should be noted that the medical intelligent agent learning and training device based on experience self-distillation mentioned above in this invention is a virtual device implemented by executing computer program instructions. The modules it contains correspond one-to-one with the steps in the medical intelligent agent learning and training method based on experience self-distillation, which need not be elaborated upon.

Claims

1. A medical intelligent agent learning and training method based on experience self-distillation, characterized in that, Includes the following steps: Step S1: When the user interacts with the agent, collect evidence from the user's query and the agent's retrieval, the answer generated by the agent, and the user's feedback to form trajectory data {(s(t),a(t))|t∈[1..T]}, and store the trajectory data as historical trajectory data in the trajectory database; where t is the time step number, s(t) represents the state at time step t, a(t) represents the action at time step t, and T is the total number of time steps; Step S2: Extract historical trajectory data from the trajectory database and perform empirical self-distillation to form empirical data, which is then stored in the empirical database; the empirical data includes strategy principles and a set of historical trajectory data. The strategy principles include principle type, text content, contextual information, indicator data, tag set, and medical entity set; the principle type includes at least guiding strategy principles and warning strategy principles; the guiding strategy principles indicate that the corresponding historical trajectory data is a successful case, and the warning strategy principles indicate that the corresponding historical trajectory data is a failed case; The empirical self-distillation includes the following steps: Step S21: Input the historical trajectory data into the large language model. The large language model analyzes the historical trajectory data as an expert reflector and extracts strategy principles. Step S22: Calculate the similarity between the extracted strategy principles and the strategy principles in the experience database, and find the experience data with the same principle type and the highest similarity as the extracted strategy principles as the most similar experience. Step S23: If the similarity between the strategy principle of the most similar experience and the extracted strategy principle is greater than a certain preset threshold, then the historical trajectory data is merged into the historical trajectory data set of the most similar experience and stored in the experience database; otherwise, new experience data is generated based on the extracted strategy principle and historical trajectory data and stored in the experience database. Step S3: Optimize the agent model using a group-relative strategy optimization method based on empirical data from the empirical database, including the following steps: Step S31: Calculate the overall reward for each historical trajectory in each empirical data set; Step S32: Cluster the strategy principles in each empirical data into K strategy groups; Step S33: Based on the historical trajectory corresponding to the strategy principle of each strategy group, standardize the overall reward of the historical trajectory into a relative advantage within the group. Step S34: Adjust the agent model parameters according to the relative advantages within the group.

2. The medical intelligent agent learning and training method based on empirical self-distillation according to claim 1, characterized in that, The method also includes: Step S4: When the agent reasones about the user query, it retrieves the experience database and finds the K most similar principle types that are guiding policy principles and injects them into its reasoning process.

3. The medical intelligent agent learning and training method based on empirical self-distillation according to claim 2, characterized in that, Step S4 also includes recording the number of times the strategy principle is retrieved, the number of times it is invoked into its reasoning process, and the time it is retrieved. The method also includes: Step S5: Calculate the experience score based on the number of times the recorded strategy principle is retrieved, the number of times it is invoked into its reasoning process, and the time it is retrieved. If the experience score is less than a pre-set threshold, delete the experience data corresponding to that strategy principle from the experience database. The score is calculated using the following formula: s(p)=(cj(p)+1)*exp(-ck*t(p)) / (cu(p)+2); where, p represents the strategy principle; s(p) represents the score of strategy principle p; cj(p) represents the number of times the strategy principle p is invoked into its reasoning process; cu(p) represents the number of times strategy principle p is retrieved; exp is an exponential function with respect to the natural constant e; ck is a preset parameter; t(p) is the time length from the time when strategy principle p was retrieved to the current time.

4. The medical intelligent agent learning and training method based on empirical self-distillation according to claim 1, 2, or 3, characterized in that, Step S31 also includes calculating the probability Pnew(s(t),a(t)) of the agent model generating action a(t) in the state s(t) of the historical trajectory, and calculating the probability ratio of the new and old policies Pw(t)=Pnew(s(t),a(t)) / Pold(s(t),a(t)) with the probability Pold(s(t),a(t)) calculated during the previous training. When adjusting the agent model parameters based on the relative advantage within the group in step S34, calculate... loss=sum({gama(t)*min(Pw(t)*Av,clip(Pw(t),1-ex,1+ex)*Av))|t∈[1..T]}); Then, the agent's model is optimized based on the loss; where, `sum` represents the sum of the values ​​in the set within the parentheses; gama(t) represents the time decay factor; min represents the minimum value; Av indicates relative advantage within the group; clip represents the clipping function. If Pw(t) < 1 - ex, then the value is 1 - ex; if Pw(t) > 1 + ex, then the value is 1 + ex; otherwise, the value is Pw(t). ex is the preset clipping threshold.

5. A medical intelligent agent learning and training device based on experience self-distillation, characterized in that, Includes the following modules: Module M1: When the user interacts with the agent, it collects evidence from user queries and agent retrieval, answers generated by the agent, and user feedback to form trajectory data {(s(t),a(t))|t∈[1..T]}, and stores the trajectory data as historical trajectory data in the trajectory database; where t is the time step number, s(t) represents the state at time step t, a(t) represents the action at time step t, and T is the total number of time steps; Module M2: Extracts historical trajectory data from the trajectory database, performs empirical self-distillation, and stores the resulting empirical data in the empirical database; the empirical data includes strategy principles and a set of historical trajectory data. The strategy principles include principle type, text content, contextual information, indicator data, tag set, and medical entity set; the principle type includes at least guiding strategy principles and warning strategy principles; the guiding strategy principles indicate that the corresponding historical trajectory data is a successful case, and the warning strategy principles indicate that the corresponding historical trajectory data is a failed case; The empirical self-distillation includes the following modules: Module M21: Input historical trajectory data into the large language model. The large language model analyzes the historical trajectory data as an expert reflector and extracts strategy principles. Module M22: Calculate the similarity between the extracted strategy principles and the strategy principles in the experience database, and find the experience data with the same principle type and the highest similarity as the extracted strategy principles as the most similar experience. Module M23: If the similarity between the strategy principle of the most similar experience and the extracted strategy principle is greater than a certain preset threshold, then the historical trajectory data is merged into the historical trajectory data set of the most similar experience and stored in the experience database; otherwise, new experience data is generated based on the extracted strategy principle and historical trajectory data and stored in the experience database. Module M3: A device for optimizing the agent model using group-based relative strategies based on empirical data in an empirical database, including the following modules: Module M31 calculates the overall reward for each historical trajectory in each set of empirical data. Module M32 clusters the policy principles in each empirical data into K policy groups; Module M33 standardizes the overall reward of the historical trajectory into a relative advantage within the group based on the historical trajectory corresponding to the strategy principles of each strategy group. Module M34 adjusts the agent model parameters based on the relative advantages within the group.

6. The medical intelligent agent learning and training device based on experience self-distillation according to claim 5, characterized in that, The device also includes: Module M4: When the agent reasones about a user query, it retrieves the experience database and finds the K most similar policy principles that are guiding policy principles, and injects them into its reasoning process.

7. The medical intelligent agent learning and training device based on experience self-distillation according to claim 6, characterized in that, The module M4 also includes recording the number of times the policy principle is retrieved, the number of times it is invoked into its reasoning process, and the time it is retrieved. The device also includes: Module M5: Calculates an experience score based on the number of times the recorded strategy principle is retrieved, the number of times it is invoked into its reasoning process, and the time it is retrieved. If the experience score is less than a pre-set threshold, the experience data corresponding to that strategy principle is deleted from the experience database. The score is calculated using the following formula: s(p)=(cj(p)+1)*exp(-ck*t(p)) / (cu(p)+2); where, p represents the strategy principle; s(p) represents the score of strategy principle p; cj(p) represents the number of times the strategy principle p is invoked into its reasoning process; cu(p) represents the number of times strategy principle p is retrieved; exp is an exponential function with respect to the natural constant e; ck is a preset parameter; t(p) is the time length from the time when strategy principle p was retrieved to the current time.

8. The medical intelligent agent learning and training device based on empirical self-distillation according to claim 5, 6, or 7, characterized in that, Module M31 also includes calculating the probability Pnew(s(t),a(t)) of the agent model generating action a(t) in the state s(t) of the historical trajectory, and calculating the probability ratio of the new and old policies Pw(t)=Pnew(s(t),a(t)) / Pold(s(t),a(t)) with the probability Pold(s(t),a(t)) calculated during the previous training. When module M34 adjusts the agent model parameters based on the relative advantage within the group, it calculates... loss=sum({gama(t)*min(Pw(t)*Av,clip(Pw(t),1-ex,1+ex)*Av))|t∈[1..T]}); Then, the agent's model is optimized based on the loss; where, `sum` represents the sum of the values ​​in the set within the parentheses; gama(t) represents the time decay factor; min represents the minimum value; Av indicates relative advantage within the group; clip represents the clipping function. If Pw(t) < 1 - ex, then the value is 1 - ex; if Pw(t) > 1 + ex, then the value is 1 + ex; otherwise, the value is Pw(t). ex is the preset clipping threshold.

Images