Medical large model evaluation method, device, apparatus, storage medium and program product

Abstract

Embodiments of the present specification provide a medical large model evaluation method, device, apparatus, storage medium and program product. Image data generated by a plurality of users in the process of interacting with a medical large model is obtained, and based on original dialogue data in the image data, target dialogue data covering multiple medical scenarios, multiple user subjects and cross-time dimensions is constructed to carry complex personalized memories of the model; based on the target dialogue data and the medical health archives in the image data, evaluation data in three dimensions of memory construction, memory recall and memory consumption are generated by an intelligent agent, the complex personalized memories are transmitted to the evaluation data, and the full-process decoupling evaluation of the answer generation link of the medical large model is realized through the evaluation data. If there is a deviation in the model output, it can be accurately located whether the deviation originates from the memory construction, recall or consumption link, thereby capturing the actual performance of the model when processing complex personalized memories, and providing accurate and traceable quantitative basis for model optimization.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method, equipment, device, storage medium and program product for evaluating a large medical model. Background Technology

[0002] In recent years, medical service platforms have emerged for scenarios such as intelligent consultation and health follow-up. These platforms interact with users based on a large-scale medical model, providing services such as medical consultation and diagnosis. The large-scale medical model is responsible for dynamically maintaining the patient's medical context information during the interaction, including personal information, medical history, symptom evolution, and medication records. It accurately recalls relevant information when the user asks a question, and then generates answers based on this information, providing professional and natural dialogue responses.

[0003] As smart healthcare becomes more prevalent, users may upload not only static medical records but also linked health monitoring devices. The real-time data generated by these devices, such as heart rate and blood oxygen levels, will be incorporated into the memory context. Simultaneously, the management entity expands from a single patient to multiple family members, resulting in the memory context involving multiple differentiated entities. Furthermore, the long-term dependencies generated by multi-turn interactions contribute to the temporal complexity of the memory context. This complexity not only poses a significant challenge to the personalized service capabilities of large-scale medical models but also renders traditional evaluation methods based on model outputs ineffective, as they struggle to capture the model's performance when processing complex, personalized memories.

[0004] Therefore, how to build an evaluation mechanism that can comprehensively cover complex scenarios in order to quantitatively evaluate the performance of medical big data models in processing complex personalized memories has become an urgent technical problem to be solved. Summary of the Invention

[0005] This specification provides an evaluation method, device, apparatus, storage medium, and program product for a large medical model, used to evaluate the performance of the large medical model in processing complex personalized memories and improve the evaluation accuracy.

[0006] This specification provides an evaluation method for a large-scale medical model, comprising: acquiring multiple profile data generated by multiple target users during their interaction with the large-scale medical model, the profile data including medical and health records and original dialogue data; constructing multiple target dialogue data based on the original dialogue data in the multiple profile data, the target dialogue data referring to multi-turn dialogue data covering multiple medical scenarios and multiple user subjects, and spanning time dimensions; generating multiple evaluation samples using multiple intelligent agents based on the medical and health records and multiple target dialogue data in the multiple profile data, the evaluation samples including: memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data; and quantitatively evaluating the memory construction capability, memory recall capability, and memory consumption capability of the large-scale medical model based on the memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data in the multiple evaluation samples, to obtain evaluation results.

[0007] This specification also provides a method for generating an evaluation dataset for a large medical model, comprising: acquiring multiple profile data generated by multiple target users during their interaction with the large medical model, the profile data including medical and health records and original dialogue data; constructing multiple target dialogue data based on the original dialogue data in the multiple profile data, the target dialogue data referring to multi-turn dialogue data covering multiple medical scenarios and multiple user subjects, and spanning time dimensions; and generating multiple evaluation samples using multiple intelligent agents based on the medical and health records in the multiple profile data and the multiple target dialogue data, the evaluation samples including: memory-constructed evaluation data, memory-recalled evaluation data, and memory-consumption evaluation data.

[0008] This specification also provides an evaluation device for a large-scale medical model, comprising: an acquisition module, a construction module, a generation module, and an evaluation module; the acquisition module is used to acquire multiple profile data generated by multiple target users during their interaction with the large-scale medical model, the profile data including medical and health records and original dialogue data; the construction module is used to construct multiple target dialogue data based on the original dialogue data in the multiple profile data, the target dialogue data referring to multi-turn dialogue data covering multiple medical scenarios and multiple user subjects, and spanning time dimensions; the generation module is used to generate multiple evaluation samples using multiple intelligent agents based on the medical and health records and multiple target dialogue data in the multiple profile data, the evaluation samples including: memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data; the evaluation module is used to quantitatively evaluate the memory construction capability, memory recall capability, and memory consumption capability of the large-scale medical model based on the memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data in the multiple evaluation samples, respectively, to obtain evaluation results.

[0009] This specification also provides an electronic device, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute one or more computer instructions to perform the steps in the method provided in this specification.

[0010] This specification also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps of the method provided in this specification.

[0011] This specification also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, can implement the steps of the method provided in this specification.

[0012] In this embodiment, user profile data generated during interactions with a large-scale medical model is acquired. Based on the original dialogue data within the profile data, target dialogue data covering multiple medical scenarios, multiple user subjects, and cross-time dimensions is constructed, enabling the target dialogue data to carry the model's complex personalized memories. Based on the medical and health records in the target dialogue data and profile data, multiple agents generate evaluation samples across three dimensions: memory construction, memory retrieval, and memory consumption, effectively transmitting complex personalized memories to these evaluation samples. The large-scale medical model is then evaluated using these evaluation samples, achieving a fully decoupled evaluation of the answer generation process. When deviations exist in the large-scale medical model's output, the source of these deviations can be precisely pinpointed within the memory construction, memory retrieval, and memory consumption stages. This allows for the capture of the large-scale medical model's actual performance in processing complex personalized memories, providing precise and traceable quantitative evidence for model optimization. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating the evaluation method for a large medical model provided as an exemplary embodiment of this specification.

[0014] Figure 2 This is a schematic diagram of the process for generating target dialogue data corresponding to the original dialogue data, provided in one embodiment of this specification.

[0015] Figure 3 This is a schematic diagram illustrating the generation of memory construction evaluation data provided for an exemplary embodiment of this specification.

[0016] Figure 4 This is a schematic diagram illustrating the evaluation problem in generating memory consumption evaluation data, provided as an exemplary embodiment of this specification.

[0017] Figure 5 This is a schematic diagram showing the distribution of evaluation questions and corresponding question tags provided for an exemplary embodiment of this specification.

[0018] Figure 6 This is a schematic diagram illustrating the process of generating recall assessment data and recall consumption assessment data for another exemplary embodiment of this specification.

[0019] Figure 7 This is a schematic diagram illustrating the evaluation of the memory construction ability of a large medical model, provided as another exemplary embodiment of this specification.

[0020] Figure 8 This is a schematic diagram illustrating the evaluation of the memory recall capability of a large medical model, provided as another exemplary embodiment of this specification.

[0021] Figure 9 This is a schematic diagram illustrating the evaluation of the memory consumption capacity of a large medical model, provided as another exemplary embodiment of this specification.

[0022] Figure 10 This is a schematic diagram of the structure of a medical large-scale model evaluation device provided as an exemplary embodiment of this specification.

[0023] Figure 11 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0025] It should be noted that, in the cases involving user information in this application's embodiments, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. The various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0026] In an assessment scenario, one approach for a general large-scale model is as follows: Based on a pre-defined patient role and a medical consultation event relationship diagram, a large-scale dialogue model is used to generate multi-turn dialogue content that matches the patient role. Sample questions and their labeled answers are then constructed based on the multi-turn dialogue content. The sample questions and multi-turn dialogue content are fed into the general large-scale model, which generates the target answer. By comparing the target answer with the standard answer, it is determined whether the general large-scale model has "answered correctly," thus obtaining the assessment result.

[0027] Directly applying the above assessment scheme to a large-scale medical model yields coarse-grained results, which are insufficient to support in-depth analysis and optimization of the model's memory capabilities. For example, when predicted answers deviate, the causes may involve various stages in the answer generation process within the large-scale medical model, making it difficult to pinpoint the exact source of the error and thus hindering the assessment results' ability to guide the optimization of the model.

[0028] In the embodiments of this specification, profile data generated by multiple users during their interaction with the medical big data model is acquired. Based on the original dialogue data in the profile data, target dialogue data covering multiple medical scenarios, multiple user subjects, and cross-time dimensions is constructed, enabling the target dialogue data to carry the model's complex personalized memories. Based on the medical and health records in the target dialogue data and profile data, multiple intelligent agents are used to generate evaluation samples in three dimensions: memory construction, memory retrieval, and memory consumption, effectively transmitting complex personalized memories to the evaluation samples. The medical big data model is evaluated through the above evaluation samples, achieving a fully decoupled evaluation of the answer generation chain within the medical big data model. When there is a deviation in the output of the medical big data model, the deviation can be accurately located at which stage of memory construction, memory retrieval, and memory consumption, thereby capturing the actual performance of the medical big data model in processing complex personalized memories and providing accurate and traceable quantitative basis for model optimization.

[0029] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0030] Figure 1 This is a flowchart illustrating an exemplary embodiment of the evaluation method for a large medical model provided in this specification, such as... Figure 1 The method shown includes: 101. Obtain multiple profile data generated by multiple target users during their interaction with the medical big data model. The profile data includes medical and health records and original dialogue data.

[0031] 102. Based on the original dialogue data in multiple profile data, construct multiple target dialogue data. Target dialogue data refers to multi-turn dialogue data that covers multiple medical scenarios and multiple user subjects and spans time dimensions.

[0032] 103. Based on medical and health records from multiple profile data and dialogue data from multiple targets, multiple intelligent agents are used to generate multiple evaluation samples, including: memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data.

[0033] 104. Based on memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data from multiple evaluation samples, the memory construction ability, memory recall ability, and memory consumption ability of the medical big model are quantitatively evaluated to obtain evaluation results.

[0034] In the embodiments described in this specification, medical service platforms or healthcare applications can be linked to a large-scale medical model to provide users with services such as intelligent consultation, health advice, health follow-up, and health monitoring. During the use of the large-scale medical model, users can actively upload medical and health records, bind intelligent monitoring devices, and engage in multi-round dialogues with the model.

[0035] In this embodiment, to improve the evaluation effect of the medical big data model, multiple profile data generated by multiple target users during their interaction with the medical big data model are selected as the data basis for constructing evaluation samples. The medical big data model is then evaluated based on these samples, which helps improve the accuracy of the evaluation results. It is noted that in this embodiment, in response to user authorization, or with user authorization or full authorization from all parties, dialogue data generated by the user and the medical big data model in multiple rounds of dialogue, user-uploaded medical and health records, and data detected by intelligent detection devices can be collected and integrated to form user profile data. In this embodiment, the profile data used is anonymized and / or encrypted, and the processing flow is not visible to the outside world, ensuring data security throughout the entire chain. That is, in this embodiment, the collection, use, and processing of profile data comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0036] The profile data includes: medical and health records and raw dialogue data. Medical and health records are used to record a comprehensive set of data related to medical and health throughout the user's lifecycle. Optionally, medical and health records include dynamic monitoring data and static basic information. Static basic information may include, but is not limited to: personal symptoms, past medical history, physical examination reports, medical records, medication records, vaccination records, and chronic disease management data. Dynamic monitoring data may include, but is not limited to: heart rate, steps, blood oxygen saturation, blood glucose, and blood pressure. Raw dialogue data refers to multi-turn dialogue data between the user and the medical big data model. Raw dialogue data refers to multi-turn dialogue data between the target user and the medical big data model. In the embodiments of this specification, multiple profile data generated by multiple target users in the medical big data model are obtained. One target user corresponds to one profile data. A target user refers to someone who has left a large amount of representative, multi-turn interaction data on the medical big data model, which can effectively reflect the behavioral patterns and needs of typical users. Therefore, it is particularly suitable for building or validating user profiles, evaluating medical big data models, and optimizing service processes.

[0037] In the embodiments of this specification, multiple target dialogue data are constructed based on the original dialogue data from multiple profile data sets. For example, for the original dialogue data in a profile data set, target dialogue data corresponding to that original dialogue data is constructed. Since the original dialogue data may cover a limited range of medical scenarios, a limited range of user groups, and a limited time span, the original dialogue data is expanded in terms of medical scenarios, user groups, and time series. This allows the target dialogue data to fully encompass the complex personalized memory features encountered by the large-scale medical model in practical applications, laying the foundation for the generation of subsequent evaluation data.

[0038] Target dialogue data refers to multi-turn dialogue data covering multiple medical scenarios and multiple user groups, spanning various time dimensions. Multiple medical scenarios may include, but are not limited to, chronic disease management, emergency response, health promotion, food safety, and family care. Multiple user groups may include, but are not limited to, the user themselves, family members, colleagues, and friends. "Spanning multiple time dimensions" means that the target dialogue data involves multiple time dimensions, including but not limited to, 1 day, 10 days, 1 month, and 3 months. These multiple medical scenarios, multiple user groups, and multiple time dimensions are not isolated but deeply intertwined and mutually supportive, collectively forming target dialogue data that reflects complex and personalized memory characteristics. For example, the medical scenario determines the topic, medical content, and interaction purpose of the dialogue; the user group determines the perspective, role relationships, and differences in information needs; and the time span determines the dynamic evolution of information, dependencies, and the rhythm of interaction.

[0039] For example, a target dialogue data includes multiple dialogue fragments, which involve multiple medical scenarios. Some medical scenarios involve the user itself, while others involve related entities of the user (such as family members, caregivers, colleagues, or friends). In each medical scenario, the dialogue fragments are arranged in chronological order, forming a multi-round interaction sequence with temporal dependencies. The roles of the subjects involved, the purpose of the interaction, and the information content all change dynamically as the scenario changes, thus constructing a complex personalized memory narrative with multiple subjects participating, multiple scenarios intertwined, and evolving across time.

[0040] For example, taking 13 target dialogue data sets as an example, each target dialogue data set includes at least 10 rounds of dialogue data, with a minimum time span of 10 days, an average time span of 1 month, and a maximum time span of 3 months. The dialogue frequency covers daily and weekly interactions. The 13 target dialogue data sets include 6 chronic disease management scenarios, 3 scenarios involving medical history and acute illness memories, 2 scenarios involving healthy individuals and acute illness memories, and 2 scenarios involving multiple family members. Among them, the 6 chronic disease management scenarios include: 3 diabetes scenarios, 2 hypertension scenarios, and 1 anxiety scenario.

[0041] In the embodiments of this specification, multiple evaluation samples are generated using multiple intelligent agents based on medical and health records from multiple profile data and multiple target dialogue data. The medical big data model possesses question-answering and memory capabilities. In this embodiment, the evaluation primarily focuses on the medical big data model's memory construction, memory recall, and memory consumption capabilities. Therefore, one evaluation sample in this embodiment includes: memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data. For example, a profile data set includes medical and health records and original dialogue data, which corresponds to target dialogue data. Based on the target dialogue data and the medical and health records included in the profile data, multiple evaluation samples are generated using multiple intelligent agents. The memory construction evaluation data is used to assess whether the medical big data model can accurately extract, integrate, and structure key medical information (such as medical history, medication records, and symptom changes) from medical dialogue data and medical and health records to form a reliable long-term memory representation; that is, it is used to evaluate the medical big data model's memory construction capability.

[0042] Among them, the memory recall assessment data is used to evaluate whether the medical big model can accurately recall and call some memory information (such as past diagnoses, previous suggestions, allergy history, etc.) in subsequent dialogues based on the context or the question information currently entered by the user. This verifies the accuracy and timeliness of the medical big model's cross-conversation memory, that is, it is used to evaluate the memory recall capability of the medical big model.

[0043] Among them, the memory consumption assessment data is used to evaluate how the medical big data model can effectively utilize some of the recalled memory information to generate personalized, coherent and clinically meaningful answer information (such as adjusting chronic disease management plans, warning of potential risks, and providing emotional support), reflecting the application value of memory in actual services, that is, to evaluate the memory consumption capacity of the medical big data model.

[0044] In this system, multiple agents cooperate to generate evaluation data across three dimensions: memory construction, memory retrieval, and memory consumption. For example, there could be three agents, each responsible for constructing one type of evaluation data. These three agents could be: a memory construction agent, a memory retrieval agent, and a memory consumption agent. The memory construction agent generates memory construction evaluation data based on medical and health records within the profile data, and the target dialogue data corresponding to the original dialogue data within that profile data. The memory retrieval agent retrieves the memory construction evaluation data based on the target dialogue data and the medical and health records to obtain memory retrieval evaluation data. The memory consumption agent generates the memory consumption evaluation data based on the target dialogue data, the medical and health records, and the memory retrieval evaluation data.

[0045] In this context, regardless of the specific type of intelligent agent mentioned above, an intelligent agent refers to a functional unit that can be independently packaged and possesses one or more intelligent behavioral capabilities, such as perception, reasoning, decision-making, or learning. The core characteristic of an intelligent agent is its ability to autonomously or semi-autonomously complete specific cognitive tasks based on input information and to serve as a component in building complex intelligent systems. An intelligent agent is an autonomous computer module capable of perceiving external input information, processing the perceived information, and performing goal-oriented reasoning and decision-making actions. In some embodiments, an intelligent agent may include: a perception module, a decision / reasoning engine, an action module, and a memory system. The perception module receives external input (such as text, images, and speech) as a basis for decision-making. The decision / reasoning engine plans tasks and performs complex reasoning. In some embodiments, the intelligent agent's decision / reasoning engine may be driven by a machine learning model (e.g., a language model) to perform complex tasks such as contextual understanding, logical reasoning, task decomposition, and strategy formulation based on the capabilities of the machine learning model. The action module performs operations such as outputting natural language, calling interfaces, operating tools, and generating code. The memory system relies on external storage (such as vector databases or state managers) to store short-term or long-term memories and maintain task continuity. The agent also has learning capabilities and can continuously optimize its performance through prompting engineering, in-context learning (ICL), fine-tuning, or reinforcement learning.

[0046] The aforementioned machine learning model can be a deep learning model with a relatively large number of parameters or a relatively small number of parameters. This application does not limit the number of parameters supported by the deep learning model used, aiming to meet actual needs. The deep learning model involved in this application can be a large language model, which is a natural language processing (NLP) model trained on a large scale. Large language models are typically built based on deep learning techniques and trained on large-scale training datasets, thus exhibiting powerful performance in processing natural language tasks. Given a text (i.e., context), the large language model attempts to predict the most likely next word. The number of parameters in the large language model is not limited. For example, the number of parameters in the large language model may be greater than a set threshold, typically in the millions or billions. Alternatively, the number of parameters may be less than a million. During training, these parameters are continuously adjusted and optimized based on the difference between the prediction results and the actual results of the large language model to improve its prediction accuracy. In some embodiments, large language models typically employ advanced neural network architectures, such as Transformer architectures, to build their model structures. These architectures enable large language models to capture complex patterns in text and handle long-range dependencies. After thorough training, large language models possess powerful generative capabilities, producing coherent and context-appropriate text content based on given prompts.

[0047] In the embodiments of this specification, the memory construction capability, memory recall capability, and memory consumption capability of the medical big model are quantitatively evaluated based on memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data from multiple evaluation samples, so as to obtain evaluation results.

[0048] For example, based on memory construction assessment data from multiple evaluation samples, the memory construction capability of the medical big data model is quantitatively evaluated to obtain assessment results for the memory construction capability dimension. For instance, by analyzing whether the medical big data model accurately, completely, and structurally extracts key medical information (such as medical history, medication records, symptom descriptions, and lifestyle habits) from target dialogue data and medical and health records during user interaction, the information capture rate, entity recognition accuracy, and timeline consistency of the medical big data model are calculated. This quantifies the medical big data model's ability to transform target dialogue data and medical and health records into usable long-term memory (i.e., memory construction assessment data).

[0049] For example, based on memory recall evaluation data from multiple evaluation samples, the memory recall capability of the medical big data model is quantitatively evaluated to obtain evaluation results for the memory recall capability dimension. For instance, by assessing whether the medical big data model can accurately retrieve and reference historical memory content at the correct time and according to relevant context (i.e., memory recall evaluation data), indicators such as recall accuracy, context relevance score, and false reference rate are calculated, thereby quantifying the reliability and timing ability of the medical big data model in cross-session memory retrieval.

[0050] For example, based on memory consumption assessment data from multiple evaluation samples, the memory consumption capacity of a large-scale medical model is quantitatively assessed to obtain evaluation results for the memory consumption capacity dimension. For instance, this examines how the large-scale medical model effectively applies recalled memory information (i.e., memory recall assessment data) to real-world service scenarios, such as generating personalized health recommendations, dynamically adjusting management plans, providing risk warnings, or offering emotional support. Through methods such as expert scoring, user satisfaction, and automatic assessment of recommendation rationality (e.g., clinical guideline compliance), the actual value of memory in improving service quality and user experience is quantified.

[0051] In the embodiments of this specification, profile data generated by multiple users during their interaction with the medical big data model is acquired. Based on the original dialogue data in the profile data, target dialogue data covering multiple medical scenarios, multiple user subjects, and cross-time dimensions is constructed, enabling the target dialogue data to carry the model's complex personalized memories. Based on the medical and health records in the target dialogue data and profile data, multiple intelligent agents are used to generate evaluation samples in three dimensions: memory construction, memory retrieval, and memory consumption, effectively transmitting complex personalized memories to the evaluation samples. The medical big data model is evaluated through the above evaluation samples, achieving a fully decoupled evaluation of the model's internal answer generation chain. When there is a deviation in the model output, the deviation can be accurately located at which stage of memory construction, memory retrieval, or memory consumption, thereby capturing the model's actual performance in processing complex personalized memories and providing accurate and traceable quantitative evidence for model optimization.

[0052] In one optional embodiment, the implementation method for constructing multiple target dialogue data based on original dialogue data from multiple profile data is not limited. In one example, in the medical scenario dimension, multiple preset medical scenarios are maintained, including but not limited to: chronic disease management, emergency response, health promotion, food safety, and family care. The original dialogue data may cover some or all of the preset medical scenarios. Based on this, for any original dialogue data, medical scenario identification is performed to obtain a first medical scenario; if the first medical scenario does not fully cover all preset medical scenarios, it indicates that one or more medical scenarios are missing in the original dialogue data, which can be referred to as target medical scenarios. Then, supplementary dialogue data corresponding to the target medical scenarios is generated based on the target medical scenarios not covered in the preset medical scenarios and other original dialogue data; the original dialogue data and supplementary dialogue data are integrated into the target dialogue data corresponding to the original dialogue data.

[0053] The implementation method for integrating the original dialogue data and supplementary dialogue data into the target dialogue data corresponding to the original dialogue data is not limited. For example, supplementary dialogue data includes a dialogue start timestamp, and the original dialogue data also includes a dialogue start timestamp. The original dialogue data and supplementary dialogue data can be merged according to the dialogue start timestamp order to form a continuous conversation sequence. If supplementary dialogue data is used to fill blank periods in the original dialogue data (such as no dialogue records in a certain week), it can be inserted at the corresponding time position to ensure that the overall time span (such as a short time span of 10 days or an average time span of 1 month) meets the target requirements. Another example is to check the contextual logic between adjacent dialogue rounds. If supplementary dialogue data introduces a new topic or continues an old topic, the supplementary dialogue data can be fine-tuned so that the language style, terminology, and health status description of the supplementary dialogue data are naturally connected with the original dialogue data.

[0054] Optionally, if the first medical scenario fully covers the preset medical scenarios, the original dialogue data will be directly used as the target dialogue data.

[0055] In another example, at the user subject level, if the original dialogue data includes a user subject, then related user subjects are added to supplement the dialogue data for certain medical scenarios. In yet another example, at the time level, a preset time span is maintained for each medical scenario in the original dialogue data. These preset time spans include: 1 day, 1 week, 10 days, 1 month, and 3 months, etc. If the dialogue data for a medical scenario is missing a portion of the preset time span, then the dialogue data for that missing time span is added to obtain supplementary dialogue data. The original dialogue data and supplementary dialogue data are then sequentially integrated according to their start timestamps to obtain the target dialogue data.

[0056] Optionally, the implementation method for generating supplementary dialogue data corresponding to the target medical scenario based on the target medical scenario not covered in the preset medical scenario and other original dialogue data is not limited. An example is provided below.

[0057] In one example, supplementary dialogue data corresponding to the target medical scenario is generated based on other original dialogue data. This includes: identifying the medical scenario in the other original dialogue data to obtain a second medical scenario; if the second medical scenario includes the target medical scenario, it means that the other original dialogue data includes dialogue data related to the target medical scenario; then, dialogue segments in the other original dialogue data that are associated with the target medical scenario are identified, where the user subject of the dialogue segment is another target user, or an associated user of that other target user; the user subject of the dialogue segment is rewritten to be consistent with the user subject of the original dialogue data to obtain supplementary dialogue data. For example, the user subject in the original dialogue data can be the target user itself or an associated user of the target user.

[0058] In another example, based on a target medical scenario that is not covered in the preset medical scenarios, supplementary dialogue data corresponding to the target medical scenario is generated. This includes: identifying medical scenarios from other original dialogue data to obtain a second medical scenario; if multiple second medical scenarios do not include the target medical scenario, it means that the other original dialogue data does not include dialogue data under the target medical scenario. Then, a dialogue generation agent is used to generate supplementary dialogue data based on the description information of the target medical scenario, the user subject of the original dialogue data, and multiple time dimensions.

[0059] For an introduction to dialogue agents, please refer to the preceding section on agents; it will not be repeated here. The dialogue generation agent is primarily used to generate dialogue data using a large dialogue generation model. For an introduction to the large dialogue generation model, please refer to the preceding embodiments. A large language model pre-trained on a large general dataset can be used as the base model for the large dialogue generation model. To adapt to the specific application scenario of this application, namely dialogue generation, the pre-trained large language model can be fine-tuned on a dataset formed from dialogue data in a large medical and health scenario, making the fine-tuned large dialogue generation model more suitable for performing dialogue generation tasks.

[0060] The implementation method of using a dialogue generation agent to generate supplementary dialogue data based on the description of the target medical scenario, the user subjects of the original dialogue data, and multiple time dimensions is not limited. For example, the description of the target medical scenario, the user subjects of the original dialogue data and their associated user subjects, as well as multiple time dimensions, are input into the dialogue generation agent. The dialogue generation agent calls a large dialogue generation model to generate multi-turn dialogue data covering the target medical scenario as supplementary dialogue data.

[0061] The following is a scenario-based implementation example of constructing multiple target dialogue data based on raw dialogue data from multiple profile datasets, such as... Figure 2 As shown, step 21: Select profile data. In step 21: Obtain multiple raw dialogue data from multiple (e.g., 200) candidate users generated by the medical big data model, call the profile generation agent, and generate profile data for the candidate users based on the dialogue content of the multiple raw dialogue data and the medical and health records of the candidate users. The profile data can reflect the characteristics of each candidate user, such as "chronic disease management type," "emotional anxiety type," "multiple disease comorbidity type," and can also include disease type, medication habits, and health concerns. Through the profile types that medical experts and the medical big data model focus on, multiple target users are selected from the multiple candidate users, thereby obtaining multiple profile data for multiple target users. For example, one type of target user profile data involves "elderly, hypertension, and children's concern about the illness," used to test family care scenarios; another type of target user profile data involves "young people, insomnia, and staying up late," used to test lifestyle intervention capabilities.

[0062] Step 22: Construct Target Dialogue Data. In Step 22, raw dialogue data is obtained from the target user's profile data. It is determined whether the raw dialogue data fully covers the preset scenario information. If so, the raw dialogue data is directly used as the target dialogue data. If the raw dialogue data does not fully cover the preset scenario information, the target medical scenario is missing. Further, it is determined whether the raw dialogue data of other users (e.g., candidate users or other target users) includes the missing target medical scenario. If so, supplementary dialogue data is generated for the raw dialogue data by splicing the conversation, realizing cross-user splicing of the conversation. For example, Patient A: Family member has not been diagnosed with diabetes, missing the "self-confirmation" step; Patient B: Self has been diagnosed with diabetes. The dialogue data of Patient B regarding diabetes is extracted, and the subject of Patient B's dialogue data is rewritten to Patient A. The rewritten dialogue data is used as supplementary dialogue data, and the supplementary dialogue data and Patient A's raw dialogue data are used as the target dialogue data. If the raw dialogue data does not include the target medical scenario, dialogue data under the target medical scenario is generated by the dialogue generation agent as supplementary dialogue data. For example, in a target medical scenario, scenario Q might be: "I haven't been sleeping well lately, is it related to the blood pressure medication I'm taking?", the dialogue generation agent automatically supplements the analysis logic, generating reasonable follow-up questions and answers as supplementary dialogue data. This supplementary dialogue data and the original dialogue data are then integrated into the target dialogue data.

[0063] Step 23: Review the target dialogue data. For example, label the target dialogue data with conversation tags such as "chronic disease management," "medication consultation," "emotional support," and "multiple coexisting diseases." Filter out conversation tags irrelevant to the current assessment objective (e.g., advertisements, casual conversation). Correct the target dialogue data, for example, by manually calibrating it, including: subject correction, time correction, numerical accuracy correction, and conflict correction. Subject correction ensures the speaker's identity is clear (e.g., "I" or "family member"); time correction adjusts the sequence of events to conform to medical logic; numerical accuracy correction checks, for example, whether blood pressure values ​​are reasonable; and conflict correction eliminates contradictory information in the dialogue. Finally, the target dialogue data and medical health records are obtained.

[0064] In one optional embodiment, the implementation method of generating multiple assessment samples using multiple intelligent agents based on medical and health records from multiple profile data and multiple target dialogue data is not limited. An exemplary description follows.

[0065] In one example, an implementation method for generating multiple assessment samples using multiple intelligent agents based on medical and health records from multiple profile data and multiple target dialogue data includes: taking multiple intelligent agents including a memory-building intelligent agent, a question-generating intelligent agent, and a memory-recalling intelligent agent as an example, for each target user, the memory-building intelligent agent extracts multiple key medical information from the target user's target dialogue data to form a first benchmark list, which serves as memory-building assessment data; the question-generating intelligent agent generates multiple assessment questions based on the target user's medical and health records and the first benchmark list; for any assessment question, it obtains consultation point information generated by a medical expert for the assessment question; the memory-recalling intelligent agent recalls at least a portion of the key medical information supporting the consultation point information from the first benchmark list to form a first truth value list, which serves as memory-recalling assessment data; and the memory-consuming intelligent agent generates answer point information for the assessment questions based on the first truth value list, using the assessment questions and their answer point information as memory-consuming assessment data.

[0066] In this process, for any target user, a memory-based intelligent agent is constructed. Multiple key medical information items are extracted from the target user's target dialogue data to form a first benchmark list, which serves as the memory construction evaluation data. Key medical information refers to core health-related data or facts that are substantially significant for health assessment, disease management, clinical decision-making, or personalized services. For example, key medical information may include, but is not limited to: disease diagnosis information, symptoms and chief complaints, medication and treatment information, generation methods, behavioral habits, emergencies, psychological states, and monitoring data. Key medical information can be extracted from target dialogue data and further from medical and health records. The first benchmark list serves as the ground truth in the memory construction phase. The ground truth refers to the standard answer used to test the large-scale medical model, representing a human-accepted, correct label or result, which is the target that the large-scale medical model needs to strive to match or approach. The first benchmark list includes multiple key medical information items. Each first benchmark list can serve as a memory construction evaluation data point.

[0067] Specifically, a question-generating agent is used to generate multiple assessment questions based on the target user's medical and health records and a first benchmark list. For example, by combining the target user's medical and health records with key medical information from the first benchmark list, the question-generating agent automatically generates multiple assessment questions with clinical significance, temporal context, and personalized features, focusing on dimensions such as the target user's disease characteristics, recent symptoms, medication behavior, acute illness memories, and lifestyle.

[0068] For any given assessment question, the system obtains key information from medical experts regarding the consultation points generated for that question. For example, multiple assessment questions can be provided to medical experts, who can then simulate clinical diagnostic thinking to identify the crucial information points needed to answer the question—the corresponding consultation points. Optionally, the system can also include the weights of each consultation point; the higher the weight, the more important the consultation point. For example, if the assessment question is, "My blood pressure has been fluctuating a lot lately, what should I do?", the consultation points could be: recent blood pressure records (weight 30%), whether medication is taken regularly (weight 30%), changes in diet, sleep, and mood (weight 20%), and whether there are any chronic diseases such as diabetes (weight 20%).

[0069] In this process, a memory recall agent is used to recall at least a portion of the key medical information supporting the consultation point from a first benchmark list, forming a first truth list, which serves as memory recall evaluation data. The first truth list acts as the ground truth for the memory recall phase. It includes at least a portion of the key medical information from the first benchmark list. For example, using a memory recall agent, for consultation point information provided by a medical expert for a specific evaluation question (such as "pay attention to recent blood glucose fluctuations and medication adherence"), the agent precisely retrieves and extracts key medical information directly related to the consultation point information from the first benchmark list corresponding to the target user (such as "average fasting blood glucose of 8.6 mmol / L in the past 7 days" and "occasionally missed metformin doses in the past two weeks"). This successfully recalled information is combined into a first truth list, serving as memory recall evaluation data to measure whether the system can correctly and completely retrieve the required facts from long-term memory.

[0070] Specifically, a memory-consuming intelligent agent is used to generate key answer points for assessment questions based on a first truth value list. These assessment questions and their key answer points are then used as memory-consuming assessment data. For example, using the memory-consuming intelligent agent and based on the first truth value list, an accurate, personalized, and fact-based key answer point is generated for each assessment question. This assessment question and key answer point information are then used as memory-consuming assessment data to evaluate whether the large-scale medical model can effectively utilize recalled memory content to generate clinically valuable and user-relevant answer information.

[0071] Optionally, the implementation method of using a question-generating agent to generate multiple assessment questions based on the target user's medical and health records and a first benchmark list is not limited. An example is provided below: using a question-generating agent, multiple assessment questions are generated based on preset question types and multiple key medical information from the first benchmark list. The question types include at least: single-hop, multi-hop reasoning, open-ended, and adversarial types. Dynamic monitoring data and static basic information are obtained from the target user's medical and health records; the dynamic monitoring data includes abnormal indicators. The question-generating agent then generates multiple assessment questions based on the abnormal indicators in the dynamic monitoring data and the static basic information.

[0072] The system utilizes a question-generating agent to generate multiple assessment questions based on a set of pre-defined question types and several key medical information items from a first benchmark list. These question types include at least: single-hop, multi-hop reasoning, open-ended, and adversarial. Single-hop questions can be answered by directly referencing a single key medical information item. For example, the answer to the question "What blood pressure medication are you currently taking?" comes directly from medication records. Multi-hop reasoning questions involve logical inference based on two or more key medical information items. For example, the answer to the question "Is your recent high blood sugar related to missing a metformin dose?" can link the two key medical information items: elevated blood sugar and decreased medication adherence. Open-ended questions encourage users to describe their experiences, feelings, or behaviors, with no fixed answers and requiring personalized guidance based on memory. For example, the answer to the question "What has been the biggest challenge you've faced while managing your blood sugar recently?" is based on empathetic responses provided by past conversations regarding diet, exercise, or emotions. Adversarial questions intentionally introduce contradictory, ambiguous, or challenging statements to test whether the medical model can correct misunderstandings or clarify facts based on reliable memory. For example, I heard that diabetes can be cured by dieting without medication. Is that true? The answer to this question is based on the user's medical history and medical knowledge.

[0073] The process involves acquiring dynamic monitoring data and static basic information from the target user's medical and health records. A question-generating agent is then used to generate multiple assessment questions based on abnormal indicators in the dynamic monitoring data and the static basic information. Detailed explanations of the dynamic monitoring data and static basic information can be found in the aforementioned embodiments and will not be repeated here. For example, the question-generating agent generates high-risk warning or personalized intervention questions based on the combination of abnormal indicators and static basic information in the dynamic monitoring data. Abnormal indicators refer to values ​​or trends that deviate from the normal range or individual baseline identified from the user's dynamic monitoring data (such as continuous records of blood glucose, blood pressure, heart rate, weight, sleep, etc.). Static basic information refers to the user's relatively stable and long-term unchanging health characteristics derived from their medical and health records. The combination principle of abnormal indicators and static basic information is that abnormal indicators provide "what happened," while static basic information explains "who it is important to." This combination ensures that the assessment questions not only reflect the user's current risk but also possess clinical relevance, personalization, and intervention orientation, thereby effectively testing the memory, understanding, and application capabilities of the medical big data model. For example, an assessment question could be: "Your blood pressure has recently risen significantly, exceeding 160 / 90. Considering your long history of hypertension, have you experienced headaches, chest tightness, or blurred vision?" Optionally, user question information can also be obtained from the original dialogue data as assessment questions. Specifically, the question-generating agent is invoked to identify multiple question information items and their types from the target user's original dialogue data; from these multiple question information items, the question information item whose question type matches a preset question type is selected as the assessment question. Preset question types can include, but are not limited to: single-hop, multi-hop reasoning, open-ended, and adversarial types.

[0074] Optionally, a method for using a memory-based intelligent agent to extract multiple key medical information from the target user's target dialogue data to form a first benchmark list includes: for each target user, step 31, using a memory-based intelligent agent to extract multiple key medical information from the target user's target dialogue data to form a candidate benchmark list; step 32, using a first review intelligent agent to analyze and review the candidate benchmark list based on the target dialogue data, to output a first review conclusion and a first modification suggestion; step 33, if the first review conclusion indicates failure, then providing the first modification suggestion to the memory-based intelligent agent for the memory-based intelligent agent to optimize the memory construction strategy (such as optimizing prompt words or model parameters), and repeating the above steps. Steps 31 to 33 continue until the first review conclusion of the first review agent indicates approval. The candidate benchmark list output by the memory-building agent is then used as the benchmark list to be reviewed. Step 34 involves providing the benchmark list to the target dialogue data to the annotation team for manual correction. The annotation team manually checks each item and generates a second review conclusion and a second modification suggestion. Step 35 involves obtaining the second review conclusion. If the second review conclusion indicates failure, the second modification suggestion is provided to the memory-building agent to further optimize the memory-building strategy. Steps 31 to 35 are repeated until the second review conclusion indicates approval. The candidate benchmark list output by the memory-building agent in the last round is then used as the first benchmark list. Figure 3 As shown, an example is provided for the target dialogue data, the memory-building agent, the first review agent, and the list of benchmarks to be reviewed.

[0075] Optionally, an implementation method using a question-generating agent to generate multiple assessment questions based on preset multiple question types and multiple key medical information in a first benchmark list includes: for any target user, step 41, using a memory-constructing agent to extract multiple key medical information from the target user's target dialogue data to form a first benchmark list; step 42, using the question-generating agent to generate multiple candidate questions based on preset multiple question types and multiple key medical information in the first benchmark list; step 43, using an AI doctor agent to generate multiple candidate questions based on the multiple candidate questions and the first benchmark list. The question corresponds to multiple answer points; Step 44, using a memory evaluation agent, based on the multiple answer points of multiple candidate questions, evaluates multiple candidate questions and outputs a third review conclusion and a third modification suggestion for the candidate questions; if the third review conclusion indicates that the review is not passed, the question generation strategy (such as prompt words or model parameters) of the question generation agent is optimized based on the third modification suggestion, and steps 42 and 44 are repeated until the third review conclusion indicates that the review is passed. The candidate questions output by the question generation agent in the last round are used as evaluation questions, forming a closed-loop optimization mechanism to improve the quality of questions and the matching degree of answers. Figure 4The document provides illustrative examples of target session data, memory construction agents, question generation agents, AI doctor agents, and memory assessment agents.

[0076] In addition, Figure 4 The process also includes: Step 45, using a question-generating agent to generate multiple assessment questions based on abnormal indicators and static basic information in dynamic monitoring data; and Step 46, calling a question-identification agent to identify multiple question information and their types from the target user's original dialogue data; and selecting question information whose question type matches a preset question type as the assessment question. In addition, it includes: Step 47, a manual review process for the generated assessment questions, where medical experts review the consistency, accuracy, and medical factuality of the assessment questions to prevent the spread of misleading or erroneous knowledge. The medical big data model reviews whether the distribution of assessment questions is balanced, for example, covering common diseases, rare diseases, and chronic diseases, and whether there is any duplication or redundancy, making supplements and adjustments as necessary.

[0077] In generating assessment questions, a question-generating agent can be used to generate multiple question labels for each assessment question. The distribution of these labels across multiple dimensions is then statistically analyzed to assess the comprehensiveness and representativeness of the assessment questions. Subsequent assessment results can be used to optimize the large-scale medical model based on the scores under different question labels. For example, the memory construction, memory recall, and memory consumption capabilities of assessment questions under each question label can be quantitatively evaluated to obtain a comprehensive score for each label. If the comprehensive score for a particular question label is found to be below a set score threshold, the sample data under that question label is used to optimize the large-scale medical model. Furthermore, for any given question label, scores for the three dimensions of memory construction, memory recall, and memory consumption can be calculated. If the score for any dimension is found to be below a set score threshold, optimization is performed on that dimension under that question label.

[0078] exist Figure 5 The text provides examples of the distribution of question labels across various dimensions. The question input format labels and their distribution are as follows: single-round plain text 84.21%, single-round text and image 4%, single-round plain text... Figure 2The percentage of users with multiple rounds of text-only interactions was 8%, and those with multiple rounds of text and images was 1.6%. User subject tags and their distribution were: self 74.9%, specific relationship members 8%, multiple member subjects 2%, no related members 14%, and vague members 2%. Memory dependency need tags and their distribution were: needed 77%, not needed 23%. Memory interaction status tags and their distribution were: memory combined 71.26%, memories mutually exclusive 11.74%, and no related memories 17%. Memory content type tags and their distribution were: medical history 30.24%, physiological indicators 26.35%, medication history 22.75%, test reports 10.48%, dietary preferences 5.99%, lifestyle 4.49%, surgery 2.69%, pregnancy and childbirth history 2.40%, allergy history 1.80%, current user status 1.80%, and others 4.79%. Consumer scenario tags and their distribution: 36.19% answered based on user's intrinsic memory information; 16.67% were purely science-related questions without memory reference; 11.90% involved interpretation and trend analysis of physiological indicators; 11.90% offered personalized lifestyle suggestions; 9.05% involved single-step information queries; 8.57% answered based on user's family member information; 3.33% were mutually exclusive with user's memory information; 1.43% were vague subject-related questions; and 0.95% involved no relevant role and no memory reference. Reasoning difficulty tags and their distribution: Easy 28.74%, Medium 58.7%, Difficult 12.65%.

[0079] Optionally, for example, for any assessment question, the method of obtaining the consultation points information generated by medical experts for the assessment question, such as... Figure 6 As shown, for any assessment question, step 61 involves obtaining the consultation points information generated by medical experts for the assessment question; in addition, step 62 involves using the assessment question, the first benchmark list, medical and health records, and consultation points information as inputs for the memory recall stage.

[0080] The methods for using a memory-based recall agent to recall at least a portion of key medical information supporting the key points of the consultation from a first benchmark list to form a first truth list include: Figure 6As shown, in the memory recall stage, step 63 involves using a memory recall agent to recall at least a portion of key medical information supporting the consultation key points from a first baseline list to form a candidate truth value list; step 64 involves using a second review agent to make a reasonableness judgment on the candidate truth value list, obtaining a first reasonableness judgment result and a first optimization suggestion to ensure that the candidate truth value list can support the consultation key points; if the first reasonableness judgment result indicates that it is unreasonable, the memory recall strategy of the memory recall agent (such as prompt words or model parameters of the large model in the agent) is optimized based on the first optimization suggestion shown, and steps 63 and 64 are repeated until the first reasonableness judgment result indicates that it is reasonable, and the candidate truth value list generated by the memory recall agent in the last round is taken as the first truth value list.

[0081] Optionally, using a memory-consuming intelligent agent, based on a first truth value list, key answer points for assessment questions are generated. The assessment questions and their key answer points are then used as implementation methods for memory-consuming assessment data, including: Figure 6 As shown, in step 65, the memory-consuming agent generates candidate answer key information for the assessment question based on the first truth value list; in step 66, the third review agent performs a rationality judgment on the candidate answer key information for the assessment question, obtaining a second rationality judgment result and a second optimization suggestion; if the second rationality judgment result indicates that it is unreasonable, the memory consumption strategy of the memory-consuming agent (such as prompt words or model parameters of the large model in the agent) is optimized based on the second optimization suggestion, and steps 65 and 66 are repeated until the second rationality judgment result indicates that it is reasonable, and the candidate answer key information generated by the memory-consuming agent in the last round is taken as the final answer key information.

[0082] further, Figure 6 The process also includes step 67, which provides the assessment questions, the first truth value list, and the key points of the answers to the medical experts for secondary revision, resulting in the revised key points of the answers.

[0083] In one optional embodiment, the memory construction capability, memory recall capability, and memory consumption capability of the medical big data model are quantitatively evaluated based on memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data from multiple evaluation samples, respectively, to obtain evaluation results. The implementation method is not limited. In one example, the memory construction evaluation data includes a first benchmark list, the memory recall evaluation data includes a first truth value list, and the memory consumption evaluation data includes evaluation questions and answer key information. Based on the evaluation questions, the medical big data model is triggered to use the target dialogue data corresponding to the evaluation questions as a memory context for medical question-and-answer processing. The medical question-and-answer processing includes: constructing a second benchmark list from the memory context, recalling a second truth value list from the second benchmark list, and generating answer information based on the second truth value list. Based on the first benchmark list and the second benchmark list, the first truth value list and the second truth value list, as well as the answer key information and the answer information, the memory construction capability, memory recall capability, and memory consumption capability of the medical big data model are quantitatively evaluated to obtain evaluation results.

[0084] Optionally, the implementation method for quantitatively evaluating the memory construction ability, memory recall ability, and memory consumption ability of the medical big data model based on the first and second benchmark lists, the first and second truth lists, as well as the key information of the answers and the answer information, to obtain the evaluation results is not limited. An example is provided below: based on the second and first benchmark lists, the completeness of memory construction of the medical big data model is evaluated to obtain a first evaluation result; based on the second and first truth lists, the accuracy of memory recall of the medical big data model is evaluated to obtain a second evaluation result; based on the answer information and key information of the answers, as well as the answer information and the second truth list, the effectiveness of memory consumption of the medical big data model is evaluated to obtain a third evaluation result.

[0085] Optionally, the medical big data model includes a question-answering module and a memory module. The question-answering module can be implemented as a question-answering big data model, and the memory module can be implemented as a memory big data model. The medical question-answering process can involve using the target dialogue data corresponding to the assessment question as the memory context. The memory big data model constructs a second benchmark list from the memory context and recalls a second truth value list from the second benchmark list. The second truth value list and the assessment question are then provided to the question-answering big data model. The question-answering big data model generates the answer information corresponding to the assessment question based on the second truth value list. Accordingly, memory construction assessment data is used to evaluate the memory construction capability of the memory big data model, memory recall assessment data is used to evaluate the memory recall capability of the memory big data model, and memory consumption assessment data is used to evaluate the memory consumption capability of the question-answering big data model.

[0086] Further optionally, an implementation method for evaluating the memory construction integrity of a large medical model based on a second benchmark list and a first benchmark list to obtain a first evaluation result includes: invoking a first decision agent to analyze the missing medical key information, redundant medical key information, and matching medical key information in the second benchmark list relative to the first benchmark list; calculating an integrity score based on the number of missing medical key information, redundant medical key information, and matching medical key information; and generating a first evaluation result reflecting the memory construction capability based on the integrity score.

[0087] Specifically, the first decision-making agent is invoked to analyze the second benchmark list relative to the first benchmark list for missing, redundant, and matching medical key information. Missing medical key information refers to medical key information that exists in the first benchmark list but not in the second benchmark list. Redundant medical key information refers to redundant or erroneous medical key information that exists in the second benchmark list but not in the first benchmark list; matching medical key information refers to medical key information that exists in both the first and second benchmark lists and is semantically consistent.

[0088] The integrity score is calculated based on the number of missing medical key information, redundant medical key information, and matching medical key information. The integrity score can be the accuracy, recall, and F1 score of the memory construction. The number of matching medical key information is the True Positive (TP), the number of missing medical key information is the False Negative (FN), and the number of redundant medical key information is the False Positive (FP).

[0089] Recall (R) measures completeness, i.e., how much critical medical information is covered. Recall R = TP / (TP + FN). Precision (P) measures accuracy, i.e., how much of the recalled information is correct. Precision P = TP / (TP + FP). F1 is the harmonic mean of precision and recall, e.g., F1 = 2 * (P * R) / (P + R) = (2 * TP) / (2 * TP + FP + FN). Here, "*" represents multiplication, and " / " represents division. The first assessment result generated based on the completeness score can include: a completeness score, such as a range of 0 to 100; a list of missing critical medical information and its clinical impact level; examples of redundant critical medical information and potential risk warnings; and a memory construction ability rating, such as excellent, good, and needing improvement.

[0090] like Figure 7As shown, this paper provides an example of evaluating the memory construction integrity of a large medical model based on a second benchmark list and a first benchmark list to obtain the first evaluation result. Figure 7 The first and second benchmark lists are illustrated in the example, and the execution logic within the first decision-making intelligent body is also illustrated in the example. Within the first decision-making intelligent body, it is determined whether the second benchmark list completely covers all the medical key information in the first benchmark list. If the second benchmark list does not completely cover the first benchmark list, it indicates incomplete recall. If the second benchmark list completely covers the first benchmark list, it is further determined whether the second benchmark list contains redundant medical key information relative to the first benchmark list. If the second benchmark list contains redundant medical key information, it indicates over-recall. If the second benchmark list does not contain redundant medical key information, it indicates that the second benchmark list accurately recalls the information.

[0091] Further optionally, an implementation method for evaluating the memory recall accuracy of a large medical model based on a second truth list and a first truth list to obtain a second evaluation result includes: when the second truth list and the first truth list are structured information, running an automated evaluation script to identify missing medical key information, redundant medical key information, and matching medical key information in the second truth list relative to the first truth list; calculating an accuracy score based on the number of missing medical key information, redundant medical key information, and matching medical key information; and generating a second evaluation result reflecting memory recall capability based on the accuracy score.

[0092] In this context, matched medical key information refers to medical key information that exists in both the first and second truth lists and is semantically equivalent. Missing medical key information refers to medical key information that exists in the first truth list but not in the second truth list, reflecting missed recall. Redundant medical key information refers to medical key information that exists in the second truth list but not in the first truth list, reflecting false recall or hallucination.

[0093] In cases where both the second and first truth lists are structured information, an automated evaluation script can be run to perform fine-grained comparisons based on rule matching, semantic alignment models, or ontology inference engines. This identifies missing, redundant, and matched medical key information in the second truth list relative to the first truth list. The accuracy score can be the precision, recall, and F1 score of memory recall. The number of matched medical key information items is the true positives (TP), the number of missing medical key information items is the false negatives (FN), and the number of redundant medical key information items is the false positives (FP). The implementation methods for calculating precision, recall, and F1 score can be found in the aforementioned embodiments and will not be repeated here.

[0094] The second evaluation results generated based on the accuracy score include, but are not limited to: accuracy score, such as F1 score, a value between 0 and 1 or 0 and 100%; recall capability rating, such as high accuracy, moderate accuracy, or significant risk of hallucination; a list of redundant key medical information used to debug hallucinations in the medical model; missing key medical information and its clinical significance; and a comparison trend with historical versions of the medical model to support continuous iterative evaluation. The second evaluation results directly reflect the information fidelity capability of the medical model in the recall phase and are an important indicator for measuring the reliability of the medical model.

[0095] like Figure 8 As shown, the second truth list is compared with the first truth list, and an accuracy score is calculated as an example.

[0096] Further optionally, an implementation method for evaluating the memory consumption effectiveness of a large medical model based on answer information and key answer information, as well as the answer information and a second truth list, to obtain a third evaluation result includes: using a second decision agent to identify redundant key medical information in the second benchmark list relative to the first benchmark list and marking it as interference information; determining an effectiveness score based on the information completeness of the answer information relative to the key answer information and the overlap between the answer information and the interference information; and generating a third evaluation result reflecting memory consumption ability based on the effectiveness score.

[0097] The core of the above method lies in measuring whether the medical big data model effectively utilizes the correct key medical information, avoids interfering information, and covers the key points of the answer when it "consumes" the internal memory of the medical big data model in the process of generating answer information.

[0098] Specifically, the second decision-making agent is invoked to identify critical medical information that exists in the second truth list but is redundant in the first truth list. This redundant critical medical information is marked as interference information, i.e., erroneous or irrelevant memory content, which may originate from hallucinations or noise. For example, the first truth list includes [M2, M3], and the actually recalled second truth list is [M2, M3, M4], where {M4} is interference information, i.e., a set of noise.

[0099] The effectiveness of memory consumption capacity is comprehensively reflected from two dimensions. One dimension measures whether the answer information covers the expected key points of the answer; information completeness is the ratio of the number of key points covered in the answer information to the total number of key points in the answer. This can be further determined using a semantic matching model. The other dimension assesses the medical big model's ability to consume memory information by judging the combination of the medical big model with interfering information. This involves determining the overlap between the answer information and interfering information, expressed as the ratio of the number of interfering information items in the answer information to the total number of key medical information items mentioned in the answer. In one example, the effectiveness score = information completeness × (1 - overlap). In another example, the effectiveness score is a weighted average of information completeness and overlap. If the overlap is 0, it indicates that there is no interfering information in the answer information, and information completeness is directly used as the effectiveness score. If the overlap is greater than 0, the product of a preset attenuation coefficient and information completeness is used as the effectiveness score, indicating that the effectiveness is significantly reduced when interfering information is present in the answer information.

[0100] The third assessment results, generated based on the effectiveness score, include, but are not limited to: effectiveness score (e.g., 0 to 1 or a percentage score); details of the coverage of key information in the answer; warnings about interfering information citations (e.g., "An answer incorrectly mentions that penicillin can be used for mycoplasma infection"); memory consumption capacity rating (e.g., highly reliable, requires suppression of hallucinations, serious omission of key points); and improvement suggestions (e.g., strengthening the knowledge filtering mechanism during answer generation). The third assessment results directly reflect the effectiveness and safety of the medical big data model in converting internal memory into answer information.

[0101] like Figure 9 As shown, this example provides a method using a second decision-making agent to identify redundant key medical information in a second benchmark list relative to a first benchmark list and mark it as interference information; and to determine a validity score based on the information completeness of the answer information relative to the answer's key points and the degree of overlap between the answer information and the interference information. Figure 9In this process, the system acquires the assessment question, a second truth list, a first truth list, and key answer information. Using a second decision agent, it identifies redundant medical information in the second benchmark list relative to the first benchmark list and marks it as interference. This interference information constitutes a noise set. The noise set is calculated as: second truth list - (intersection of the first and second truth lists). The system then determines whether the first truth list is empty. If empty, it indicates that the assessment question can generate answer information without relying on specific memory. It then determines whether the answer information is combined with the medical information in the noise set. If combined, the validity score is 0; otherwise, it is 2. If the first truth list is not empty, the system determines whether the answer information fully incorporates the medical information in the first truth list. If fully incorporated, it further determines whether the answer information is combined with the information in the noise set. If so, the validity score is 0; otherwise, it is 2. If the answer information does not fully incorporate the key medical information in the first truth list, but only incorporates a portion of it, further evaluation is needed to determine whether the answer information is combined with information concentrated in the noise pool. If yes, the validity score is 0; if no, the validity score is 1. A validity score of 2 indicates accurate use of the required memory without misusing noise; a validity score of 1 indicates omission of key memory but no excessive combination with noise; and a validity score of 0 indicates over-combination.

[0102] In addition to providing the above-mentioned evaluation method for large medical models, this specification also provides a method for generating evaluation datasets for large medical models, which includes: Acquire multiple profile data generated by multiple target users during their interaction with the medical big data model. The profile data includes medical and health records and original dialogue data. Based on the raw dialogue data from multiple profile data, multiple target dialogue data are constructed. Target dialogue data refers to multi-turn dialogue data that covers multiple medical scenarios and multiple user subjects and spans multiple time dimensions. Based on medical and health records from multiple profile data and dialogue data from multiple targets, multiple intelligent agents are used to generate multiple evaluation samples, including: memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data.

[0103] For detailed descriptions of the implementation methods and effects of each step in the above method embodiments, please refer to the foregoing embodiments, which will not be repeated here.

[0104] The methods described in the embodiments of this specification provide the following technical effects: In terms of dialogue data generation, the original dialogue data of users on the medical big data model is used to supplement the original dialogue data with scenarios, expand to multiple user subjects, and extend the time dimension. This constructs multi-turn dialogue data covering multiple medical scenarios and multiple user subjects, spanning the time dimension. This allows the target dialogue data to carry the complex personalized memory of the medical big data model, ensuring the medical consistency of the dialogue data. This makes it suitable for evaluating the personalized question-and-answer effect of the medical big data model. In particular, maintaining multiple user subjects breaks through the limitations of single static profile data and establishes a complex health management assessment sample library that includes multi-user subject interaction (i.e., multi-family member interaction).

[0105] In terms of assessment question generation, multiple intelligent agents collaborate—such as a memory creation agent, a question generation agent, and an AI doctor agent—to conduct closed-loop verification, generating assessment questions that are both professional and adapted to user expression. Question tags are generated for these assessment questions, and the distribution of these tags allows for monitoring of scenario coverage, providing direction for subsequent optimization of the large-scale medical model. The assessment questions cover multi-subject scenarios, including the individual, family members, multiple subjects, and ambiguous subjects. From a consumer scenario perspective, in addition to single-hop information queries, the assessments increasingly cover scenarios combining patient ontology information and family member information, better meeting the demands of personalized effectiveness assessments rather than simple memory information queries.

[0106] In the dimensions of generating the first benchmark list and the first truth list (i.e., Ground Truth), a hybrid annotation system with two rounds of iterative calibration is implemented through medical expert annotation, memory-building agent annotation, and AI doctor agent review; memory-recall agent annotation and first-review agent review; and memory-consuming agent annotation and second-review agent review. Independent verification standards are established for the three stages of memory building, memory recall, and response consumption. In the memory consumption stage, the annotation process is reconstructed through medical consultation logic. For example, medical experts reverse-engineer consultation key information based on assessment questions, use consultation key points and perform memory matching verification to align response key points with clinical thinking.

[0107] In terms of evaluation, the system comprehensively covers the three dimensions of memory construction, memory recall, and memory consumption, providing a holistic assessment of the effectiveness of the memory generation, recall, and consumption environments. At the memory construction layer, a large model is used for decision-making, calculating completeness scores based on a first and second benchmark list, such as precision, recall, and F1 score. At the memory recall layer, an automated script runs to perform precise matching based on the identifiers (ids) of the first and second truth lists, calculating accuracy scores, such as precision, recall, and F1 score. At the memory consumption layer, an automated evaluation is conducted using both answer key information and a noise set as dual reference factors to calculate the effectiveness score of personalized question-answering capabilities.

[0108] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; or the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0109] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0110] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving data sent by B, or it can be understood as A indirectly receiving data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending data directly to A, or it can be understood as B indirectly sending data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0111] Figure 10 A schematic diagram of the structure of a medical large-scale model evaluation device provided as an exemplary embodiment of this specification, such as... Figure 10 As shown, the device includes: an acquisition module 1001, a construction module 1002, a generation module 1003, and an evaluation module 1004; The acquisition module 1001 is used to acquire multiple profile data generated by multiple target users during their interaction with the medical big data model. The profile data includes medical and health records and original dialogue data.

[0112] Module 1002 is used to construct multiple target dialogue data based on the original dialogue data in multiple profile data. The target dialogue data refers to multi-turn dialogue data that covers multiple medical scenarios and multiple user subjects and spans time dimensions.

[0113] The generation module 1003 is used to generate multiple evaluation samples based on medical and health records and multiple target dialogue data from multiple profile data, using multiple intelligent agents. The evaluation samples include: memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data.

[0114] The evaluation module 1004 is used to quantitatively evaluate the memory construction ability, memory recall ability, and memory consumption ability of the medical big model based on memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data from multiple evaluation samples, so as to obtain evaluation results.

[0115] For detailed descriptions of the implementation methods and effects of the above-mentioned device, please refer to the foregoing embodiments, which will not be repeated here.

[0116] Figure 11 This specification illustrates a schematic diagram of an electronic device provided in an exemplary embodiment. This electronic device is applicable to the medical large-scale model evaluation method and the medical large-scale model evaluation dataset generation method provided in the foregoing embodiments. Figure 11 As shown, the electronic device 700 mainly consists of a communication interface 702, a user interface 704, a processor 706, and a memory 708. These components are interconnected and communicate with each other through a system bus, network, or other connection mechanism 410. The communication interface 702 enables the device 700 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 702 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 702 can also be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi (Wireless Fidelity), Bluetooth, Global Positioning System (GPS), or wide-area wireless interface such as WiMAX (Wireless Maximum) or LTE (Long Term Evolution). Of course, the communication interface 702 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 702 may also include multiple physical communication interfaces, such as a Wi-Fi interface, a Bluetooth interface, and a wide-area wireless interface.

[0117] User interface 704 includes receiving user input and providing output to the user. Therefore, user interface 704 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), LED (Light Emitting Diode), display using DLP (Digital Light Processing) technology, printer, and other known or future similar devices. User interface 704 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other known or future similar devices. In some embodiments, user interface 704 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, electronic device 700 may support remote access from other devices via communication interface 702 or another physical interface (not shown). User interface 704 can be configured to receive user input, the position and movement of which can be indicated by an indicator or cursor described herein. User interface 704 can also be configured as a display device for rendering or displaying text fragments.

[0118] Processor 706 may include one or more general-purpose processors and / or special-purpose processors. Memory 708 may include one or more volatile and / or non-volatile memory components and may be integrated wholly or partially with processor 706. Memory 708 may include removable and non-removable components.

[0119] The processor 706 is capable of executing program instructions 718 (e.g., compiled or uncompiled program logic and / or machine code) stored in memory 708 to perform the various functions described herein.

[0120] Memory 708 may contain non-transitory computer-readable media, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Memory 708 stores program instructions that, when executed by device 700, enable device 700 to perform any of the methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Processor 706 executing program instructions 718 may cause processor 706 to use data 712.

[0121] For example, program instructions 718 may include an operating system 722 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 700 and one or more applications 720 (e.g., a browser, social application, or game application). Similarly, data 712 may include operating system data 716 and application data 714. Operating system data 716 is primarily accessible to the operating system 722, while application data 714 is primarily accessible to one or more applications 720. Application data 714 may reside in a file system visible or hidden from the user of device 700.

[0122] Application 720 can communicate with operating system 722 through one or more application programming interfaces (APIs). These APIs help application 720 read and / or write application data 714, transmit or receive information via communication interface 702, receive or display information on user interface 704, etc.

[0123] In some terminology, application 720 may be simply referred to as "app". Furthermore, application 720 can be downloaded to device 700 through one or more online app stores or app markets. However, applications can also be installed on device 700 in other ways, such as through a web browser or a physical interface on electronic device 700 (e.g., a USB port).

[0124] Accordingly, embodiments of this specification also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Accordingly, embodiments of this specification also provide a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above-described method embodiments. It should be understood that each step or combination of steps in the above-described method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above-described method embodiments.

[0125] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes said element.

[0126] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0127] The terminology used in the embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “said,” and “the” used in the embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one. “A plurality” generally includes at least two, but does not exclude the inclusion of at least one.

[0128] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0129] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for evaluating a large-scale medical model, characterized in that, include: Acquire multiple profile data generated by multiple target users during their interaction with the medical big data model, the profile data including medical and health records and original dialogue data; Based on the original dialogue data in the multiple profile data, multiple target dialogue data are constructed. The target dialogue data refers to multi-turn dialogue data that covers multiple medical scenarios and multiple user subjects and spans time dimensions. Based on the medical and health records in the multiple profile data and the multiple target dialogue data, multiple evaluation samples are generated using multiple intelligent agents. The evaluation samples include: memory construction evaluation data, memory recall evaluation data and memory consumption evaluation data. Based on the memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data from the multiple evaluation samples, the memory construction ability, memory recall ability, and memory consumption ability of the medical big model are quantitatively evaluated to obtain the evaluation results.

2. The method according to claim 1, characterized in that, Based on the raw dialogue data from the multiple profile data, multiple target dialogue data are constructed, including: For any given original dialogue data, medical scene recognition is performed on the original dialogue data to obtain a first medical scene; If the first medical scenario does not fully cover the preset medical scenario, then supplementary dialogue data corresponding to the target medical scenario is generated based on the target medical scenario that is not covered in the preset medical scenario and other original dialogue data. The original dialogue data and the supplementary dialogue data are integrated into the target dialogue data corresponding to the original dialogue data.

3. The method according to claim 2, characterized in that, Based on the target medical scenario not covered in the preset medical scenario and other original dialogue data, supplementary dialogue data corresponding to the target medical scenario is generated, including: Medical scene recognition is performed on the other original dialogue data to obtain a second medical scene; If the second medical scenario includes the target medical scenario, then identify the dialogue segments in the other original dialogue data that are associated with the target medical scenario; rewrite the user subject of the dialogue segment to be consistent with the user subject of the original dialogue data to obtain the supplementary dialogue data; If the second medical scenario does not include the target medical scenario, then a dialogue generation agent is used to generate the supplementary dialogue data based on the description information of the target medical scenario, the user subject of the original dialogue data, and multiple time dimensions.

4. The method according to claim 1, characterized in that, Based on the medical and health records in the multiple profile data and the multiple target dialogue data, multiple assessment samples are generated using multiple intelligent agents, including: For any target user, a memory-based intelligent agent is constructed, and multiple key medical information is extracted from the target user's target dialogue data to form a first benchmark list, which serves as the memory construction evaluation data. Using a question-generating intelligent agent, multiple assessment questions are generated based on the target user's medical and health records and the first benchmark list; For any assessment question, obtain the key consultation points generated by medical experts for that assessment question; Using a memory recall agent, at least some key medical information that supports the consultation key information is recalled from the first benchmark list to form a first truth value list, which serves as memory recall evaluation data. Using a memory-consuming intelligent agent, the key points of the answers to the evaluation questions are generated based on the first truth value list, and the evaluation questions and their key points of answers are used as memory-consuming evaluation data.

5. The method according to claim 4, characterized in that, Using a question-generating agent, based on the target user's medical and health records and the first benchmark list, multiple assessment questions are generated, including: The problem-generating agent generates multiple assessment questions based on a variety of preset problem types and multiple key medical information in the first benchmark list. Dynamic monitoring data and static basic information are obtained from the target user's medical and health records, wherein the dynamic monitoring data includes abnormal indicators; A problem-generating agent is used to generate multiple evaluation questions based on abnormal indicators in the dynamic monitoring data and the static basic information.

6. The method according to claim 5, characterized in that, Also includes: The question-generating agent is invoked to identify multiple question information and their types from the target user's original dialogue data; From the multiple sets of question information, select the question information whose question type matches the preset question type as the evaluation question.

7. The method according to any one of claims 1-6, characterized in that, The memory construction evaluation data includes a first benchmark list, the memory recall evaluation data includes a first truth value list, and the memory consumption evaluation data includes evaluation questions and key answer information. Based on the memory construction assessment data, memory recall assessment data, and memory consumption assessment data from the multiple assessment samples, the memory construction ability, memory recall ability, and memory consumption ability of the medical big data model are quantitatively assessed to obtain assessment results, including: Based on the assessment question, the medical big data model is triggered to use the target dialogue data corresponding to the assessment question as a memory context for medical question-and-answer processing. The medical question-and-answer processing includes: constructing a second benchmark list from the memory context, recalling a second truth value list from the second benchmark list, and generating answer information based on the second truth value list. Based on the first benchmark list and the second benchmark list, the first truth value list and the second truth value list, as well as the answer key information and the answer information, the memory construction ability, memory recall ability and memory consumption ability of the medical big model are quantitatively evaluated to obtain the evaluation results.

8. The method according to claim 7, characterized in that, Based on the first and second benchmark lists, the first and second truth lists, and the key information of the responses and the answer information, the memory construction ability, memory recall ability, and memory consumption ability of the medical big data model are quantitatively evaluated to obtain evaluation results, including: Based on the second benchmark list and the first benchmark list, the memory construction integrity of the medical big model is evaluated to obtain the first evaluation result; Based on the second truth list and the first truth list, the memory recall accuracy of the medical big model is evaluated to obtain a second evaluation result; Based on the answer information and the key points of the answer, as well as the answer information and the second truth value list, the memory consumption effectiveness of the medical big model is evaluated to obtain a third evaluation result.

9. The method according to claim 8, characterized in that, Based on the second benchmark list and the first benchmark list, the memory construction integrity of the medical large model is evaluated to obtain a first evaluation result, including: The first decision agent is invoked to analyze the missing medical key information, redundant medical key information, and matching medical key information in the second benchmark list relative to the first benchmark list. Based on the number of missing medical key information, redundant medical key information, and matching medical key information, a completeness score is calculated. Based on the integrity score, a first assessment result reflecting the ability to build memory is generated.

10. The method according to claim 8, characterized in that, Based on the second truth list and the first truth list, the memory recall accuracy of the medical big data model is evaluated to obtain a second evaluation result, including: When the second truth list and the first truth list are structured information, an automated evaluation script is run to identify medical key information missing, redundant, and matching in the second truth list relative to the first truth list. An accuracy score is calculated based on the number of missing medical key information, redundant medical key information, and matching medical key information. Based on the accuracy score, a second assessment result reflecting memory recall ability is generated.

11. The method according to claim 8, characterized in that, Based on the answer information and the key points of the answer, as well as the answer information and the second truth value list, the memory consumption effectiveness of the medical big data model is evaluated to obtain a third evaluation result, including: Using a second decision-making agent, redundant medical key information in the second benchmark list relative to the first benchmark list is identified and marked as interference information; A validity score is determined based on the completeness of the answer information relative to the key points of the answer, and the degree of overlap between the answer information and the distracting information. Based on the aforementioned effectiveness score, a third assessment result reflecting memory consumption ability is generated.

12. A method for generating an evaluation dataset for a large medical model, characterized in that, include: Acquire multiple profile data generated by multiple target users during their interaction with the medical big data model, the profile data including medical and health records and original dialogue data; Based on the original dialogue data in the multiple profile data, multiple target dialogue data are constructed. The target dialogue data refers to multi-turn dialogue data that covers multiple medical scenarios and multiple user subjects and spans time dimensions. Based on the medical and health records in the multiple profile data and the multiple target dialogue data, multiple evaluation samples are generated using multiple intelligent agents. The evaluation samples include: memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data.

13. A testing device for a large medical model, characterized in that, include: Acquisition module, construction module, generation module, and evaluation module; The acquisition module is used to acquire multiple profile data generated by multiple target users during their interaction with the medical big data model. The profile data includes medical and health records and original dialogue data. The construction module is used to construct multiple target dialogue data based on the original dialogue data in the multiple profile data. The target dialogue data refers to multi-turn dialogue data that covers multiple medical scenarios and multiple user subjects and spans time dimensions. The generation module is used to generate multiple evaluation samples based on the medical and health records in the multiple profile data and the multiple target dialogue data using multiple intelligent agents. The evaluation samples include: memory construction evaluation data, memory recall evaluation data and memory consumption evaluation data. The evaluation module is used to quantitatively evaluate the memory construction ability, memory recall ability, and memory consumption ability of the medical big model based on the memory construction evaluation data, memory recall evaluation data, and memory consumption evaluation data in the multiple evaluation samples, so as to obtain the evaluation results.

14. An electronic device, characterized in that, include: A memory and a processor; the memory being used to store one or more computer instructions; the processor being used to execute the one or more computer instructions for: performing the steps of the method according to any one of claims 1-11 and 12.

15. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it is able to perform the steps of the method described in any one of claims 1-11 and 12.

16. A computer program product, characterized in that, include: A computer program / instruction that, when executed by a processor, enables the implementation of the steps in the method described in any one of claims 1-11 and 12.

Images