Large medical model testing method and device

Abstract

The invention provides a medical big model testing method and device, and relates to the technical field of data processing, and the method comprises the steps: constructing a dynamic medical knowledge graph, calculating the knowledge half-life period of a medical knowledge entity based on the dynamic medical knowledge graph, and obtaining a medical big model according to the relation among the current time, the release time attribute and the knowledge half-life period; calculating the timeliness score of each medical knowledge entity, screening the medical knowledge entities with the timeliness scores lower than a preset timeliness threshold value, and generating a dynamic test case; inputting the dynamic test case into a to-be-tested medical large model to perform a multi-modal timeliness test, screening out a high-value failure sample from a test result, and extracting an output conclusion of the medical large model for the high-value failure sample, performing semantic matching on the output conclusion and each historical version of medical knowledge entity in the medical knowledge entity version sequence, and positioning a failure knowledge node; and performing a pressure test on the medical large model.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for testing large medical models. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale medical models have shown enormous application potential in areas such as clinical assisted diagnosis, medical question answering, and medical record analysis. These models typically rely on massive amounts of medical data for pre-training to acquire general knowledge and reasoning abilities in the medical field. However, medical knowledge is highly time-sensitive and dynamically evolving; clinical treatment guidelines, drug indications, disease classification standards, and surgical procedures are constantly being updated and iterated.

[0003] Currently, the testing and evaluation of large-scale medical models mainly rely on static benchmark datasets. These test sets are usually constructed manually or extracted from existing question banks, and their content remains fixed after construction.

[0004] This testing method has significant limitations. First, static test sets cannot reflect the timely updates of medical knowledge and are difficult to test the model's performance when faced with newly published medical consensus or obsolete knowledge. For example, the model may provide outdated treatment recommendations based on outdated guidelines, and static testing cannot identify such timeliness errors. Second, relying on manual updates of test cases is inefficient and costly, and it is difficult to cover the rapidly growing literature and new knowledge in various medical sub-disciplines. In addition, existing testing methods usually only focus on the accuracy of the model's output results and lack a systematic assessment of the model's cognitive adaptability when faced with conflicts between new and old knowledge or the evolution of medical viewpoints.

[0005] Therefore, how to efficiently and automatically test the timeliness of large medical models and evaluate their ability to adapt to knowledge evolution is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] This invention provides a method and apparatus for testing large medical models, which solves the problem in the prior art of how to efficiently and automatically test the timeliness of large medical models and evaluate their ability to adapt to knowledge evolution.

[0007] This invention provides a method for testing large-scale medical models, comprising: A dynamic medical knowledge graph is constructed, which contains medical knowledge entities. Each medical knowledge entity has a publication time attribute and an evidence level attribute. Different medical knowledge entities of the same knowledge point constitute a version sequence. The medical knowledge entities in the version sequence are associated through an evolution path. Based on the dynamic medical knowledge graph, the knowledge half-life of the medical knowledge entities is calculated. According to the relationship between the current time, the publication time attribute and the knowledge half-life, the timeliness score of each medical knowledge entity is calculated, and medical knowledge entities with timeliness scores lower than the preset timeliness threshold are filtered out to generate dynamic test cases. The dynamic test cases are input into the medical big model to be tested for multimodal timeliness testing. High-value failure samples are selected from the test results. The output conclusions of the medical big model for the high-value failure samples are extracted. The version sequence corresponding to the high-value failure samples is determined. The output conclusions are semantically matched with the medical knowledge entities of each historical version in the version sequence to locate the failure knowledge nodes. Based on the evolution path of the medical knowledge entities corresponding to the high-value failure samples, a cognitive evolution stress test environment is constructed to perform stress tests on the medical big model and evaluate the cognitive resilience index of the medical big model. Based on the cognitive resilience index of the medical big data model and the failed knowledge nodes, suggestions for repairing the timeliness defects of the medical big data model are generated.

[0008] According to a medical large-scale model testing method provided by the present invention, the construction of a dynamic medical knowledge graph includes: Unstructured text is obtained from multiple medical data sources, medical knowledge entities are parsed out through relation extraction, and the publication time and evidence level of each medical knowledge entity are extracted as metadata. At the same time, the medical discipline classification to which each medical knowledge entity belongs is identified. Based on the impact factor of the source journal, the authority level of the publishing institution, and the document type of the medical knowledge entity, the temporal influence weight of each medical knowledge entity is calculated. The medical knowledge entities, their metadata, medical discipline classifications, and temporal influence weights are stored in the graph database. The same knowledge point published at different times in the graph database is identified, and version sequences and evolution paths between the medical knowledge entities are established to form the dynamic medical knowledge graph.

[0009] According to a medical large-scale model testing method provided by the present invention, based on the dynamic medical knowledge graph, the knowledge half-life of the medical knowledge entities is calculated, including: From the dynamic medical knowledge graph, the historical update frequency of medical knowledge entities under each medical discipline category is statistically analyzed, and the discipline update coefficient corresponding to each medical discipline category is determined based on the historical update frequency. The theoretical half-life of the medical knowledge entity is calculated based on the discipline update coefficient, benchmark period, and evidence level attribute of the medical knowledge entity to which it belongs. By statistically analyzing the actual average time from the publication to the update of medical knowledge entities under each medical discipline category at a fixed period, the actual half-life of each medical discipline category is obtained. When there is a deviation between the actual half-life and the theoretical half-life of the previous cycle, the subject update coefficient is calibrated according to the deviation ratio, and the knowledge half-life of the medical knowledge entity is recalculated using the calibrated subject update coefficient.

[0010] According to the medical large-scale model testing method provided by the present invention, the process of calculating the knowledge half-life further includes: The number and update speed of medical knowledge entities under the medical discipline classification to which the medical knowledge entities belong are statistically analyzed from the dynamic medical knowledge graph. The temporal influence weights of each medical knowledge entity under the medical discipline classification are extracted from the dynamic medical knowledge graph, and the consensus subversion degree is calculated by comparing the conclusion differences between the current version of the medical knowledge entity and the historical version of the medical knowledge entity for the same knowledge point. Based on the update quantity, update speed, temporal influence weight, and consensus subversion degree, calculate the knowledge update intensity index of the medical discipline classification; The knowledge update intensity index is monitored in real time. When the knowledge update intensity index exceeds a preset threshold, the current knowledge half-life of each medical knowledge entity under the medical discipline classification is shortened and adjusted using the knowledge update intensity index. Alternatively, when feedback is received regarding the high-value failed sample, the medical knowledge entity corresponding to the high-value failed sample is extracted, the knowledge half-life of the medical knowledge entity is shortened and adjusted, and the timeliness score of the medical knowledge entity is recalculated.

[0011] According to the medical large-scale model testing method provided by the present invention, the step of screening high-value failure samples from the test results includes: For single-modal data, calculate the semantic novelty between the output content of the medical big model and the current version of medical knowledge entities in the dynamic medical knowledge graph; When the semantic novelty is lower than the modality-specific novelty threshold, and the self-confidence of the medical big model in the output content is higher than the preset confidence threshold, the dynamic test case is determined to have failed. Alternatively, for multimodal data, the output conclusions of the medical model on the same medical problem under different modalities are compared. When the output conclusions of different modalities point to contradictory treatment plans, the dynamic test case is determined to have failed. Dynamic test cases that are determined to have failed are marked as high-value failure samples.

[0012] According to a medical large-scale model testing method provided by the present invention, the dynamic test cases are input into the medical large-scale model to be tested for multimodal timeliness testing, including: When the dynamic test cases are text data, the dynamic test cases containing text data are input into the medical big model to obtain the output content. A dual-channel text analysis network is constructed, and the output content and the text content of the corresponding current version medical knowledge entity in the dynamic medical knowledge graph are input respectively. The semantic novelty score between the two is calculated through an attention mechanism. When the dynamic test case is image data, the dynamic test case containing image data is input into the medical big model to obtain image operation suggestions. Key feature points are extracted from the image operation suggestions. The key feature points are spatiotemporally aligned and compared with the standard surgical procedure features corresponding to the current version of the medical knowledge entity in the dynamic medical knowledge graph, and the feature matching degree is calculated. When the dynamic test case is diagnosis-related data, the dynamic test case containing the diagnosis description is input into the medical big model to obtain the diagnosis description. The diagnosis description is then mapped to multiple versions of the standard system through a version converter and compared with the standard code corresponding to the current version of the medical knowledge entity in the dynamic medical knowledge graph.

[0013] According to a medical large-scale model testing method provided by the present invention, the method for locating failure knowledge nodes includes: Extract the output conclusions of the medical big model from the high-value failure samples; Following the evolution path of the medical knowledge entities corresponding to the high-value failed samples in the dynamic medical knowledge graph, the output conclusion is semantically matched with the content of the medical knowledge entities in each historical version of the version sequence. The historical version of medical knowledge entity with the highest semantic match to the output conclusion is marked as the invalid knowledge node.

[0014] According to the medical large-scale model testing method provided by the present invention, the method for constructing a cognitive evolution stress testing environment based on the evolution path of the medical knowledge entities corresponding to the high-value failure samples includes: From the dynamic medical knowledge graph, based on the evolution path of the medical knowledge entities corresponding to the high-value failed samples, the version sequence of each medical knowledge entity under the medical discipline classification to which the medical knowledge entity belongs is extracted; Based on the release time attributes and content changes of each medical knowledge entity in the version sequence, fit the cognitive evolution curve of the medical discipline classification; The time points in the cognitive evolution curve where version changes occur are identified as key time nodes.

[0015] According to the medical large-scale model testing method provided by the present invention, the construction of the cognitive evolution stress testing environment further includes: The version replacement frequency of each medical knowledge entity under the medical discipline classification is statistically analyzed from the dynamic medical knowledge graph to obtain the knowledge evolution speed. The degree of consensus subversion is obtained by calculating the degree of difference between the conclusions of the current version of medical knowledge entities and the historical version of medical knowledge entities from the evolutionary path. Based on the speed of knowledge evolution and the degree of consensus disruption, the knowledge pressure parameter corresponding to each key time node is calculated.

[0016] According to the medical large-scale model testing method provided by the present invention, the construction of the cognitive evolution stress testing environment further includes: The knowledge pressure parameter is compared with a preset pressure level threshold. Key time points where the knowledge pressure parameter is below the first threshold are classified as mild pressure levels. Key time points where the knowledge pressure parameter falls between the first and second thresholds are classified as medium-pressure levels. Key time points where the knowledge pressure parameter exceeds the second threshold are classified as severe pressure levels.

[0017] According to a medical large-scale model testing method provided by the present invention, stress testing is performed on the medical large-scale model, including: For the key time points of the mild stress level, medical knowledge entities of adjacent versions before and after the key time points are extracted from the evolution path to construct knowledge accumulation test questions; For the key time points of the moderate pressure level, medical knowledge entities with differing conclusions at the key time points are extracted from the evolutionary path, and academic controversy test questions are constructed. For the key time points of the severe pressure level, extract the medical knowledge entities from the evolution path where the conclusions of the key time points are overturned, and construct paradigm shift test problems. According to the time sequence of the key time nodes, the corresponding test questions are input into the medical big model in turn to achieve stress testing of the medical big model.

[0018] According to a medical large-scale model testing method provided by the present invention, the evaluation of the cognitive resilience index of the medical large-scale model includes: The knowledge transfer rate of the medical big model is obtained by statistically analyzing the proportion of test rounds in which the current version of medical knowledge entity conclusions are adopted in the stress test out of the total test rounds. The conflict resolution accuracy of the medical model is obtained by statistically analyzing the proportion of times the current version of the medical knowledge entity is selected as the answer in the test questions corresponding to the severe stress level, relative to the total number of test questions under the severe stress level.

[0019] According to a medical large-scale model testing method provided by the present invention, the cognitive resilience index for evaluating the medical large-scale model further includes: The medical knowledge entity versions corresponding to the conclusions output by the medical big model for each test question during the stress test are arranged in chronological order according to the key time nodes to form a cognitive trajectory. By comparing the cognitive trajectory with the cognitive evolution curve, key time points in which the output conclusions of the medical big data model lag behind the cognitive evolution curve are identified as cognitive lag points. Based on the knowledge transfer rate, the conflict resolution accuracy, and the cognitive lag node, the cognitive resilience index of the medical big model is obtained.

[0020] According to a medical large-scale model testing method provided by the present invention, the step of generating timeliness defect repair suggestions for the medical large-scale model includes: Obtain the publication time attribute and the medical discipline classification to which the failed knowledge node belongs; At the data level, it is recommended to supplement the training data with medical knowledge entities published after the publication time attribute that belong to the same medical discipline classification as the expired knowledge node. At the model level, incremental training is planned for the same medical discipline category as the aforementioned failed knowledge nodes; At the knowledge base level, the successor version of the medical knowledge entity associated with the failed knowledge node is obtained from the evolution path, and an update of the successor version of the medical knowledge entity is prompted.

[0021] The present invention also provides a large-scale medical model testing device, comprising: A construction module is used to construct a dynamic medical knowledge graph. The dynamic medical knowledge graph contains medical knowledge entities. Each medical knowledge entity has a publication time attribute and an evidence level attribute. Different medical knowledge entities of the same knowledge point constitute a version sequence. Each medical knowledge entity in the version sequence is associated through an evolution path. The calculation module is used to calculate the knowledge half-life of the medical knowledge entities based on the dynamic medical knowledge graph, calculate the timeliness score of each medical knowledge entity according to the relationship between the current time, the publication time attribute and the knowledge half-life, and filter out medical knowledge entities with timeliness scores lower than a preset timeliness threshold, and generate dynamic test cases. The localization module is used to input the dynamic test cases into the medical big model to be tested for multimodal timeliness testing, filter out high-value failure samples from the test results, extract the output conclusions of the medical big model for the high-value failure samples, determine the version sequence corresponding to the high-value failure samples, and perform semantic matching between the output conclusions and the medical knowledge entities of each historical version in the version sequence to locate the failure knowledge nodes. The testing module is used to construct a cognitive evolution stress testing environment based on the evolution path of the medical knowledge entities corresponding to the high-value failure samples, perform stress testing on the medical big model, and evaluate the cognitive resilience index of the medical big model. The repair module is used to generate repair suggestions for the timeliness defects of the medical big model based on the cognitive elasticity index of the medical big model and the failed knowledge nodes.

[0022] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the medical large model testing method as described above.

[0023] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the medical large-scale model testing method as described above.

[0024] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the medical large-scale model testing method as described above.

[0025] The medical large-scale model testing method and apparatus provided by this invention constructs a dynamic medical knowledge graph containing version sequences and evolution paths, and introduces a knowledge half-life calculation mechanism. This allows for the automatic selection of easily expired knowledge and the generation of dynamic test cases based on the timeliness scores of medical entities, thus eliminating the lag of manually maintaining static test sets and achieving precise capture of new and expired medical knowledge. By semantically matching high-value failed samples with historical versions of medical knowledge entities, the specific obsolete knowledge nodes causing model failure can be accurately located. Furthermore, the cognitive evolution stress testing environment constructed based on the evolution path can simulate real-world scenarios of medical knowledge replacement, quantitatively evaluate the model's cognitive resilience in the face of knowledge iteration, and generate targeted suggestions for repairing timeliness defects, effectively improving the safety, reliability, and knowledge update response speed of the medical large-scale model in clinical applications. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the medical large-scale model testing method provided by the present invention; Figure 2 This is a schematic diagram of the medical large-scale model testing device provided by the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The medical large-scale model testing method provided in this invention can be executed by an electronic device with data processing capabilities, such as a server, computer, workstation, or other computing device with a processor and memory. In some embodiments, the executing entity of this method can be a server cluster deployed with a testing system, or a standalone device configured with corresponding software modules.

[0030] Figure 1 This is a flowchart illustrating the medical large-scale model testing method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 110: Construct a dynamic medical knowledge graph. The dynamic medical knowledge graph contains medical knowledge entities. Each medical knowledge entity has a publication time attribute and an evidence level attribute. Different medical knowledge entities of the same knowledge point constitute a version sequence. Each medical knowledge entity in the version sequence is associated through an evolution path. In this application, a dynamic medical knowledge graph refers to a structured knowledge storage system that reflects the evolution of medical knowledge over time. Unlike traditional static knowledge graphs, dynamic medical knowledge graphs not only store the medical knowledge itself, but also record the temporal dimension information and evolutionary relationships of the knowledge.

[0031] Medical knowledge entities are the basic building blocks of a dynamic medical knowledge graph, and may include, but are not limited to: disease entities, such as hypertension and diabetes; drug entities, such as antihypertensive drugs and antibiotics; treatment plan entities, such as surgical plans and medication plans; diagnostic standard entities, such as disease diagnostic thresholds and grading standards; and medical guideline entities, such as clinical practice guidelines and expert consensus. Each medical knowledge entity carries specific medical knowledge content and is accompanied by metadata information describing that knowledge.

[0032] The publication date attribute refers to the official publication or effective date of the original medical literature, guidelines, or standards corresponding to the medical knowledge entity. This attribute is used to identify the time point in the creation of medical knowledge, providing a time benchmark for subsequent timeliness assessments.

[0033] The level of evidence refers to the degree of reliability of the clinical evidence upon which a medical knowledge entity relies. In the medical field, different types of research provide varying degrees of evidence strength.

[0034] For example, randomized controlled trials typically provide a higher level of evidence than observational studies, which in turn provide a higher level of evidence than expert opinions. The level of evidence can be categorized using a grading system to characterize the credibility of medical knowledge.

[0035] A version sequence refers to a sequence of multiple medical knowledge entities published at different times for the same medical knowledge point, arranged in chronological order.

[0036] More specifically, the version sequence includes the current version of medical knowledge entities and historical version medical knowledge entities.

[0037] The current version of medical knowledge entities refers to medical knowledge that is currently recognized as correct and authoritative by the medical community in the version sequence. It represents the latest state of understanding of the knowledge point. For example, the latest diagnostic criteria for hypertension (systolic blood pressure ≥130 mmHg) released in 2023 is the current version.

[0038] Historical medical knowledge entities refer to older versions of medical knowledge that were once considered correct in the version sequence, but have been updated or revised as medical research has progressed. This knowledge was valid at the time of its publication, but has now been replaced by newer versions and archived. For example, the diagnostic criteria for hypertension (systolic blood pressure ≥140 mmHg) before 2017 are historical versions.

[0039] The current version of medical knowledge entities and the historical version of medical knowledge entities together constitute a complete version sequence of the same knowledge point. The evolution path records the process of medical knowledge changing from old to new. The current version is used to determine whether the model output is outdated, while the historical version is used to locate which period of old knowledge the model uses.

[0040] Medical knowledge is dynamic and evolving; the same knowledge point may be updated, revised, or even overturned as medical research deepens.

[0041] For example, the diagnostic criteria for a certain disease may be published in different versions of guidelines in different years, and the medical knowledge entities corresponding to these different versions of guidelines constitute the version sequence of that knowledge point.

[0042] Evolutionary paths refer to the relationships established between medical knowledge entities in adjacent or related versions within a version sequence.

[0043] Evolutionary paths record the trajectory of medical knowledge from old to new versions, including changes in knowledge content, revisions to conclusions, and updates to standards. By tracing these evolutionary paths, the historical evolution of medical knowledge can be tracked, providing a basis for subsequent timeliness testing and defect attribution.

[0044] Step 120: Based on the dynamic medical knowledge graph, calculate the knowledge half-life of the medical knowledge entities, calculate the timeliness score of each medical knowledge entity according to the relationship between the current time, the publication time attribute and the knowledge half-life, and filter out medical knowledge entities with timeliness scores lower than the preset timeliness threshold, and generate dynamic test cases. In this application, knowledge half-life is an indicator used to quantify the timeliness of medical knowledge, representing the time period required for a medical knowledge entity to decrease in credibility to half of its initial level after publication.

[0045] The concept of knowledge half-life, borrowed from the idea of ​​the half-life of radioactive elements in physics, is used to describe the process by which medical knowledge gradually loses its timeliness over time. The rate of knowledge updating varies across different medical fields; therefore, the knowledge half-life differs for different medical knowledge entities.

[0046] When calculating the half-life of knowledge, factors such as the medical discipline classification to which the medical knowledge entity belongs and the level of evidence attributes can be taken into account.

[0047] Medical disciplines are classified to reflect the updating characteristics of the knowledge in their respective fields. For example, knowledge in fields such as oncology and infectious diseases is updated relatively quickly, while knowledge in basic medical fields such as anatomy and physiology is relatively stable. The level of evidence reflects the reliability of knowledge; knowledge with a high level of evidence typically has a longer validity period.

[0048] Timeliness score is a quantitative indicator based on the knowledge half-life calculation, used to assess the validity of medical knowledge entities at the current point in time. In one possible implementation, the timeliness score can be calculated using the following formula: Score(t) = E × [1 - (t_current - t_publish) / T_half] × C; Among them, Score(t) represents the timeliness score, E represents the evidence level coefficient, t_current represents the current time, t_publish represents the publication time of the medical knowledge entity, T_half represents the knowledge half-life, and C represents the confidence decay factor. The value rule of the confidence decay factor C is as follows: when the knowledge has not passed the half-life (i.e., t_current - t_publish < T_half), C = 1; when the knowledge exceeds the half-life, the value of C starts to decrease from 1, indicating that its credibility decreases slowly rather than disappearing instantly.

[0049] When a medical knowledge entity is newly published, its timeliness score is relatively high; as time goes by, the timeliness score gradually decreases; when the elapsed time approaches or exceeds the knowledge half-life, the timeliness score will decrease significantly.

[0050] The preset timeliness threshold is the critical value used to judge whether a medical knowledge entity needs to be tested. When the timeliness score of a medical knowledge entity is lower than the preset timeliness threshold, it indicates that the medical knowledge entity may have become obsolete or is about to become obsolete, and needs to be focused on and tested.

[0051] Dynamic test cases are test cases automatically generated according to the medical knowledge entities with relatively low timeliness scores screened out.

[0052] Dynamic test cases can include test questions, test scenarios, etc. related to medical knowledge entities, and are used to verify whether the medical large model to be tested has mastered the latest medical knowledge.

[0053] The dynamic characteristics of dynamic test cases are reflected in that the test cases are dynamically generated according to the real-time state of the knowledge graph and the timeliness score, rather than a pre-fixed static test set.

[0054] Step 130, input the dynamic test cases into the medical large model to be tested for multi-modal timeliness testing, screen out high-value failure samples from the test results, extract the output conclusions of the medical large model for the high-value failure samples, determine the version sequence corresponding to the high-value failure samples, and perform semantic matching between the output conclusions and each historical version medical knowledge entity in the version sequence to locate the failed knowledge nodes; In this application, multi-modal timeliness testing refers to timeliness verification testing for different types of medical data.

[0055] Medical data is multimodal, including textual data such as medical literature and treatment guidelines; imaging data such as medical images and surgical videos; and coded data such as ICD disease codes and surgical codes. Multimodal timeliness testing designs corresponding test schemes and evaluation indicators for different modalities of data.

[0056] High-value failure samples refer to test cases that are judged to have failed during the testing process and have important analytical value.

[0057] The characteristic of high-value failure samples is that the medical big model gives incorrect or outdated output conclusions for these test cases, and these errors are representative and can reflect the typical defects of the medical big model in terms of knowledge timeliness.

[0058] The output conclusions refer to the response content given by the medical big data model to dynamic test cases, which may include diagnostic suggestions, treatment plan recommendations, drug use guidance, disease code output, etc.

[0059] Semantic matching refers to comparing the semantic similarity between the output conclusion of a large medical model and the content of each historical version of medical knowledge entities in a sequence of medical knowledge entity versions. Semantic matching is not a simple string matching process; rather, it uses natural language processing techniques to understand the semantic meaning of the output conclusion and the content of historical versions, and to determine the degree of semantic similarity between the two.

[0060] Historical version medical knowledge entities refer to medical knowledge entities in the version sequence that are not the current version medical knowledge entities. When a new version of medical knowledge is released, the old version medical knowledge entities are archived and saved, forming historical version medical knowledge entities.

[0061] Failed knowledge nodes refer to historical versions of medical knowledge entities that are located through semantic matching and that best match the output conclusions of the large medical model.

[0062] The results of locating the failed knowledge nodes indicate that the large medical model may still be using the medical knowledge from a historical version, rather than the knowledge of the current version of the medical knowledge entity, thus causing the test to fail.

[0063] Step 140: Based on the evolution path of the medical knowledge entities corresponding to the high-value failure samples, construct a cognitive evolution stress test environment, perform stress tests on the medical big model, and evaluate the cognitive resilience index of the medical big model. In this application, the cognitive evolution stress testing environment refers to a testing scenario that simulates the dynamic evolution of medical knowledge. This testing environment is constructed based on the evolutionary paths of medical knowledge entities corresponding to high-value failure samples, reproducing the evolutionary process of medical knowledge from its generation and development to its updating and disruption, and is used to test the adaptability of a large-scale medical model in the face of knowledge changes.

[0064] Stress testing refers to applying varying intensities of knowledge evolution stress to a large-scale medical model within a cognitive evolution stress testing environment, and then observing and evaluating the model's response performance. The purpose of stress testing is to assess the robustness and adaptability of the large-scale medical model under scenarios such as rapid knowledge changes, controversial academic viewpoints, and shifts in medical paradigms.

[0065] Cognitive resilience is a set of indicators used to quantitatively assess the ability of large-scale medical models to cope with knowledge evolution. It reflects the ability of large-scale medical models to promptly adopt new knowledge, correctly handle conflicts between old and new knowledge, and keep pace with the evolution of medical cognition when faced with updates to medical knowledge.

[0066] Step 150: Based on the cognitive resilience index of the medical big model and the failed knowledge nodes, generate suggestions for repairing the timeliness defects of the medical big model.

[0067] In this embodiment, the timeliness defect repair suggestions are generated based on test results and are used to guide the improvement of the medical big data model. These suggestions are generated based on the assessment results of the cognitive resilience index and the location results of failed knowledge nodes, providing targeted repair directions and improvement measures for specific problems exposed in the medical big data model regarding knowledge timeliness.

[0068] Suggestions for fixing timeliness defects can cover multiple levels, including the data level, the model level, and the knowledge base level.

[0069] This application constructs a dynamic medical knowledge graph incorporating temporal information and evolutionary relationships. Based on the knowledge half-life mechanism, it achieves quantitative assessment of the timeliness of medical knowledge and automatic generation of dynamic test cases. Through multimodal timeliness testing and failure knowledge node localization, it accurately identifies the timeliness defects of knowledge in large-scale medical models. Cognitive evolution stress testing enables in-depth assessment of the cognitive resilience of large-scale medical models, ultimately generating targeted remediation suggestions. This method solves the problem that traditional static testing methods cannot assess the timeliness of knowledge in large-scale medical models, providing technical support for the continuous optimization of large-scale medical models and clinical risk control.

[0070] Optionally, the construction of the dynamic medical knowledge graph includes: Unstructured text is obtained from multiple medical data sources, medical knowledge entities are parsed out through relation extraction, and the publication time and evidence level of each medical knowledge entity are extracted as metadata. At the same time, the medical discipline classification to which each medical knowledge entity belongs is identified. Based on the impact factor of the source journal, the authority level of the publishing institution, and the document type of the medical knowledge entity, the temporal influence weight of each medical knowledge entity is calculated. The medical knowledge entities, their metadata, medical discipline classifications, and temporal influence weights are stored in the graph database. The same knowledge point published at different times in the graph database is identified, and version sequences and evolution paths between the medical knowledge entities are established to form the dynamic medical knowledge graph.

[0071] In this application, multiple medical data sources are first connected to obtain raw unstructured medical text; then, relation extraction technology is used to parse medical knowledge entities from the text; next, the publication time and evidence level of each medical knowledge entity are extracted from the text as metadata, and its corresponding medical discipline classification is identified.

[0072] Medical data sources refer to information sources that provide medical knowledge content. These can include medical journal databases, such as PubMed and CNKI Medical Literature Database; clinical practice guideline publishing platforms, such as NCCN Guidelines and Chinese Medical Association Guideline Database; medical standard publishing organizations, such as the World Health Organization ICD Standards Publishing Channel; and official websites of authoritative medical institutions.

[0073] The testing system can obtain data from multiple medical data sources through API calls, data subscriptions, and other methods.

[0074] Unstructured text refers to raw text content obtained from medical data sources, including full-text medical papers, guideline documents, and standard specifications. Unstructured text does not have a predefined data format and requires parsing using natural language processing techniques.

[0075] Relation extraction is a natural language processing technique used to identify and extract entities and the relationships between them from unstructured text. In the medical field, relation extraction can identify disease entities, drug entities, symptom entities, treatment plan entities, etc., and extract the relationships between entities; for example, drug A is used to treat disease B, and symptom C is a typical manifestation of disease D, etc.

[0076] Metadata is data that describes the attributes of medical knowledge entities, including publication time and level of evidence. Publication time can be extracted from information such as the publication date of a document, the publication date of a guideline, and the effective date of a standard.

[0077] The level of evidence can be determined based on factors such as the type of literature, such as randomized controlled trials, cohort studies, case reports, expert opinions, and the quality of the study design.

[0078] Medical disciplines refer to the medical professional fields to which medical knowledge entities belong, such as oncology, cardiology, neurology, surgery, and infectious diseases.

[0079] Medical disciplines can be classified by analyzing information such as the journals in which the literature is published, keywords, and subject terms, or automatically determined by a classification model based on the text content.

[0080] In this embodiment, information such as the source journal impact factor, the authority level of the publishing institution, and the document type of each medical knowledge entity is obtained. Then, these factors are combined through a weighted calculation method to obtain the time-series influence weight of each medical knowledge entity.

[0081] Time-series influence weight is an indicator used to quantify the influence of medical knowledge entities within the medical field. It considers multiple factors: The impact factor of a source journal reflects the academic influence of the journal publishing that medical knowledge. Journals with higher impact factors typically have more rigorous peer review mechanisms, ensuring higher quality research.

[0082] The authority level of the issuing organization reflects the authority of the institution that publishes the medical knowledge. For example, knowledge published by the World Health Organization, national medical associations, and top medical research institutions usually carries higher authority.

[0083] The type of literature reflects the type of research that produces that medical knowledge. In the evidence-based medicine system, different types of studies provide different strengths of evidence: systematic reviews and meta-analyses are generally stronger than randomized controlled trials, randomized controlled trials are stronger than observational studies, and observational studies are stronger than expert opinions.

[0084] The temporal influence weight can be obtained by weighting the above factors. The higher the weight value, the greater the influence of the medical knowledge entity.

[0085] In this embodiment, the graph database is a database system used to store dynamic medical knowledge graph data, and can be implemented using a graph database or a relational database that supports time-series queries.

[0086] The graph database stores medical knowledge entities and their metadata, subject classifications, time-series influence weights, and other information.

[0087] When identifying medical knowledge entities that share the same knowledge point but were published at different times, the testing system can use methods such as semantic similarity calculation, knowledge point identifier matching, and citation relationship analysis to determine whether different medical knowledge entities belong to different versions of the same knowledge point.

[0088] When establishing the version sequence, medical knowledge entities belonging to the same knowledge point are arranged in chronological order of publication. When establishing the evolution path, the test system creates associations between adjacent versions in the version sequence and records content change information between versions.

[0089] This application implements a technical process for automatically constructing a dynamic medical knowledge graph from multi-source medical data, laying a data foundation for subsequent knowledge timeliness assessment and test case generation.

[0090] Optionally, based on the dynamic medical knowledge graph, the knowledge half-life of the medical knowledge entities is calculated, including: From the dynamic medical knowledge graph, the historical update frequency of medical knowledge entities under each medical discipline category is statistically analyzed, and the discipline update coefficient corresponding to each medical discipline category is determined based on the historical update frequency. The theoretical half-life of the medical knowledge entity is calculated based on the discipline update coefficient, benchmark period, and evidence level attribute of the medical knowledge entity to which it belongs. By statistically analyzing the actual average time from the publication to the update of medical knowledge entities under each medical discipline category at a fixed period, the actual half-life of each medical discipline category is obtained. When there is a deviation between the actual half-life and the theoretical half-life of the previous cycle, the subject update coefficient is calibrated according to the deviation ratio, and the knowledge half-life of the medical knowledge entity is recalculated using the calibrated subject update coefficient.

[0091] In this application, the version sequence in the dynamic medical knowledge graph is traversed, the historical update frequency of medical knowledge entities under each medical discipline category is statistically analyzed, and then the initial value of the discipline update coefficient corresponding to each medical discipline category is determined based on the statistical results.

[0092] In this embodiment, historical update frequency refers to the frequency with which medical knowledge entities under a specific medical discipline are updated within a historical time period. Historical update frequency can be calculated by statistically analyzing the time intervals between version changes in a version sequence.

[0093] The subject update coefficient is a parameter reflecting the speed at which medical knowledge is updated. A higher coefficient indicates faster knowledge updates within the discipline. The initial value of the subject update coefficient can be determined based on historical update frequencies or set by field experts according to the characteristics of the discipline.

[0094] More specifically, the subject update coefficient of the medical discipline to which the medical knowledge entity belongs, the preset baseline period, and the evidence level attribute of the medical knowledge entity are obtained, and then the theoretical half-life of the medical knowledge entity is calculated according to the preset calculation formula.

[0095] In this embodiment, the baseline period is the fundamental time parameter for calculating the half-life of knowledge, representing the reference validity period of medical knowledge under standard conditions. The baseline period can be set according to the overall knowledge update characteristics of the medical field.

[0096] The theoretical half-life is the expected half-life of a medical knowledge entity, calculated based on the subject update coefficient, baseline period, and evidence level attribute. The theoretical half-life can be calculated using the following formula: T = T_base / (K × E); Where T represents the theoretical half-life, T_base represents the baseline period, K represents the subject update coefficient, and E represents the evidence level coefficient. The evidence level coefficient E is a parameter determined based on the evidence level attribute. A higher evidence level corresponds to a smaller E value, resulting in a longer theoretical half-life; a lower evidence level corresponds to a larger E value, resulting in a shorter theoretical half-life.

[0097] More specifically, statistical tasks are performed at fixed time intervals to calculate the actual average time from the release of medical knowledge entities under each medical discipline category to their update to a new version, thereby obtaining the actual half-life of each medical discipline category.

[0098] In this embodiment, the fixed period is the time interval at which the system performs periodic statistics, such as once a year or once every six months.

[0099] The actual half-life is obtained by statistically analyzing real data and reflects the true average time from the publication of a medical knowledge entity to its update. The calculation of the actual half-life is based on historical data in the dynamic medical knowledge graph, which statistically analyzes the average survival time of medical knowledge entities under each medical discipline before they are replaced by a new version.

[0100] When there is a deviation between the actual half-life and the theoretical half-life of the previous cycle, the subject update coefficient is calibrated according to the deviation ratio, and the knowledge half-life of the medical knowledge entity is recalculated using the calibrated subject update coefficient.

[0101] The testing system compares the actual half-life with the theoretical half-life used in the previous cycle, calculates the deviation ratio between the two, adjusts the subject update coefficient according to the deviation ratio, and finally recalculates the knowledge half-life of each medical knowledge entity using the calibrated subject update coefficient.

[0102] In this embodiment, the testing system compares the actual half-life T_actual with the theoretical half-life T_theoretical calculated in the previous cycle based on the then-current subject update coefficient K_old. The subject update coefficient can be calibrated using the following formula: K_new = K_old × (T_theoretical / T_actual) Where K_new represents the calibrated subject update coefficient, K_old represents the uncalibrated subject update coefficient, T_theoretical represents the theoretical half-life of the previous period, and T_actual represents the statistically obtained actual half-life.

[0103] When the theoretical half-life is greater than the actual half-life (i.e., T_theoretical / T_actual > 1), it indicates that the system underestimates the knowledge update speed of the discipline (it assumes that knowledge is more durable, but this is not the case). Therefore, the K value is increased in order to shorten the half-life in the future and make the testing more frequent.

[0104] When the theoretical half-life is less than the actual half-life (i.e., T_theoretical / T_actual < 1), it indicates that the system has overestimated the knowledge update speed of the discipline. Therefore, the K value is lowered and the testing cycle is appropriately extended to optimize resources.

[0105] In this embodiment, the testing system compares the actual half-life with the theoretical half-life used in the previous cycle. If there is a deviation between the two, the subject update coefficient is adjusted according to the deviation ratio.

[0106] When the theoretical half-life is greater than the actual half-life, it indicates that the knowledge update speed of the discipline has been underestimated, and the discipline update coefficient needs to be increased to shorten the half-life in subsequent calculations.

[0107] When the theoretical half-life is less than the actual half-life, it indicates that the system has overestimated the knowledge update speed of the discipline, and the discipline update coefficient needs to be lowered to extend the half-life of subsequent calculations.

[0108] After calibrating the subject update coefficients, the testing system uses the calibrated subject update coefficients to recalculate the knowledge half-life of each medical knowledge entity, resulting in a more accurate timeliness assessment result.

[0109] This embodiment realizes a dynamic calculation and adaptive calibration mechanism for knowledge half-life. By employing a cyclical mechanism—determining the subject update coefficient through statistical analysis of historical update frequencies, calculating the theoretical half-life based on multiple factors, statistically analyzing the actual half-life, and calibrating the subject update coefficient according to deviations—the timeliness assessment can be adjusted to follow the actual update patterns of medical knowledge, improving the accuracy of the assessment and the system's adaptability.

[0110] Optionally, the process of calculating the knowledge half-life further includes: The number and update speed of medical knowledge entities under the medical discipline classification to which the medical knowledge entities belong are statistically analyzed from the dynamic medical knowledge graph. The temporal influence weights of each medical knowledge entity under the medical discipline classification are extracted from the dynamic medical knowledge graph, and the consensus subversion degree is calculated by comparing the conclusion differences between the current version of the medical knowledge entity and the historical version of the medical knowledge entity for the same knowledge point. Based on the update quantity, update speed, temporal influence weight, and consensus subversion degree, calculate the knowledge update intensity index of the medical discipline classification; The knowledge update intensity index is monitored in real time. When the knowledge update intensity index exceeds a preset threshold, the current knowledge half-life of each medical knowledge entity under the medical discipline classification is shortened and adjusted using the knowledge update intensity index. Alternatively, when feedback is received regarding the high-value failed sample, the medical knowledge entity corresponding to the high-value failed sample is extracted, the knowledge half-life of the medical knowledge entity is shortened and adjusted, and the timeliness score of the medical knowledge entity is recalculated.

[0111] In this application, the number of medical knowledge entities that have undergone version updates within a specific time period under a specific medical discipline category is statistically analyzed from the dynamic medical knowledge graph, as well as the frequency or rate of version updates, to obtain the update quantity and update speed, respectively.

[0112] In this embodiment, the update count refers to the number of medical knowledge entities under a medical discipline category that have undergone version updates within a specific time period. The update count reflects the breadth of knowledge changes within that discipline during that time period.

[0113] Update speed refers to the frequency or rate at which medical knowledge entities under a medical discipline are updated. Update speed reflects the pace of change in the knowledge of that discipline.

[0114] The testing system extracts the temporal influence weights of each medical knowledge entity under the medical discipline classification from the dynamic medical knowledge graph. Then, for the version sequence of the same knowledge point, it compares the degree of difference between the conclusions of the current version of the medical knowledge entity and the historical version of the medical knowledge entity, and calculates the consensus subversion degree accordingly.

[0115] In this embodiment, consensus subversion is an indicator used to quantify the extent of updates to medical knowledge. Consensus subversion is calculated by comparing the differences in conclusions between the current version of a medical knowledge entity and its historical versions for the same knowledge point.

[0116] If the conclusions of the new version differ only slightly from those of the old version, and are considered a gradual update, then the degree of consensus disruption is low. If the conclusions of the new version differ fundamentally from those of the old version, or are even completely opposite, and are considered a disruptive update, then the degree of consensus disruption is high.

[0117] More specifically, the testing system uses the number of updates, update speed, temporal influence weight, and consensus subversion degree as input parameters, and calculates the knowledge update intensity index of medical disciplines through weighted calculation.

[0118] In this embodiment, the knowledge update intensity index is an indicator that comprehensively reflects the activity level of knowledge updates within a medical discipline. The knowledge update intensity index is calculated through weighted averages, taking into account factors such as the number of updates, update speed, the temporal influence weight of relevant medical knowledge entities, and the degree of consensus disruption. A higher knowledge update intensity index indicates that the discipline is currently in a period of rapid knowledge updates.

[0119] The testing system continuously monitors the knowledge update intensity index of each medical discipline. When the knowledge update intensity index of a certain medical discipline exceeds the preset threshold, the system adjusts the current knowledge half-life of each medical knowledge entity under that medical discipline based on the value of the knowledge update intensity index.

[0120] In this embodiment, when the knowledge update intensity index UI_Index of a certain domain exceeds a preset threshold, the testing system can temporarily and quickly shorten its current half-life using the following formula: T_new = T_old × (1 - K_adj × UI_Index); Where T_new represents the adjusted knowledge half-life, T_old represents the unadjusted knowledge half-life, K_adj represents the adjustment coefficient, and UI_Index represents the knowledge update intensity index. This formula enables the system to respond immediately to knowledge bursts and accelerate the testing frequency in relevant fields.

[0121] In this embodiment, the testing system monitors the knowledge update intensity index in real time. When the knowledge update intensity index of a certain medical discipline exceeds a preset threshold, it indicates that the discipline is in a period of knowledge explosion or major update, and it is necessary to temporarily shorten the knowledge half-life of each medical knowledge entity under the discipline in order to accelerate the testing frequency and promptly identify potential timeliness issues of the large medical model.

[0122] The testing system receives feedback of high-value failure samples generated during the testing process, extracts the medical knowledge entities corresponding to the high-value failure samples, shortens and adjusts the knowledge half-life of these medical knowledge entities, and then recalculates the adjusted timeliness score.

[0123] In this embodiment, the testing system also supports a half-life adjustment mechanism based on test feedback. When a high-value invalid sample is found during the test, it indicates that the corresponding medical knowledge entity may be outdated. The testing system can shorten the knowledge half-life of the medical knowledge entity and related medical knowledge entities, and recalculate the timeliness score to improve the relevance of subsequent tests.

[0124] This embodiment introduces a knowledge update intensity index and a testing feedback mechanism, enabling real-time dynamic adjustment of the knowledge half-life. The knowledge update intensity index is calculated by comprehensively considering the number of updates, update speed, influence weight, and consensus disruption level. Based on this index and testing feedback, the knowledge half-life is dynamically adjusted, allowing testing resources to be prioritized in areas of most active knowledge change. This improves testing efficiency and relevance, forming a closed-loop feedback mechanism between test results and the evaluation system.

[0125] Optionally, the step of selecting high-value failure samples from the test results includes: For single-modal data, calculate the semantic novelty between the output content of the medical big model and the current version of medical knowledge entities in the dynamic medical knowledge graph; When the semantic novelty is lower than the modality-specific novelty threshold, and the self-confidence of the medical big model in the output content is higher than the preset confidence threshold, the dynamic test case is determined to have failed. Alternatively, for multimodal data, the output conclusions of the medical model on the same medical problem under different modalities are compared. When the output conclusions of different modalities point to contradictory treatment plans, the dynamic test case is determined to have failed. Dynamic test cases that are determined to have failed are marked as high-value failure samples.

[0126] In this application, for single-modal test data, the test system obtains the output content of the large medical model, retrieves the current version of the medical knowledge entity content of the corresponding knowledge point from the dynamic medical knowledge graph, and then calculates the semantic novelty between the two through semantic analysis technology.

[0127] In this embodiment, semantic novelty is an indicator used to measure the degree of semantic consistency between the output content of the medical big model and the latest medical knowledge.

[0128] Semantic novelty is calculated using natural language processing techniques, analyzing the semantic similarity between the output of the large medical model and the corresponding knowledge points in the current version of the medical knowledge entities in the dynamic medical knowledge graph.

[0129] The higher the semantic novelty, the closer the output of the medical big data model is to the latest medical knowledge; the lower the semantic novelty, the more likely the output of the medical big data model is based on outdated knowledge.

[0130] The testing system compares the calculated semantic novelty with the modality-specific novelty threshold, and at the same time obtains the self-confidence of the medical big model on the output content and compares it with the preset confidence threshold. When the semantic novelty is lower than the modality-specific novelty threshold and the self-confidence is higher than the preset confidence threshold, the dynamic test case is judged to have failed.

[0131] In this embodiment, the modality-specific novelty threshold is a semantic novelty threshold set for different modal data. Different modal data have different characteristics, so different thresholds can be set.

[0132] Self-confidence refers to the degree of certainty that a large medical model has about its output. When a large medical model outputs content, it can simultaneously output a confidence score that represents the degree of certainty.

[0133] The logic for determining test failure is as follows: if the semantic novelty of the output content of the medical big data model is low compared to the latest knowledge, but the model itself has high confidence in that output content, the test is considered a failure. This situation indicates that the medical big data model outputs content inconsistent with the latest knowledge with high confidence, which is a typical manifestation of timeliness defects.

[0134] For multimodal test data, the test system inputs different modal test cases for the same medical problem into the medical big model, obtains the output conclusions under each modality, compares the treatment plans pointed to by the output conclusions of different modalities, and judges the dynamic test case as failing when there are contradictory situations.

[0135] In this embodiment, for testing multimodal data, the testing system inputs different modal test cases for the same medical problem into the medical big model and compares the output conclusions of the medical big model under different modalities.

[0136] When the output conclusions of different modalities point to contradictory treatment plans (e.g., recommending treatment plan A based on text input, while recommending treatment plan B based on image input, and plans A and B are mutually exclusive), the test is deemed a failure. This indicates a problem with the knowledge consistency of the large medical model across different modalities.

[0137] The testing system marks dynamic test cases that are determined to have failed by the above judgment rules as high-value failure samples and stores them in the failure sample library for subsequent analysis.

[0138] In this embodiment, dynamic test cases that are determined to have failed are marked as high-value failure samples for subsequent failure knowledge node location and stress testing.

[0139] In this application, a test failure determination mechanism based on semantic novelty and intermodal consistency is established. By using two methods—joint determination of semantic novelty and confidence in a single modality, and determination of conclusion consistency in a multimodal model—timeliness defects in large medical models can be accurately identified, providing high-quality failure samples for subsequent defect analysis, while avoiding the misjudgment of low-confidence outputs as timeliness defects.

[0140] Optionally, the dynamic test cases are input into the large medical model to be tested for multimodal timeliness testing, including: When the dynamic test cases are text data, the dynamic test cases containing text data are input into the medical big model to obtain the output content. A dual-channel text analysis network is constructed, and the output content and the text content of the corresponding current version medical knowledge entity in the dynamic medical knowledge graph are input respectively. The semantic novelty score between the two is calculated through an attention mechanism. When the dynamic test case is image data, the dynamic test case containing image data is input into the medical big model to obtain image operation suggestions. Key feature points are extracted from the image operation suggestions. The key feature points are spatiotemporally aligned and compared with the standard surgical procedure features corresponding to the current version of the medical knowledge entity in the dynamic medical knowledge graph, and the feature matching degree is calculated. When the dynamic test case is diagnosis-related data, the dynamic test case containing the diagnosis description is input into the medical big model to obtain the diagnosis description. The diagnosis description is then mapped to multiple versions of the standard system through a version converter and compared with the standard code corresponding to the current version of the medical knowledge entity in the dynamic medical knowledge graph.

[0141] In this application, the testing system inputs dynamic test cases containing text data into a large medical model to obtain the output content. At the same time, it obtains the text content of the corresponding current version of medical knowledge entities from a dynamic medical knowledge graph. These two parts of text are input into two channels of a dual-channel text analysis network for encoding. Then, the semantic similarity between the two is calculated through an attention mechanism, and the semantic novelty score is output.

[0142] In this embodiment, the dual-channel text analysis network is a neural network model used to calculate the semantic similarity between two text segments. The dual-channel text analysis network contains two parallel text encoding channels, which encode the output content of the large medical model and the text content of the current version of the medical knowledge entity, respectively, to generate their respective semantic vector representations.

[0143] Attention mechanisms are techniques in deep learning used to model the relationships between sequential data. In dual-channel text analysis networks, attention mechanisms are used to calculate the similarity between the semantic vectors of two texts, generating a semantic novelty score. A higher semantic novelty score indicates that the output of the large medical model is more consistent with the latest medical knowledge.

[0144] The testing system inputs dynamic test cases containing image data into a large medical model to obtain image operation suggestions, extracts key feature points from the image operation suggestions, obtains the standard surgical procedure features corresponding to the current version of medical knowledge entities from the dynamic medical knowledge graph, performs spatiotemporal alignment comparison between the key feature points and the standard surgical procedure features, and calculates the feature matching degree.

[0145] In this embodiment, the image operation suggestions are processing suggestions from the medical big data model for the output of image data, which may include surgical operation suggestions, image diagnosis suggestions, interventional treatment path planning, etc.

[0146] Key feature points are the core feature information extracted from the image operation suggestions, such as the location of the surgical incision, key nodes of the interventional path, and the location of the lesion marking.

[0147] Standard procedure features are the standardized operational features corresponding to the current version of medical knowledge entities in the dynamic medical knowledge graph, representing the standard operations recognized in the current medical field.

[0148] Spatiotemporal alignment comparison is the process of aligning and comparing key feature points with standard surgical features in terms of spatial location and time series.

[0149] Feature matching degree is an indicator that reflects the degree of consistency between key feature points and standard surgical procedure features. The higher the feature matching degree, the more the imaging operation suggestions of the medical big data model conform to the latest medical standards.

[0150] In this application, the testing system inputs dynamic test cases containing diagnostic descriptions into a large medical model to obtain the output diagnostic descriptions. The diagnostic descriptions are then mapped to multiple versions of a standard coding system via a version converter. The corresponding standard codes for the current version of medical knowledge entities are obtained from the dynamic medical knowledge graph, and the mapping results are compared with the standard codes.

[0151] In this embodiment, the version converter is a tool used to map diagnostic descriptions to different versions of standard coding systems. In the medical field, there are multiple versions of standard systems for disease classification and diagnostic coding (such as ICD-10 and ICD-11). The version converter can map the diagnostic descriptions output by the large medical model to these different versions of standard systems.

[0152] Standard coding is the standardized coding corresponding to the current version of medical knowledge entities in the dynamic medical knowledge graph, representing the currently effective coding standard.

[0153] By comparing the diagnostic descriptions output by the medical big data model with the standard coding of medical knowledge entities in the current version, it can be determined whether the medical big data model has adopted the latest coding standard, thereby assessing its timeliness at the coding level.

[0154] This embodiment designs differentiated testing schemes and evaluation indicators for medical data of different modalities, such as text, images, and diagnostic codes. Text data employs a dual-channel text analysis network and attention mechanism to calculate semantic novelty; image data uses key feature point extraction and spatiotemporal alignment comparison to calculate feature matching degree; and diagnostic code data uses a version converter and standard code comparison. This achieves comprehensive multimodal timeliness testing, enabling the evaluation of the knowledge timeliness performance of large-scale medical models from multiple dimensions.

[0155] Optionally, the location of the failed knowledge node includes: Extract the output conclusions of the medical big model from the high-value failure samples; Following the evolution path of the medical knowledge entities corresponding to the high-value failed samples in the dynamic medical knowledge graph, the output conclusion is semantically matched with the content of the medical knowledge entities in each historical version of the version sequence. The historical version of medical knowledge entity with the highest semantic match to the output conclusion is marked as the invalid knowledge node.

[0156] In this application, the testing system obtains the output content of the medical big model for the test case from high-value failure samples, and extracts the core output conclusions from it.

[0157] In this embodiment, the testing system extracts the output conclusions of the medical big data model from high-value failure samples. The output conclusions are the core part of the response content given by the medical big data model to dynamic test cases, representing the embodiment of the medical knowledge mastered by the medical big data model.

[0158] The testing system locates the medical knowledge entities corresponding to high-value failed samples in the dynamic medical knowledge graph, traverses each historical version of the medical knowledge entities in the version sequence along its evolution path, and sequentially performs semantic matching between the output conclusion of the medical big model and the content of each historical version of the medical knowledge entities, recording the semantic matching degree of each match.

[0159] In this embodiment, the testing system sequentially performs semantic matching between the output conclusion of the medical big data model and the content of medical knowledge entities in each historical version of the version sequence, following the evolution path of the corresponding medical knowledge entities in the dynamic medical knowledge graph. The purpose of semantic matching is to find the historical version that is closest to the output conclusion of the medical big data model.

[0160] The testing system compares the semantic matching degree between medical knowledge entities in each historical version and the output conclusion, identifies the medical knowledge entity in the historical version with the highest semantic matching degree, and marks it as an invalid knowledge node.

[0161] In this embodiment, the testing system marks the historical version of medical knowledge entities with the highest semantic match degree with the output conclusion of the medical big data model as invalid knowledge nodes during the semantic matching process. The location results of invalid knowledge nodes indicate that the medical big data model may still be using the historical version of medical knowledge, and its knowledge updates are lagging behind the current version of medical knowledge entities.

[0162] This embodiment achieves precise localization from test failure cases to specific knowledge failure nodes. By extracting output conclusions, traversing evolution paths, performing semantic matching, and determining the version with the highest matching degree, the specific historical time nodes of knowledge lag in the large medical model are clearly identified, providing an accurate basis for subsequent defect analysis and remediation suggestion generation.

[0163] Optionally, the construction of a cognitive evolution stress testing environment based on the evolutionary path of the medical knowledge entities corresponding to the high-value failure samples includes: From the dynamic medical knowledge graph, based on the evolution path of the medical knowledge entities corresponding to the high-value failed samples, the version sequence of each medical knowledge entity under the medical discipline classification to which the medical knowledge entity belongs is extracted; Based on the release time attributes and content changes of each medical knowledge entity in the version sequence, fit the cognitive evolution curve of the medical discipline classification; The time points in the cognitive evolution curve where version changes occur are identified as key time nodes.

[0164] In this application, the testing system determines the corresponding medical knowledge entity and its evolution path based on high-value failure samples, identifies the medical discipline classification to which the medical knowledge entity belongs, and then extracts the version sequence of all medical knowledge entities under the medical discipline classification from the dynamic medical knowledge graph.

[0165] In this embodiment, the testing system determines the medical discipline category to which the high-value failed samples belong based on the corresponding medical knowledge entities, and then extracts the version sequence of each medical knowledge entity under that medical discipline category from the dynamic medical knowledge graph. The extracted version sequence covers the entire historical evolution of knowledge in that discipline.

[0166] The testing system analyzes the release time attributes and content change information of each medical knowledge entity in the version sequence, and fits a cognitive evolution curve that reflects the evolution of knowledge in this medical discipline classification with time as the horizontal axis and knowledge status as the vertical axis.

[0167] In this embodiment, the cognitive evolution curve is a curve describing the evolution of medical knowledge over time. The testing system fits a curve reflecting the evolution of knowledge within a medical discipline based on the release time attributes and content change information of each medical knowledge entity in the version sequence. The cognitive evolution curve illustrates the trajectory of medical knowledge from its initial establishment, gradual development, updating and revision, to the potential for paradigm shifts.

[0168] The testing system analyzes the cognitive evolution curve, identifies the time points on the curve where version changes occur, and marks these time points as key time nodes.

[0169] In this embodiment, key time points refer to the points on the cognitive evolution curve where version changes occur. At key time points, medical knowledge undergoes version updates, which may involve corrections, additions, or even complete overhauls of the knowledge. Key time points are important reference points for constructing stress tests.

[0170] This embodiment achieves cognitive evolution curve fitting and key time node identification based on historical evolution paths. By extracting version sequences, analyzing release times and content changes, fitting cognitive evolution curves, and identifying key time nodes, a time-dimensional reference framework is provided for subsequent stress tests, enabling stress tests to be designed in accordance with the actual evolution of medical cognition.

[0171] Optionally, the construction of the cognitive evolution stress testing environment further includes: The version replacement frequency of each medical knowledge entity under the medical discipline classification is statistically analyzed from the dynamic medical knowledge graph to obtain the knowledge evolution speed. The degree of consensus subversion is obtained by calculating the degree of difference between the conclusions of the current version of medical knowledge entities and the historical version of medical knowledge entities from the evolutionary path. Based on the speed of knowledge evolution and the degree of consensus disruption, the knowledge pressure parameter corresponding to each key time node is calculated.

[0172] In this application, the testing system statistically analyzes the version replacement frequency of each medical knowledge entity under the medical discipline classification from the dynamic medical knowledge graph, and calculates the knowledge evolution speed that reflects the rhythm of knowledge change in the discipline.

[0173] The speed of knowledge evolution is an indicator reflecting how quickly knowledge versions are replaced within a medical discipline. It is calculated by statistically analyzing version replacement frequency; the more frequent the version replacements, the faster the knowledge evolution.

[0174] The testing system obtains the content of the current version of medical knowledge entities and the medical knowledge entities of each historical version from the evolution path, compares the degree of difference in conclusions between the current version of medical knowledge entities and the historical version of medical knowledge entities, and calculates the consensus subversion degree.

[0175] In this embodiment, the testing system calculates the degree of difference in conclusions between the current version of medical knowledge entities and historical versions of medical knowledge entities from the evolutionary path, obtaining the consensus subversion degree. The consensus subversion degree reflects the degree of impact of knowledge updates on the original medical consensus.

[0176] The testing system calculates the knowledge pressure parameters corresponding to each key time node by comprehensively considering the knowledge evolution speed and consensus disruption degree of that time node.

[0177] In this embodiment, the knowledge pressure parameter is used to quantify the intensity of knowledge evolution pressure at key time points. The knowledge pressure parameter is calculated by comprehensively considering the speed of knowledge evolution and the degree of consensus disruption. The faster the knowledge evolution speed and the higher the degree of consensus disruption, the larger the knowledge pressure parameter, indicating stronger knowledge evolution pressure corresponding to that key time point.

[0178] This embodiment calculates knowledge pressure parameters for each key time point. The process of calculating knowledge pressure parameters by statistically analyzing version change frequency to determine the speed of knowledge evolution, calculating the degree of consensus disruption based on the difference in calculated conclusions, and combining both to provide a quantitative basis for subsequent pressure level classification and differentiated testing. This allows pressure testing to be designed in a tiered manner based on the actual pressure intensity of knowledge evolution.

[0179] Optionally, the construction of the cognitive evolution stress testing environment further includes: The knowledge pressure parameter is compared with a preset pressure level threshold. Key time points where the knowledge pressure parameter is below the first threshold are classified as mild pressure levels. Key time points where the knowledge pressure parameter falls between the first and second thresholds are classified as medium-pressure levels. Key time points where the knowledge pressure parameter exceeds the second threshold are classified as severe pressure levels.

[0180] In this application, the testing system compares the knowledge pressure parameters of each key time node with a preset first threshold and a second threshold, and classifies the key time nodes into three levels: mild pressure level, moderate pressure level, or severe pressure level based on the comparison results.

[0181] In this embodiment, the testing system divides key time nodes into different pressure levels based on the magnitude of the knowledge pressure parameter. The first threshold and the second threshold are preset critical values ​​for dividing the pressure levels.

[0182] Mild stress levels correspond to key time points with low knowledge stress parameters. These time points typically correspond to the gradual accumulation and updating of knowledge, where there is no fundamental conflict between new and old knowledge.

[0183] The moderate stress level corresponds to key time points with moderate knowledge stress parameters. These time points usually correspond to knowledge updates that are subject to academic controversy, where different viewpoints differ but have not yet reached a consensus.

[0184] The severe stress level corresponds to key time points with high knowledge stress parameters. These time points usually correspond to disruptive updates or paradigm shifts in knowledge, where the new version of knowledge has fundamental differences from or even completely opposite conclusions to the old version.

[0185] This embodiment achieves the classification of stress levels for key time nodes. By comparing knowledge stress parameters with preset thresholds, key time nodes are divided into three stress levels: mild, moderate, and severe. This provides a basis for designing differentiated tests for different stress levels, enabling stress tests to conduct targeted assessments for different types of knowledge evolution scenarios.

[0186] Optionally, stress testing is performed on the large medical model, including: For the key time points of the mild stress level, medical knowledge entities of adjacent versions before and after the key time points are extracted from the evolution path to construct knowledge accumulation test questions; For the key time points of the moderate pressure level, medical knowledge entities with differing conclusions at the key time points are extracted from the evolutionary path, and academic controversy test questions are constructed. For the key time points of the severe pressure level, extract the medical knowledge entities from the evolution path where the conclusions of the key time points are overturned, and construct paradigm shift test problems. According to the time sequence of the key time nodes, the corresponding test questions are input into the medical big model in turn to achieve stress testing of the medical big model.

[0187] In this application, the testing system extracts medical knowledge entities from the evolution path of the adjacent versions before and after the key time nodes of the mild stress level, and constructs knowledge accumulation test questions based on the content differences of these medical knowledge entities.

[0188] In this embodiment, the knowledge accumulation test problem is designed for key time points of mild stress levels. The knowledge accumulation test problem is constructed based on medical knowledge entities from adjacent versions before and after the key time points, and is used to test whether the large medical model can correctly grasp the progressively updated knowledge content.

[0189] The testing system targets key time points at moderate stress levels, extracts medical knowledge entities with differing conclusions at those time points from the evolutionary path, and constructs academic controversy-type test questions based on these differing contents.

[0190] In this embodiment, the academic controversy test problem is designed for key time points at the moderate stress level. The academic controversy test problem is constructed based on medical knowledge entities with differing conclusions, and is used to test the medical large-scale model's ability to handle academic controversies and its judgment accuracy.

[0191] The testing system targets key time points of severe stress levels, extracts medical knowledge entities from the evolutionary path that lead to overturned conclusions at those time points, and constructs paradigm shift test questions based on these disruptive changes.

[0192] In this embodiment, the paradigm shift test problem is designed for key time points at the level of severe stress. Based on medical knowledge entities where conclusions are overturned, the paradigm shift test problem is used to test whether a large-scale medical model can update its understanding and adopt new medical consensus in a timely manner when faced with a knowledge paradigm shift.

[0193] The testing system inputs the corresponding test questions into the medical big data model in chronological order according to the key time nodes, records the output results of the medical big data model on each test question, and completes the stress test of the medical big data model.

[0194] In this embodiment, the testing system inputs the constructed test questions into the medical big data model sequentially according to the chronological order of key time nodes. This chronological testing method simulates the real evolution of medical knowledge and can assess whether the medical big data model's cognition is synchronized with the historical evolution of medical knowledge.

[0195] This embodiment designs differentiated test questions for key time points at different stress levels. Mild stress levels use knowledge accumulation-type test questions, moderate stress levels use academic controversy-type test questions, and severe stress levels use paradigm shift-type test questions. These tests are executed sequentially in chronological order, achieving a progressive stress test from mild to severe, comprehensively evaluating the performance and adaptability of the medical big data model under different knowledge evolution scenarios.

[0196] Optionally, the cognitive resilience index for evaluating the medical big data model includes: The knowledge transfer rate of the medical big model is obtained by statistically analyzing the proportion of test rounds in which the current version of medical knowledge entity conclusions are adopted in the stress test out of the total test rounds. The conflict resolution accuracy of the medical model is obtained by statistically analyzing the proportion of times the current version of the medical knowledge entity is selected as the answer in the test questions corresponding to the severe stress level, relative to the total number of test questions under the severe stress level.

[0197] In this application, the testing system counts the number of times the medical big model outputs conclusions that adopt the current version of medical knowledge entity conclusions in all test rounds of stress testing, calculates the proportion of this number to the total test rounds, and obtains the knowledge transfer rate.

[0198] In this embodiment, the knowledge transfer rate is an indicator reflecting the medical big data model's ability to adopt the latest knowledge. The formula for calculating the knowledge transfer rate is: Knowledge Transfer Rate = (Number of test rounds adopting the current version of medical knowledge entity conclusions / Total number of test rounds) × 100%. The higher the knowledge transfer rate, the more timely the medical big data model is in updating knowledge and adopting the latest medical consensus.

[0199] The testing system counts the number of times the answer corresponding to the current version of medical knowledge entity is selected in the test questions corresponding to the severe stress level of the medical big model, calculates the proportion of this number to the total number of test questions under the severe stress level, and obtains the conflict resolution accuracy.

[0200] In this embodiment, conflict resolution accuracy is an indicator reflecting the medical big data model's ability to correctly select the latest knowledge when faced with conflicts between old and new knowledge. Conflict resolution accuracy focuses on the test results at the severe stress level, as this level corresponds to a disruptive knowledge update scenario and best tests the medical big data model's ability to handle conflicts between old and new knowledge. The formula for calculating conflict resolution accuracy is: Conflict Resolution Accuracy = (Number of times the current version of medical knowledge entity was selected as the answer under the severe stress level / Total number of test questions under the severe stress level) × 100%.

[0201] This embodiment establishes two core cognitive resilience indicators: knowledge transfer rate and conflict resolution accuracy. Knowledge transfer rate reflects the overall ability of the medical big data model to adopt new knowledge, while conflict resolution accuracy reflects its ability to make correct choices in extreme conflict scenarios. These two indicators quantitatively assess the medical big data model's ability to cope with knowledge evolution from different dimensions, providing quantifiable and comparable support for the evaluation results.

[0202] Optionally, the cognitive resilience index for evaluating the medical big data model further includes: The medical knowledge entity versions corresponding to the conclusions output by the medical big model for each test question during the stress test are arranged in chronological order according to the key time nodes to form a cognitive trajectory. By comparing the cognitive trajectory with the cognitive evolution curve, key time points in which the output conclusions of the medical big data model lag behind the cognitive evolution curve are identified as cognitive lag points. Based on the knowledge transfer rate, the conflict resolution accuracy, and the cognitive lag node, the cognitive resilience index of the medical big model is obtained.

[0203] In this application, the testing system records the conclusions output by the medical big data model for each test question during the stress test, determines the version of the medical knowledge entity corresponding to each output conclusion, and arranges these version information in chronological order according to key time nodes to form a cognitive trajectory reflecting the changes in the cognitive state of the medical big data model.

[0204] In this embodiment, the cognitive trajectory describes the changes in the cognitive state of the medical big data model during stress testing. The testing system records the versions of medical knowledge entities corresponding to the output conclusions of the medical big data model in each test question, and arranges them in chronological order according to key time nodes to form a trajectory line reflecting the cognitive evolution of the medical big data model.

[0205] The testing system overlays and compares the cognitive trajectory of the medical big data model with the cognitive evolution curve fitted based on the dynamic medical knowledge graph, identifies key time points where the cognitive trajectory lags behind the cognitive evolution curve, and marks these points as cognitive lag nodes.

[0206] In this embodiment, the testing system compares the cognitive trajectory of the medical big data model with a cognitive evolution curve fitted based on a dynamic medical knowledge graph. When the version of the medical knowledge entity corresponding to the output conclusion of the medical big data model at a certain key time point lags behind the version of the cognitive evolution curve at that time point, that key time point is identified as a cognitive lag node. A cognitive lag node indicates that the medical big data model's cognition at that time point has failed to keep up with the actual evolution of medical knowledge.

[0207] The test system integrates the knowledge transfer rate, conflict resolution accuracy, and a list of cognitive lag nodes to form the cognitive resilience index evaluation results of the medical big data model.

[0208] In this embodiment, the test system integrates knowledge transfer rate, conflict resolution accuracy, and cognitive lag node information to obtain the cognitive resilience index of the medical big data model. The cognitive resilience index is a comprehensive evaluation result of the medical big data model's ability to cope with knowledge evolution, and may include the values ​​of various indicators and a list of cognitive lag nodes, providing a basis for generating subsequent repair suggestions.

[0209] This embodiment introduces the concepts of cognitive trajectory and cognitive lag nodes. By recording version information corresponding to output conclusions to form a cognitive trajectory, comparing the cognitive trajectory with the cognitive evolution curve to identify lag nodes, and integrating multiple indicators to form a cognitive resilience index, this process achieves visualized tracking of the cognitive evolution state of a large-scale medical model and precise location of lag nodes. It also enriches the evaluation dimensions of the cognitive resilience index, making the evaluation results more comprehensive and intuitive.

[0210] Optionally, the generation of timeliness defect repair suggestions for the medical big model includes: Obtain the publication time attribute and the medical discipline classification to which the failed knowledge node belongs; At the data level, it is recommended to supplement the training data with medical knowledge entities published after the publication time attribute that belong to the same medical discipline classification as the expired knowledge node. At the model level, incremental training is planned for the same medical discipline category as the aforementioned failed knowledge nodes; At the knowledge base level, the successor version of the medical knowledge entity associated with the failed knowledge node is obtained from the evolution path, and an update of the successor version of the medical knowledge entity is prompted.

[0211] In this application, the testing system obtains the publication time attribute and the medical discipline classification information of the located failure knowledge nodes as the basis for generating repair suggestions.

[0212] In this embodiment, the testing system acquires the publication time attribute and the medical discipline classification information of the failed knowledge nodes located through semantic matching. This information is used to determine the scope of the remediation recommendations.

[0213] Based on the publication time attribute of the failed knowledge node, the testing system filters out medical knowledge entities published after that time that belong to the same medical discipline category as the failed knowledge node from the dynamic medical knowledge graph, and generates recommendations to use these medical knowledge entities as supplementary training data for repair.

[0214] In this embodiment, the data-level repair suggestions target the training data of the large-scale medical model. Based on the publication time attribute of the failed knowledge node, the testing system recommends supplementing it with new training data containing medical knowledge entities published after that time that belong to the same medical discipline category as the failed knowledge node. This ensures that the large-scale medical model can learn updated medical knowledge in subsequent training.

[0215] The testing system retrieves information about the successor versions of medical knowledge entities associated with failed knowledge nodes from the evolution path, and generates suggestions to check and update the corresponding entries in the external knowledge base.

[0216] In this embodiment, the repair suggestions at the knowledge base level target external knowledge bases that the large medical model may rely on. The test system obtains the subsequent version medical knowledge entity information associated with the failed knowledge node from the evolution path, prompting relevant personnel to check and update the corresponding entries in the external knowledge base to ensure that the knowledge base stores the medical knowledge of the current version of the medical knowledge entity.

[0217] This embodiment generates timely defect remediation suggestions covering three levels: data, model, and knowledge base. At the data level, it recommends supplementing with newly released training data; at the model level, it plans targeted incremental training; and at the knowledge base level, it prompts updates to relevant entries. This provides a comprehensive, specific, and actionable guidance plan for targeted improvements to the large-scale medical model, forming a complete closed-loop process from testing to remediation, supporting the continuous optimization and iteration of the large-scale medical model.

[0218] In summary, the medical large-scale model testing method provided by this invention achieves comprehensive, systematic, and accurate testing and evaluation of the timeliness of medical large-scale model knowledge by constructing a dynamic medical knowledge graph, calculating knowledge half-life, generating dynamic test cases, performing multimodal timeliness testing, locating failed knowledge nodes, constructing a cognitive evolution stress testing environment, evaluating cognitive resilience indicators, and generating repair suggestions. This method solves the technical problems of traditional static testing methods being unable to capture the dynamic evolution of medical knowledge and assess the timeliness of model knowledge, providing important technical support for the continuous optimization and safe clinical application of medical large-scale models.

[0219] The medical large-scale model testing device provided by the present invention is described below. The medical large-scale model testing device described below can be referred to in correspondence with the medical large-scale model testing method described above.

[0220] Figure 2 This is a schematic diagram of the medical large-scale model testing device provided by the present invention, as shown below. Figure 2 As shown, it includes: The construction module 210 is used to construct a dynamic medical knowledge graph, which contains medical knowledge entities. Each medical knowledge entity has a publication time attribute and an evidence level attribute. Different medical knowledge entities of the same knowledge point constitute a version sequence. Each medical knowledge entity in the version sequence is associated through an evolution path. The calculation module 220 is used to calculate the knowledge half-life of the medical knowledge entities based on the dynamic medical knowledge graph, calculate the timeliness score of each medical knowledge entity according to the relationship between the current time, the publication time attribute and the knowledge half-life, and filter out medical knowledge entities with timeliness scores lower than a preset timeliness threshold, and generate dynamic test cases. The positioning module 230 is used to input the dynamic test cases into the medical big model to be tested for multimodal timeliness testing, filter out high-value failure samples from the test results, extract the output conclusions of the medical big model for the high-value failure samples, determine the version sequence corresponding to the high-value failure samples, and perform semantic matching between the output conclusions and the medical knowledge entities of each historical version in the version sequence to locate the failure knowledge nodes. The testing module 240 is used to construct a cognitive evolution stress test environment based on the evolution path of the medical knowledge entities corresponding to the high-value failure samples, perform stress tests on the medical big model, and evaluate the cognitive resilience index of the medical big model. The repair module 250 is used to generate timeliness defect repair suggestions for the medical big model based on the cognitive elasticity index of the medical big model and the failed knowledge nodes.

[0221] In this application, by constructing a dynamic medical knowledge graph containing version sequences and evolution paths, and introducing a knowledge half-life calculation mechanism, dynamic test cases can be generated by automatically filtering easily expired knowledge based on the timeliness score of medical entities. This eliminates the lag of manually maintaining static test sets and achieves accurate capture of new and expired medical knowledge. By semantically matching the high-value failure samples detected in the tests with historical versions of medical knowledge entities, the specific obsolete knowledge nodes that cause model failure can be accurately located. Furthermore, the cognitive evolution stress test environment built based on the evolution path can simulate real-world scenarios of medical knowledge replacement, quantitatively evaluate the cognitive resilience index of the model when facing knowledge iteration, and generate targeted suggestions for timeliness defect repair, effectively improving the safety, reliability, and knowledge update response speed of the large medical model in clinical applications.

[0222] Figure 3 Figure 3 shows a schematic diagram of the electronic device provided by this invention. The electronic device may include a processor 310, a communication interface 320, a memory 330, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a large-scale medical model testing method. This method includes: constructing a dynamic medical knowledge graph, which contains medical knowledge entities. Each medical knowledge entity has a publication time attribute and an evidence level attribute. Different medical knowledge entities of the same knowledge point constitute a version sequence, and the medical knowledge entities in the version sequence are associated through an evolutionary path. Based on the dynamic medical knowledge graph, the knowledge half-life of the medical knowledge entities is calculated. According to the relationship between the current time, the publication time attribute and the knowledge half-life, the timeliness score of each medical knowledge entity is calculated, and medical knowledge entities with timeliness scores lower than the preset timeliness threshold are filtered out to generate dynamic test cases. The dynamic test cases are input into the medical big model to be tested for multimodal timeliness testing. High-value failure samples are selected from the test results. The output conclusions of the medical big model for the high-value failure samples are extracted. The version sequence corresponding to the high-value failure samples is determined. The output conclusions are semantically matched with the medical knowledge entities of each historical version in the version sequence to locate the failure knowledge nodes. Based on the evolution path of the medical knowledge entities corresponding to the high-value failure samples, a cognitive evolution stress test environment is constructed to perform stress tests on the medical big model and evaluate the cognitive resilience index of the medical big model. Based on the cognitive resilience index of the medical big data model and the failed knowledge nodes, suggestions for repairing the timeliness defects of the medical big data model are generated.

[0223] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0224] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the medical large-scale model testing method provided by the above methods. The method includes: constructing a dynamic medical knowledge graph, which contains medical knowledge entities. Each medical knowledge entity has a publication time attribute and an evidence level attribute. Different medical knowledge entities of the same knowledge point constitute a version sequence. Each medical knowledge entity in the version sequence is associated through an evolution path. Based on the dynamic medical knowledge graph, the knowledge half-life of the medical knowledge entities is calculated. According to the relationship between the current time, the publication time attribute and the knowledge half-life, the timeliness score of each medical knowledge entity is calculated, and medical knowledge entities with timeliness scores lower than the preset timeliness threshold are filtered out to generate dynamic test cases. The dynamic test cases are input into the medical big model to be tested for multimodal timeliness testing. High-value failure samples are selected from the test results. The output conclusions of the medical big model for the high-value failure samples are extracted. The version sequence corresponding to the high-value failure samples is determined. The output conclusions are semantically matched with the medical knowledge entities of each historical version in the version sequence to locate the failure knowledge nodes. Based on the evolution path of the medical knowledge entities corresponding to the high-value failure samples, a cognitive evolution stress test environment is constructed to perform stress tests on the medical big model and evaluate the cognitive resilience index of the medical big model. Based on the cognitive resilience index of the medical big data model and the failed knowledge nodes, suggestions for repairing the timeliness defects of the medical big data model are generated.

[0225] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the medical large-scale model testing method provided by the above methods. The method includes: constructing a dynamic medical knowledge graph, wherein the dynamic medical knowledge graph contains medical knowledge entities, each of the medical knowledge entities having a publication time attribute and an evidence level attribute, different medical knowledge entities of the same knowledge point constitute a version sequence, and each medical knowledge entity in the version sequence establishes an association through an evolution path; Based on the dynamic medical knowledge graph, the knowledge half-life of the medical knowledge entities is calculated. According to the relationship between the current time, the publication time attribute and the knowledge half-life, the timeliness score of each medical knowledge entity is calculated, and medical knowledge entities with timeliness scores lower than the preset timeliness threshold are filtered out to generate dynamic test cases. The dynamic test cases are input into the medical big model to be tested for multimodal timeliness testing. High-value failure samples are selected from the test results. The output conclusions of the medical big model for the high-value failure samples are extracted. The version sequence corresponding to the high-value failure samples is determined. The output conclusions are semantically matched with the medical knowledge entities of each historical version in the version sequence to locate the failure knowledge nodes. Based on the evolution path of the medical knowledge entities corresponding to the high-value failure samples, a cognitive evolution stress test environment is constructed to perform stress tests on the medical big model and evaluate the cognitive resilience index of the medical big model. Based on the cognitive resilience index of the medical big data model and the failed knowledge nodes, suggestions for repairing the timeliness defects of the medical big data model are generated.

[0226] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0227] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0228] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for testing large-scale medical models, characterized in that, include: A dynamic medical knowledge graph is constructed, which contains medical knowledge entities. Each medical knowledge entity has a publication time attribute and an evidence level attribute. Different medical knowledge entities of the same knowledge point constitute a version sequence. The medical knowledge entities in the version sequence are associated through an evolutionary path. Based on the dynamic medical knowledge graph, the knowledge half-life of the medical knowledge entities is calculated. According to the relationship between the current time, the publication time attribute and the knowledge half-life, the timeliness score of each medical knowledge entity is calculated, and medical knowledge entities with timeliness scores lower than the preset timeliness threshold are filtered out to generate dynamic test cases. The dynamic test cases are input into the medical big model to be tested for multimodal timeliness testing. High-value failure samples are selected from the test results. The output conclusions of the medical big model for the high-value failure samples are extracted. The version sequence corresponding to the high-value failure samples is determined. The output conclusions are semantically matched with the medical knowledge entities of each historical version in the version sequence to locate the failure knowledge nodes. Based on the evolution path of the medical knowledge entities corresponding to the high-value failure samples, a cognitive evolution stress test environment is constructed to perform stress tests on the medical big model and evaluate the cognitive resilience index of the medical big model. Based on the cognitive resilience index of the medical big data model and the failed knowledge nodes, suggestions for repairing the timeliness defects of the medical big data model are generated.

2. The medical large-scale model testing method according to claim 1, characterized in that, The construction of the dynamic medical knowledge graph includes: Unstructured text is obtained from multiple medical data sources, medical knowledge entities are parsed out through relation extraction, and the publication time and evidence level of each medical knowledge entity are extracted as metadata. At the same time, the medical discipline classification to which each medical knowledge entity belongs is identified. Based on the impact factor of the source journal, the authority level of the publishing institution, and the document type of the medical knowledge entity, the temporal influence weight of each medical knowledge entity is calculated. The medical knowledge entities, their metadata, medical discipline classifications, and temporal influence weights are stored in the graph database. The same knowledge point published at different times in the graph database is identified, and version sequences and evolution paths between the medical knowledge entities are established to form the dynamic medical knowledge graph.

3. The medical large-scale model testing method according to claim 2, characterized in that, Based on the dynamic medical knowledge graph, the knowledge half-life of the medical knowledge entities is calculated, including: From the dynamic medical knowledge graph, the historical update frequency of medical knowledge entities under each medical discipline category is statistically analyzed, and the discipline update coefficient corresponding to each medical discipline category is determined based on the historical update frequency. The theoretical half-life of the medical knowledge entity is calculated based on the discipline update coefficient, baseline period, and evidence level attribute of the medical knowledge entity to which it belongs. By statistically analyzing the actual average time from the publication to the update of medical knowledge entities under each medical discipline category at a fixed period, the actual half-life of each medical discipline category is obtained. When there is a deviation between the actual half-life and the theoretical half-life of the previous cycle, the subject update coefficient is calibrated according to the deviation ratio, and the knowledge half-life of the medical knowledge entity is recalculated using the calibrated subject update coefficient.

4. The medical large-scale model testing method according to claim 3, characterized in that, The process of calculating the half-life of the knowledge also includes: The number and update speed of medical knowledge entities under the medical discipline classification to which the medical knowledge entities belong are statistically analyzed from the dynamic medical knowledge graph. The temporal influence weights of each medical knowledge entity under the medical discipline classification are extracted from the dynamic medical knowledge graph, and the consensus subversion degree is calculated by comparing the conclusion differences between the current version of the medical knowledge entity and the historical version of the medical knowledge entity for the same knowledge point. Based on the update quantity, update speed, temporal influence weight, and consensus subversion degree, calculate the knowledge update intensity index of the medical discipline classification; The knowledge update intensity index is monitored in real time. When the knowledge update intensity index exceeds a preset threshold, the current knowledge half-life of each medical knowledge entity under the medical discipline classification is shortened and adjusted using the knowledge update intensity index. Alternatively, when feedback is received regarding the high-value failed sample, the medical knowledge entity corresponding to the high-value failed sample is extracted, the knowledge half-life of the medical knowledge entity is shortened and adjusted, and the timeliness score of the medical knowledge entity is recalculated.

5. The medical large-scale model testing method according to claim 1, characterized in that, The process of selecting high-value failure samples from the test results includes: For single-modal data, calculate the semantic novelty between the output content of the medical big model and the current version of medical knowledge entities in the dynamic medical knowledge graph; When the semantic novelty is lower than the modality-specific novelty threshold, and the self-confidence of the medical big model in the output content is higher than the preset confidence threshold, the dynamic test case is determined to have failed. Alternatively, for multimodal data, the output conclusions of the medical model on the same medical problem under different modalities are compared. When the output conclusions of different modalities point to contradictory treatment plans, the dynamic test case is determined to have failed. Dynamic test cases that are determined to have failed are marked as high-value failure samples.

6. The medical large-scale model testing method according to claim 1, characterized in that, The dynamic test cases are input into the large medical model to be tested for multimodal timeliness testing, including: When the dynamic test cases are text data, the dynamic test cases containing text data are input into the medical big model to obtain the output content. A dual-channel text analysis network is constructed, and the output content and the text content of the corresponding current version medical knowledge entity in the dynamic medical knowledge graph are input respectively. The semantic novelty score between the two is calculated through an attention mechanism. When the dynamic test case is image data, the dynamic test case containing image data is input into the medical big model to obtain image operation suggestions. Key feature points are extracted from the image operation suggestions. The key feature points are spatiotemporally aligned and compared with the standard surgical procedure features corresponding to the current version of the medical knowledge entity in the dynamic medical knowledge graph, and the feature matching degree is calculated. When the dynamic test case is diagnosis-related data, the dynamic test case containing the diagnosis description is input into the medical big model to obtain the diagnosis description. The diagnosis description is then mapped to multiple versions of the standard system through a version converter and compared with the standard code corresponding to the current version of the medical knowledge entity in the dynamic medical knowledge graph.

7. The medical large-scale model testing method according to claim 1, characterized in that, The knowledge nodes for locating failures include: Extract the output conclusions of the medical big model from the high-value failure samples; Following the evolution path of the medical knowledge entities corresponding to the high-value failed samples in the dynamic medical knowledge graph, the output conclusion is semantically matched with the content of the medical knowledge entities in each historical version of the version sequence. The historical version of medical knowledge entity with the highest semantic match to the output conclusion is marked as the invalid knowledge node.

8. The medical large-scale model testing method according to claim 1, characterized in that, The cognitive evolution stress test environment is constructed based on the evolution path of the medical knowledge entities corresponding to the high-value failed samples, including: From the dynamic medical knowledge graph, based on the evolution path of the medical knowledge entities corresponding to the high-value failed samples, the version sequence of each medical knowledge entity under the medical discipline classification to which the medical knowledge entity belongs is extracted; Based on the release time attributes and content changes of each medical knowledge entity in the version sequence, fit the cognitive evolution curve of the medical discipline classification; The time points in the cognitive evolution curve where version changes occur are identified as key time nodes.

9. The medical large-scale model testing method according to claim 8, characterized in that, The construction of the cognitive evolution stress testing environment also includes: The version replacement frequency of each medical knowledge entity under the medical discipline classification is statistically analyzed from the dynamic medical knowledge graph to obtain the knowledge evolution speed. The degree of consensus subversion is obtained by calculating the degree of difference between the conclusions of the current version of medical knowledge entities and the historical version of medical knowledge entities from the evolutionary path. Based on the speed of knowledge evolution and the degree of consensus disruption, the knowledge pressure parameter corresponding to each key time node is calculated.

10. The medical large-scale model testing method according to claim 9, characterized in that, The construction of the cognitive evolution stress testing environment also includes: The knowledge pressure parameter is compared with a preset pressure level threshold. Key time points where the knowledge pressure parameter is below the first threshold are classified as mild pressure levels. Key time points where the knowledge pressure parameter falls between the first and second thresholds are classified as medium-pressure levels. Key time points where the knowledge pressure parameter exceeds the second threshold are classified as severe pressure levels.

11. The medical large-scale model testing method according to claim 10, characterized in that, Perform stress tests on the aforementioned large-scale medical model, including: For the key time points of the mild stress level, medical knowledge entities of adjacent versions before and after the key time points are extracted from the evolution path to construct knowledge accumulation test questions; For the key time points of the moderate pressure level, medical knowledge entities with differing conclusions at the key time points are extracted from the evolutionary path, and academic controversy test questions are constructed. For the key time points of the severe pressure level, extract the medical knowledge entities from the evolution path where the conclusions of the key time points are overturned, and construct paradigm shift test problems. According to the time sequence of the key time nodes, the corresponding test questions are input into the medical big model in turn to achieve stress testing of the medical big model.

12. The medical large-scale model testing method according to claim 11, characterized in that, The cognitive resilience indicators for evaluating the medical big data model include: The knowledge transfer rate of the medical big model is obtained by statistically analyzing the proportion of test rounds in which the current version of medical knowledge entity conclusions are adopted in the stress test out of the total test rounds. The conflict resolution accuracy of the medical model is obtained by statistically analyzing the proportion of times the current version of the medical knowledge entity is selected as the answer in the test questions corresponding to the severe stress level, relative to the total number of test questions under the severe stress level.

13. The medical large-scale model testing method according to claim 12, characterized in that, The cognitive resilience indicators for evaluating the medical big data model also include: The medical knowledge entity versions corresponding to the conclusions output by the medical big model for each test question in the stress test are arranged in chronological order according to the key time nodes to form a cognitive trajectory. By comparing the cognitive trajectory with the cognitive evolution curve, key time points in which the output conclusions of the medical big data model lag behind the cognitive evolution curve are identified as cognitive lag points. Based on the knowledge transfer rate, the conflict resolution accuracy, and the cognitive lag node, the cognitive resilience index of the medical big model is obtained.

14. The medical large-scale model testing method according to claim 1, characterized in that, The generation of timeliness defect repair suggestions for the large medical model includes: Obtain the publication time attribute and the medical discipline classification to which the failed knowledge node belongs; At the data level, it is recommended to supplement the training data with medical knowledge entities published after the publication time attribute that belong to the same medical discipline classification as the expired knowledge node. At the model level, incremental training is planned for the same medical discipline category as the aforementioned failed knowledge nodes; At the knowledge base level, the successor version of the medical knowledge entity associated with the failed knowledge node is obtained from the evolution path, and an update of the successor version of the medical knowledge entity is prompted. The subsequent version of the medical knowledge entity is a medical knowledge entity whose publication time is later than that of the expired knowledge node.

15. A medical large-scale model testing device, characterized in that, include: A construction module is used to construct a dynamic medical knowledge graph. The dynamic medical knowledge graph contains medical knowledge entities. Each medical knowledge entity has a publication time attribute and an evidence level attribute. Different medical knowledge entities of the same knowledge point constitute a version sequence. Each medical knowledge entity in the version sequence is associated through an evolution path. The calculation module is used to calculate the knowledge half-life of the medical knowledge entities based on the dynamic medical knowledge graph, calculate the timeliness score of each medical knowledge entity according to the relationship between the current time, the publication time attribute and the knowledge half-life, and filter out medical knowledge entities with timeliness scores lower than a preset timeliness threshold, and generate dynamic test cases. The positioning module is used to input the dynamic test cases into the medical big model to be tested for multimodal timeliness testing, filter out high-value failure samples from the test results, extract the output conclusions of the medical big model for the high-value failure samples, determine the version sequence corresponding to the high-value failure samples, and perform semantic matching between the output conclusions and the medical knowledge entities of each historical version in the version sequence to locate the failure knowledge nodes. The testing module is used to construct a cognitive evolution stress testing environment based on the evolution path of the medical knowledge entities corresponding to the high-value failure samples, perform stress testing on the medical big model, and evaluate the cognitive resilience index of the medical big model. The repair module is used to generate repair suggestions for the timeliness defects of the medical big model based on the cognitive elasticity index of the medical big model and the failed knowledge nodes.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the medical large-scale model testing method as described in any one of claims 1 to 14.

17. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the medical large-scale model testing method as described in any one of claims 1 to 14.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the medical large-scale model testing method as described in any one of claims 1 to 14.

Images