Medical large model fine tuning optimization method based on knowledge guidance and related device

Abstract

The invention belongs to a medical large model fine-tuning optimization method, and provides a medical large model fine-tuning optimization method based on knowledge guidance and a related device aiming at the technical problems of incomplete diagnosis result, low diagnosis result precision and indefinite logical reasoning path of the existing medical large model. And obtaining a thinking guidance information part in the logic chain prompt by means of the knowledge guidance model, inputting the thinking guidance information part into the medical large model to obtain output normalized by the chain prompt, and further performing fine tuning optimization on the medical large model. Wherein the knowledge guidance module comprises a multi-modal-multi-level connection map fusion module, an expert consensus driving module, a analogy deduction module, an explicit derivation rule calculation module and a progressive thinking guidance module.

Description

Technical Field

[0001] The present application relates to a method for fine-tuning and optimizing a large medical model, and specifically to a method for fine-tuning and optimizing a large medical model based on knowledge guidance and related devices. Background Art

[0002] As the global population ages, the number of patients with neuropsychiatric diseases has increased significantly, bringing tremendous pressure to society and the medical system. Represented by diseases such as Alzheimer's disease, depression, and anxiety, more and more elderly people and patients with neuropsychiatric diseases have an urgent need for efficient and accurate medical services.

[0003] In this context, artificial intelligence technology, especially medical big models, has emerged, providing powerful tools and solutions for the medical field. By integrating multimodal data (such as MRI, PET, genomic data, electronic medical records, etc.), medical big models can provide support for the early diagnosis and precise treatment of complex diseases. However, most existing medical big models have three main problems: (1) They only use single-modality data and fail to fully integrate the advantages of multimodal data, resulting in incomplete diagnostic results or insufficient diagnostic accuracy. (2) Because the medical big models themselves rely on complex deep learning architectures, they cannot provide a clear logical reasoning path, making the conclusions difficult for doctors and patients to trust. (3) Due to the large differences between individual patients, the accuracy of existing medical big models in early diagnosis is still not ideal. Summary of the invention

[0004] This application aims to address the technical problems of incomplete diagnostic results, low diagnostic accuracy, and unclear logical reasoning paths in current large medical models, and provides a knowledge-guided fine-tuning and optimization method and related devices for large medical models.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions: In the first aspect, the present application proposes a knowledge-guided fine-tuning optimization method for a large medical model, comprising: Acquire multimodal data to be diagnosed; Input the multimodal data to be diagnosed into the knowledge guidance model to obtain the thinking guidance information part in the logic chain prompt, which is used to input into the medical big model to obtain the output after the chain prompt standardization; The knowledge guidance model includes a multi-modal-multi-level connection graph fusion module, an expert consensus driving module, an analogy deduction module, an explicit deduction rule calculation module and a progressive thinking guidance module; The multi-modal-multi-level connection graph fusion module is used to obtain the overall embedded features as the state according to the modal data graph of each modal data of the disease to be diagnosed; The expert consensus driving module is used to extract expert consensus and clinical experience from the expert knowledge base of the disease to be diagnosed, combine the state semantic labels corresponding to the overall embedded features, determine the state relationship labels between the overall embedded features, and extract the embedded feature representation of the relationship from the state relationship labels, and then combine the overall embedded features, state semantic labels, state relationship labels and the embedded feature representation of the relationship to construct a state hierarchical map, and then combine the modal data map of the modal data to obtain a global multimodal-multi-level connection map; The analogy deduction module is used to construct an individual multimodal-multi-level connection map based on multiple groups of triples through analogy deduction according to the global multimodal-multi-level connection map; the triples include two overall embedded features and corresponding embedded feature representations; The explicit derivation rule calculation module is used to obtain complete inference rules for all nodes according to each node of the individual multimodal-multi-level connection graph; The progressive thinking guidance module is used to combine the global multi-modal-multi-level connection graph and the modal types in the complete reasoning rules of all nodes to obtain the thinking guidance information part in the logical chain prompt.

[0006] Furthermore, according to the modal data graph of each modal data of the disease to be diagnosed, the overall embedded features as the state are obtained, including: Based on the feature aggregation layer of the graph neural network, through the aggregation function Aggregate The neighborhood information of each node in the modal data graph of each modal data of the disease to be diagnosed is aggregated to form a global representation of the node, and the overall embedding features of the modal data graph of each modal data of the disease to be diagnosed are extracted.

[0007] Furthermore, the individual multimodal-multi-level connection graph is constructed based on multiple groups of triples through analogical deduction, including: Obtain example triples based on the global multimodal-multi-level connection graph; The predicted triples and example triples are embedded in the analog feature embedding layer for representation learning. The feature mapping between the modalities is learned through the cross-modal adaptive interaction layer to obtain the embedding representation corresponding to each modality. The multimodal information fusion layer integrates the embedded representations corresponding to each modality, and then obtains the predicted triples and feature representations of the predicted triples after prediction through the reasoning consistency normalization layer and the prediction output layer; According to the predicted triples and the feature representations of the predicted triples, an individual multimodal-multi-level connection graph is constructed.

[0008] Furthermore, according to each node of the individual multimodal-multi-level connection graph, a complete inference rule for all nodes is obtained, including: Inputting each node of the individual multimodal-multi-level connection graph into the strategy function of inference rule calculation in sequence; The strategy function includes a state coding layer, multiple feature extraction layers and a strategy selection layer connected in sequence, an activation layer is arranged before the state coding layer, the multiple feature extraction layers and the strategy selection layer, and the output of the strategy selection layer also includes normalization processing.

[0009] Furthermore, the combination of the global multi-modal-multi-level connection graph and the modal types in the complete reasoning rules of all nodes obtains the thinking guidance information part in the logic chain prompt, including: Representative data of different modalities are extracted from the state hierarchical graph of the global multimodal-multi-level connection graph, and a question-answering example library with progressive reasoning is constructed for different modal data combinations. Then, according to the modal types in the complete reasoning rules of all nodes, corresponding question-answering examples are extracted from the question-answering example library as the thinking guidance information part of the logical chain prompt.

[0010] Furthermore, the loss function used in the training of the analogy deduction module is:

[0011] in, is the relaxed loss function, is the total number of triplets in the training set, is the sine similarity function, is the hidden feature representation of the example triplet, To predict the hidden feature representation of the triple, To maximize the calculation formula, and They are the hidden feature representations of the head and tail entities in the prediction pair, respectively.

[0012] Furthermore, the reward function used in the training of the explicit derivation rule calculation module is:

[0013]

[0014]

[0015]

[0016] in, is the total reward function, Reward for global goals, is the path efficiency reward, Reward for path diversity, is the last node of the individual multimodal-multi-level connection graph. is the path length, is the set of explored paths, is the cosine similarity function, for The coefficient of for The coefficient of for The coefficient of The Monte Carlo policy gradient method is used to update the policy function parameters during the training of the explicit derivation rule calculation module.

[0017] Furthermore, the loss function used in the training of the knowledge-guided model is:

[0018] in, is the overall loss function when training the knowledge-guided model, The loss function involved in updating the policy function parameters using the Monte Carlo policy gradient method is recorded as the path optimization loss. Calculate the semantic loss function for the reasoning process and labels of the thinking guidance information part in the logical chain prompt:

[0019] in, is the reasoning process label, for Each embedding representation produced by the semantic encoding model, for Each embedding representation produced by the semantic encoding model, Provides a logical chain to guide the reasoning process of the information part.

[0020] In the second aspect, the present application proposes a knowledge-guided medical large model fine-tuning optimization system, comprising: A data acquisition module, used to acquire multimodal data to be diagnosed; The prompt output module is used to input the multimodal data to be diagnosed into the knowledge guidance model to obtain the thinking guidance information part of the logic chain prompt, which is used to input into the medical big model to obtain the output after the chain prompt standardization; The knowledge guidance model includes a multi-modal-multi-level connection graph fusion module, an expert consensus driving module, an analogy deduction module, an explicit deduction rule calculation module and a progressive thinking guidance module; The multi-modal-multi-level connection graph fusion module is used to obtain the overall embedded features as the state according to the modal data graph of each modal data of the disease to be diagnosed; The expert consensus driving module is used to extract expert consensus and clinical experience from the expert knowledge base of the disease to be diagnosed, combine the state semantic labels corresponding to the overall embedded features, determine the state relationship labels between the overall embedded features, and extract the embedded feature representation of the relationship from the state relationship labels, and then combine the overall embedded features, state semantic labels, state relationship labels and the embedded feature representation of the relationship to construct a state hierarchical map, and then combine the modal data map of the modal data to obtain a global multimodal-multi-level connection map; The analogy deduction module is used to construct an individual multimodal-multi-level connection map based on multiple groups of triples through analogy deduction according to the global multimodal-multi-level connection map; the triples include two overall embedded features and corresponding embedded feature representations; The explicit derivation rule calculation module is used to obtain complete inference rules for all nodes according to each node of the individual multimodal-multi-level connection graph; The progressive thinking guidance module is used to combine the global multi-modal-multi-level connection graph and the modal types in the complete reasoning rules of all nodes to obtain the thinking guidance information part in the logical chain prompt.

[0021] In a third aspect, the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned knowledge-guided medical large model fine-tuning optimization.

[0022] Compared with the prior art, this application has the following beneficial effects: This application proposes a method for fine-tuning and optimizing a large medical model based on knowledge guidance, which uses the knowledge guidance model to obtain the thinking guidance information part in the logical chain prompt, which is used to input into the large medical model to obtain the output after the chain prompt standardization, and then fine-tune and optimize the large medical model. Among them, the knowledge guidance module includes a multi-modal-multi-level connection graph fusion module, an expert consensus driving module, an analogy deduction module, an explicit deduction rule calculation module and a progressive thinking guidance module. The multimodal-multi-level connection graph fusion module converts medical data of different modalities into graph structured data, extracts node features of different modal data, and realizes multimodal information complementarity; the expert consensus driven module generates semantic labels of different modal states and relationship labels between them by extracting medical consensus from the expert knowledge base; the analogy deduction module realizes feature mapping and fusion between different modalities, supports analogy deduction of multimodal data, and improves reasoning ability and individualized diagnosis accuracy; the explicit deduction rule calculation module extracts the explicit deduction rules of the disease, provides a logical chain prompt logical reasoning rule part, provides explicit logical support for the reasoning of the medical big model, and enhances the interpretability of the medical big model; the progressive thinking guidance module can use chain prompt technology to realize sparse sample learning of the medical big model, guide the medical big model to perform step-by-step reasoning, and ultimately improve the diagnostic accuracy of the medical big model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A schematic diagram of a process flow of a knowledge-guided medical large model fine-tuning optimization method of this application; Figure 2 The second flow chart of the knowledge-guided medical large model fine-tuning optimization method of this application Figure 3 This is a principle diagram corresponding to the second flow diagram of the knowledge-guided medical large model fine-tuning optimization method of this application; Figure 4 It is a schematic diagram of the principle of the analogy deduction module based on the multi-modal-multi-level connection graph in the embodiment of the present application; Figure 5 Schematic diagram of the principle of an explicit derivation rule calculation module based on a multi-modal-multi-level connection graph in an embodiment of the present application; Figure 6 This is a schematic diagram of the knowledge-guided medical large model fine-tuning optimization system of this application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0027] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0028] As the global population ages, the number of patients with neuropsychiatric diseases has increased significantly, bringing tremendous pressure to society and the medical system. Represented by diseases such as Alzheimer's disease, depression, and anxiety, more and more elderly people and patients with neuropsychiatric diseases have put forward urgent demands for efficient and accurate medical services. The increase in the number of these patients has not only affected people's health, but also aggravated the shortage of social medical resources.

[0029] In this context, the rise of artificial intelligence technology, especially medical big models, has provided powerful tools and solutions for the medical field. By integrating multimodal data (such as MRI, PET, genomic data, electronic medical records, etc.), medical big models can provide support for the early diagnosis and precise treatment of complex diseases. Compared with traditional diagnosis methods based on a single data modality, the fusion of multimodal data can more comprehensively reflect the complex mechanism of the disease and improve the accuracy of diagnosis.

[0030] However, medical big models face a major challenge in their application, namely the "black box" problem. The decision-making process of medical big models usually relies on complex deep learning architectures and a large number of parameters, based on data-driven implicit learning models rather than explicit rules. Although these models perform well in processing complex data and prediction tasks, their decision-making process lacks credibility and cannot provide clear explanations for doctors and patients. This makes the model's reasoning process difficult to understand, limiting its application in clinical practice.

[0031] In the medical field, interpretability is crucial to the application of models. Medical large models must be interpretable so that doctors can trust the model's diagnostic results and make clinical decisions based on the information it provides. Interpretability not only helps ensure the diagnostic safety and reliability of the model, but also helps medical institutions improve the treatment effect of patients through the output of the model. Especially in the diagnosis and treatment of neuropsychiatric diseases, due to the complexity of the disease itself and the differences between patients, models that lack interpretability may lead to inaccurate diagnosis and even bring medical risks. Although existing interpretability methods (such as LIME and SHAP) can provide some explanations, these methods are usually applicable to simpler models. When faced with the complex nonlinear relationships and high-dimensional data of medical large models, they can only provide local or fragmented explanations, and it is difficult to cope with the complex logic and large-scale data processing involved in medical large models. Therefore, existing interpretability technologies cannot meet the needs of medical large models.

[0032] Therefore, the current medical big model mainly has the following problems: Current medical big models rely on single-modality data when dealing with complex disease diagnosis. However, single-modality data often cannot fully reflect the multidimensional characteristics of the disease, resulting in limited diagnostic accuracy. Multimodal data (such as brain imaging, EEG, genome, and electronic medical records) can provide complementary sources of information, but most current medical big models find it difficult to effectively integrate these different data sources, resulting in incomplete diagnosis. In addition, the current medical big models are highly complex in structure, with a large number of parameters, and rely on processing and transmitting high-dimensional data through nonlinear transformations and multi-layer networks. This implicit learning mechanism makes the reasoning and decision-making process of the model difficult to be understood by the outside world, resulting in a "black box" problem, which seriously affects the interpretability of medical big models and limits their application in medical clinics. Although existing interpretability technologies (such as LIME and SHAP) can provide local explanations for relatively simple models, when dealing with highly complex nonlinear relationships and multidimensional data, these technologies can only provide fragmented explanations and are difficult to cover the overall decision logic of the model, resulting in insufficient explanation support when dealing with complex medical big models. Furthermore, although existing large medical models have certain diagnostic capabilities when dealing with complex diseases, their diagnostic accuracy is still not ideal due to the lack of deep understanding of the disease field and effective reasoning mechanisms.

[0033] The relevant specific studies mainly include: The Chinese invention patent with publication number CN118802369A proposes a collaborative enhancement method for APT knowledge graphs and large language models. Specifically disclosed is a collaborative enhancement method for APT knowledge graphs and large language models, including the following steps: S1. Constructing an APT knowledge graph focusing on the field of network security; S2. Designing chain prompts based on the questions input by the user and passing them to the large language model in sequence for step-by-step answers. The large language model uses the knowledge provided by the APT knowledge graph to query and locate the questions; S3. Combining the located APT knowledge nodes with the question content input by the user, using the strongly related nodes retrieved from the subgraph to enrich the context-aware prompts; S4. The large language model generates a comprehensive answer for the user based on the context-aware prompts. This patent application enhances the semantic understanding, reasoning and prediction capabilities of the large language model in APT scenarios. The large language model can better analyze and infer attacker behavior and attack paths, improve accuracy in complex network environments, and enhance the defense and response capabilities against APT attacks. Furthermore, the Chinese patent application with publication number CN118674056A proposes an intelligent solution method, system, device and medium for multi-step reasoning problems, and specifically designs a multi-step reasoning problem-solving framework, which generates explanation steps in a step-by-step manner, as a multi-step thinking chain to inspire model reasoning, enhances multi-step reasoning capabilities, and has strong practical value in realizing strong artificial intelligence. In addition, the application establishes a knowledge fusion mechanism and a relevance evaluation mechanism in the multi-step reasoning problem-solving framework, which helps to improve the logical rationality and factual accuracy of the generated reasoning process, thereby improving the accuracy of problem solving. At the same time, the external knowledge introduced by the knowledge fusion mechanism can provide additional information for many question-answering related machine learning tasks, and the evaluation of the problem-solving steps by the relevance evaluation mechanism can assist the education platform in making reasonable evaluations and feedback for the solution process, and can provide more personalized online guidance services based on this.

[0034] However, current studies have the following problems: Most existing medical models only use single-modality data (such as brain imaging, EEG data, genetic data or electronic medical records), and fail to fully integrate the advantages of multimodal data. When dealing with complex diseases, single-modality data is difficult to fully capture the multidimensional characteristics of the disease, especially diseases such as Alzheimer's disease, which often require multimodal data to fully reveal the biomarkers and development mechanisms of the disease. The use of single-modality data leads to incomplete diagnostic results and insufficient diagnostic accuracy. Although the existing medical big models perform well in dealing with complex data, because the model itself relies on a complex deep learning architecture and cannot provide a clear logical reasoning path, its conclusions are difficult to be trusted by doctors and patients. Especially in the diagnosis of diseases, doctors cannot clearly understand the reasoning basis of the model through the decision-making process of the model, which affects the credibility and application of the diagnostic results. When facing nonlinear, multi-dimensional complex data, the medical big model fails to provide sufficient explicit reasoning basis, and the "black box" nature of the model remains a major problem. Due to the complex pathogenesis of neuropsychiatric diseases (such as Alzheimer's disease, depression, etc.) and the large differences between individual patients, the accuracy of existing models in early diagnosis is still not ideal. Current methods often rely on limited pathological data and pathological signs and cannot provide high-accuracy diagnosis in the early stages of the disease. At the same time, existing models fail to fully utilize expert knowledge and medical consensus to enhance diagnostic accuracy, lack a systematic explicit reasoning mechanism, and result in a lack of logical explanation and support for diagnostic results.

[0035] In view of the shortcomings of the above-mentioned prior art, this application proposes a knowledge-guided fine-tuning optimization method for a large medical model and related devices, and constructs a method for enhancing the interpretability of a large medical model based on "multimodal data + expert knowledge", which focuses on improving the diagnostic accuracy and interpretability of the model in the field of diseases. The application is described in detail below in conjunction with the embodiments and drawings.

[0036] like Figure 1 As shown, it is a flow chart of the knowledge-guided medical large model fine-tuning optimization method of the present application, which may include: S101, obtaining multimodal data to be diagnosed.

[0037] S102, input the multimodal data to be diagnosed into the knowledge guidance model, obtain the thinking guidance information part in the logic chain prompt, which is used to input into the medical big model to obtain the output standardized by the chain prompt.

[0038] Among them, the knowledge guidance model includes a multimodal-multi-level connection graph fusion module, an expert consensus driving module, an analogy deduction module, an explicit deduction rule calculation module and a progressive thinking guidance module.

[0039] The multimodal-multi-level connection graph fusion module is used to obtain the overall embedded features as the state according to the modal data graph of each modal data of the disease to be diagnosed. The multimodal-multi-level connection graph fusion module is the basis of the entire optimization method and is responsible for integrating multimodal data from different medical examination methods (such as CT, MRI images, pathological reports, physiological monitoring data, etc.). These data are each represented in the form of a graph and contain rich disease information. The fused graph not only retains the unique information of each modal data, but also captures the association and complementarity between them, providing a comprehensive description of the disease state for subsequent analysis. The multimodal-multi-level connection graph fusion module integrates multimodal data sources to provide comprehensive disease characteristics, overcomes the problems of missing information and insufficient diagnostic accuracy brought by single-modal data, and also effectively reveals the complementary information between different modal data through the construction of multi-level connection graphs, comprehensively captures the multi-dimensional characteristics of the disease, and solves the limitations of single-modal data.

[0040] The expert consensus driven module is used to extract expert consensus and clinical experience from the expert knowledge base of the disease to be diagnosed, combine the state semantic labels corresponding to the overall embedded features, determine the state relationship labels between the overall embedded features, and extract the embedded feature representation of the relationship from the state relationship labels, and then combine the overall embedded features, state semantic labels, state relationship labels and embedded feature representation of the relationship to construct a state hierarchical map, and then combine the modal data map of the modal data to obtain a global multimodal-multi-level connection map. The expert consensus driven module can rely on a huge expert knowledge base, which contains the consensus, clinical experience and research results of medical experts on various diseases. First, based on the overall embedded features extracted from the multimodal data of the disease to be diagnosed, the relevant expert consensus and clinical experience are searched in the knowledge base. Through semantic matching and similarity calculation, accurate state semantic labels and state relationship labels are assigned to these features. These labels not only describe the state of the disease, but also reveal the logical relationship between states. Finally, the module uses these labels and relationships to construct a state hierarchical map, which reflects the possible paths and stages of disease development. The expert consensus driven module introduces expert knowledge to ensure that the inference rules extracted during the model reasoning process are explicit.

[0041] The analogy deduction module is used to construct an individual multimodal-multi-level connection map based on the global multimodal-multi-level connection map and multiple groups of triples through analogy deduction; the triples include two overall embedded features and corresponding embedded feature representations. Analogy deduction is a logical reasoning method that infers new triples based on known triples (i.e., two overall embedded features and the relationship between them). Through multiple iterations and reasoning, a map reflecting the disease status of individual patients is gradually constructed. This map not only contains the patient's specific condition information, but also reflects the personalized characteristics of disease development.

[0042] The explicit derivation rule calculation module is used to obtain complete reasoning rules for all nodes according to each node of the individual multimodal-multi-level connection map. The explicit derivation rule calculation module generates complete reasoning rules based on the individualized multimodal-multi-level connection map. These rules are expressed in the form of logical expressions and describe the causal relationship and evolution law between disease states. The explicit derivation rule calculation module can extract all logical relationships by traversing the nodes and edges in the map and convert them into executable reasoning rules. These rules provide clear guidance for subsequent decision-making. An explicit derivation rule calculation module based on a multimodal-multi-level connection map is designed to extract explicit reasoning rules from the connection map and provide them to the medical big model, thereby enhancing the interpretability of the model and solving the problem of low credibility of diagnostic results caused by the "black box" problem of the medical big model. The analogy deduction module and explicit inference rule calculation module based on the multimodal-multi-level connection graph mine the paths between different data modalities and diagnostic conclusions from the multimodal-multi-level connection graph, extract the key explicit reasoning rules in the decision-making process, and make up for the shortcomings of existing technologies in interpreting complex models.

[0043] The progressive thinking guidance module is used to combine the modal types in the global multimodal-multi-level connection map and the complete reasoning rules of all nodes to obtain the thinking guidance information part in the logical chain prompt. The progressive thinking guidance module is the final output link of the optimization method. It combines the global multimodal-multi-level connection map and the complete reasoning rules of all nodes to generate thinking guidance information in the logical chain prompt. This information is presented to doctors or users in a structured manner to guide them to think and make decisions according to the logical chain. The design of a progressive thinking guidance module guides the large medical model to gradually generate diagnostic results, enhances the diagnostic accuracy of the model in the scenario of neuropsychiatric diseases, and ensures that the output of the model is logical and interpretable, so that the accuracy of early diagnosis has been significantly improved, helping it to better utilize the knowledge related to the disease and enhance the accuracy and reliability of diagnosis.

[0044] This application proposes a method for constructing logical chain prompts based on "multimodal data + expert knowledge". By constructing a connection graph and calculating the path of explicit reasoning rules, the prompts of the medical large model are further constructed, which improves the diagnostic accuracy of the medical large model disease and the interpretability of the diagnostic results. The multimodal-multi-level connection graph fusion module aggregates the graph structure data of medical data of different modalities (such as brain imaging, EEG, genome, electronic medical records, etc.) and extracts node features of neuropsychiatric disease data of different modalities through a graph neural network. Multimodal information complementarity is achieved. The expert consensus-driven module extracts medical consensus from the expert knowledge base, generates semantic labels of different modal states and relationship labels between them, and uses the natural language processing model to extract the semantic embedding features of the relationship, providing semantic support for the connection graph of multimodal data. The mechanism of combining expert knowledge with multimodal connection graphs, and the mechanism of combining modal hierarchical graphs with state hierarchical graphs, solves the problem of how semantic embedding models can effectively process expert knowledge and organically combine it with the feature embedding of multimodal data, ensuring that the multidimensional features of different modalities can be integrated into a unified multimodal-multi-level connection graph. The analogy deduction module based on the multimodal-multi-level connection graph realizes feature mapping and fusion between different modalities through the design of cross-modal adaptive interaction layers, and supports the analogy deduction of multimodal data. This module can predict the relationship between states and further construct an individualized multimodal-multi-level connection graph to improve reasoning ability and individualized diagnostic accuracy. The explicit deduction rule calculation module based on the multimodal-multi-level connection graph extracts explicit deduction rules for neuropsychiatric diseases from the state hierarchical graph based on deep reinforcement learning technology, provides a logical chain prompt logical reasoning rule part, provides explicit logical support for the reasoning of large medical models, and enhances the interpretability of large models.

[0045] Existing medical big models often rely on single-modal data, resulting in an incomplete understanding of the model and insufficient diagnostic accuracy. This application proposes a multimodal-multi-level connection graph fusion module to integrate multiple modal data and extract and aggregate features through a graph neural network. Compared with existing single-modal or simple multimodal fusion technologies, this application can more comprehensively reflect the complex mechanisms of neuropsychiatric diseases and improve the accuracy and comprehensiveness of diagnosis. Existing medical big models have a "black box" problem and cannot clearly explain their decision-making process, which makes it difficult for the model to gain the trust of doctors in clinical practice. Existing interpretability methods have certain applications in simple models, but they can only provide fragmented explanations when dealing with complex nonlinear relationships and high-dimensional data. This application introduces an expert consensus-driven module, an analogy deduction module based on a multimodal-multi-level connection graph, and an explicit derivation rule calculation module, combined with the medical consensus in the expert knowledge base, to generate explicit reasoning rules between states and states, and construct an explicit reasoning path, which significantly improves the interpretability of the model, so that doctors can understand and trust the diagnostic results of the model. Most existing medical models use standard training methods and lack the ability to reason step by step for complex tasks. Especially in the field of neuropsychiatric diseases, due to data scarcity, it is difficult for models to learn effectively through traditional methods. This application proposes a progressive thinking guidance module, which combines the professional knowledge of clinicians to build a question-and-answer example library with progressive reasoning, and guides large medical models to perform sparse sample learning through chain prompts. This technology enables the model to reason step by step given a small amount of training data, thereby improving its learning efficiency and diagnostic ability in the diagnosis of neuropsychiatric diseases.

[0046] like Figure 2 As shown, it is a second flow chart of the knowledge-guided medical large model fine-tuning optimization method of this application, as shown Figure 3 The figure is a principle diagram corresponding to the second flow diagram of the knowledge-guided medical large model fine-tuning optimization method of the present application. Specifically, it may include: S201, model building The model constructed in this application mainly includes five modules: multimodal-multi-level connection map fusion module, expert consensus driving module, analogy deduction module based on multimodal-multi-level connection map, explicit deduction rule calculation module based on multimodal-multi-level connection map and progressive thinking guidance module. The specific construction methods of each model are as follows: (1) Constructing a multi-modal and multi-level connection graph fusion module Step a1: Map the modal data of a certain mode As input, it is input into the input layer of the multimodal-multi-level connection graph fusion module.

[0047] It should be noted that the modality in this application specifically refers to different medical data, such as brain images, EEG data, electronic medical records, etc., which are all modal data. Modal data can be represented as a graph structure, and a modal data map is formed through a graph structure composed of nodes and edges.

[0048] Step a2: Feature aggregation layer based on graph neural network through aggregation function Aggregate the neighborhood information of the node to form a global representation of the node, and then extract the overall embedding features of the modal data graph (hereinafter also referred to as "status"): . Graph neural network is a deep learning method that specializes in processing graph data. It can perform end-to-end learning and reasoning on graph structured data. The core components of graph neural network include graph convolution layer, message passing mechanism and multi-layer perceptron. The graph convolution layer is responsible for updating the node features in the graph by aggregating the features of neighboring nodes to the central node and applying linear transformation and nonlinear activation function to update the features of the central node. The message passing mechanism allows information to be transmitted and aggregated between nodes, thereby capturing the global graph structure information. The multi-layer perceptron is used to further process and transform the node features to improve the expressiveness of the model.

[0049] Step a3: Modal data spectrum in step a1 That is, the modal level graph, the overall embedding feature obtained by step a2 It is a node of the state hierarchy graph. and The output of this modality, together with the output of the expert consensus driven module, constitutes a multimodal-multi-level connection graph.

[0050] (2) Building an expert consensus-driven module Step b1: Taking Alzheimer's disease as an example, extract expert consensus and clinical experience from the expert knowledge base in the field of Alzheimer's disease, and use the state semantic label corresponding to the overall embedded feature obtained in step a2 (For example, in structural imaging modalities, the overall embedding feature The overall feature represented by can be "hippocampal atrophy", and the corresponding state semantic label is "hippocampal atrophy"), find the relationship labels between states ,Relationship labels can be medical logics such as "caused by pathological changes", "symptoms and pathology association", etc. The form of expert knowledge extraction is text description.

[0051] Step b2: Use a semantic extraction model trained on a large-scale medical text dataset , with relationship tags The semantic features of are used as the embedding features of the relationship and are expressed as:

[0052] in, Embedded features representing relations, i.e., state relations.

[0053] Step b3: Combine the overall embedding features from step a3 , the state semantic label in step b1 , state relationship label , and the state relationship obtained in step b2 Building a state hierarchy map Finally, we get a global multi-modal-multi-level connection graph .

[0054] (3) Construct an analogy deduction module based on a multimodal and multi-level connection graph.

[0055] like Figure 4 As shown, it is a schematic diagram of the principle of the analogy deduction module based on the multimodal-multi-level connection graph.

[0056] Step c1: When training the analogy deduction module based on multimodal-multi-level connection graph, the input is the state pair , and the relationship between them The training triples , and the state pair , and the relationship between them Example triples formed When making predictions in actual applications, the input is the example triple , and the state pair , and the relationship to be predicted between them (masked as [MASK]) .

[0057] Step c2: Take the prediction process as an example to illustrate this step. The input training triples and example triples are embedded in the analog feature embedding layer for representation learning, and the feature mapping between the modalities is learned through the cross-modal adaptive interaction layer. The embedded representation is obtained , , , .

[0058] Step c3: After generating the node embedding representation, the multimodal data is integrated through the multimodal information fusion layer. Then, the predicted triple (predicted triple after prediction) and the feature representation of the predicted triple are obtained through the reasoning consistency normalization layer and the prediction output layer. , .

[0059] Step c4: Based on the predicted triples , combined with feature representation , , construct an individual multimodal-multi-level connection map.

[0060] (4) Construct an explicit inference rule calculation module based on a multimodal and multi-level connection graph.

[0061] like Figure 5 , which is a schematic diagram of the principle of the explicit inference rule calculation module based on the multi-modal-multi-level connection graph.

[0062] Step d1: Connect the nodes of the individual multimodal-multi-level connection graph Input into the model.

[0063] Step d2: The strategy function for calculating the inference rule is , which represents the current state Next, the next step is to select the relationship, the function is a learnable parameter. The policy function consists of a state encoding layer, several feature extraction layers, and a policy selection layer. An activation layer is added after each layer, and the output of the policy selection layer is normalized. Step d3: After the policy selection, the model obtains the next node of the inference rule path .from Start, repeat steps d1 and d2 until the output node of step d2 is , you can get arrive The complete inference rules.

[0064] (5) Constructing a progressive thinking guidance module Step e1: Extract representative data of different modalities from the state hierarchical map of the global multimodal-multi-level connection map, and build a question-answering example library with progressive reasoning for different modal data combinations under the guidance of clinicians. Each example should contain a simulated input part and a simulated output part. The simulated input part includes: simulated instruction requirements, simulated patient information description, simulated strong logical evidence chain, and simulated weak logical evidence chain. The simulated output part includes: simulated detailed diagnostic reports and personalized intervention strategies, simulated reasoning processes, and simulated thinking logic trees.

[0065] Step e2: According to the modal type in the inference rule obtained in step d3, the corresponding question and answer examples are extracted from the question and answer example library with progressive reasoning constructed in step e1 as the thinking guidance information part in the logical chain prompt.

[0066] S202, model training (1) Constructing a global multimodal-multi-level connection graph Step f1: Collect multimodal data of several Alzheimer's patients and divide them into training set, validation set and test set according to the proportion. And set basic training parameters, including the number of training times, etc.

[0067] Step f2: Preprocess the multimodal data of the training set in step f1 to obtain the graph structure data corresponding to each modal data of each individual in the training set. , can be executed according to the content of the multi-modal-multi-level connection graph fusion module in step S201, to obtain Corresponding status .

[0068] Step f3: Execute the steps in the expert consensus driven module of step S201 to obtain the global multimodal-multi-level connection map composed of the training set data .

[0069] (2) Constructing individual multimodal-multi-level connection maps Step g1: The graph structure data of each modality of the individuals obtained in the training set of step f2 Corresponding status , forming several triplets As the prediction triple. The global multimodal-multi-level connection map obtained from step f3 Extract triples from as training triplets.

[0070] Step g2: and Input into the analogy deduction module based on multimodal-multi-level connection map, execute the specific process in the analogy deduction module based on multimodal-multi-level connection map in step S201, and obtain the predicted triple . The states corresponding to each modal data of the individual in step f2 are All triples obtained after reasoning are combined to obtain an individual multimodal-multi-level connection graph .

[0071] Step g3: Construct a relaxed loss that minimizes the example With prediction Hidden feature representation of , while maximizing the hidden feature representation of the head and tail entities in the prediction pair accomplish:

[0072] (3) Mining explicit inference rules Step h1: Individual multimodal-multi-level connection map obtained in step g2 The features of the disease detection standard modality in the , such as "scale shows cognitive impairment" as the termination node , other nodes as starting nodes .

[0073] Step h2: Input the explicit inference rule calculation module based on multi-modal-multi-level connection graph. During the training of the policy function, the three reward functions defined are as follows:

[0074]

[0075]

[0076] Among them, the path efficiency reward Improve reasoning efficiency by limiting the path length of reinforcement learning interactions with the environment, Represents the path length. Path diversity reward Negative cosine similarity is used to increase the diversity of paths found by the model.

[0077] The total reward function is The Monte Carlo policy gradient is used to update the policy function parameters as follows:

[0078] The loss function involved in this step is It can be expressed as: .

[0079] Step h3: Repeat step d3 to obtain arrive The complete inference rules.

[0080] Step h4: Repeat step h3 to get all starting nodes To the end node The reasoning path of the strong logic evidence chain and the weak logic evidence chain as part of the explicit reasoning rules.

[0081] (4) Tips for building a logical chain Step i1: Execute the steps of the progressive thinking guidance module in step S201 to obtain progressive thinking guidance information.

[0082] Step i2: Construct a patient information description based on the patient (individual) medical records in step g2. The instructions should include detailed requirements for diagnostic reports and personalized intervention strategies in clinical applications.

[0083] Step i3: The instruction requirements in step i2, the patient information description, the explicit reasoning rules obtained in step h4, and the progressive thinking guidance information obtained in step i1 are combined to form a logical chain prompt.

[0084] (5) Model optimization Step j1: Select the medical macro model and execute (1) to (4) in step S202 to obtain a logic chain prompt. In the above steps, the dimensions of all latent space variables are aligned with the semantic space of the macro model.

[0085] Step j2: Input the logic chain prompts into the medical big model to obtain the output after chain prompt normalization.

[0086] Step j3: Status of the inference rule part of the logic chain prompt and relationship are replaced by their corresponding text labels and , taking it as the overall reasoning process label The reasoning process of the output part in step j2 The semantic loss score calculated with this label is:

[0087] in, Respectively Each embedding representation produced after passing through the semantic encoding model.

[0088] Step j4: The loss function of the entire model is:

[0089] All modules and latent space parameters are updated according to the overall loss function.

[0090] Step j5: According to the number of training times in step f1, the model is trained with the training set, and the validation set is used for validation in each iteration. After the iteration, the best model is selected based on the validation results.

[0091] S203, Model Testing Step k1: According to the test set individual data in step f1, execute the steps in the multimodal-multi-level connection graph fusion module in step S201 to obtain the states corresponding to each modality data of the test individual .

[0092] Step k2: Execute steps (2) to (4) in step S202 to obtain logic chain prompts.

[0093] Step k3: Input the logic chain prompts into the medical big model selected in step j1 to obtain an interpretable diagnostic report and personalized intervention strategy for the patient (test individual).

[0094] It should be noted that the above only uses Alzheimer's disease as an example to illustrate this application. This application has wide applicability and can be extended to other complex disease fields, such as oncology, cardiovascular disease, etc. Specifically, in the application process of the model, it is only necessary to replace the multimodal data of the corresponding disease (such as tumor imaging, cardiac ultrasound, blood biomarkers, etc.) as input in the construction stage of the multimodal-multi-level connection map fusion module, and it can be applied to the diagnosis and treatment decisions in other medical fields. By inputting multimodal data of different diseases into this method, this application can construct personalized multimodal connection maps for the specific pathological mechanisms of various complex diseases, and generate corresponding explicit reasoning rules and diagnostic prompts, thereby providing effective support for the precise diagnosis and treatment of different diseases.

[0095] like Figure 6 As shown, it is a schematic diagram of the knowledge-guided medical large model fine-tuning optimization system of the present application, which may include: A data acquisition module, used to acquire multimodal data to be diagnosed; The prompt output module is used to input the multimodal data to be diagnosed into the knowledge guidance model to obtain the thinking guidance information part of the logic chain prompt, which is used to input into the medical big model to obtain the output after the chain prompt standardization; The knowledge guidance model includes a multi-modal-multi-level connection graph fusion module, an expert consensus driving module, an analogy deduction module, an explicit deduction rule calculation module and a progressive thinking guidance module; The multi-modal-multi-level connection graph fusion module is used to obtain the overall embedded features as the state according to the modal data graph of each modal data of the disease to be diagnosed; The expert consensus driving module is used to extract expert consensus and clinical experience from the expert knowledge base of the disease to be diagnosed, combine the state semantic labels corresponding to the overall embedded features, determine the state relationship labels between the overall embedded features, and extract the embedded feature representation of the relationship from the state relationship labels, and then combine the overall embedded features, state semantic labels, state relationship labels and the embedded feature representation of the relationship to construct a state hierarchical map, and then combine the modal data map of the modal data to obtain a global multimodal-multi-level connection map; The analogy deduction module is used to construct an individual multimodal-multi-level connection map based on multiple groups of triples through analogy deduction according to the global multimodal-multi-level connection map; the triples include two overall embedded features and corresponding embedded feature representations; The explicit derivation rule calculation module is used to obtain complete inference rules for all nodes according to each node of the individual multimodal-multi-level connection graph; The progressive thinking guidance module is used to combine the global multi-modal-multi-level connection graph and the modal types in the complete reasoning rules of all nodes to obtain the thinking guidance information part in the logical chain prompt.

[0096] It should be noted that in the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of each module is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another device, or some features can be ignored or not executed. The module described as a separate component may or may not be physically separated. The component displayed as a module may be a physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed in multiple different places. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0097] In addition, each module in each embodiment of the present invention may be integrated into a processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0098] The present application also proposes a computer program product, which includes instructions, and when the instructions are executed by a processor, the steps in the above-mentioned knowledge-guided medical large model fine-tuning optimization method embodiment are implemented.

[0099] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in a memory and executed by the processor to accomplish the present invention.

[0100] The carrier for implementing the above-mentioned computer program product may be a computer device, or the computer program product may be stored in a computer storage medium.

[0101] The computer device may be a desktop computer, a notebook computer, a palm computer, a cloud server, etc. The computer device may include, but is not limited to, a processor and a memory.

[0102] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0103] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0104] If the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0105] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A knowledge-guided fine-tuning optimization method for a large medical model, characterized in that: include: Acquire multimodal data to be diagnosed; Input the multimodal data to be diagnosed into the knowledge guidance model to obtain the thinking guidance information part in the logic chain prompt, which is used to input into the medical big model to obtain the output after the chain prompt standardization; The knowledge guidance model includes a multi-modal-multi-level connection graph fusion module, an expert consensus driving module, an analogy deduction module, an explicit deduction rule calculation module and a progressive thinking guidance module; The multi-modal-multi-level connection graph fusion module is used to obtain the overall embedded features as the state according to the modal data graph of each modal data of the disease to be diagnosed; The expert consensus driving module is used to extract expert consensus and clinical experience from the expert knowledge base of the disease to be diagnosed, combine the state semantic labels corresponding to the overall embedded features, determine the state relationship labels between the overall embedded features, and extract the embedded feature representation of the relationship from the state relationship labels, and then combine the overall embedded features, state semantic labels, state relationship labels and the embedded feature representation of the relationship to construct a state hierarchical map, and then combine the modal data map of the modal data to obtain a global multimodal-multi-level connection map; The analogy deduction module is used to construct an individual multimodal-multi-level connection map based on the global multimodal-multi-level connection map and multiple groups of triples through analogy deduction; the triples include two overall embedded features and corresponding embedded feature representations; The explicit derivation rule calculation module is used to obtain complete inference rules for all nodes according to each node of the individual multimodal-multi-level connection graph; The progressive thinking guidance module is used to combine the global multi-modal-multi-level connection graph and the modal types in the complete reasoning rules of all nodes to obtain the thinking guidance information part in the logical chain prompt.

2. The knowledge-guided fine-tuning and optimization method for a large medical model according to claim 1 is characterized in that: According to the modal data graph of each modal data of the disease to be diagnosed, the overall embedded features as the state are obtained, including: Based on the feature aggregation layer of the graph neural network, through the aggregation function Aggregate The neighborhood information of each node in the modal data graph of each modal data of the disease to be diagnosed is aggregated to form a global representation of the node, and the overall embedding features of the modal data graph of each modal data of the disease to be diagnosed are extracted.

3. The knowledge-guided fine-tuning and optimization method for a large medical model according to claim 1 is characterized in that: The individual multimodal-multi-level connection graph is constructed based on multiple sets of triples through analogical deduction, including: Obtain example triples based on the global multimodal-multi-level connection graph; The predicted triples and example triples are embedded in the analog feature embedding layer for representation learning. The feature mapping between the modalities is learned through the cross-modal adaptive interaction layer to obtain the embedding representation corresponding to each modality. The multimodal information fusion layer integrates the embedded representations corresponding to each modality, and then obtains the predicted triples and feature representations of the predicted triples after prediction through the reasoning consistency normalization layer and the prediction output layer; According to the predicted triples and the feature representations of the predicted triples, an individual multimodal-multi-level connection graph is constructed.

4. The knowledge-guided fine-tuning and optimization method for a large medical model according to claim 1 is characterized in that: According to each node of the individual multimodal-multi-level connection graph, a complete inference rule for all nodes is obtained, including: Inputting each node of the individual multimodal-multi-level connection graph into the strategy function of inference rule calculation in sequence; The strategy function includes a state coding layer, multiple feature extraction layers and a strategy selection layer connected in sequence, an activation layer is arranged before the state coding layer, the multiple feature extraction layers and the strategy selection layer, and the output of the strategy selection layer also includes normalization processing.

5. The knowledge-guided medical large model fine-tuning optimization method according to claim 1 is characterized in that: The above-mentioned combination of the global multi-modal-multi-level connection graph and the modal types in the complete reasoning rules of all nodes obtains the thinking guidance information part in the logic chain prompt, including: Representative data of different modalities are extracted from the state hierarchical graph of the global multimodal-multi-level connection graph, and a question-answering example library with progressive reasoning is constructed for different modal data combinations. Then, according to the modal types in the complete reasoning rules of all nodes, corresponding question-answering examples are extracted from the question-answering example library as the thinking guidance information part of the logical chain prompt.

6. The knowledge-guided medical large model fine-tuning optimization method according to claim 1 is characterized in that: The loss function used in the training of the analogy deduction module is: in, is the relaxed loss function, is the total number of triplets in the training set, is the sine similarity function, is the hidden feature representation of the example triplet, To predict the hidden feature representation of the triple, To maximize the calculation formula, and They are the hidden feature representations of the head and tail entities in the prediction pair, respectively.

7. The knowledge-guided fine-tuning and optimization method for a large medical model according to claim 1 is characterized in that: The reward function used in the training of the explicit derivation rule calculation module is: in, is the total reward function, Reward for global goals, is the path efficiency reward, Reward for path diversity, is the last node of the individual multimodal-multi-level connection graph. is the path length, is the set of explored paths, is the cosine similarity function, for The coefficient of for The coefficient of for The coefficient of The Monte Carlo policy gradient method is used to update the policy function parameters during the training of the explicit derivation rule calculation module.

8. The knowledge-guided medical large model fine-tuning optimization method according to claim 1 is characterized in that: The loss function used in the training of the knowledge-guided model is: in, is the overall loss function when training the knowledge-guided model, The loss function involved in updating the policy function parameters using the Monte Carlo policy gradient method is recorded as the path optimization loss. Calculate the semantic loss function for the reasoning process and labels of the thinking guidance information part in the logical chain prompt: in, is the reasoning process label, for Each embedding representation produced by the semantic encoding model, for Each embedding representation produced by the semantic encoding model, Provides a logical chain to guide the reasoning process of the information part.

9. A knowledge-guided medical large model fine-tuning optimization system, characterized in that: include: A data acquisition module, used to acquire multimodal data to be diagnosed; The prompt output module is used to input the multimodal data to be diagnosed into the knowledge guidance model to obtain the thinking guidance information part of the logic chain prompt, which is used to input into the medical big model to obtain the output after the chain prompt standardization; The knowledge guidance model includes a multi-modal-multi-level connection graph fusion module, an expert consensus driving module, an analogy deduction module, an explicit deduction rule calculation module and a progressive thinking guidance module; The multi-modal-multi-level connection graph fusion module is used to obtain the overall embedded features as the state according to the modal data graph of each modal data of the disease to be diagnosed; The expert consensus driving module is used to extract expert consensus and clinical experience from the expert knowledge base of the disease to be diagnosed, combine the state semantic labels corresponding to the overall embedded features, determine the state relationship labels between the overall embedded features, and extract the embedded feature representation of the relationship from the state relationship labels, and then combine the overall embedded features, state semantic labels, state relationship labels and the embedded feature representation of the relationship to construct a state hierarchical map, and then combine the modal data map of the modal data to obtain a global multimodal-multi-level connection map; The analogy deduction module is used to construct an individual multimodal-multi-level connection map based on the global multimodal-multi-level connection map and multiple groups of triples through analogy deduction; the triples include two overall embedded features and corresponding embedded feature representations; The explicit derivation rule calculation module is used to obtain complete inference rules for all nodes according to each node of the individual multimodal-multi-level connection graph; The progressive thinking guidance module is used to combine the global multi-modal-multi-level connection graph and the modal types in the complete reasoning rules of all nodes to obtain the thinking guidance information part in the logical chain prompt.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the knowledge-guided fine-tuning and optimization of a large medical model as described in any one of claims 1 to 8 is implemented.

Images