A medical diagnosis and treatment auxiliary method based on multimodal knowledge graph and reinforcement learning collaborative optimization

Abstract

The application relates to a medical diagnosis and treatment auxiliary method based on multimodal knowledge graph and reinforcement learning collaborative optimization, which comprises the following steps: firstly, acquiring multimodal medical data and constructing a multimodal knowledge graph; secondly, deeply fusing the graph knowledge into a multimodal encoder through knowledge injection type representation learning; then, constructing a scanning-drilling dual-mode dynamic reasoning navigation mechanism to generate a multi-step reasoning track containing a visual focus track and reasoning text; finally, constructing a collaborative optimization framework which fuses a knowledge graph supervision signal and a reinforcement learning reward signal to perform strategy optimization. Through the collaborative optimization of the multimodal knowledge graph and the reinforcement learning, the application realizes a medical diagnosis and treatment auxiliary decision with high accuracy and strong interpretability, and effectively solves the technical problems of not deep multimodal data fusion, an uninterpretable reasoning process and strategy optimization collapse.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, specifically a medical diagnosis and treatment assistance method based on the collaborative optimization of multimodal knowledge graphs and reinforcement learning. Background Technology

[0002] With the continuous improvement of medical informatization, massive amounts of multi-source heterogeneous data have accumulated in clinical diagnosis and treatment, including medical images, electronic medical records, laboratory test results, and genomic information. How to effectively integrate and utilize this multimodal data to assist clinical decision-making has become a research hotspot in the field of medical artificial intelligence.

[0003] Currently, artificial intelligence methods for assisting medical diagnosis and treatment mainly face the following technical challenges:

[0004] First, multimodal data fusion is not deep enough. Existing methods mostly focus on single-modal data processing, such as diagnostic reasoning based solely on medical images or structured medical records. However, clinical diagnosis essentially relies on the comprehensive judgment of multi-source information—imaging signs must be combined with clinical symptoms, medical history, and laboratory tests to arrive at an accurate conclusion. Traditional methods struggle to unify medical information from different modalities into a single semantic space for joint reasoning.

[0005] Second, there is a gap between the symbolic representation of medical knowledge and the numerical reasoning of deep learning models. Medical knowledge graphs can provide structured medical concepts, relationships, and logical constraints, but how to effectively inject this symbolic knowledge into neural network models so that they follow medical logic during reasoning remains an unsolved problem. Existing knowledge augmentation methods mostly adopt retrieval-enhanced generative architectures, using knowledge graphs as external retrieval sources, but this approach lacks sufficient depth of knowledge injection and makes it difficult to achieve knowledge-driven reasoning constraints.

[0006] Third, the interpretability of diagnostic support systems is insufficient. The "black box" nature of deep learning makes it difficult for models to provide reasoning that meets the needs of clinical decision-making. Doctors not only need the final diagnostic conclusion, but also need to understand the reasoning chain behind the conclusion—including intermediate steps such as lesion localization, feature analysis, and differential diagnosis. Existing methods still have significant shortcomings in generating visual reasoning trajectories and readable diagnostic evidence.

[0007] Fourth, strategy optimization has limitations. Methods based on purely online reinforcement learning are prone to the "advantage collapse" problem, generating seemingly reasonable but clinically erroneous reasoning paths. Traditional supervised fine-tuning methods, on the other hand, rely on large-scale labeled reasoning data, which is costly to label and difficult to cover rare cases.

[0008] Therefore, in view of the above situation, there is an urgent need to develop a medical diagnosis and treatment assistance method based on the collaborative optimization of multimodal knowledge graph and reinforcement learning, so as to overcome the shortcomings in current practical applications. Summary of the Invention

[0009] The purpose of this invention is to provide a medical diagnosis and treatment assistance method based on the collaborative optimization of multimodal knowledge graph and reinforcement learning, so as to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] A medical diagnostic assistance method based on the collaborative optimization of multimodal knowledge graph and reinforcement learning includes the following steps:

[0012] Step 1: Obtain multimodal medical data of the target object. The multimodal medical data includes medical imaging data, electronic medical record data, and structured examination data. Perform entity recognition and relationship extraction on the multimodal medical data to construct a multimodal knowledge graph.

[0013] Step 2: Based on the multimodal knowledge graph, perform knowledge-injected representation learning, embedding entities from the knowledge graph into the feature representation of the multimodal encoder to generate a knowledge-enhanced multimodal representation model;

[0014] Step 3: Based on the multimodal representation model, construct a dual-modal dynamic reasoning navigation mechanism that includes scanning mode and drilling mode, and generate a multi-step reasoning trajectory that includes visual attention region and reasoning text;

[0015] Step 4: Based on the inference trajectory, construct a collaborative optimization framework that integrates knowledge graph supervision signals and reinforcement learning reward signals, use a dual-source dominance function for policy optimization, and update the model parameters;

[0016] Step 5: Based on the optimized model, reason about the input multimodal data to be diagnosed and generated a structured output containing multi-step reasoning trajectories and treatment suggestions.

[0017] As a further aspect of the present invention: the construction of the multimodal knowledge graph in step one includes: performing feature extraction and entity recognition on medical data of different modalities respectively, mapping each modal entity to a unified semantic space, and constructing a multimodal knowledge graph with multimodal medical entities as nodes and semantic relationships between entities as edges based on a heterogeneous graph attention network.

[0018] As a further aspect of the present invention: the knowledge-injected representation learning in step two includes: using a multimodal encoder to extract visual features of medical images, semantic features of clinical texts, and structural features of knowledge graph entities; embedding knowledge graph entities into the corresponding visual and text features through a cross-attention mechanism; and optimizing the parameters of each encoder through a multi-task collaborative training objective.

[0019] As a further aspect of the present invention: the dual-modal dynamic reasoning navigation mechanism in step three includes: in scanning mode, the model performs a global scan of the medical image and outputs a set of abnormal candidate regions; in drilling mode, the model generates targeted queries for the candidate regions and performs refined analysis; and the switching between scanning mode and drilling mode is controlled by the confidence level change value.

[0020] As a further aspect of the present invention: the collaborative optimization framework in step four includes: constructing an offline guided trajectory library and generating a policy exploration trajectory; designing a composite reward function that includes accuracy rewards, knowledge graph supervision rewards, diversity rewards, and format standardization rewards; calculating the advantage functions of the offline guided trajectory and the policy exploration trajectory respectively; and updating the model parameters by pruning the policy gradient in the trust domain.

[0021] As a further aspect of the present invention, the knowledge graph supervision reward is calculated in the following way: the reasoning conclusion of each step in the reasoning trajectory is converted into a candidate triple, the consistency of the triple is verified by querying the multimodal knowledge graph, and the knowledge supervision reward value is calculated based on the knowledge consistency score.

[0022] As a further aspect of the present invention: the structured output in step five includes a dynamic visual focusing trajectory diagram, step-by-step reasoning basis text, diagnostic conclusions, and confidence assessment.

[0023] This invention also proposes a medical diagnosis and treatment assistance system based on the collaborative optimization of multimodal knowledge graphs and reinforcement learning, comprising:

[0024] The multimodal data acquisition module is used to acquire multimodal medical data of the target object;

[0025] The knowledge graph construction module is used to extract multimodal medical entities and relationships from the multimodal medical data and construct a multimodal knowledge graph.

[0026] The multimodal representation learning module is used to perform knowledge-injected representation learning based on the multimodal knowledge graph to generate a knowledge-enhanced multimodal representation model.

[0027] The dynamic reasoning navigation module is used to generate a multi-step reasoning trajectory that includes visual focusing trajectory and reasoning text based on the multimodal representation model;

[0028] The collaborative optimization training module is used to optimize the strategy based on the inference trajectory by fusing knowledge graph supervision and reinforcement learning.

[0029] The diagnosis and treatment output module is used to generate structured diagnosis and treatment reports containing inference trajectories and treatment suggestions based on the optimized model.

[0030] As a further aspect of the present invention: the collaborative optimization training module includes an offline guided trajectory library construction unit, a strategy exploration trajectory generation unit, a composite reward calculation unit, a dual-source advantage function calculation unit, and a strategy update unit.

[0031] Compared with the prior art, the beneficial effects of the present invention are:

[0032] 1. Deep integration of multimodal knowledge. By constructing a multimodal knowledge graph and injecting knowledge into the underlying representation learning, the problem of insufficient multimodal data integration and inadequate knowledge utilization in traditional methods is solved, realizing unified semantic representation and deep association mining of medical images, clinical texts and structured data.

[0033] 2. Knowledge-driven interpretable reasoning. Through a gaze-guided dynamic reasoning navigation mechanism, the system simulates the progressive diagnostic thinking of clinicians, generating a visual reasoning path that includes visual focus trajectories and reasoning text. This significantly improves the interpretability of the diagnostic and treatment support system, meeting the needs of clinical traceability and quality control.

[0034] 3. Stability of policies through collaborative optimization. By integrating knowledge graph supervision signals with reinforcement learning reward signals into a collaborative optimization framework, the "advantage collapse" problem that easily occurs in purely online reinforcement learning is effectively solved. Policy exploration under knowledge constraints ensures both the consistency of medical logic in reasoning and improves the model's generalization ability.

[0035] 4. Reduced dependence on labeled data. By using knowledge graph supervision signals and reinforcement learning autonomous exploration mechanisms, the reliance on large-scale labeled inference data is reduced, enabling the model to learn effectively under conditions of limited labeled data.

[0036] 5. Dynamic self-evolution capability. Through a dual-source optimization mechanism of online exploration trajectory and offline guided trajectory, the model can continuously optimize its inference strategy during use, and newly generated diagnosis and treatment records and follow-up treatment effects can be fed back to the system for incremental updates. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the overall process of the medical diagnosis and treatment assistance method based on multimodal knowledge graph and reinforcement learning collaborative optimization provided by the present invention.

[0038] Figure 2 This is a schematic diagram of the multimodal knowledge graph construction process provided by the present invention.

[0039] Figure 3 This is a diagram of the knowledge-injected multimodal representation learning model architecture provided by the present invention.

[0040] Figure 4 A schematic diagram of the "scan-drill" dual-modal dynamic reasoning navigation mechanism provided by the present invention.

[0041] Figure 5 This is a schematic diagram of the collaborative optimization training framework provided by the present invention. Detailed Implementation

[0042] The technical solution of this application will be further described in detail below with reference to specific embodiments.

[0043] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0044] This invention provides a medical diagnostic and treatment assistance method based on the collaborative optimization of multimodal knowledge graphs and reinforcement learning, such as... Figure 1 As shown, it includes the following steps:

[0045] Step 1: Multimodal Medical Data Acquisition and Knowledge Graph Construction

[0046] Acquire multimodal medical data of the target object, including: medical imaging data (including radiological images, pathological slides, endoscopic images, etc.), electronic medical record data (including chief complaint, present illness, past medical history, physical examination, etc.), and structured examination data. Perform entity recognition and relation extraction on the multimodal medical data to construct a multimodal knowledge graph. Figure 2 As shown, the specific steps include the following:

[0047] 1.1 Medical Image Feature Extraction

[0048] A 3D convolutional neural network is used to extract hierarchical features from medical image data. First, the input raw images are preprocessed, including resampling to a uniform resolution, grayscale normalization, and preliminary delineation of regions of interest. Then, through multi-layer 3D convolution and pooling operations, low-level features (such as edges and textures), mid-level features (such as lesion morphology and density distribution), and high-level semantic features (such as lesion type and staging signs) are extracted. Finally, the 3D contour representation of the lesion region, density distribution feature vector, and texture evolution feature sequence are output.

[0049] 1.2 Clinical Text Entity Recognition

[0050] A named entity recognition method based on pre-trained language models is used to extract key medical entities from electronic medical records. The pre-trained language models are BioBERT or PubMedBERT models, which are specifically designed for the biomedical field. These models are pre-trained on large-scale biomedical corpora and have better semantic understanding capabilities for medical terms.

[0051] Entity recognition specifically includes: first, segmenting the electronic medical record text into sentences and words; then, inputting the processed text into a pre-trained language model to obtain the context-related vector representation of each word; finally, using a conditional random field layer to perform sequence labeling on the word vectors to identify disease entities, symptom entities, examination entities, treatment entities, and drug entities.

[0052] Building upon entity recognition, a relation classification model based on an attention mechanism is further employed to identify semantic relationships between entities, including relationship types such as "disease-have-symptoms", "examination-detection-disease", and "drug-treatment-disease".

[0053] 1.3 Structured Data Processing

[0054] The structured examination data is standardized. This includes: converting the units of different test indicators to standard units; filling missing values ​​using multiple imputation methods; identifying and labeling outliers based on medical reference ranges; and extracting the temporal variation characteristics of key indicators, including trends (rising, falling, stable), rates of change, and the time points of extreme values.

[0055] 1.4 Construction of Multimodal Knowledge Graph

[0056] The entities extracted from the different modalities are mapped to a unified semantic space. An independent encoder is designed for each modality to convert image features, text entities, and structured indicators into fixed-length semantic vectors. Then, by contrastive learning objectives, the semantic vectors of the same medical concept in different modalities are brought closer together in space, while the semantic vectors of different concepts are moved further apart.

[0057] Based on a unified semantic space, a multimodal knowledge graph is constructed using a heterogeneous graph attention network. In this knowledge graph, nodes represent medical entities of different modalities, including image sign nodes, disease nodes, symptom nodes, examination nodes, and drug nodes; edges represent semantic relationships between entities, and the type of edge is determined by the relation classification results. The heterogeneous graph attention network can employ different attention calculation methods for different types of nodes and edges, thus better modeling the complex interactions between multimodal medical entities.

[0058] Through the above processing, medical information from different modalities is unified into the same semantic space, and the inherent correlations of disease-related factors in multimodal data sources are explored, laying the foundation for subsequent knowledge injection and reasoning.

[0059] Step Two: Knowledge-Injected Multimodal Representation Learning

[0060] Knowledge-injected representation learning is performed based on the multimodal knowledge graph, embedding entities from the knowledge graph into the feature representation of the multimodal encoder to generate a knowledge-enhanced multimodal representation model. For example... Figure 3 As shown, the specific steps include the following:

[0061] 2.1 Construction of Multimodal Encoder

[0062] The image encoder, text encoder, and knowledge encoder are integrated to form a unified multimodal coding architecture.

[0063] The image encoder employs the Vision Transformer architecture, dividing medical images into a sequence of fixed-size image patches. It extracts global dependencies between these patches using a self-attention mechanism, ultimately outputting a visual feature map of the image. Compared to traditional convolutional neural networks, the Vision Transformer is better able to capture long-range spatial relationships within the image.

[0064] The text encoder employs a pre-trained biomedical language model to encode clinical text into context-aware semantic vector sequences. This encoder can process clinical texts of varying lengths and styles, including conversational chief complaints, specialized examination reports, and structured medical records.

[0065] The knowledge encoder employs a graph neural network to encode entities and relationships in a multimodal knowledge graph into structured embedding vectors. Specifically, it uses a graph attention network architecture where each knowledge graph node updates its representation by aggregating information from its neighbors. Simultaneously, it utilizes a meta-path-guided aggregation method to effectively propagate cross-modal association information between different entity types.

[0066] 2.2 Collaborative Training Strategy

[0067] A multi-task collaborative training strategy is adopted, updating each encoder by sharing underlying parameters and jointly optimizing the objective function. The training objective consists of three parts:

[0068] The first task is masked image modeling, which randomly masks parts of an image patch region. The model is trained to predict the features of the masked region based on the context, thereby learning the internal structure and semantic information of the image.

[0069] The second task is masked language modeling, which involves randomly masking some words in clinical texts and training the model to predict the masked medical entities or terms based on the context, thereby enhancing the model's ability to understand medical language.

[0070] The third task is knowledge graph link prediction, which involves randomly covering some triples in the knowledge graph and training the model to predict missing elements based on the graph structure, thereby strengthening the model's understanding of medical knowledge associations.

[0071] The three tasks share the underlying parameters of the multimodal encoder and are trained through alternating optimization, so that the representations of each modality can enhance each other within a unified framework.

[0072] 2.3 Knowledge Injection Mechanism

[0073] The design incorporates a knowledge injection layer, which embeds the entities output by the knowledge encoder into the corresponding image and text features, thereby achieving deep fusion of symbolic knowledge and numerical features.

[0074] For lesion regions in medical images, the spatial location of the lesion in the image is first located by an object detection network. Then, disease entity embeddings that match the features of the lesion are retrieved from the multimodal knowledge graph. The visual features and knowledge features are then fused through multiple rounds of interaction via a cross-attention mechanism to finally generate a knowledge-enhanced image region representation.

[0075] For medical entities in clinical texts, the original word embeddings are replaced or enhanced with knowledge graph entity embeddings. Specifically, when the text encoder identifies a medical entity, it retrieves the corresponding structured embedding from the knowledge graph and adaptively weights and combines the knowledge embedding with the context word vectors through a gating fusion mechanism.

[0076] 2.4 Multimodal Feature Fusion

[0077] This paper utilizes a cross-modal attention mechanism to fuse image features, text features, and knowledge features through multiple rounds of interaction. A multi-head cross-attention structure is employed, enabling image features to focus on relevant information within the text and knowledge, and vice versa. After multiple rounds of interaction, a unified multimodal representation vector is generated through aggregation operations, serving as input for subsequent inference processes.

[0078] Step 3: Dynamic Reasoning Navigation and Trajectory Generation

[0079] Based on the aforementioned multimodal representation model, a bimodal dynamic reasoning navigation mechanism incorporating scanning and drilling modes is constructed to generate a multi-step reasoning trajectory including visual attention regions and reasoning text. For example... Figure 4 As shown, the specific steps include the following:

[0080] 3.1 Dual-modal exploration mechanism

[0081] The design incorporates a dual-modal exploration mechanism called "scan-drill" to simulate the diagnostic process of clinicians who first perform a global scan and then focus on a local area.

[0082] In scanning mode, the model performs a global scan of the medical image, traversing the entire image using a sliding window or attention scan to locate all potentially abnormal regions. Each candidate region is assigned an initial confidence score, indicating the likelihood that the region contains an abnormality.

[0083] In drilling mode, the model performs refined analysis on candidate regions identified during the scanning phase. Specifically, the model generates targeted queries, such as "Does this region have spiculation?" or "Are the lesion edges clear?", and performs targeted analysis by combining local magnification features of the image with disease-symptom association knowledge in the knowledge graph.

[0084] 3.2 Gaze-Guided Reasoning Navigation Layer

[0085] Based on the aforementioned bimodal exploration mechanism, a gaze-guided reasoning navigation layer is constructed to control the timing and direction of switching between the two modes.

[0086] The navigation layer determines its next action by monitoring changes in the confidence level of each candidate region. When the confidence level of a candidate region significantly improves after drilling analysis, the navigation layer continues drilling mode to perform more detailed, multi-round analysis on that region. When the confidence level of a candidate region shows limited improvement after drilling, or when other more suspicious regions emerge, the navigation layer switches back to scanning mode and performs a global scan again.

[0087] 3.3 Inference Trajectory Generation

[0088] The system records the coordinates of the visual attention area and the corresponding inference text for each round of exploration, generating a multi-step inference trajectory. Each trajectory contains three elements: the currently focused spatial area, the inference judgment for that step, and the confidence score for that step.

[0089] The entire reasoning process begins with a global scan, goes through multiple rounds of alternating "scanning-drilling" until the model has fully analyzed all suspicious areas and formed a final diagnostic conclusion, thus forming a complete multi-step reasoning trajectory.

[0090] Step 4: Collaborative Optimization Based on Knowledge Graph Supervision and Reinforcement Learning

[0091] Based on the aforementioned reasoning trajectory, a collaborative optimization framework integrating knowledge graph supervision signals and reinforcement learning reward signals is constructed. A dual-source dominance function is used for policy optimization to update model parameters. For example... Figure 5 As shown, the specific steps include the following:

[0092] 4.1 Construction of Offline Guide Track Library

[0093] By leveraging medical knowledge from knowledge graphs, a high-quality expert reasoning trajectory database is generated through a confidence sampling method.

[0094] First, the inference navigation mechanism from step three is run on the validation set to obtain a large number of candidate inference trajectories. Then, the trajectories are sorted according to the average confidence score of each step, and the trajectories with the highest confidence scores are selected as high-quality candidates. A kernel sampling algorithm is used for sampling, with the sampling probability proportional to the confidence score, and the sampling concentration is controlled by a temperature parameter. Finally, medical experts review and confirm the selected trajectories to build an offline guided trajectory library.

[0095] 4.2 Strategy Exploration Trajectory Generation

[0096] During training, the policy model autonomously generates multiple inference trajectories. Unlike offline guided trajectories, policy exploration trajectories are generated autonomously by the model without external guidance, allowing the model to try different inference paths and decision-making strategies.

[0097] Each exploration trajectory records the model's complete reasoning process, including the selection of visual attention areas at each step, the output of reasoning text, and the final diagnostic conclusion.

[0098] 4.3 Design of Composite Reward Function

[0099] Design a composite reward function to comprehensively evaluate the quality of the inference trajectory from multiple dimensions:

[0100] Accuracy rewards are calculated based on the degree of match between the diagnostic result and the gold standard label. For labeled data confirmed by pathology or follow-up results, exact matching is used to assess the correctness of the diagnostic conclusion; for samples lacking a gold standard, the internal consistency of the diagnostic logic is verified through knowledge graph reasoning.

[0101] The knowledge graph-based supervision and reward system checks whether the conclusion of each step in the reasoning trajectory conforms to the medical knowledge constraints in the knowledge graph. Each conclusive statement in the reasoning text is converted into a structured triple, and the multimodal knowledge graph is queried to verify whether the triple exists or can be derived through a graph path. Trajectories with higher knowledge consistency receive higher rewards.

[0102] Diversity rewards encourage models to cover diverse diagnostic approaches across different reasoning attempts by calculating the spatial separation of the explored regions and the semantic differences in the reasoning text.

[0103] The format standardization reward verifies whether the reasoning text conforms to the preset structured format requirements, ensuring that the output has good readability and clinical usability.

[0104] The final total reward is the weighted sum of the four rewards mentioned above.

[0105] 4.4 Calculation of Dual-Source Dominance Function

[0106] A hybrid optimization method is employed to calculate the advantage function for both offline guided trajectories and strategy-based exploration trajectories. For trajectories from the offline guided trajectory library, normalization is performed based on the mean and standard deviation of the comprehensive reward of all trajectories in the offline trajectory library. For exploration trajectories generated autonomously by the model, normalization is performed based on the mean and standard deviation of the comprehensive reward of all trajectories in the current exploration round.

[0107] 4.5 Strategy Update

[0108] Based on the calculated advantage function, a policy gradient pruning method with a trust region is used to update model parameters. By limiting the magnitude of each update, drastic changes in the model during a single optimization are avoided. The ratio of the probability of the new policy and the old policy taking the same action under the same state is calculated. When this ratio exceeds a preset pruning interval, the optimization objective is pruned. Simultaneously, a KL divergence penalty term is added to the objective function to further constrain the magnitude of policy changes.

[0109] Step 5: Generation and Interpretation of Diagnostic and Treatment Support Results

[0110] The optimized model performs inference on the input multimodal data to be diagnosed, generating a structured output containing multi-step inference trajectories and treatment suggestions. Specifically, this includes the following sub-steps:

[0111] 5.1 Multimodal Input Processing

[0112] The system receives medical images, clinical texts, and structured data of patients to be diagnosed, and extracts fusion features through a multimodal encoder in step two.

[0113] 5.2 Dynamic Reasoning

[0114] The dynamic inference navigation mechanism from step three is invoked to perform a complete diagnostic inference on the patient data. The model starts with a global scan, autonomously determining the timing of switching between scanning and drilling, as well as the area of ​​focus, ultimately generating a complete personalized inference trajectory.

[0115] 5.3 Generation of Treatment Recommendations

[0116] Based on the final diagnostic status derived from the reasoning trajectory, a structured treatment plan is generated. This plan not only includes the disease name and classification, but also recommends corresponding examination and verification plans, treatment options, and follow-up suggestions based on treatment-related knowledge in the knowledge graph.

[0117] 5.4 Interpretable Output

[0118] The output includes a visualized medical report containing the following:

[0119] First, a dynamic visual focus trajectory map is created by overlaying a sequence of attention heatmaps onto the original medical images, presenting the temporal changes in the model's focus area during the diagnostic process in a step-by-step manner.

[0120] Second, the step-by-step reasoning is presented in a structured chain of thought, showing the diagnostic reasoning process and knowledge basis for each step. Each step is marked with the imaging features and medical knowledge cited.

[0121] Third, confidence assessment: assign a confidence score to the diagnostic conclusion and list the main factors that affect the confidence level.

[0122] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these should also be considered within the scope of protection of the present invention. These will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. A medical diagnosis and treatment assistance method based on the collaborative optimization of multimodal knowledge graph and reinforcement learning, characterized in that, Includes the following steps: Step 1: Obtain multimodal medical data of the target object. The multimodal medical data includes medical imaging data, electronic medical record data, and structured examination data. Perform entity recognition and relationship extraction on the multimodal medical data to construct a multimodal knowledge graph. Step 2: Based on the multimodal knowledge graph, perform knowledge-injected representation learning, embedding entities from the knowledge graph into the feature representation of the multimodal encoder to generate a knowledge-enhanced multimodal representation model; Step 3: Based on the multimodal representation model, construct a dual-modal dynamic reasoning navigation mechanism that includes scanning mode and drilling mode, and generate a multi-step reasoning trajectory that includes visual attention region and reasoning text; Step 4: Based on the inference trajectory, construct a collaborative optimization framework that integrates knowledge graph supervision signals and reinforcement learning reward signals, use a dual-source dominance function for policy optimization, and update the model parameters; Step 5: Based on the optimized model, reason about the input multimodal data to be diagnosed and generated a structured output containing multi-step reasoning trajectories and treatment suggestions.

2. The medical diagnosis and treatment assistance method based on multimodal knowledge graph and reinforcement learning collaborative optimization according to claim 1, characterized in that, The construction of the multimodal knowledge graph in step one includes: extracting features and recognizing entities from medical data of different modalities, mapping entities of each modality to a unified semantic space, and constructing a multimodal knowledge graph with multimodal medical entities as nodes and semantic relationships between entities as edges based on a heterogeneous graph attention network.

3. The medical diagnosis and treatment assistance method based on multimodal knowledge graph and reinforcement learning collaborative optimization according to claim 1, characterized in that, The knowledge-injected representation learning in step two includes: using a multimodal encoder to extract visual features of medical images, semantic features of clinical texts, and structural features of knowledge graph entities; embedding knowledge graph entities into corresponding visual and textual features through a cross-attention mechanism; and optimizing the parameters of each encoder through a multi-task collaborative training objective.

4. The medical diagnosis and treatment assistance method based on multimodal knowledge graph and reinforcement learning collaborative optimization according to claim 1, characterized in that, The dual-modal dynamic reasoning and navigation mechanism in step three includes: in scanning mode, the model performs a global scan of the medical image and outputs a set of abnormal candidate regions; in drilling mode, the model generates targeted queries for the candidate regions and performs refined analysis; and the switching between scanning mode and drilling mode is controlled by the confidence level change value.

5. The medical diagnosis and treatment assistance method based on multimodal knowledge graph and reinforcement learning collaborative optimization according to claim 1, characterized in that, The collaborative optimization framework in step four includes: constructing an offline guided trajectory library and generating policy exploration trajectories; designing a composite reward function that includes accuracy rewards, knowledge graph supervision rewards, diversity rewards, and format standardization rewards; calculating the dominance functions of the offline guided trajectories and policy exploration trajectories respectively; and updating model parameters by pruning the policy gradient in the trust domain.

6. The medical diagnosis and treatment assistance method based on multimodal knowledge graph and reinforcement learning collaborative optimization according to claim 5, characterized in that, The knowledge graph supervision reward is calculated as follows: the reasoning conclusion of each step in the reasoning trajectory is converted into a candidate triple, the consistency of the triple is verified by querying the multimodal knowledge graph, and the knowledge supervision reward value is calculated based on the knowledge consistency score.

7. The medical diagnosis and treatment assistance method based on multimodal knowledge graph and reinforcement learning collaborative optimization according to claim 1, characterized in that, The structured output in step five includes a dynamic visual focusing trajectory diagram, step-by-step reasoning text, diagnostic conclusions, and confidence assessments.

8. A medical diagnostic assistance system based on the collaborative optimization of multimodal knowledge graph and reinforcement learning, characterized in that, include: The multimodal data acquisition module is used to acquire multimodal medical data of the target object; The knowledge graph construction module is used to extract multimodal medical entities and relationships from the multimodal medical data and construct a multimodal knowledge graph. The multimodal representation learning module is used to perform knowledge-injected representation learning based on the multimodal knowledge graph to generate a knowledge-enhanced multimodal representation model. The dynamic reasoning navigation module is used to generate a multi-step reasoning trajectory that includes visual focusing trajectory and reasoning text based on the multimodal representation model; The collaborative optimization training module is used to optimize the strategy based on the inference trajectory by fusing knowledge graph supervision and reinforcement learning. The diagnosis and treatment output module is used to generate structured diagnosis and treatment reports containing inference trajectories and treatment suggestions based on the optimized model.

9. The medical diagnosis and treatment assistance system based on multimodal knowledge graph and reinforcement learning collaborative optimization according to claim 4, characterized in that, The collaborative optimization training module includes an offline guided trajectory library construction unit, a strategy exploration trajectory generation unit, a composite reward calculation unit, a dual-source advantage function calculation unit, and a strategy update unit.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images