Computer system based on multi-modal intelligent diagnosis and treatment large model and training method thereof

Abstract

The invention provides a computer system based on a multi-modal intelligent diagnosis and treatment large model and a training method thereof, and belongs to the technical field of medical algorithm models.The computer system comprises a text processing module used for receiving clinical text data and extracting structured text features from the clinical text data; the image processing module is used for receiving the medical image data and extracting image feature representation from the medical image data; the multi-modal fusion module is used for mapping the structured text feature and the image feature representation to a shared medical semantic-image hidden space through an isomorphic projection bridging layer to obtain a comprehensive representation vector; and the decision output module is used for reasoning multiple clinical decision tasks based on the comprehensive representation vector and outputting a task reasoning result. According to the method, conservation of medical semantics and spatial topology can be kept while multi-modal data are fused, so that information submergence and modal bias are reduced, and the reasoning accuracy and robustness of a multi-modal intelligent diagnosis and treatment large model are improved.

Description

Technical Field

[0001] This application relates to the field of medical algorithm model technology, and in particular to a computer system based on a multimodal intelligent diagnosis and treatment large model and its training method. Background Technology

[0002] With the development of deep learning technology, more and more research is exploring how to use convolutional neural networks, visual attention models, and other technologies to analyze medical imaging data, and how to combine multi-source information such as clinical text and laboratory test results for comprehensive diagnosis. Interstitial lung disease (ILD) is a highly heterogeneous and deadly lung disease, encompassing more than 200 different subtypes. In the field of ILD, most existing multimodal intelligent diagnostic and treatment models focus on single-modality data, such as analyzing only high-resolution CT (HRCT) images to assess the degree of fibrosis, and lack models capable of comprehensively analyzing multi-source data such as clinical text, imaging, and pathology.

[0003] While some studies have attempted to fuse multimodal data to simulate expert decision-making processes, in practical applications, these models often employ simple concatenation or fully connected layers for feature fusion. This approach suffers from information overload and modality bias when processing multimodal data. Furthermore, existing techniques fail to adequately consider the conservation of medical semantics and spatial topology during the fusion process, leading to the loss of fine-grained features of each modality during cross-modal distribution alignment, thus impacting the model's diagnostic accuracy and robustness.

[0004] Therefore, there is an urgent need to propose a new computer system based on a multimodal intelligent diagnosis and treatment model to solve the above problems. Summary of the Invention

[0005] The purpose of this application is to provide a computer system based on a multimodal intelligent diagnosis and treatment large model and its training method to solve the above problems.

[0006] To achieve the above objectives, firstly, this application proposes a computer system based on a multimodal intelligent diagnosis and treatment large-scale model, the computer system comprising: The text processing module is used to receive clinical text data of the target object and extract structured text features from the clinical text data; The image processing module is used to receive medical image data of the target object and extract image feature representations from the medical image data; The multimodal fusion module is used to map the structured text features and the image feature representations to a shared medical semantic-image latent space through an isomorphic projection bridging layer to obtain a comprehensive representation vector; The decision output module is used to reason about multiple clinical decision tasks based on the comprehensive representation vector and output the task reasoning results for the target object.

[0007] In some implementations, the multimodal fusion module is further configured to fuse the mapped structured text features and image feature representations in the shared medical semantic-image latent space using a hierarchical cross-modal attention architecture to obtain a comprehensive representation vector. In some implementations, the hierarchical cross-modal attention architecture includes: The underlying submodule is used to extract text modal features and image modal features from the mapped structured text features and image feature representations, respectively; The middle-layer submodule is used to realize the interaction between the text modal features and the image modal features, and output the preliminary fusion features; The high-level submodule is used to combine the preliminary fusion features with fragments of the medical knowledge graph of interstitial lung disease and output a comprehensive representation vector. In some implementations, the multiple clinical decision-making tasks include: a first task of classifying the disease type of interstitial lung disease, a second task of predicting the prognosis of the target population, and a third task of determining the corresponding clauses from the treatment guidelines; the task reasoning results include the disease classification results corresponding to the first task, the prognosis prediction results corresponding to the second task, and the clause matching results corresponding to the third task. In some implementations, the decision output module includes a task-adaptive gating unit for adjusting the feature-shared weights and feature-specific weights among various clinical decision tasks based on the real-time gradient distribution. In some embodiments, the computer system further includes: The uncertainty assessment module is used to assess the confidence level of the task inference results output by the decision output module using Monte Carlo Dropout and temperature scaling techniques. When the confidence assessment result does not belong to the preset confidence interval, a warning sign is generated. In some embodiments, the computer system further includes: The evidence chain display module is used to generate a decision evidence chain that supports the reasoning results of the task. The decision evidence chain includes image heatmaps, indicator deviation lists, and related content in the diagnosis and treatment guidelines.

[0008] Secondly, to achieve the above objectives, this application also proposes a training method for a multimodal intelligent diagnosis and treatment large-scale model, the method comprising: A training set containing multimodal data of interstitial lung disease is constructed, wherein each case sample in the training set includes clinical text information, medical imaging information, disease type, prognostic events and guideline content; The text processing module and image processing module in the multimodal intelligent diagnosis and treatment model are pre-trained using the clinical text information and the medical image information, respectively. The pre-trained text processing module and image processing module are connected to the multimodal fusion module and decision output module. Based on the training set, the multimodal intelligent diagnosis and treatment model is jointly trained with multimodal data and multi-task branches. The model parameters of the multi-task branches are optimized simultaneously through backpropagation until the preset iteration termination condition is met, and the trained multimodal intelligent diagnosis and treatment model is obtained.

[0009] In some embodiments, the method further includes: During training, the medical knowledge graph of interstitial lung disease is used as a new knowledge attention head or new input to the multimodal intelligent diagnosis and treatment model. This allows the multimodal intelligent diagnosis and treatment model to dynamically retrieve medical knowledge graph fragments of interstitial lung disease corresponding to case samples through the new knowledge attention head or new input, and embed the corresponding medical knowledge graph fragments into the model reasoning process. In some embodiments, the method further includes: The multimodal intelligent diagnosis and treatment model under training is evaluated using validation data independent of the training set, and the hyperparameter combination corresponding to the evaluation result is determined by the AutoML-Bayes controller. The multimodal intelligent diagnosis and treatment model under training is optimized by the hyperparameter combination until the preset iteration termination condition is met, and the trained multimodal intelligent diagnosis and treatment model is obtained.

[0010] Compared with the prior art, the beneficial effects of this application include: Firstly, by employing an isomorphic projection bridging layer, this application can better integrate feature data from different modalities while maintaining the conservation of medical semantics and spatial topology, thereby reducing information overload and modal bias, and improving the inference accuracy and robustness of the multimodal intelligent diagnosis and treatment model.

[0011] Secondly, through multimodal feature data fusion and reasoning of multiple clinical decision-making tasks, this application can simulate the comprehensive analysis process of experts, provide a comprehensive judgment on difficult cases of interstitial lung disease, and improve the generalization ability of the multimodal intelligent diagnosis and treatment model under different conditions. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.

[0013] Figure 1 This is a schematic diagram of the model architecture of a multimodal intelligent diagnosis and treatment model in one embodiment; Figure 2 This is a flowchart illustrating the training method for a multimodal intelligent diagnosis and treatment large model in one embodiment. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0015] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0016] For example, the terms "first," "second," etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For instance, without departing from the scope of this application, a first task may be referred to as a second task, and similarly, a second task may be referred to as a first task. Both the first task and the second task are tasks, but they are not the same task.

[0017] For example, the terms "comprising" or "including" used in this application indicate the presence of features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0018] As mentioned earlier, although some studies have attempted to fuse multimodal data to simulate expert decision-making processes, in practical applications, these models often employ simple concatenation or fully connected layers for feature fusion. This approach suffers from information overload and modality bias when processing multimodal data. Information overload refers to the phenomenon where, during multimodal data fusion, the vastly different data volumes and information densities of different modalities can cause information from some modalities to be "overwhelmed" by information from others after fusion. For example, if image data contains rich detail while text data is relatively scarce, simple feature concatenation or fusion may allow image data to dominate the final decision, ignoring or masking useful information from the text data. In such cases, the model may not be able to fully utilize information from all modalities, leading to information loss during the decision-making process and affecting the model's diagnostic accuracy and robustness. Modality bias refers to the tendency of a model to rely on information from a specific modality in multimodal data processing, while neglecting or underutilizing information from other modalities. This may be because data from a particular modality is easier for the model to learn, or that information from that modality is more prominent in the training data, causing the model to over-rely on that modality during decision-making. For example, if a model discovers during training that image data is particularly effective for classification tasks, it might rely primarily on image features during prediction, ignoring features from text or other modalities, even if those features might be crucial in certain situations. Modality bias leads to an imbalance in how the model handles multimodal data, affecting its generalization ability and adaptability to different types of data. Furthermore, existing technologies have failed to adequately consider the conservation of medical semantics and spatial topology during the fusion process. This leads to the loss of fine-grained features of each modality during cross-modal distribution alignment, affecting the model's diagnostic accuracy and robustness. Conservation of medical semantics refers to maintaining the medical meaning and contextual information of data during data processing and model training. For example, in processing medical images, important lesion features or key symptom descriptions in text data should not be lost or misinterpreted during fusion. Conservation of spatial topology means that in multimodal fusion, not only should features in the image be preserved, but also the spatial relationships and structure between different parts of the image. To this end, this application proposes a computer system and its training method based on a multimodal intelligent diagnosis and treatment model, which can better integrate feature data from different modalities while maintaining the conservation of medical semantics and spatial topology, thereby reducing information overload and modality bias, and improving the inference accuracy and robustness of the multimodal intelligent diagnosis and treatment model.

[0019] like Figure 1As shown in the figure, this application provides a computer system based on a multimodal intelligent diagnosis and treatment model. The computer system includes a text processing module, an image processing module, a multimodal fusion module, and a decision output module.

[0020] In this embodiment, the text processing module refers to the module in the multimodal intelligent diagnosis and treatment model used to process and analyze text content. It can be a pre-trained language model in the interstitial lung disease domain, used to receive clinical text data of the target object (such as case text, clinical records, etc.) and extract structured text features (such as symptoms, signs, test results and medical history, etc.) from the clinical text data.

[0021] In some implementations, the pre-trained language model for the interstitial lung disease domain can be based on a general medical language model, which is then further trained in the interstitial lung disease domain. By retraining the general medical language model with a large corpus of Chinese medical literature, clinical guidelines, and case texts related to interstitial lung disease, the retrained language model for the interstitial lung disease domain can master the specialized terminology and contextual understanding capabilities specific to lung diseases, particularly in the interstitial lung disease domain.

[0022] In this embodiment, the image processing module refers to the module in the multimodal intelligent diagnosis and treatment model used for processing and analyzing image-related content. It can be a 3D-CNN (3D Convolutional Neural Network) / ViT (Vision Transformer, a computer vision model based on the Transformer architecture) parallel network architecture used to receive medical image data of the target object (such as CT images, pathological slide images, etc.) and extract image feature representations from the medical image data. 3D-CNN is a deep learning model specifically designed for processing three-dimensional data, such as CT images, and can extract image feature representations. ViT refers to a visual model applied to pathological slide images, which can process pathological slide images in blocks and capture global image dependencies through the Transformer architecture.

[0023] In this embodiment, the multimodal fusion module is used to map the structured text features and the image feature representations to a shared medical semantic-image latent space through an isomorphic projection bridging layer to obtain a comprehensive representation vector.

[0024] It should be noted that the isomorphic projection bridging layer is a technique used to map data features from different modalities to a shared feature space. It can employ a learnable orthogonal projection matrix to ensure that structured text features and image feature representations are aligned across modal distributions during fusion without losing their respective fine-grained features. The shared medical semantic-image latent space is a multi-dimensional feature space used to represent and store fused features from different modalities (such as text and images). Compared to existing techniques that use "simple stitching + fully connected fusion," this application can explicitly maintain semantic and spatial topological conservation during the projection process, thereby avoiding information overload and modal bias, and improving the diagnostic accuracy and robustness of the multimodal intelligent diagnosis and treatment model for interstitial lung disease.

[0025] In some implementations, the multimodal fusion module is also used to fuse the mapped structured text features and image feature representations in the shared medical semantic-image latent space using a hierarchical cross-modal attention architecture to obtain a comprehensive representation vector.

[0026] Specifically, the hierarchical cross-modal attention architecture includes a bottom-level submodule, a middle-level submodule, and a high-level submodule. The bottom-level submodule extracts text modal features and image modal features from the mapped structured text features and image feature representations, respectively. The middle-level submodule enables the interaction between the text modal features and the image modal features, and outputs preliminary fusion features. For example, it highlights the correlation between the "honeycomb lung" sign in CT images and UIP (Usual Interstitial Pneumonia) lesions in pathological images. Through feature interaction, the middle-level submodule enhances the correlation between different modal features, providing richer information for subsequent feature fusion. The high-level submodule combines the preliminary fusion features with fragments of a medical knowledge graph of interstitial lung disease and outputs a comprehensive representation vector. By introducing a Knowledge Attention Head (KG-Attn Head) into the high-level submodule, the multimodal fusion module can dynamically retrieve fragments of the medical knowledge graph (diagnostic criteria, image-pathology correspondence) for interstitial lung disease during training and embed them into Query-Key computation. This allows the multimodal intelligent diagnosis and treatment model to not only learn data patterns but also undergo explicit calibration against clinical causal logic. Explicit calibration means that during the training process of the multimodal intelligent diagnosis and treatment model, it relies not only on data-driven learning (i.e., automatically learning features and patterns from data) but also combines rule-driven methods (i.e., using prior knowledge or rules to guide the learning process). This approach ensures that the output of the multimodal intelligent diagnosis and treatment model not only conforms to data patterns but also to medical logic and causal relationships. This achieves a two-way constraint of "data-driven + rule-driven," significantly improving diagnostic consistency for rare subtypes and long-tail cases of interstitial lung disease.

[0027] In some embodiments, the computer system further includes a sequence processing module for encoding time-series data into time-series feature vectors. The time-series data includes structured test data and time-series physiological signals, such as lung function monitoring data and vital sign monitoring data. A multimodal fusion module then maps structured text features, image feature representations, and time-series feature vectors to a shared medical semantic-image latent space through an isomorphic projection bridging layer to obtain a comprehensive representation vector.

[0028] Specifically, the multimodal fusion module is used to fuse mapped structured text features, image feature representations, and time-series feature vectors in the shared medical semantic-image latent space using a hierarchical cross-modal attention architecture to obtain a comprehensive representation vector. By introducing a sequence processing module, the ability to process time-series data is enhanced. This time-series data includes structured test data and time-series physiological signals, such as lung function monitoring data and vital sign monitoring data. The sequence processing module encodes this time-series data into time-series feature vectors, enabling the model to capture and analyze physiological and pathological information that changes over time, thereby improving the accuracy and robustness of model inference.

[0029] The decision output module in this embodiment is used to reason about multiple clinical decision tasks based on the comprehensive representation vector, and output the task reasoning results for the target object. The multiple clinical decision tasks may include: a first task of classifying interstitial lung disease types, a second task of predicting the prognosis of the target object, and a third task of determining corresponding clauses from treatment guidelines. The task reasoning results include the disease classification result corresponding to the first task, the prognosis prediction result corresponding to the second task, and the clause matching result corresponding to the third task.

[0030] It should be noted that prognostic prediction refers to predicting the possible course and outcome of a disease based on its developmental patterns. Prognostic prediction results include time clues such as judgments of specific consequences and the probability of a certain outcome occurring.

[0031] In some implementations, the decision output module sets up three decoding branches for the first, second, and third tasks, with the branches sharing a comprehensive representation vector through a Task-Adaptive Gate (TAG). The Task-Adaptive Gate is used to adjust the feature-sharing weights and feature-specific weights among the various clinical decision tasks according to the real-time gradient distribution, avoiding multi-task negative transfer (i.e., performance degradation caused by inter-task interference in multi-task learning).

[0032] Real-time gradient distribution refers to the process where each parameter of the model is updated based on the gradient of the loss function during training. The gradient distribution reflects the model's performance and learning progress across different tasks. Shared feature weights are the weights of features that are helpful for all tasks, meaning multiple tasks can share this feature information to improve overall performance. Specific feature weights are the weights of features that are helpful only for a specific task, meaning some features may only have a significant impact on a particular task. For example, if a feature has a positive impact on multiple tasks, TAG increases the shared weight of that feature; conversely, if a feature has a significant impact on only one task, TAG increases the specific weight of that feature.

[0033] This embodiment uses TAG to precisely control feature-shared weights and feature-specific weights, achieving collaborative optimization between tasks rather than mutual interference. This helps the model allocate and utilize features more flexibly in a multi-task environment, thereby improving the overall performance and generalization ability of the model.

[0034] In some embodiments, the computer system further includes an uncertainty assessment module, which assesses the confidence level of the task inference results output by the decision output module using Monte Carlo Dropout and temperature scaling techniques, and generates a warning sign when the confidence assessment result does not fall within a preset confidence interval.

[0035] It's important to note that Monte Carlo Dropout is a method that uses Dropout during model inference to estimate prediction uncertainty. Dropout itself is a regularization technique that prevents overfitting by randomly "dropping" some neurons in the network during training. Monte Carlo Dropout extends this concept to the inference stage, generating a set of multiple model predictions by preserving the randomness of the Dropout layer during inference, thereby estimating prediction uncertainty.

[0036] Temperature scaling is a technique for adjusting the probability distribution of a model's output. In the softmax function, by adjusting the temperature parameter T (usually T=1 representing the original distribution), the "sharpness" of the output probability distribution can be controlled. A temperature T less than 1 makes the distribution more concentrated, while a temperature T greater than 1 makes the distribution smoother. By adjusting the temperature parameter, the confidence level of the model's output can be better controlled. By using Monte Carlo Dropout, the output of each clinical decision task is predicted multiple times, generating a set of prediction results. Temperature scaling is then applied to this set of prediction results for each clinical decision task, using different temperature parameters to adjust the sharpness of the output probability distribution. Based on the adjusted output probability distribution for each clinical decision task, a corresponding confidence interval, such as standard deviation or confidence interval width, is calculated as the confidence assessment result. If the confidence assessment result does not fall within the preset confidence interval, an alert is generated, indicating that human expert review is required. In some implementations, when inference results from different modal data are inconsistent, it indicates that the model's interpretation of the input data is uncertain, or that the model's generalization ability is limited in certain situations. This will trigger the generation of warning indicators to alert the model to potential problems in multimodal data processing, thereby ensuring the accuracy and reliability of the output of the multimodal intelligent diagnosis and treatment model.

[0037] In some embodiments, the computer system further includes an evidence chain display module for generating a decision evidence chain that supports the reasoning results of the task, the decision evidence chain including image heatmaps, indicator deviation lists, and related content in treatment guidelines.

[0038] Image heatmaps are a visualization technique used to highlight key areas or features in medical images. In this embodiment, the image heatmap can be generated by the image processing module to identify areas in the image relevant to disease diagnosis. For example, in lung CT images, the heatmap can highlight potential lesions such as tumors, inflammation, or fibrosis.

[0039] A deviation list refers to a list of medical indicators that significantly differ from the normal range or expected value, and may include laboratory test results, physiological measurements, and other clinical parameters. In this embodiment, the deviation list can be generated by the text processing module or sequence processing module when processing clinical text data or time-series data.

[0040] The related content in the treatment guidelines refers to the clauses in the treatment guidelines that are associated with the reasoning results of the clinical decision-making task. In this embodiment, the related content in the treatment guidelines can be the clauses used by the decision output module during reasoning.

[0041] The evidence chain display module automatically generates and displays decision-making evidence chains that support the task's reasoning results, including image heatmaps, indicator deviation lists, and related content from treatment guidelines. The decision-making evidence chains are clickable, allowing doctors to view detailed image heatmaps, indicator explanations, and guideline content.

[0042] It should be noted that the computer system based on a multimodal intelligent diagnosis and treatment model proposed in this application can be independently deployed and run in a hospital information system environment, providing decision support for clinical practice. During deployment, the computer system based on the multimodal intelligent diagnosis and treatment model is encapsulated as a service interface. Clinicians submit patient information through the intelligent diagnosis and treatment system interface, requesting diagnostic suggestions from the computer system based on the multimodal intelligent diagnosis and treatment model. The computer system uses its trained multimodal intelligent diagnosis and treatment model to perform reasoning on the input multimodal data, outputting task reasoning results and corresponding decision evidence chains. The computer system based on a multimodal intelligent diagnosis and treatment model proposed in this application overcomes the shortcomings of existing technologies, such as strong expert dependence and poor interpretability, through unified multimodal and multi-task modeling, medical knowledge fusion and integration, and closed-loop confidence management, and has significant clinical practical value.

[0043] For example, a pulmonologist inputs data on a complex case of ILD into the system. The data includes electronic medical records, laboratory test results, the patient's high-resolution CT images, and a possible lung biopsy pathology report. The computer system outputs the following reasoning result based on the data: "Diagnosis: Idiopathic pulmonary fibrosis (IPF), differential diagnosis: Chronic allergic pneumonia. Reasoning: The patient presents with progressive dyspnea and dry cough; HRCT shows peripheral honeycomb shadows in both lungs; pathological findings are UIP-type changes. These findings meet the diagnostic criteria for IPF. Given the patient's history of pigeon ownership, chronic allergic pneumonia needs to be differentiated." The system also indicates that this conclusion has a high confidence level. The physician can further query the system-annotated honeycomb lung regions on the images and extracted pathological points to understand the diagnostic basis.

[0044] In the computer system based on a multimodal intelligent diagnosis and treatment model proposed in this application embodiment, in the first aspect, by adopting an isomorphic projection bridging layer, this application can better integrate feature data of different modalities, while maintaining the conservation of medical semantics and spatial topology, thereby reducing information overload and modal bias, and improving the inference accuracy and robustness of the multimodal intelligent diagnosis and treatment model.

[0045] Secondly, through multimodal feature data fusion and reasoning of multiple clinical decision-making tasks, this application can simulate the comprehensive analysis process of experts, provide a comprehensive judgment on difficult cases of interstitial lung disease, and improve the generalization ability of the multimodal intelligent diagnosis and treatment model under different conditions.

[0046] Thirdly, the uncertainty assessment module and the evidence chain display module not only improve the interpretability of the multimodal intelligent diagnosis and treatment model, but also enhance doctors' trust in the task reasoning results of the multimodal intelligent diagnosis and treatment model, thereby promoting the application of artificial intelligence technology in clinical diagnosis and treatment.

[0047] In one embodiment, such as Figure 2 As shown in the figure, this application provides a training method for a multimodal intelligent diagnosis and treatment large model, the method including the following steps: Step S10: Construct a training set containing multimodal data of interstitial lung disease.

[0048] The dataset in this embodiment comes from confirmed ILD cases from multiple hospitals, containing clinical data from at least 500 patients. For each case sample, complete multimodal data were collected, including clinical textual information (such as medical history, physical examination records, laboratory test results, pulmonary function test results, etc.), medical imaging information (such as chest HRCT images and corresponding professional interpretation reports, etc.), pathological slide images (if available) and disease type (obtainable from diagnostic records), genomic data (if applicable), and guideline content (obtainable from subsequent treatment plans) and prognostic events (obtainable from follow-up results). All data underwent anonymization and standardization, using a unified data format and medical terminology coding (e.g., conforming to the HL7FHIR standard) to ensure that data from different sources can be fused and analyzed.

[0049] It should be noted that for unstructured data such as imaging information and pathological slide images, it is also necessary to organize multidisciplinary experts to perform manual annotation, such as outlining fibrotic lesion areas on HRCT images and marking typical feature areas on pathological images, in order to generate high-quality, labeled training samples.

[0050] Step S20: The text processing module and image processing module in the multimodal intelligent diagnosis and treatment model are pre-trained using the clinical text information and the medical image information, respectively.

[0051] In this embodiment, the text processing module is further trained on the interstitial lung disease domain based on a general medical language model. The general medical language model is retrained using clinical text information and a large amount of relevant Chinese medical literature, clinical guidelines, case texts, and other corpora. This retrained language model for the interstitial lung disease domain can master the professional terminology and contextual understanding capabilities specific to lung diseases, particularly interstitial lung disease.

[0052] Meanwhile, the image processing module is pre-trained using medical image databases (such as public ILDHRCT image sets) and collected ILD medical image information to learn the recognition of common lesions.

[0053] In some implementations, a heterogeneous parallel strategy is used to train the text processing module and the image processing module. The text processing module is on GPUs in blocks 0-N, and the image processing module is on GPUs in blocks N+1-M; pipelined streaming is performed within the same mini-batch of training data, which can improve throughput by more than 40%.

[0054] A progressive layer freezing (P-Layer Freezing) approach can also be introduced: first freeze the high-level text processing module and train the image processing module separately; then fine-tune the text processing and image processing modules together, with only 10% of the parameters being fine-tuned globally. This strategy reduces the total GPU training time by approximately 35% while maintaining performance.

[0055] Step S30: Connect the pre-trained text processing module and image processing module with the multimodal fusion module and decision output module, and perform joint training of each module in the multimodal intelligent diagnosis and treatment model with multimodal data and multi-task branches based on the training set. Simultaneously optimize the model parameters of the multi-task branches through backpropagation until the preset iteration termination condition is met, and obtain the trained multimodal intelligent diagnosis and treatment model.

[0056] In this embodiment, a multi-objective combined loss function can be preset, and the loss terms of multiple task branches can be integrated by weighted summation to optimize the model parameters for multi-modal joint training. The multi-objective combined loss function is L_total = α·L_cls + β·L_reg + γ·L_rec + δ·L_KG.

[0057] Where L_cls is the classification loss for the first task, used to measure the accuracy of class prediction (e.g., cross-entropy loss). L_reg is the regression loss for the second task, used to measure the prediction error of continuous values. L_rec is the reconstruction loss, used to measure the difference between the input and the reconstructed output. L_KG is the knowledge graph loss for the third task, used to measure the consistency between the model output and the guideline terms. α, β, γ, and δ are weighting coefficients used to control the proportion of each loss term's contribution to the total loss.

[0058] Step S30 improves the overall performance of the multimodal intelligent diagnosis and treatment model in multi-clinical decision-making scenarios (such as complex tasks like multimodal fusion and structured prediction) by jointly optimizing classification, regression, reconstruction and knowledge graph constraints.

[0059] During training, an early stopping strategy is used to monitor various metrics on validation data independent of the training set. When the performance of the multimodal intelligent diagnosis and treatment model no longer improves, the training is stopped when the preset iteration termination condition is met, in order to prevent overfitting.

[0060] In some implementations, a contrastive learning strategy can also be introduced, which forces the multimodal intelligent diagnosis and treatment model to learn to distinguish subtle differences between cases by constructing a similar case comparison task, thereby improving the discriminative power of feature representation.

[0061] In some implementations, transfer learning can also be used to acquire knowledge from other related tasks. For example, the image processing module of a multimodal intelligent diagnosis and treatment model can be initialized with pre-trained weights on large-scale image data (such as RadImageNet) to accelerate convergence and improve image discrimination capabilities.

[0062] In some embodiments, the method further includes: evaluating the multimodal intelligent diagnosis and treatment model under training using validation data independent of the training set, determining the hyperparameter combination corresponding to the evaluation result through an AutoML-Bayes controller, optimizing the multimodal intelligent diagnosis and treatment model under training through the hyperparameter combination until a preset iteration termination condition is met, thereby obtaining a trained multimodal intelligent diagnosis and treatment model.

[0063] The AutoML-Bayes controller is an automated machine learning tool that uses Bayesian optimization algorithms to search for optimal hyperparameter combinations. It searches for hyperparameters such as α, β, γ, δ, learning rate, regularization strength, network width of the fusion layer, and weight decay in real time, using the Pareto front on the validation set as the convergence criterion to prevent overfitting and task imbalance.

[0064] Through training and optimization using the various implementation methods described above, a high-performance and stable multimodal intelligent diagnosis and treatment model was finally obtained. In a laboratory testing environment, this model achieved a diagnostic accuracy of over 90% for major ILD subtypes, with sensitivity and specificity both exceeding 85%, and a clause matching rate exceeding 85% in terms of consistency with expert guidelines. These indicators are significantly better than the average level of general clinicians, demonstrating the practical clinical auxiliary value of this multimodal intelligent diagnosis and treatment model. Furthermore, the algorithm framework and data used by the multimodal intelligent diagnosis and treatment model are controlled by the local team, allowing the model to be deployed and run independently on a local server, without relying on external third-party models or cloud services, thus ensuring the security of sensitive medical data.

[0065] In some embodiments, the method further includes: During training, the medical knowledge graph of interstitial lung disease is used as a new knowledge attention head or new input to the multimodal intelligent diagnosis and treatment model. This allows the multimodal intelligent diagnosis and treatment model to dynamically retrieve medical knowledge graph fragments of interstitial lung disease corresponding to case samples through the new knowledge attention head or new input, and embed the corresponding medical knowledge graph fragments into the model reasoning process.

[0066] The medical knowledge graph of interstitial lung disease in this embodiment is a compilation of authoritative domestic and international ILD diagnosis and treatment guidelines, standard diagnostic pathways, treatment plans and the latest research results into a graph format. The graph nodes include medical concepts such as disease, symptoms, imaging features, pathological types, and therapeutic drugs, as well as the relationships between them.

[0067] In model training, the medical knowledge graph of interstitial lung disease (ILD) is integrated into the multimodal data fusion process. By introducing a Knowledge Attention Head (KG-Attn Head), the multimodal fusion module can dynamically retrieve fragments of the ILD medical knowledge graph (diagnostic criteria, image-pathology correspondences) during training and embed them into the Query-Key calculation. This allows the multimodal intelligent diagnostic model to not only learn data patterns but also undergo explicit calibration against clinical causal logic. Explicit calibration means that the training process of the multimodal intelligent diagnostic model relies not only on data-driven learning (i.e., automatically learning features and patterns from data) but also incorporates rule-driven methods (i.e., using prior knowledge or rules to guide the learning process). This method ensures that the output of the multimodal intelligent diagnostic model conforms not only to data patterns but also to medical logic and causal relationships. This achieves a two-way constraint of "data-driven + rule-driven," significantly improving diagnostic consistency for rare subtypes and long-tail cases of ILD.

[0068] In model training, the medical knowledge graph of interstitial lung disease is integrated into the model's reasoning process. This can be achieved by introducing knowledge constraints into the model's attention mechanism or by adding knowledge-related features as new inputs to the training samples, ensuring that the model's decisions are based on sound reasoning. This knowledge fusion design improves the model's interpretability and medical credibility, making the model's decision-making process consistent with medical causal logic.

[0069] In the training method of the multimodal intelligent diagnosis and treatment model proposed in this application, firstly, by constructing a multimodal data training set containing clinical text information, medical image information, disease types, prognostic events, and treatment plans, the model can learn richer and more comprehensive feature representations. This fusion of multimodal data enables the model to understand medical data from different perspectives, thereby improving the diagnostic accuracy of complex diseases such as interstitial lung disease.

[0070] Secondly, by adopting a joint training method using multimodal data and multi-task branches, the model can learn multiple tasks simultaneously. This not only improves the model's learning efficiency and reduces training time, but also enhances the model's understanding of the correlation between different tasks, thereby improving the overall diagnostic and treatment effect.

[0071] Thirdly, the model is evaluated using validation data independent of the training set, and automated hyperparameter optimization is performed using the AutoML-Bayes controller, ensuring that the multimodal intelligent diagnosis and treatment model maintains good performance even on unseen data. This automated hyperparameter search method reduces the need for manual adjustments, improves the efficiency and accuracy of model training, and further optimizes model performance by searching for the optimal hyperparameter combination in real time.

[0072] Fourthly, by setting preset iteration termination conditions, the model training process is automated and standardized, avoiding overfitting and resource waste. This setting of iteration termination conditions ensures the sufficiency and effectiveness of model training, while also guaranteeing the model's generalization ability.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0074] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, any of the embodiments or implementations claimed above can be used in any combination. The information disclosed in this background section is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

Claims

1. A computer system based on a multimodal intelligent diagnosis and treatment model, characterized in that, The computer system includes: The text processing module is used to receive clinical text data of the target object and extract structured text features from the clinical text data; The image processing module is used to receive medical image data of the target object and extract image feature representations from the medical image data; The multimodal fusion module is used to map the structured text features and the image feature representations to a shared medical semantic-image latent space through an isomorphic projection bridging layer to obtain a comprehensive representation vector; The decision output module is used to reason about multiple clinical decision tasks based on the comprehensive representation vector and output the task reasoning results for the target object.

2. The computer system based on a multimodal intelligent diagnosis and treatment large model according to claim 1, characterized in that, The multimodal fusion module is also used to fuse the mapped structured text features and image feature representations in the shared medical semantic-image latent space using a hierarchical cross-modal attention architecture to obtain a comprehensive representation vector.

3. The computer system based on a multimodal intelligent diagnosis and treatment large model according to claim 2, characterized in that, The hierarchical cross-modal attention architecture includes: The underlying submodule is used to extract text modal features and image modal features from the mapped structured text features and image feature representations, respectively; The middle-layer submodule is used to realize the interaction between the text modal features and the image modal features, and output the preliminary fusion features; The high-level submodule is used to combine the preliminary fusion features with fragments of the medical knowledge graph of interstitial lung disease and output a comprehensive representation vector.

4. The computer system based on a multimodal intelligent diagnosis and treatment large model according to claim 1, characterized in that, The multiple clinical decision-making tasks include: a first task of classifying the disease types of interstitial lung disease, a second task of predicting the prognosis of the target population, and a third task of determining the corresponding clauses from the treatment guidelines; the task reasoning results include the disease classification results corresponding to the first task, the prognosis prediction results corresponding to the second task, and the clause matching results corresponding to the third task.

5. The computer system based on a multimodal intelligent diagnosis and treatment large model according to claim 1, characterized in that, The decision output module includes a task adaptive gating unit, which is used to adjust the feature-shared weights and feature-specific weights among various clinical decision tasks according to the real-time gradient distribution.

6. The computer system based on a multimodal intelligent diagnosis and treatment large model according to claim 1, characterized in that, The computer system also includes: The uncertainty assessment module is used to assess the confidence level of the task inference results output by the decision output module using Monte Carlo Dropout and temperature scaling techniques. When the confidence assessment result does not belong to the preset confidence interval, a warning sign is generated.

7. The computer system based on a multimodal intelligent diagnosis and treatment large model according to claim 1, characterized in that, The computer system also includes: The evidence chain display module is used to generate a decision evidence chain that supports the reasoning results of the task. The decision evidence chain includes image heatmaps, indicator deviation lists, and related content in the diagnosis and treatment guidelines.

8. A training method for a multimodal intelligent diagnosis and treatment large-scale model, characterized in that, The method includes: A training set containing multimodal data of interstitial lung disease is constructed, wherein each case sample in the training set includes clinical text information, medical imaging information, disease type, prognostic events and guideline content; The text processing module and image processing module in the multimodal intelligent diagnosis and treatment model are pre-trained using the clinical text information and the medical image information, respectively. The pre-trained text processing module and image processing module are connected to the multimodal fusion module and decision output module. Based on the training set, the multimodal intelligent diagnosis and treatment model is jointly trained with multimodal data and multi-task branches. The model parameters of the multi-task branches are optimized simultaneously through backpropagation until the preset iteration termination condition is met, and the trained multimodal intelligent diagnosis and treatment model is obtained.

9. The training method for the multimodal intelligent diagnosis and treatment large model according to claim 8, characterized in that, The method further includes: During training, the medical knowledge graph of interstitial lung disease is used as a new knowledge attention head or new input to the multimodal intelligent diagnosis and treatment model. This allows the multimodal intelligent diagnosis and treatment model to dynamically retrieve medical knowledge graph fragments of interstitial lung disease corresponding to case samples through the new knowledge attention head or new input, and embed the corresponding medical knowledge graph fragments into the model reasoning process.

10. The training method for the multimodal intelligent diagnosis and treatment large model according to claim 8, characterized in that, The method further includes: The multimodal intelligent diagnosis and treatment model under training is evaluated using validation data independent of the training set, and the hyperparameter combination corresponding to the evaluation result is determined by the AutoML-Bayes controller. The multimodal intelligent diagnosis and treatment model under training is optimized by the hyperparameter combination until the preset iteration termination condition is met, and the trained multimodal intelligent diagnosis and treatment model is obtained.

Images