A clinical auxiliary decision system for HER2-positive breast cancer based on multi-modal analysis and an application method thereof

Abstract

The application aims to provide a HER2-positive breast cancer clinical auxiliary decision system based on multi-modal analysis, comprising: a data acquisition layer; a data processing and storage layer for cleaning, format standardization and feature extraction of the patient multi-modal data; an AI model analysis layer comprising a multi-modal deep learning model of a fusion convolutional neural network (CNN), a graph convolutional network (GCN), a multi-layer perceptron (MLP) and a Transformer text encoder, which outputs risk assessment of patient prognosis and treatment response according to multi-source features obtained from the data processing and storage layer; a decision support layer for generating individualized treatment path recommendations according to the risk assessment results generated by the AI model analysis layer, combining clinical guidelines and a rule engine, and presenting the recommended scheme and prediction basis to the clinician through a visual interactive interface; a feedback learning layer for application in the actual clinical execution process.

Description

Technical Field

[0001] This invention belongs to the field of cancer prediction, specifically relating to a clinical auxiliary decision-making system for HER2-positive breast cancer based on multimodal analysis and its application method. Background Technology

[0002] Breast cancer is one of the most prevalent malignant tumors worldwide, with both high incidence and mortality rates. According to molecular subtyping, HER2 (human epidermal growth factor receptor 2) positive breast cancer accounts for approximately 15%–20% of all breast cancer cases. This subtype is characterized by its highly aggressive biological behavior, high recurrence rate, and poor prognosis. However, with the widespread use of HER2-targeted therapies (such as anti-HER2 drugs like trastuzumab, pertuzumab, and T-DM1), patients' disease-free survival and overall survival have been significantly prolonged, representing an important advancement in personalized breast cancer treatment.

[0003] In clinical practice, physicians typically make treatment decisions based on multimodal data such as pathology reports, imaging results, genetic testing reports, and medical records. However, current clinical decision support methods rely heavily on human experience and guideline recommendations, which have the following significant shortcomings:

[0004] Information silos and the difficulty of data integration: Patient data comes from diverse sources, including electronic medical records, imaging data, pathological slides, laboratory tests, and genomic data. Existing systems mostly use distributed storage and lack unified data standards and integration mechanisms, making it difficult for doctors to obtain a comprehensive and accurate description of the patient's condition in a short time, thus affecting the efficiency and quality of treatment decisions.

[0005] Lack of intelligent efficacy prediction models: Although some studies have attempted to use AI technology to assess breast cancer prognosis, most focus on single data modalities (such as imaging or genes), resulting in poor model generalization ability and difficulty in accurately reflecting the response differences and risk distribution of HER2-positive individuals during actual treatment. Existing systems generally fail to effectively integrate radiomics, pathological images, clinical indicators, and molecular biological data, limiting their clinical application in targeted efficacy assessment.

[0006] Difficulty in supporting personalized treatment pathway recommendations: Treatment of HER2-positive breast cancer involves multiple drug combinations and timing options (such as neoadjuvant, adjuvant, and conversion therapy), while most existing decision-making systems only provide static treatment plans or links to general guidelines, lacking the ability to dynamically adjust pathway recommendations based on patient characteristics and risks, and thus failing to meet the requirements of precision medicine.

[0007] Without a closed-loop feedback mechanism, the model cannot adaptively evolve: Traditional AI-assisted platforms mostly output models only once, lacking a mechanism for dynamic adjustment and self-optimization as the patient's treatment progresses. Even if the patient experiences treatment deviations or drug resistance development, the system cannot detect and update the recommended path in a timely manner, posing potential risks.

[0008] Weak support for HER2-targeted therapy subgroups: HER2-related pathological information (such as IHC 3+, FISH amplification, ERBB2 mutation, etc.) often exists in unstructured form in clinical information systems. Traditional AI models have difficulty effectively extracting or modeling related features, resulting in poor model performance on HER2 subtypes and an inability to provide targeted treatment strategy recommendations for these high-risk patients.

[0009] Therefore, there is an urgent need to build an AI-assisted decision-making platform with multimodal data fusion capabilities, the ability to predict treatment efficacy and recommend treatment pathways, and a closed-loop feedback self-learning mechanism. This platform would address the shortcomings of existing technologies in information integration, individual difference assessment, targeted efficacy prediction, and dynamic pathway regulation, and truly serve the precision treatment management of HER2-positive breast cancer. Summary of the Invention

[0010] The purpose of this invention is to provide a clinical decision support system for HER2-positive breast cancer based on multimodal analysis, applicable to treatment decisions for HER2-positive breast cancer. The system is characterized by comprising:

[0011] The data acquisition layer is used to collect multimodal patient data from hospital information systems and medical devices, including electronic medical record texts, laboratory test results, gene sequencing data, medical images, and pathological images.

[0012] The data processing and storage layer is used to clean, standardize, and extract features from the patient's multimodal data, and to store it using a combination of structured database and data lake.

[0013] The AI ​​model analysis layer includes a multimodal deep learning model that integrates convolutional neural networks (CNN), graph convolutional networks (GCN), multilayer perceptrons (MLP), and a Transformer text encoder. This model outputs a risk assessment of patient prognosis and treatment response based on multi-source features obtained from the data processing and storage layer.

[0014] The decision support layer is used to generate personalized treatment path recommendations based on the risk assessment results generated by the AI ​​model analysis layer, combined with clinical guidelines and rule engine, and present the recommended plans and prediction basis to clinicians through a visual interactive interface.

[0015] The feedback learning layer is used to collect patient follow-up information and physician feedback on treatment plan adjustments during actual clinical implementation. This information is then used for model updates and dynamic adjustments to treatment pathways to form a continuously iterative closed-loop decision-making mechanism.

[0016] Furthermore, the data acquisition layer pays special attention to HER2-related information, including the HER2 receptor expression status in the pathology report (IHC score or FISH test results), HER2 staining results in the pathological slide images, ERBB2 copy number variation or mutation status in gene testing, and records of drugs and efficacy of the patient's previous HER2-targeted therapy.

[0017] Furthermore, the data processing and storage layer preprocesses different types of data separately, wherein:

[0018] Key fields such as "HER2 positive (3+)" were extracted from text medical record data using natural language processing.

[0019] Preliminary feature extraction was performed on medical and pathological images using CNNs.

[0020] Gene feature matrices are generated from genomic data through bioinformatics analysis.

[0021] The laboratory data was normalized, and the processed multimodal data was indexed and managed by patient.

[0022] Furthermore, in the AI ​​model analysis layer:

[0023] CNN encoders are used to extract features from medical and pathological images;

[0024] The GCN encoder aggregates copy number variations or mutation information of the HER2 signaling pathway and related genes based on biomolecular interaction maps.

[0025] The MLP encoder performs non-linear mapping of structured clinical indicators (such as age, tumor stage, HER2 status, etc.);

[0026] The Transformer text encoder performs deep semantic analysis on texts such as medical records, pathology and imaging reports, and concatenates or fuses the feature vectors output by each encoder in the multimodal feature fusion layer to finally generate risk scores and efficacy prediction results.

[0027] Furthermore, the risk score is trained using a partial likelihood loss of the Cox proportional hazards model, enabling the model to learn the disease-free survival or recurrence probability distribution of patients with survival time as a supervisory signal.

[0028] Furthermore, the decision support layer automatically generates individualized treatment plans that include HER2-targeted drugs based on the risk scores or efficacy prediction results output by the AI ​​model and in combination with clinical guidelines. The prognostic indicators and main basis of different plans can be displayed on the interface for doctors to select or adjust.

[0029] Furthermore, the decision support layer's recommended protocols for HER2-positive breast cancer patients include:

[0030] Standard regimen: Trastuzumab combined with chemotherapy;

[0031] Enhanced regimen: Trastuzumab combined with pertuzumab and chemotherapy;

[0032] If signs of drug resistance are detected, it is recommended to switch to T-DM1 or other second-line regimens;

[0033] If insufficient pathological response is clearly demonstrated during the perioperative period or after neoadjuvant therapy, it suggests escalating the treatment intensity or changing the medication.

[0034] Furthermore, the closed-loop mechanism of the feedback learning layer includes:

[0035] Monitor imaging assessments, tumor markers, and molecular testing results during patient treatment;

[0036] If the efficacy is found to deviate from the model prediction or drug resistance progresses, the model will be triggered to re-evaluate the new patient data in real time and provide an adjustment plan.

[0037] Real-world cases are regularly collected into the database, and batch learning is used to retrain and validate the multimodal model, iteratively updating the model parameters.

[0038] Furthermore, during model updates, newly added HER2 treatment-related patient samples are weighted, making the model more focused on identifying and predicting efficacy and drug resistance patterns in HER2-positive individuals.

[0039] Furthermore, an application method for a clinical decision support system for HER2-positive breast cancer based on multimodal analysis is characterized by the following steps:

[0040] Acquire and preprocess patient multimodal data;

[0041] The processed features are input into a multimodal deep learning model to obtain patient risk scores and efficacy sensitivity predictions;

[0042] A preliminary treatment pathway is generated based on the prediction results and clinical guidelines, and then confirmed and implemented by the physician.

[0043] During the treatment process, the patient's actual therapeutic effect is monitored regularly, and the treatment plan is dynamically adjusted based on the monitoring results;

[0044] The adjustment process and final treatment outcome are fed back to the system, and the model is periodically optimized and updated, thereby achieving individualized precision treatment for HER2-positive breast cancer patients.

[0045] This invention provides an AI-powered multimodal clinical decision support platform that, targeting the treatment needs of HER2-positive breast cancer, integrates multi-source heterogeneous clinical data and combines deep learning models with path recommendation mechanisms, achieving the following significant beneficial effects:

[0046] Breaking down information silos and enabling multimodal medical data fusion management, this invention's platform supports unified access to both structured and unstructured data, encompassing electronic medical record texts, image data, pathological slide images, gene sequencing results, and laboratory indicators. It particularly enhances the identification and extraction of HER2-related data (such as IHC scores, FISH amplification status, and ERBB2 mutations). Through standardization and normalization mechanisms in the data processing layer, the efficiency and accuracy of data fusion are improved, providing a complete feature foundation for subsequent AI modeling.

[0047] A multimodal deep learning model is established to accurately predict treatment efficacy and prognostic risk. The platform incorporates image CNN, graph convolutional network (GCN), multilayer perceptron (MLP), and Transformer text encoding modules to construct feature subspaces for different types of medical data and uniformly represent individualized disease characteristics through a feature fusion layer. Compared to traditional unimodal models, the fusion model of this invention can more comprehensively capture the biological diversity of HER2-positive breast cancer and significantly improve the prediction accuracy of responses to anti-HER2 drugs (such as trastuzumab and T-DM1).

[0048] This invention enables personalized treatment pathway recommendations and dynamic adjustments. Based on risk scores and efficacy assessments output by the model, the platform automatically generates multiple treatment pathway options and optimizes recommendations by combining clinical guidelines and individualized characteristics. The system supports interactive adjustments to recommended pathways by physicians and records clinical choices and their motivations, achieving personalized treatment planning through human-machine collaboration. Compared to static, template-based platforms, this invention achieves dynamic pathway construction driven by data and supported by rules, making it particularly suitable for clinical scenarios where HER2 treatment strategies are diverse and frequently adjusted.

[0049] A closed-loop feedback learning mechanism is constructed to support model self-evolution. The platform has functions such as continuous follow-up information collection, efficacy and side effect recording, and treatment response biomarker monitoring (e.g., ctDNA). When discrepancies in efficacy or potential drug resistance signals are detected, risk reassessment and pathway adjustment can be proactively triggered. Simultaneously, real-world data is aggregated into the model training pool for periodic retraining and version optimization. Through this feedback mechanism, the platform can continuously enhance its ability to identify and adapt to HER2 treatment subgroups over time, meeting the requirements of precision medicine with significant individual differences.

[0050] Improving the efficiency and consistency of treatment decisions and facilitating refined clinical management, the platform allows physicians to quickly obtain model efficacy scores and recommendation rationale for various treatment pathways, and make rapid decisions based on interface prompts. This effectively enhances clinical efficiency and reduces the risk of misjudgment due to lack of experience. Through model evaluation and guideline rules linkage, the platform provides standardized and quantifiable treatment decision-making basis for medical institutions at different levels nationwide, particularly helping to improve the capacity of primary healthcare institutions in managing HER2-positive breast cancer.

[0051] The platform facilitates the intelligent evaluation and promotion of novel HER2 treatment options (such as ADC drugs). As new drugs (such as trastuzumab deruxtecan and other ADC drugs) are put into use, the platform can quickly incorporate their relevant characteristics and real-world response patterns. Based on the model's learning capabilities, it can predict their potential advantages in specific HER2 mutation subtypes, thereby providing data support for the individualized indication selection and pathway design of new drugs.

[0052] In summary, this invention not only represents a significant technological advancement in system structure, model algorithms, and path generation, but also achieves intelligent, closed-loop, and evolvable precision decision support in the challenging field of HER2-positive breast cancer treatment, demonstrating significant clinical value and promising prospects for wider application. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the overall structure of the present invention;

[0054] Figure 2 This is a structural diagram of the AI ​​multimodal deep learning model of the present invention. Detailed Implementation

[0055] Example 1 presents the overall structure of an AI-powered multimodal clinical decision support system and describes its workflow. The system architecture is designed around the clinical needs of HER2-targeted therapy for breast cancer. It integrates multi-source heterogeneous medical data, predicts the efficacy of anti-HER2 drugs (such as trastuzumab and T-DM1), and supports the development and adjustment of individualized treatment pathways. The system features a layered architecture with closely collaborative modules, forming a closed loop from data acquisition and model analysis to decision support to meet key needs in the treatment management of HER2-positive patients, such as HER2 status extraction, targeted drug sensitivity prediction, and drug resistance monitoring.

[0056] System architecture design: The system mainly consists of the following functional modules:

[0057] Data Acquisition Layer: Connecting the hospital information system and medical equipment, this layer collects multimodal patient data, including electronic medical record texts, laboratory test results, gene sequencing data, and medical and pathological images. For HER2-targeted therapy, data acquisition focuses particularly on HER2-related information sources, such as HER2 receptor expression status in pathology reports (IHC score or FISH test results), H&E and HER2-stained pathological slide images of tumor tissue, ERBB2 (HER2 gene) copy number variations or mutations in gene testing, and patient records of previous targeted therapy treatments. The system aggregates this multi-source data in real time through standardized interfaces, ensuring that important HER2 indicators are fully extracted for subsequent analysis.

[0058] Data Processing and Storage Layer: This layer preprocesses and fuses the collected data, storing multimodal information using a combination of structured databases and data lakes. Preprocessing steps include data cleaning, format standardization, and feature extraction. For example, text medical records undergo natural language processing to identify key fields (such as "HER2 positive (3+)"); medical and pathological images are processed through image processing and preliminary CNN feature extraction; gene sequencing data undergoes bioinformatics analysis to obtain a gene feature matrix (containing HER2 gene status); and laboratory data is normalized to obtain numerical features. All processed data is indexed by patient and stored in a patient feature database to prepare for AI model analysis.

[0059] AI Model Analysis Layer: The core AI engine of the system resides in this layer, composed of multimodal deep learning models (see Example 2 for details). The model receives multi-source features from the data processing layer and outputs predictions for patient prognosis and treatment response. In the HER2 treatment scenario, the model integrates the patient's pathological image features (reflecting HER2 protein expression intensity, etc.), genomic features (such as HER2 amplification status), clinical numerical features (such as HER2 positive / negative, molecular subtyping, tumor stage), and physician report text information to generate targeted predictions, such as the patient's efficacy prediction for trastuzumab treatment (sensitivity or resistance tendency) and disease-free survival / relapse risk. These predictions provide a basis for subsequent treatment pathway decisions.

[0060] Decision Support Layer: Based on the predictions output by the AI ​​model, the system generates personalized clinical pathway recommendations. This layer includes a rule engine and a visual interactive interface: the rule engine combines model predictions with clinical guidelines to formulate optimized treatment plans, while the interface presents the recommended plans and the basis for the predictions to clinicians. In the case of HER2-positive breast cancer patients, the decision support layer prioritizes plans that include HER2-targeted drugs, such as recommending "trastuzumab combined with chemotherapy as first-line treatment," and suggests whether to intensify treatment (e.g., combining with pertuzumab) or prepare a second-line plan (e.g., T-DM1) based on the model's predicted recurrence risk level. Doctors can intuitively see the system's prognostic scores for different plans (e.g., 5-year survival probability, reduction in recurrence risk) and key HER2-related drivers on the interface, thus assisting their decision-making.

[0061] Feedback Learning Layer: The system is designed with a closed-loop feedback mechanism (see Example 3 for details). This layer is responsible for collecting subsequent data and physician feedback during the patient's actual treatment process, and using this data for model updates and pathway adjustments. Specifically, the system continuously monitors follow-up information during patient treatment (such as regular tumor imaging assessment results, changes in tumor marker levels, molecular detection from repeat biopsies, etc.), paying particular attention to HER2-targeted therapy efficacy indicators and drug resistance monitoring signals. If signs of disease progression or drug resistance are detected (e.g., tumor growth shown in imaging after a period of treatment or detection of new HER2 mutations in liquid biopsy), the feedback layer will trigger the model to reassess the risk and adjust the treatment pathway accordingly. Simultaneously, this real-world data is aggregated for periodic model retraining, improving the model's predictive accuracy and generalization ability in HER2 treatment scenarios, forming a mechanism for the AI ​​model's self-evolution.

[0062] Workflow: When the system is running, it provides clinical decision support according to the following workflow (taking HER2 targeted therapy as an example):

[0063] Patient Data Entry: When a breast cancer patient seeks medical attention, their relevant data is first entered into the system. This includes basic demographic information, clinical stage, tumor pathology report, gene testing results, and imaging data. For HER2-positive patients, this stage particularly focuses on obtaining key information such as HER2 status (e.g., IHC test 3+ or FISH confirming amplification) and previous anti-HER2 treatment history. The system automatically extracts the "HER2 positive" field from the medical record text, analyzes the HER2 protein expression intensity from the pathological images, and records data such as whether the patient has received trastuzumab, and the pathological response status of neoadjuvant / adjuvant therapy.

[0064] Multimodal data processing: The system preprocesses and extracts features from data from different sources: key values ​​(such as Ki-67 index, ER / PR status) are directly read from structured data; tumor biomarker descriptions and physician comments are extracted from text data using an NLP model; deep features are extracted from medical images (such as MRI and ultrasound) and pathological slides using a pre-trained CNN; and mutation lists and pathway enrichment features (including HER2 pathway activity) are extracted from genomic data through an analysis pipeline. Subsequently, the system fuses features from all modalities according to patient ID and interfaces them with the AI ​​model.

[0065] AI model prediction: The fused patient multimodal feature vectors are input into a deep learning model of the system (refer to the model structure in Example 2). The model calculates the predicted results for the patient, including prognostic risk scores and efficacy response assessments. Specifically, the model uses the representations learned during training to estimate the patient's disease-free survival distribution (or relapse probability curve) and combines the patient's HER2-related features to assess their sensitivity to anti-HER2 drugs. For example, the model may output that the patient's 5-year disease-free survival rate is 80% (under standard HER2-targeted therapy) and predicts good efficacy against first-line trastuzumab (e.g., the model classification result is "sensitive," with a probability ≥0.9). If the model detects high-risk signals (e.g., combined with multimodal information to determine that the patient has hidden drug resistance risk factors), the risk score will increase, indicating the need to strengthen subsequent treatment strategies.

[0066] Personalized treatment pathway recommendation: Based on model predictions and existing clinical guidelines, the system automatically generates preliminary treatment pathway suggestions. The system lists several options and their expected model outcomes. For example, for HER2-positive early breast cancer patients, Option A: Standard regimen (anthracycline / taxane chemotherapy + trastuzumab), the model predicts a 5-year disease-free survival rate of 80%; Option B: Enhanced regimen (adding pertuzumab to Option A), the model predicts a 5-year disease-free survival rate of 85%, especially beneficial for patients identified by the model as having a high risk of recurrence; Option C: If there are residual lesions after surgery, switch to T-DM1 therapy, the model predicts it can further reduce the risk of recurrence. Considering both model scores and safety, the system recommends Option B as the preferred pathway. Information on all options is presented to physicians through a decision support interface, including key nodes in each pathway (such as the specific drugs and order of neoadjuvant therapy -> surgery -> adjuvant therapy) and the model-predicted efficacy indicators. This pathway recommendation fully demonstrates the application of HER2-targeted therapy in the system: the system automatically considers that HER2-positive patients should use anti-HER2 drugs, and quantifies the expected benefits of different options through models, helping doctors make the best decision among multiple choices.

[0067] Clinical Decision-Making and Implementation: The multidisciplinary team of physicians refers to the system's recommended pathway and, in conjunction with the patient's specific circumstances, finalizes the treatment plan. If the physician adopts the system's recommendation (e.g., choosing trastuzumab combined with chemotherapy and planning to switch to T-DM1 under specific circumstances), the plan implementation phase begins. If the physician adjusts the plan based on experience (e.g., avoiding a certain drug due to the patient's comorbidities), this adjustment and the reasons are also recorded in the system as feedback data. After the plan is finalized, the patient begins treatment according to the pathway, and the system enters monitoring mode.

[0068] Efficacy Monitoring and Feedback: During patient treatment, the system continuously acquires follow-up data. For example, imaging assessment results after every 2-3 treatment cycles, periodic laboratory tests, and adverse reaction reports are input into the system. For HER2 treatment, key monitoring indicators include imaging evidence of tumor shrinkage or enlargement, circulating tumor markers (such as CEA and CA15-3) levels, and newly emerging molecular biomarkers (such as drug resistance-related gene mutations detected by ctDNA). The system's feedback mechanism analyzes this data and compares it with the model's previous predictions: if the patient's efficacy meets expectations (e.g., imaging shows significant tumor shrinkage, consistent with the model's prediction of trastuzumab sensitivity), the system continues the current pathway and uses this positive result to strengthen model confidence; if deviations are found (e.g., the tumor does not shrink after treatment but progresses, suggesting possible drug resistance, inconsistent with the model's optimistic prediction), the system immediately triggers an alert and enters the pathway adjustment process (see Example 3 for details). These real-world data and their differences from model predictions are recorded as data points for subsequent model retraining.

[0069] Treatment Pathway Adjustment and Model Updates: When feedback indicates a need to adjust treatment, the system reassesses the patient's condition and updates the recommended treatment path based on the latest data. For example, if potential resistance is detected during trastuzumab treatment, the system will run the AI ​​model again using the latest tumor characteristics (which may include resistance-related mutations) to predict new risk and efficacy indicators, and then suggest an adjusted treatment path, such as timely switching to T-DM1 or other second-line regimens. After physician confirmation, the new path is implemented. Simultaneously, the system adds the case to its learning library and incorporates it during regular model training, updating model parameters to enable more accurate pattern learning on datasets containing resistance cases. Through this closed loop, the system continuously accumulates practical experience in HER2-targeted therapy, and model performance improves iteratively, enabling early prediction of resistance and adjustment of treatment plans when similar patients emerge in the future. The entire workflow achieves a cycle from data -> model -> decision -> feedback -> re-decision, continuously validated and optimized in clinical practice to guide precision treatment for HER2-positive breast cancer patients.

[0070] In summary, the system structure and process described in Example 1 fully reflect the clinical needs in the HER2-targeted therapy scenario: it obtains key information such as HER2 status from the data level, performs in-depth analysis through AI models, predicts the efficacy of anti-HER2 therapy, and integrates this prediction into the intelligent recommendation and dynamic adjustment of treatment pathways, providing clinicians with a reliable decision support tool.

[0071] Example 2 describes the AI ​​model structure and technical details used in the system. This model is a multimodal deep learning model, comprising sub-modules such as Convolutional Neural Networks (CNN), Graph Convolutional Networks (GCN), Multilayer Perceptrons (MLP), and Transformers. It processes different types of medical data and merges multimodal information through a fusion layer to output predicted results for patient prognosis and treatment response. During model training, techniques such as the Cox proportional hazards model loss function are used to ensure the clinical validity of the prediction results. The following details the components of the model and their role in HER2 efficacy prediction (original technical details and formulas are retained).

[0072] Image Modality CNN Encoder: For medical and pathological image data, the model uses a convolutional neural network to extract high-dimensional features. The CNN encoder takes an input image X and progressively extracts spatial structural features and semantic information through multiple convolutional and pooling layers to form an image feature representation F. img The calculation of each convolution layer can be represented as:

[0073] Where * denotes the convolution operation, W l and bl Let f(⋅) be the weight and bias of the convolution kernel in layer l, and f(⋅) be the nonlinear activation function. The system can use a classic CNN architecture (such as ResNet, DenseNet, etc.) to ensure the effectiveness of feature extraction. In the HER2-targeted therapy scenario, the CNN encoder can process two important types of image data: (a) pathological tissue slide images, including H&E stained slides and HER2 immunohistochemical (IHC) stained slides. Features extracted by CNN can quantify histological morphology and HER2 protein expression levels, such as identifying the proportion of strongly positive (3+) tumor cells in HER2 (IHC) slides, the degree of tumor infiltration of lymphocytes, and other prognostic image patterns. (b) radiological images (such as breast MRI, CT, etc.), where CNN can capture tumor size, spread, and metastatic features. These image features provide the model with intuitive information on tumor burden and biological invasiveness, which helps to assess the patient's response to HER2-targeted drugs (for example, patients with a high baseline tumor burden may require a more intensive treatment regimen, which is reflected in the model as the influence of image features on risk prediction).

[0074] Molecular Feature GCN Encoder: For genomics and molecular feature data, the model introduces a Graph Convolutional Neural Network (GCN) to extract features from biomolecular relationship networks. First, a molecular interaction graph is constructed based on biological knowledge. Nodes represent key genes or molecules (including the HER2 gene ERBB2 and its signaling pathway members, such as PI3KCA and PTEN), and edges represent interactions between genes (such as protein-protein interactions, regulatory relationships, etc.). Each patient has a corresponding molecular feature vector (such as gene expression levels, copy number variations, or mutation states). These features are assigned to the corresponding nodes in the graph as the initial input feature matrix H. (0) GCN learns a high-level representation of each node by iteratively aggregating information from neighboring nodes. For example, a layer update in GCN can be represented as:

[0075] Where A~ is the adjacency matrix of the graph (including self-loops to consider the characteristics of the nodes themselves), D~ is the degree matrix of A~, and W (l) Let be the trainable weight matrix of the l-th layer, and σ be the activation function. Through the above updates, GCN fuses the gene network topology and features layer by layer, ultimately producing the embedding representation of each node. Pooling the embeddings of all nodes as needed (e.g., pooling for disease-related genes or reading the representation of a specific node) yields the overall molecular representation vector F for the patient. gcnIn HER2 efficacy prediction applications, the GCN encoder can highlight biological signals of the HER2 pathway: for example, if a patient's tumor exhibits HER2 gene amplification (significantly increased ERBB2 copy number) or abnormalities in related pathway genes, the values ​​of that node and its neighboring nodes in the initial features will be high. GCN amplifies and transmits this information through neighborhood aggregation, and the generated molecular feature representation will reflect the activation state of the HER2 pathway. This helps the model recognize that the patient's tumor may be more sensitive to HER2-targeted drugs (because HER2 is the main driver gene) or may have an inherent tendency to resist drugs if accompanied by PI3K pathway mutations. In summary, the GCN module enables the model to utilize biological network structures to more effectively capture gene-level synergies and mutation patterns, providing crucial molecular-dimensional information in multimodal fusion.

[0076] Clinical Data MLP Encoder: For structured clinical data, both numerical and categorical, the model employs a multilayer perceptron (MLP) for encoding. Input clinical features include patient demographic information (e.g., age), tumor clinical stage (TNM stage), histological grade, hormone receptor status (ER, PR), HER2 status (positive / negative), and past treatment information (whether chemotherapy, radiotherapy, or targeted therapy have been received). These features, after one-hot encoding or normalization, are input to several fully connected layers for hierarchical mapping, resulting in a fixed-length vector representation F. clinical The computation process of MLP is as follows: Given the input feature vector x, the first layer calculates... Second floor Through several layers of nonlinear transformation, the final output F is obtained. clinical =h n (W) i ,b i (This involves weights and biases for each layer). In this process, the model can learn the weights of different clinical variables on prognosis. For example, the input feature of HER2 positivity will influence the output through the weighted connections of the MLP (after training, if the overall prognosis of HER2-positive patients is better with targeted therapy, the model may assign a risk-reducing weight to the HER2-positive feature; conversely, it reflects a higher risk for HER2-positive patients who do not receive targeted therapy). Similarly, traditional prognostic factors such as age, tumor size, and lymph node status are also represented with corresponding weights in the MLP. Through MLP encoding, the model obtains a condensed representation of clinical features, combining information from various independent variables to supplement subsequent fusion. It is worth noting that if a patient has previously received HER2-targeted therapy and failed (disease progression), the MLP can encode the "drug resistance history" as a risk-increasing signal, thereby alerting the model to potential drug resistance.

[0077] Text Modality Transformer Encoder: Medical scenarios involve a large amount of unstructured text data, such as doctors' medical records, pathology and imaging reports, and discharge summaries. This text often contains important information (such as family history, medication reactions, and side effects) that is difficult to fully quantify into a structured table. Therefore, the model introduces the Transformer architecture to encode this text data. The Transformer uses a self-attention mechanism to model the input sequence. Its basic principle is: first, the text is segmented and embedded to obtain a word vector sequence; then, the Query, Key, and Value matrices are calculated for attention weights. The hidden representation H of the input text sequence... text The self-attention calculation form is:

[0078]

[0079] Where Q=H text W Q K=H text W K V=H text W V (W) Q W K W V (for trainable parameters), d k This refers to the dimension of the key vector. The stacking of multi-head attention and feedforward network layers enables the Transformer to efficiently extract global features and semantic relationships from long texts. Through pre-trained or fine-tuned Transformer models (such as variants like BERT) on medical text, the system transforms each text into a fixed-length vector representation Ftext, reflecting the useful information implicit in the patient's text record. In the HER2 treatment scenario, the Transformer encoder can extract key points from complex descriptions such as: "Postoperative pathology report indicates HER2 3+, Ki-67 30%, and liver metastasis was found at 6-month follow-up," supplementing details not covered by structured data. It can also capture subjective evaluations from physicians in the medical record (such as "good tolerance to trastuzumab but no significant tumor shrinkage"), which directly or indirectly indicate treatment efficacy. Through deep text analysis by the Transformer, the model gains a more comprehensive understanding of the patient's condition, allowing free textual information, including HER2-related content, to be incorporated into decision-making.

[0080] Multimodal feature fusion and output layer: The representation vectors (F) generated by the above-mentioned CNN, GCN, MLP, and Transformer modules img ,F gcn ,F clinical ,F textThe features are merged at the fusion layer. Fusion methods can involve concatenating vectors followed by a fully connected layer, or more advanced methods such as cross-modal attention. However, in this embodiment, a simple and effective strategy is chosen: concatenating the feature vectors of each modality into a comprehensive feature.

[0081] Then, through further nonlinear transformation using several fully connected layers, the final comprehensive patient representation F is obtained. fusion This fusion representation encapsulates key patient information across different modalities, achieving a complementary effect of "1+1>2". For example, for HER2-positive patients, F fusion This will simultaneously reflect: strong HER2 protein expression shown in pathological images, HER2 pathway activity signals extracted by GCN, HER2-positive markers encoded by MLP and previous treatment information, and records of HER2 status and efficacy in text understood by the Transformer. This integrated vector is more comprehensive and robust than any single modality, and can be used to more accurately predict clinical outcomes. Finally, the model's output layer is compared to F... fusion Mapping is performed to generate target predictions. The main prediction task is patient survival risk assessment, and the model outputs a risk score, or log-risk g(x) (where xxx represents all input features of the patient). A higher risk score means a greater risk of the patient experiencing an adverse outcome (relapse or death), and vice versa. To use this output for survival analysis, the Cox proportional hazards model's partial likelihood loss is used to optimize the model parameters during training. Its negative log-likelihood loss function is defined as:

[0082] Where index i iterates through all patients who experienced an event (such as relapse or death), E i For event indicator variables (event occurrence E) i =1, censor E i =0), t i Let L be the survival or follow-up time of patient i. By minimizing L, the model learns to assign higher risk scores to patients who experience events earlier, compared to those with longer survival times, thus maintaining statistical consistency with the Cox model. L2 regularization can also be introduced during training to avoid overfitting. This Cox model loss allows the model to be trained directly with survival time as a supervisory signal, eliminating the need for discrete classification and fully utilizing follow-up data to improve prediction accuracy.

[0083] In the context of HER2 efficacy prediction, with appropriate feature design, the model output can not only represent the overall prognostic risk but also be used to predict the sensitivity of targeted drug efficacy. For example, by explicitly including the variable of whether or not HER2-targeted therapy is received in the input features (this information is known for the patients in the training set), the model will learn the impact of this treatment on risk. When it is necessary to assess the potential benefit of a new patient to targeted drugs, we can compare the model's output under different assumptions: one input uses "assume targeted therapy" as a feature, and another input uses "assume not receiving" as a feature. By comparing the risk scores obtained from each, we can infer whether the patient's risk can be significantly reduced by using targeted drugs. This model-based simulation provides a means of efficacy prediction: if the addition of trastuzumab significantly reduces the risk predicted by the model, it indicates that the model judges the patient to be sensitive to trastuzumab and can benefit significantly; if the risk does not change much or even increases, it suggests possible drug resistance or limited benefit. In terms of model implementation, a dedicated classification output node can be added to predict the response rate of anti-HER2 treatment (using data with clear efficacy assessment endpoints for supervision). For example, during training, the pathological complete remission (pCR) results of patients receiving neoadjuvant HER2 targeted therapy can be labeled as "responding" or "not responding," and the binary cross-entropy loss can be calculated using the sigmoid output for training. Under this multi-task learning strategy, the model simultaneously optimizes the Cox survival loss and efficacy classification loss, achieving a balanced predictive ability for long-term survival and short-term efficacy. Regardless of the specific implementation used, the model can provide a data-driven evaluation of the effectiveness of HER2 targeted therapy based on the fusion of multimodal information.

[0084] Model Training and Validation: During the development phase, the system used large-scale historical breast cancer patient cohort data to train the aforementioned multimodal model, which included a significant number of HER2-positive patients and their detailed follow-up information. Training employed stochastic gradient descent (e.g., the Adam optimizer) to minimize the aforementioned loss function. To accommodate imbalanced data distributions (e.g., fewer HER2-positive cases or limited event occurrence), appropriate weighting of different categories of samples was applied to the loss function, or a batch equalization sampling strategy was employed. During model training, survival model evaluation metrics such as the consistency index (C-index), as well as time-dependent AUC and error rate on the validation set, were continuously monitored to select the optimal model hyperparameters. Experimental results show that the multimodal model has significantly higher predictive accuracy than the single-modal model, particularly in the HER2-targeted therapy subgroup, where it better distinguishes the prognosis of different patients: for example, for patients receiving trastuzumab adjuvant therapy, the model-predicted risk score was highly correlated with the actual recurrence rate (the 5-year disease-free survival rate of the high-risk group was significantly lower than that of the low-risk group), demonstrating that the model effectively captures the characteristic patterns associated with HER2 efficacy.

[0085] Through the model structure design in this embodiment, the system organically integrates information from four aspects: images, genes, clinical data, and text, establishing a comprehensive feature representation for each patient, and using this as a basis for survival and efficacy prediction. In the application of HER2-positive breast cancer, this means the model can intelligently predict the patient's response to anti-HER2 regimens and long-term benefits by combining key information such as HER2 receptor expression and gene amplification, providing precise quantitative references for clinical decision-making.

[0086] Example 3 illustrates how the system recommends treatment pathways (clinical decision-making processes) based on model predictions, and how it continuously optimizes decision quality and model performance through feedback mechanisms in practical applications. The system's pathway recommendation module transforms the prediction results obtained in Example 2 into specific clinical action plans, allowing for dynamic adjustments based on the patient's subsequent condition. The entire process fully considers the characteristics of HER2-targeted therapy, embedding professional knowledge into the decision-making process and meeting actual clinical needs through feedback learning.

[0087] Personalized Treatment Pathway Recommendation: The system utilizes the output of an AI model to customize a personalized treatment path for each patient. A treatment path refers to a series of decision-making nodes and specific treatment options for a patient from the initial diagnosis to subsequent treatment stages, including determining the initial treatment plan, setting treatment evaluation nodes, and adjusting the treatment plan according to established criteria. The core objective of path recommendation is to select the plan that optimizes the patient's prognosis while adhering to clinical guidelines and the patient's individual conditions. The specific recommendation process is as follows:

[0088] Treatment Plan Generation: The system first generates a set of feasible treatment options based on the patient's disease status and past information. This generation process references clinical guidelines and expert experience rules. For example, for HER2-positive breast cancer patients, initial feasible plans typically include: "Standard Plan A: Chemotherapy with trastuzumab," "Intensive Plan B: Chemotherapy with trastuzumab and pertuzumab," and "Alternative Plan C: If contraindicated, switch to the small molecule targeted drug lapatinib," etc. For each plan, the system defines its key milestones (such as medications used in each stage of neoadjuvant therapy -> surgery -> adjuvant therapy), expected timeline, and contingency plans (such as rescheduling plans in case of disease progression).

[0089] Efficacy Prediction: For each alternative treatment plan, the system invokes an AI model to simulate and predict the prognosis after implementing that plan. This is achieved by adjusting the variables related to the "treatment plan" in the input features (similar to the risk comparison method mentioned in Example 2). For example, when simulating plan A, the model input is set to the patient receiving trastuzumab + standard chemotherapy, and the model outputs the corresponding risk score gAg_AgA and 5-year survival rate SAS_ASA; when simulating plan B, the input includes dual-targeted (trastuzumab + pertuzumab) information, and the outputs the risk gBg_BgB and 5-year survival rate SBS_BSB, and so on. These outputs of the model serve as quantitative indicators to assess the efficacy and risk levels of different plans. In addition, the model can also provide more granular predictions, such as the probability of no relapse within one year and the risk of specific adverse events, for comprehensive comparison of the merits of different plans.

[0090] Recommended Treatment Plan: The system ranks and filters treatment plans based on predictive metrics. The primary goal is generally to maximize the main clinical outcome (e.g., long-term survival / disease-free survival), while also considering risk and patient preference. If a particular plan clearly offers the highest survival rate or the lowest risk score, it is recommended as the first-line plan. For example, in high-risk HER2-positive patients (high tumor burden, high recurrence risk), if the model predicts that enhanced plan B significantly improves survival compared to standard plan A, the system will recommend plan B as the first choice. If multiple plans have similar predictive results, secondary factors (e.g., side effects, economic costs) are considered, or physician preferences are sought. During the recommendation process, the system also generates decision explanations, including key features and their contributions (e.g., "HER2 amplification and high Ki-67 increase the risk of recurrence under plan A; plan B, through dual targeting, better inhibits the HER2 pathway; the model predicts a 5% increase in 5-year disease-free survival"). These explanations, along with the recommended plans, are presented to physicians to enhance the transparency and credibility of the decision-making process.

[0091] Clinical Confirmation and Implementation: After receiving the system's recommendation, the physician team will finalize the treatment path based on the patient's specific condition. If the system-recommended plan is adopted, it enters the implementation phase. If the physician makes adjustments based on other considerations (e.g., the patient's comorbidity makes them intolerant to a certain drug, and an alternative plan is used), the system supports the physician in modifying the recommended path and records the reasons for the modification. Taking HER2-targeted therapy as an example, if the system recommends "dual-targeted therapy combined with chemotherapy," but the physician considers the patient's low cardiac function and decides not to use pertuzumab, opting only for trastuzumab, then after the physician adjusts the plan on the system interface, the new path will be the actual implementation plan. The system treats the physician's modifications as human feedback and stores them in the system for future use in improving the recommendation algorithm (see the feedback mechanism below). Through the above process, the system transforms complex model predictions into concrete and executable clinical pathways. Especially in HER2-positive breast cancer, the pathway recommendation module ensures that all decisions are optimized around HER2-targeted therapy: whether it is the selection of first-line treatment or the replacement of the treatment plan after drug resistance occurs, the system plans ahead and makes the best guess based on data, ensuring that the patient obtains the maximum efficacy.

[0092] Treatment process monitoring and dynamic adjustment: Once the treatment pathway begins, the system enters the efficacy monitoring and pathway dynamic adjustment phase. During HER2-targeted therapy, close monitoring of the patient's response to the drug and disease progression is necessary. The system acquires information through preset monitoring nodes:

[0093] Regular assessment milestones: Following the recommended pathway, at key time points (e.g., after 2-3 cycles of neoadjuvant therapy, during surgery, mid-adjuvant therapy, each imaging follow-up), the system prompts healthcare professionals to input assessment results, including imaging findings of tumor shrinkage, pathological assessment results (e.g., postoperative residual tumor size, pathological complete remission), and treatment-related toxicity levels. For HER2-positive patients, important milestones include: clinical and pathological response assessment after neoadjuvant therapy (which determines whether escalation is needed; for example, the KATHERINE study showed that T-DM1 should be initiated if pathological complete remission is not achieved), and imaging follow-up at 6 months of treatment. The system will input this new data into the AI ​​model in real time to update the patient's characteristic representation.

[0094] Real-time model recalculation: Whenever new clinical data is acquired, the system uses the latest information to recalculate the patient's model in real time, obtaining updated risk predictions. If the patient's treatment is effective (e.g., imaging shows significant tumor shrinkage, and the model adjusts its risk assessment accordingly), the system maintains the original treatment plan and marks this stage as successful. If poor treatment is observed (e.g., no change or even worsening of imaging, with a significantly increased model risk score), the system generates an alert, suggesting that the treatment strategy may need to be adjusted. Furthermore, the system assists in judgment by detecting specific resistance signals. For example, for patients receiving trastuzumab, if new tumor lesions appear during follow-up or ctDNA detects HER2 intracellular domain mutations (known to cause trastuzumab resistance), the model will reflect this information as an increased risk, even if current imaging changes are not significant. Combining the model and rule thresholds, the system can identify potential resistance or progression early, thereby promptly suggesting changes to the treatment pathway.

[0095] Pathway Adjustment Decisions: When adjustment criteria are met (model-predicted deterioration, clear evidence of progression, or intolerable toxicity), the system automatically references a library of alternative treatments to develop a new pathway for the patient. For example, in cases of progression after first-line trastuzumab treatment, the system will recommend switching to a second-line regimen: commonly T-DM1 monotherapy or other new drugs from clinical trials. The system will also predict the efficacy of the new regimen to ensure that the adjusted pathway still offers the greatest expected benefit. The entire adjustment recommendation process is similar to the initial recommendation, except that the patient's response to previous treatments and resistance characteristics must be considered (this information is already in the model input). For HER2 treatment, this means that if trastuzumab resistance is detected, the pathway adjustment will tend to include drugs with different mechanisms of action (such as the ADC drug T-DM1 or small molecule tyrosine kinase inhibitors) to overcome resistance. The system will notify the physician of the adjustment recommendation, and the physician will implement the new pathway after confirmation. Through this dynamic adjustment, the system can provide individualized intervention based on real-time changes in the patient, ensuring that the treatment strategy always matches the patient's current disease status.

[0096] Feedback-learning mechanism: While serving individual patients, the system also accumulates knowledge at a broader level, continuously improving itself through feedback learning. This mechanism includes two levels: immediate feedback and offline learning.

[0097] Real-time feedback applications: During each path execution and adjustment, the actual results obtained by the system are immediately used to correct decision parameters. For example, if the system predicts a patient is sensitive to trastuzumab and recommends continued treatment, but the actual outcome shows early progression, the system can lower its confidence in similar patterns for that patient, becoming more conservative in subsequent decisions (e.g., triggering a treatment switch earlier). This real-time feedback optimizes the current patient's path through rule engines or model fine-tuning. Furthermore, subjective feedback from doctors is also included: doctors can rate or comment on the reasonableness of each recommendation on the system interface. The system uses these evaluations as a decision-making reference, making adjustments when encountering similar situations again (e.g., if a recommended treatment is repeatedly rejected by doctors, the system will lower its priority).

[0098] Offline Model Updates: The system aggregates a large amount of accumulated patient data (including input features, model predictions and actual outcomes, treatment pathways, efficacy results, etc.) into a central database, and regularly retrains and validates the AI ​​model. This is a batch learning process, which may occur every few months. As new data is added, the model training set expands, especially with the continuous enrichment of new cases of HER2-targeted therapy. Through retraining, the multimodal model can learn the latest patterns. For example, when a new HER2-targeted drug (such as a newly approved ADC drug) is introduced, with sufficient case data accumulated, the model will be able to incorporate the drug's efficacy factors, thus considering the new drug in future pathway recommendations and accurately predicting its efficacy. This mechanism ensures that the system does not remain in a static state at the time of training, but evolves continuously with medical advancements and local practice data. After each model update, its performance is evaluated on an independent validation set (e.g., C-index improvement, prediction improvement for important subgroups such as HER2-positive patients). Only when the new model outperforms the old model on key metrics and has no significant bias is it deployed online for clinical decision-making to ensure safety and effectiveness.

[0099] Through a feedback mechanism, the system becomes "smarter with use." Particularly in HER2 efficacy prediction and treatment optimization, each patient's treatment experience provides valuable learning material for the model. For example, if patients with a certain HER2-positive subtype (e.g., those with PI3K pathway mutations) generally do not respond to first-line trastuzumab, but respond well to second-line T-DM1, then as more similar cases are input, the model will gradually identify this pattern. When encountering new patients with this subtype in the future, the system may consider more aggressive approaches (such as adding a PI3K inhibitor or early T-DM1) in the initial pathway recommendation, thereby improving efficacy. This demonstrates the integration of analysis and clinical knowledge: AI automatically summarizes implicit patterns through data feedback, improving decision-making.

[0100] In summary, the system path recommendation and feedback mechanism demonstrated in Example 3 enables the AI ​​multimodal clinical decision support system to form a closed loop in practical applications: prediction drives decision-making, and results feed back into the model. This mechanism ensures that decisions related to HER2-targeted therapy can be dynamically optimized for individual circumstances, while the system itself becomes increasingly intelligent and reliable under the continuous nourishment of real-world data, providing strong technical support for anti-HER2 therapy and even broader personalized cancer treatment.

[0101] The circuits and controls involved in this invention are all existing technologies and will not be described in detail here.

[0102] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A clinical decision support system for HER2-positive breast cancer based on multimodal analysis, applied to treatment decisions for HER2-positive breast cancer, characterized in that, The system includes: The data acquisition layer is used to collect multimodal patient data from hospital information systems and medical devices, including electronic medical record texts, laboratory test results, gene sequencing data, medical images, and pathological images. The data processing and storage layer is used to clean, standardize, and extract features from the patient's multimodal data, and to store it using a combination of structured database and data lake. The AI ​​model analysis layer includes a multimodal deep learning model that integrates convolutional neural networks (CNN), graph convolutional networks (GCN), multilayer perceptrons (MLP), and a Transformer text encoder. This model outputs a risk assessment of patient prognosis and treatment response based on multi-source features obtained from the data processing and storage layer. The decision support layer is used to generate personalized treatment path recommendations based on the risk assessment results generated by the AI ​​model analysis layer, combined with clinical guidelines and rule engine, and present the recommended plans and prediction basis to clinicians through a visual interactive interface. The feedback learning layer is used to collect patient follow-up information and physician feedback on treatment plan adjustments during actual clinical implementation. This information is then used for model updates and dynamic adjustments to treatment pathways to form a continuously iterative closed-loop decision-making mechanism.

2. The clinical decision support system for HER2-positive breast cancer based on multimodal analysis according to claim 1, characterized in that, The data acquisition layer pays special attention to HER2-related information, including the HER2 receptor expression status in the pathology report (IHC score or FISH test results), HER2 staining results in pathological slide images, ERBB2 copy number variation or mutation status in gene testing, and records of drugs and efficacy of the patient's previous HER2-targeted therapy.

3. The clinical decision support system for HER2-positive breast cancer based on multimodal analysis according to claim 1, characterized in that, The data processing and storage layer preprocesses different types of data, wherein: Key fields such as "HER2 positive (3+)" were extracted from text medical record data using natural language processing. Preliminary feature extraction was performed on medical and pathological images using CNNs. Gene feature matrices are generated from genomic data through bioinformatics analysis; The laboratory data was normalized, and the processed multimodal data was indexed and managed by patient.

4. The clinical decision support system for HER2-positive breast cancer based on multimodal analysis according to claim 1, characterized in that, In the AI ​​model analysis layer: CNN encoders are used to extract features from medical and pathological images; The GCN encoder aggregates copy number variations or mutation information of the HER2 signaling pathway and related genes based on biomolecular interaction maps. The MLP encoder performs non-linear mapping of structured clinical indicators (such as age, tumor stage, HER2 status, etc.); The Transformer text encoder performs deep semantic analysis on texts such as medical records, pathology and imaging reports, and concatenates or fuses the feature vectors output by each encoder in the multimodal feature fusion layer to finally generate risk scores and efficacy prediction results.

5. A clinical decision support system for HER2-positive breast cancer based on multimodal analysis according to claim 4, characterized in that, The risk score is trained using a partial likelihood loss of the Cox proportional hazards model, enabling the model to learn the distribution of the patient's disease-free survival or recurrence probability with survival time as a supervisory signal.

6. A clinical decision support system for HER2-positive breast cancer based on multimodal analysis according to claim 1, characterized in that, The decision support layer automatically generates individualized treatment plans that include HER2-targeted drugs based on the risk scores or efficacy predictions output by the AI ​​model and combined with clinical guidelines. The prognostic indicators and main evidence of different plans can be displayed on the interface for doctors to select or adjust.

7. A clinical decision support system for HER2-positive breast cancer based on multimodal analysis according to claim 1, characterized in that, The decision support layer's recommendations for HER2-positive breast cancer patients include: Standard regimen: Trastuzumab combined with chemotherapy; Enhanced regimen: Trastuzumab combined with pertuzumab and chemotherapy; If signs of drug resistance are detected, it is recommended to switch to T-DM1 or other second-line regimens; If insufficient pathological response is clearly demonstrated during the perioperative period or after neoadjuvant therapy, it suggests escalating the treatment intensity or changing the medication.

8. A clinical decision support system for HER2-positive breast cancer based on multimodal analysis according to claim 1, characterized in that, The closed-loop mechanism of the feedback learning layer includes: Monitor imaging assessments, tumor markers, and molecular testing results during patient treatment; If the efficacy is found to deviate from the model prediction or drug resistance progresses, the model will be triggered to re-evaluate the new patient data in real time and provide an adjustment plan. Real-world cases are regularly collected into the database, and batch learning is used to retrain and validate the multimodal model, iteratively updating the model parameters.

9. A clinical decision support system for HER2-positive breast cancer based on multimodal analysis according to claim 8, characterized in that, When updating the model, newly added HER2 treatment-related patient samples are weighted to make the model focus more on the identification and prediction of efficacy and drug resistance patterns in HER2-positive individuals.

10. An application method of the HER2-positive breast cancer clinical auxiliary decision-making system based on multimodal analysis according to claim 1, characterized in that, The method includes the following steps: Acquire and preprocess patient multimodal data; The processed features are input into a multimodal deep learning model to obtain patient risk scores and efficacy sensitivity predictions; A preliminary treatment pathway is generated based on the prediction results and clinical guidelines, and then confirmed and implemented by the physician. During the treatment process, the patient's actual therapeutic effect is monitored regularly, and the treatment plan is dynamically adjusted based on the monitoring results; The adjustment process and final treatment outcome are fed back to the system, and the model is periodically optimized and updated, thereby achieving individualized precision treatment for HER2-positive breast cancer patients.

Images