A method, device and electronic equipment for predicting the efficacy of a pulmonary granulomatous disease

Abstract

The application provides a lung granuloma disease drug efficacy prediction method and device and electronic equipment, relates to the technical field of intelligent medical treatment, and the scheme comprises the following steps: acquiring multi-modal case data of a target object; preprocessing original lung tissue pathological images, original chest CT images and original clinical index data respectively to obtain corresponding pathological image data, CT image data and clinical index data; inputting the pathological image data, the CT image data and the clinical index data into a pre-trained multi-modal drug efficacy prediction model to obtain probability data corresponding to a plurality of preset drug efficacy label categories; and generating and outputting drug efficacy prediction data of the target object according to the probability data corresponding to each preset drug efficacy label category. The scheme adopts multi-modal deep learning to fuse pathology, imaging and clinical indexes, automatically extracts microscopic, macroscopic and functional characteristics, objectively outputs drug efficacy prediction data, improves prediction accuracy and generalization ability, and assists clinical medication decision-making.

Description

Technical Field

[0001] This invention relates to the field of smart medical technology, and in particular to a method, device and electronic equipment for predicting the efficacy of drugs for pulmonary granulomatous diseases. Background Technology

[0002] Pulmonary sarcoidosis is a group of systemic pulmonary granulomatous diseases with unknown etiologies and highly heterogeneous natural courses, most commonly affecting the lungs and intrathoracic lymph nodes. Glucocorticoids, as first-line systemic treatment for symptomatic pulmonary sarcoidosis, can rapidly control active inflammation and improve patients' clinical symptoms, pulmonary function, and imaging abnormalities. However, in clinical practice, patients' responses to hormone therapy vary significantly. Some patients are prone to disease relapse or hormone dependence after dose reduction. Long-term use of hormones can also cause various systemic adverse reactions such as metabolic abnormalities, bone damage, infection, and adrenal axis suppression. Therefore, accurately identifying hormone-sensitive individuals before or early in treatment, and predicting the risk of relapse and the degree of organ function reversibility, is crucial for individualized treatment, optimizing hormone tapering regimens, and reducing drug toxicity and side effects. This is also a pressing clinical challenge. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, and electronic device for predicting the efficacy of drugs for pulmonary granulomatous diseases, so as to solve at least one of the above-mentioned technical problems existing in the prior art.

[0004] In a first aspect, to solve the above-mentioned technical problems, the present invention provides a method for predicting the efficacy of drugs for pulmonary granulomatous diseases, comprising: Acquire multimodal case data of the target object; the multimodal case data includes at least the original lung tissue pathological images, original chest CT images, and original clinical indicator data; preprocess the original lung tissue pathological images, original chest CT images, and original clinical indicator data respectively to obtain the corresponding pathological image data, CT image data, and clinical indicator data; input the pathological image data, CT image data, and clinical indicator data into a pre-trained multimodal drug efficacy prediction model to obtain probability data corresponding to multiple preset drug efficacy label categories; generate and output drug efficacy prediction data of the target object based on the probability data corresponding to each preset drug efficacy label category.

[0005] Optionally, the original lung tissue pathological images, original chest CT images, and original clinical indicator data are preprocessed to obtain corresponding pathological image data, CT image data, and clinical indicator data, including: identifying and retaining the effective tissue region in the original lung tissue pathological images; dividing the effective tissue region into image blocks of fixed size at a preset magnification; and generating pathological image data based on each image block; extracting the lung parenchyma region from the original chest CT images; truncating the CT image values ​​of the lung parenchyma region to a preset range; resampling and adjusting them to a preset input size to generate CT image data; and performing layered cleaning on the original clinical indicator data to generate clinical indicator data; the layered cleaning includes at least one of outlier truncation, missing value imputation, dimensional difference elimination, and one-hot coding.

[0006] Optionally, the multimodal drug efficacy prediction model is pre-trained as follows: An initial multimodal drug efficacy prediction model is constructed, comprising a pathological image feature extraction sub-model, a CT image feature extraction sub-model, a clinical indicator feature extraction sub-model, a feature fusion sub-model, and an efficacy prediction sub-model; historical multimodal case data and historical drug efficacy result data corresponding to multiple historical objects are acquired; the historical multimodal case data includes at least historical pathological image data, historical CT image data, and historical clinical indicator data; the historical drug efficacy result data includes drug efficacy label categories corresponding to the historical objects; the historical pathological image data is input into the pre-trained pathological image feature extraction sub-model to obtain historical pathological image features; the historical CT image data is input into the pre-trained CT image feature extraction sub-model. The image feature extraction sub-model obtains historical CT image features; historical clinical indicator data is input into a pre-trained clinical indicator feature extraction sub-model to obtain historical clinical indicator features; the model parameters of the pathological image feature extraction sub-model, CT image feature extraction sub-model, and clinical indicator feature extraction sub-model are frozen, and the initial feature fusion sub-model and initial effect prediction sub-model are trained based on historical pathological image features, historical CT image features, historical clinical indicator features, and drug efficacy label categories to obtain trained feature fusion sub-model and effect prediction sub-model; based on the trained pathological image feature extraction sub-model, CT image feature extraction sub-model, clinical indicator feature extraction model, feature fusion sub-model, and effect prediction sub-model, a multimodal drug efficacy prediction model is obtained.

[0007] Optionally, based on historical pathological image features, historical CT image features, historical clinical indicator features, and drug efficacy label categories, the initial feature fusion sub-model and the initial effect prediction sub-model are trained to obtain the trained feature fusion sub-model and effect prediction sub-model. This includes: constructing the initial feature fusion sub-model and the initial effect prediction sub-model; the initial feature fusion sub-model includes a linear adaptation unit, a routing control unit, multiple pathological feature processing expert sub-networks, a cross-modal interaction unit, and a case-level attention aggregation unit; and inputting historical pathological image features, historical CT image features, and historical clinical indicator features into the linear adaptation unit, so that the linear adaptation unit... The system generates consistent historical pathology adaptation features, historical CT adaptation features, and historical clinical adaptation features. The historical CT and clinical adaptation features are input into the routing control unit, which generates routing control features based on these features. The unit then calculates dynamic routing weights for each of the multiple pathology feature processing expert sub-networks based on these routing control features. Finally, the historical pathology adaptation features are input into each of the multiple pathology feature processing expert sub-networks to obtain multiple historical expert pathology output features. These historical expert pathology output features are then weighted and aggregated according to the dynamic routing weights to generate historical route aggregated pathology features. The process involves: inputting historical route aggregated pathological features, historical CT adaptation features, and historical clinical adaptation features into a cross-modal interaction unit. The cross-modal interaction unit performs at least element-wise multiplication and element-wise difference operations, and then performs non-linear mapping on the results after feature concatenation to generate historical cross-modal joint interaction features. These features are then input as historical features to be aggregated into a case-level attention aggregation unit. The case-level attention aggregation unit calculates the case-level attention weights corresponding to each historical feature to be aggregated. Finally, each historical feature to be aggregated is assigned a case-level attention weight based on its case-level attention weight. A weighted summation is performed to generate a historical global multimodal representation, which is then input into the initial effect prediction sub-model to generate historical prediction probabilities corresponding to multiple preset drug efficacy label categories. A fusion classification loss is constructed based on the historical prediction probabilities and drug efficacy label categories, and a regularization loss is constructed based on the dynamic routing weights. A fusion prediction training loss is constructed based on the fusion classification loss and the regularization loss, and the initial feature fusion sub-model and the initial effect prediction sub-model are trained based on the fusion prediction training loss until the validation set loss meets the preset stopping condition, resulting in the trained feature fusion sub-model and effect prediction sub-model.

[0008] Optionally, the historical pathological image data includes multiple historical image patches; the pathological image feature extraction sub-model is pre-trained as follows: an initial pathological image feature extraction sub-model is constructed, which includes a pre-trained encoder unit, a global representation branch unit, a detail-aware branch unit, and an attention multi-instance aggregation unit; each historical image patch is input into the pre-trained encoder unit to extract the historical depth feature vector corresponding to each historical image patch, and a historical image patch feature set is generated based on each historical depth feature vector; the historical image patch feature set is input into the global representation branch unit and the detail-aware branch unit respectively, so that the global representation branch unit outputs historical global representation features, and the detail-aware branch unit outputs historical local morphological features related to granuloma structure, inflammatory cell infiltration, fibrosis, or extracellular matrix changes; the historical local morphological features are mapped to the same feature space as the historical global representation features, and based on the mapped historical local morphological features and historical global representation features, the feature extraction sub-model is generated. The system constructs a one-way distillation constraint loss based on local representation features. Historical global representation features are input into an attention multi-instance aggregation unit to calculate the attention weights corresponding to each historical image patch feature. These historical global representation features are then weighted and aggregated according to their attention weights to generate historical slice-level pathological image features. The historical image patch features are sorted from high to low attention weights, and the top K features are selected to form a high-response set, while the bottom K features are selected to form a low-response set. The historical image patch features in both the high-response and low-response sets are weighted and aggregated to obtain high-response and low-response features, respectively. A high-low response contrast loss is then constructed based on these features. A classification loss is constructed based on the historical slice-level pathological image features and the drug efficacy label category. Finally, a total training loss is constructed based on the classification loss, the one-way distillation constraint loss, and the high-low response contrast loss. The initial pathological image feature extraction sub-model is then jointly supervised and trained based on this total training loss to obtain the pathological image feature extraction sub-model.

[0009] Optionally, the CT image feature extraction sub-model is pre-trained as follows: An initial CT image feature extraction sub-model is constructed, comprising a 3D voxel block embedding unit, a 3D position encoding unit, a 3D transformer encoding unit, a multi-scale feature fusion unit, a global pooling unit, and a CT-assisted classification unit; each historical CT image data is divided into multiple fixed-size historical 3D voxel blocks, and each historical 3D voxel block is input into the 3D voxel block embedding unit, which maps each historical 3D voxel block to a fixed-dimensional historical CT feature vector; the historical CT feature vectors are input into the 3D position encoding unit to add 3D position encoding to each historical CT feature vector, generating a historical CT feature sequence; the historical CT feature sequence is input into the 3D transformer encoding unit, which, based on 3D... A self-attention mechanism calculates the spatial relationships between different historical 3D voxel blocks and outputs historical CT coding features at multiple levels. Historical shallow CT image features, historical intermediate CT image features, and historical deep CT image features are extracted from these historical CT coding features at multiple levels and input into a multi-scale feature fusion unit to generate historical multi-scale CT representations. These historical multi-scale CT representations are then input into a global pooling unit to generate historical patient-level CT image features. The historical patient-level CT image features are input into a CT-assisted classification unit to generate historical CT prediction probabilities corresponding to multiple preset drug efficacy label categories. The CT classification loss is calculated based on the historical CT prediction probabilities and drug efficacy label categories. The initial CT image feature extraction sub-model is then trained under supervision based on the CT classification loss to obtain the CT image feature extraction sub-model.

[0010] Optionally, the clinical indicator feature extraction sub-model is pre-trained as follows: An initial clinical indicator feature extraction sub-model is constructed, comprising a clinical indicator projection unit, an anomaly degree encoding unit, a clinical prior combination unit, a clinical feature encoding unit, and a clinical auxiliary classification unit; historical clinical indicator data is input into the clinical indicator projection unit, which projects features onto the historical clinical indicator data to generate historical basic clinical features; historical clinical indicator data is input into the anomaly degree encoding unit, which calculates the deviation of continuous clinical indicators from the normal reference range based on a preset clinical reference range, generating historical indicator anomaly degree features; historical basic clinical features are input into the clinical prior combination unit, which calculates the deviation of continuous clinical indicators from the normal reference range based on a preset clinical reference range, generating historical indicator anomaly degree features; and historical basic clinical features are input into the clinical prior combination unit, which projects features onto the historical clinical indicator data based on a preset clinical reference range, generating historical indicator anomaly degree features. A clinical prior rule is established to combine features corresponding to at least two clinical indicators to generate historical multi-indicator joint representation features. Historical basic clinical features, historical indicator anomaly degree features, and historical multi-indicator joint representation features are concatenated to generate historical enhanced clinical features. These enhanced clinical features are then input into a clinical feature encoding unit, which extracts the correlation information between different clinical indicators through nonlinear transformation and outputs historical clinical indicator features. The historical clinical indicator features are then input into a clinical auxiliary classification unit to generate historical clinical prediction probabilities corresponding to multiple preset drug efficacy label categories. A clinical classification loss is constructed based on the historical clinical prediction probabilities and drug efficacy label categories. The initial clinical indicator feature extraction sub-model is then trained under supervision based on the clinical classification loss to obtain the clinical indicator feature extraction sub-model.

[0011] Optionally, based on the probability data corresponding to each preset efficacy label category, generating and outputting efficacy prediction data for the target object includes: sorting multiple preset efficacy label categories according to the size of the probability data; determining the preset efficacy label category with the largest probability data as the target prediction category; generating first prediction data from the target prediction category and its corresponding probability data; generating second prediction data from each preset efficacy label category and its corresponding probability data; and generating and outputting efficacy prediction data for the target object based on the first prediction data and / or the second prediction data.

[0012] Secondly, based on the same inventive concept, this application also provides a device for predicting the efficacy of pulmonary granulomatous diseases, specifically comprising: a data acquisition module for acquiring multimodal case data of the target object; the multimodal case data includes at least original lung tissue pathological images, original chest CT images, and original clinical indicator data; a data processing module for preprocessing the original lung tissue pathological images, original chest CT images, and original clinical indicator data respectively to obtain corresponding target pathological image block datasets, target three-dimensional CT data, and target clinical data; a model inference module for inputting the pathological image data, CT image data, and clinical indicator data into a pre-trained multimodal efficacy prediction model to obtain probability data corresponding to multiple preset efficacy label categories; and a result generation module for generating and outputting the efficacy prediction data of the target object based on the probability data corresponding to each preset efficacy label category.

[0013] Thirdly, based on the same inventive concept, this application also provides an electronic device, specifically including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-mentioned method for predicting the efficacy of pulmonary granulomatous diseases.

[0014] By employing the above technical solution, the present invention has the following beneficial effects: It utilizes a multimodal deep learning architecture that integrates pathological images, chest CT images, and clinical indicators, and leverages a collaborative expert routing mechanism to achieve cross-modal high-order interactive fusion, overcoming the limitations of traditional methods that rely on physician experience or single indicators; it automatically extracts microscopic tissue features, macroscopic imaging features, and clinical functional states, outputting objective glucocorticoid efficacy label categories and corresponding predicted probabilities, providing auxiliary basis for individualized clinical medication decisions; simultaneously, through multimodal complementary collaboration, it effectively reduces the uncertainty of manual judgment, improves the accuracy and generalization ability of efficacy prediction, thereby improving the decision support level for glucocorticoid treatment of pulmonary sarcoidosis. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the drug efficacy prediction method for pulmonary granulomatous diseases provided in an embodiment of the present invention; Figure 2This is a schematic diagram of the structure of the feature fusion sub-model and the effect prediction sub-model provided in the embodiments of the present invention; Figure 3 A schematic diagram illustrating the construction process of the contrast loss provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the construction process of the multimodal pharmacodynamic prediction model provided in an embodiment of the present invention; Figure 5 A schematic diagram of a pulmonary granulomatous disease efficacy prediction device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0017] It should be noted that the method provided in this embodiment of the invention processes pre-collected and stored pathological image data, imaging data, and structured numerical data. The execution of the method does not involve any operation on living human or animal bodies. The drug efficacy prediction data output by this method is an intermediate processing result for medical personnel's reference; it cannot directly lead to a diagnosis of the disease, nor does it directly constitute a treatment plan. Whether to use glucocorticoid treatment and the specific dosing regimen must still be determined independently by a qualified physician based on the patient's complete clinical data.

[0018] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0020] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0021] In related technologies, pulmonary sarcoidosis is a common pulmonary granulomatous disease. Predictive techniques for glucocorticoid efficacy in pulmonary sarcoidosis often rely on physicians' subjective experience or single-modality detection indicators, resulting in strong subjectivity and low accuracy and stability. Clinically, treatment efficacy is routinely predicted using high-resolution chest CT staging, inflammatory burden, serum angiotensin-converting enzyme (ACE) levels, or single-timepoint pulmonary function parameters. However, serum ACE is only a rough surrogate indicator of disease activity, easily affected by factors such as the patient's condition and medications, and has limited sensitivity. Furthermore, single imaging or pulmonary function indicators cannot distinguish between inflammation-dominated and fibrosis-dominated pathological states, making it difficult to adapt to the highly heterogeneous course of pulmonary sarcoidosis and accurately differentiate between patient remission and progression trends, thus failing to meet the clinical need for precise prediction. Simultaneously, current technologies lack automated prediction methods that integrate microscopic pathological tissue features with macroscopic CT imaging features. The core of hormone therapy efficacy in pulmonary sarcoidosis depends on the balance between the reversibility of inflammatory lesions and the fibrotic burden. Active granulomatous inflammation responds well to hormones, while extensive fibrotic lesions are difficult to reverse with hormones. Pathological images accurately reflect microscopic gold standard information such as granulomatous infiltration and fibrosis, while chest CT scans demonstrate macroscopic imaging phenotypes of lesion distribution, inflammation, and fibrosis. However, in clinical practice, these two types of data are interpreted independently and fragmented. Currently, there is no automated technology to fuse microscopic pathological features and macroscopic CT features across scales, failing to fully exploit the diagnostic and therapeutic value of dual-source data and significantly limiting the accuracy of efficacy prediction. Furthermore, the pathological images, CT images, and multidimensional clinical indicators required for predicting the efficacy of pulmonary sarcoidosis are typical heterogeneous, multi-scale data, and existing technologies lack effective feature extraction and deep fusion schemes. Multi-source data differ significantly in physical scale, information density, and semantic level. Simple feature splicing methods cannot uncover complementary correlations between modalities, easily introducing redundant noise and leading to model overfitting and decreased generalization ability. Currently, there are no modeling and prediction methods suitable for the characteristics of pulmonary sarcoidosis, making it impossible to effectively achieve cross-scale feature extraction and deep fusion of multi-source heterogeneous data, and hindering stable and accurate prediction of hormone efficacy.

[0022] To at least partially address one or more of the aforementioned problems and other potential issues, this invention proposes a method for predicting the efficacy of medications in pulmonary granulomatous diseases. This invention employs a multimodal deep learning architecture that integrates pathological images, chest CT images, and clinical indicators. It utilizes a collaborative expert routing mechanism to achieve cross-modal high-order interactive fusion, overcoming the limitations of traditional methods that rely on physician experience or single indicators. By automatically extracting microscopic tissue features, macroscopic imaging features, and clinical functional states, it outputs objective glucocorticoid efficacy label categories and corresponding prediction probabilities, providing auxiliary basis for individualized clinical medication decisions. Simultaneously, through multimodal complementary collaboration, it effectively reduces the uncertainty of human judgment, improves the accuracy and generalization ability of efficacy prediction, thereby improving the decision support level for glucocorticoid treatment of pulmonary sarcoidosis.

[0023] The present invention will be further explained below with reference to specific embodiments.

[0024] Example 1: This invention provides a method for predicting the efficacy of drugs for pulmonary granulomatous diseases, such as... Figure 1 As shown, the method for predicting the efficacy of drugs for pulmonary granulomatous diseases includes: Step S101: Obtain multimodal case data of the target object; the multimodal case data includes at least the original lung tissue pathological images, the original chest CT images, and the original clinical indicator data.

[0025] Step S102: Preprocess the original lung tissue pathological images, original chest CT images, and original clinical indicator data to obtain the corresponding pathological image data, CT image data, and clinical indicator data.

[0026] Step S103: Input pathological image data, CT image data and clinical indicator data into a pre-trained multimodal drug efficacy prediction model to obtain probability data corresponding to multiple preset drug efficacy label categories.

[0027] Step S104: Generate and output the drug efficacy prediction data of the target object based on the probability data corresponding to each preset drug efficacy label category.

[0028] The target group refers to a single patient or subject who is undergoing the analysis process of this method for predicting the efficacy of pulmonary granulomatous diseases.

[0029] The original clinical indicator data refers to the numerical and classification information of patients' original medical records directly exported from the hospital's electronic medical record system, laboratory system, and pulmonary function testing system, without normalization, screening, or feature engineering transformation. For example, the original clinical indicator data may include demographic characteristics, pulmonary function indicators, serum indicators, etc.

[0030] In this embodiment of the invention, multimodal case data of the target subject should be obtained within a four-week early time window before or after the start of glucocorticoid treatment. Further, lung biopsy specimens can be collected from the target subject, and after slide preparation and staining, a panoramic scan of the pathological slides can be performed using a digital pathology scanner to directly export digital pathology scan images; a chest CT scanner can be used to perform a chest scan on the target subject, and the CT scan data output by the device can be obtained; baseline medical data of the target subject can be retrieved from the hospital's electronic medical records, laboratory system, and pulmonary function testing system, and various original test values ​​and clinical classification information can be directly extracted and summarized to form a clinical data set. Then, the digital pathology scan images, CT scan data, and clinical data set are subjected to privacy anonymization processing. For example, privacy anonymization processing can remove personal identification information such as the target subject's name, ID number, and contact information, and assign a unique anonymous number to the target subject. Finally, the privacy-anonymized case data can be quality-screened to remove unqualified data, obtaining the original lung tissue pathology images, original chest CT images, and original clinical indicator data as the target subject's multimodal case data. For example, the quality screening operation is to remove unqualified data, which may include: unqualified pathological slides, CT images with serious artifacts, clinical indicators with a missing proportion exceeding a preset threshold (preferably 30%), and case data with unclear follow-up outcomes.

[0031] In this embodiment of the invention, the preprocessing process may involve background removal, image segmentation, staining correction, noise reduction, and pixel normalization of the original lung tissue pathological images to eliminate irrelevant areas of the slices, differences in staining batches, and scanning noise, outputting pathological image data with uniform specifications; performing three-dimensional reconstruction, spatial resampling, Haugh window width truncation, extraction of lung regions of interest, artifact noise suppression, and three-dimensional size normalization on the original chest CT images to unify the spatial scale of the images and focus on the effective diagnostic area of ​​the lungs, obtaining CT image data; and performing anomaly cleaning, missing value completion, classification feature encoding, continuous feature standardization, and redundant feature screening on the original clinical indicator data to eliminate problems such as missing data, inconsistent dimensions, and text discretization, generating standardized clinical indicator data.

[0032] Among them, the multimodal drug efficacy prediction model is an artificial intelligence model that has been pre-trained offline based on a large amount of multimodal case data of historical subjects. It can extract cross-scale features from multi-source heterogeneous case data, mine the correlation and complementary information between different modal features, automatically learn the mapping relationship between multi-source data and the effect of glucocorticoid medication on pulmonary sarcoidosis, and finally output the model of drug efficacy prediction data.

[0033] Among them, the efficacy label category is a predefined classification label used to characterize the different medication outcomes of glucocorticoids in patients with pulmonary sarcoidosis. It is the true annotation standard used during model training and also the prediction dimension corresponding to the model's inference output, representing the probability of the target object belonging to each medication outcome. For example, the efficacy label category can be pre-set to multiple categories such as high response and low response, or treatment effective and ineffective, or complete response, partial response and no response.

[0034] In this embodiment of the invention, preprocessed pathological image data, CT image data, and clinical indicator data are input into a pre-trained multimodal drug efficacy prediction model. The multimodal drug efficacy prediction model extracts features from the three types of preprocessed data, then fuses the extracted features in a multimodal manner to obtain fused features. Finally, the fused features are used for prediction to obtain probability data corresponding to multiple preset drug efficacy label categories. The probability data represents the probability that the target subject will achieve the outcome identified by the corresponding drug efficacy label after medication.

[0035] Among them, the efficacy prediction data is the conclusion information obtained by analyzing and integrating the probability data corresponding to each preset efficacy label category output by the model. It can include the probability distribution corresponding to each category, or combine preset judgment rules to output the glucocorticoid efficacy category to which the target object is most likely to belong.

[0036] In this embodiment of the invention, probability data corresponding to multiple preset efficacy label categories are associated and organized with their corresponding efficacy labels, and a dataset with a one-to-one association between probability data and efficacy labels is output as efficacy prediction data for the target object. In particular, a judgment threshold can be preset, all output probability data can be traversed, and all probability data greater than or equal to the threshold can be filtered out. The set of these probability data and their corresponding efficacy labels is then output as efficacy prediction data for the target object. If the probability of all labels is lower than the threshold, an empty set or "no predicted efficacy" can be output.

[0037] The technical solution of this invention adopts a multimodal deep learning architecture and fusion strategy that integrates pathological images, chest CT images, and clinical indicators to achieve cross-modal high-order interactive fusion, thus overcoming the limitations of traditional methods that rely on physician experience or single indicators. It automatically extracts microscopic tissue features, macroscopic imaging features, and clinical functional status, outputting objective glucocorticoid efficacy label categories and corresponding predicted probabilities, providing auxiliary basis for individualized clinical medication decisions. At the same time, through multimodal complementarity and synergy, it effectively reduces the uncertainty of human judgment, improves the accuracy and generalization ability of efficacy prediction, and thus improves the decision support level for glucocorticoid treatment of pulmonary sarcoidosis.

[0038] In some embodiments, the original lung tissue pathological image, the original chest CT image, and the original clinical indicator data are preprocessed to obtain corresponding pathological image data, CT image data, and clinical indicator data, including: identifying and retaining the effective tissue region in the original lung tissue pathological image; dividing the effective tissue region into image blocks of fixed size at a preset magnification; and generating pathological image data based on each image block; extracting the lung parenchyma region from the original chest CT image; truncating the CT image value of the lung parenchyma region to a preset range; resampling and adjusting it to a preset input size to generate CT image data; and performing layered cleaning on the original clinical indicator data to generate clinical indicator data; the layered cleaning includes at least one of outlier truncation, missing value imputation, dimensional difference elimination, and one-hot encoding.

[0039] In this embodiment of the invention, a tissue region detection model can first be used to identify the effective tissue regions in the original lung tissue pathological image, and remove non-tissue regions such as blank backgrounds and stains. For example, the tissue region detection model can be a convolutional neural network (U-Net) based on deep learning. Then, at a preset magnification, the effective tissue regions are divided into image blocks of fixed size. Preferably, a magnification of 20 is selected, and the segmentation size is 256×256 pixels. Finally, the staining depth differences between different image blocks are corrected, and the color space and resolution are unified to obtain standardized pathological image data. For example, the Macenko staining normalization method can be used to correct the staining depth differences between different image blocks and unify the color space and resolution.

[0040] In this embodiment of the invention, the original chest CT images can first undergo integrity checks, spatial orientation unification, and slice order correction. Then, a lung parenchyma segmentation model is used to extract the lung parenchyma region, removing interference from extra-lung regions such as the thoracic skeleton and mediastinal soft tissue. For example, nnU-Net can be used as the lung parenchyma segmentation model. Next, the CT values ​​of the lung parenchyma region are truncated to a preset range and then standardized, preferably from -1000HU to 200HU. Finally, the images are resampled to a uniform spatial resolution and slice spacing, preferably with a voxel size of 1mm × 1mm × 1mm, and adjusted to a preset input size through cropping, padding, or scaling transformation to obtain the CT image data.

[0041] In this embodiment of the invention, the original clinical indicator data can be stratified and cleaned to obtain the clinical indicator data. Specifically, for continuous original clinical indicator data, outliers can first be truncated using the interquartile range method, and then missing data can be filled using the multivariate chain equation imputation (MICE) method, and Z-score standardization can be used to eliminate dimensional differences; for discrete original clinical indicator data, one-hot encoding can be used, and then missing items can be distinguished by setting independent identifiers, thus completing the standardization and format uniformity of the clinical data.

[0042] Thus, by designing differentiated preprocessing workflows for the three types of data characteristics—original lung tissue pathological images, original chest CT images, and original clinical indicators—namely, pathological tissue region identification and staining normalization, CT lung parenchyma segmentation and three-dimensional resampling, and clinical outlier processing and standardized coding, the aim is to eliminate noise interference such as staining differences, spatial heterogeneity, and dimensional differences among the original data of each modality, provide standardized and high-quality input for subsequent feature extraction, and enhance the effectiveness of feature representation.

[0043] In some embodiments, the multimodal drug efficacy prediction model is pre-trained as follows: An initial multimodal drug efficacy prediction model is constructed, comprising a pathological image feature extraction sub-model, a CT image feature extraction sub-model, a clinical indicator feature extraction sub-model, a feature fusion sub-model, and an efficacy prediction sub-model; historical multimodal case data and historical drug efficacy result data corresponding to multiple historical objects are acquired; the historical multimodal case data includes at least historical pathological image data, historical CT image data, and historical clinical indicator data; the historical drug efficacy result data includes drug efficacy label categories corresponding to the historical objects; historical pathological image data is input into the pre-trained pathological image feature extraction sub-model to obtain historical pathological image features; historical CT image data is input into the pre-trained... A CT image feature extraction sub-model is used to obtain historical CT image features. Historical clinical indicator data is input into a pre-trained clinical indicator feature extraction sub-model to obtain historical clinical indicator features. The model parameters of the pathological image feature extraction sub-model, CT image feature extraction sub-model, and clinical indicator feature extraction sub-model are frozen. Based on historical pathological image features, historical CT image features, historical clinical indicator features, and drug efficacy label categories, the initial feature fusion sub-model and the initial effect prediction sub-model are trained to obtain the trained feature fusion sub-model and effect prediction sub-model. Based on the trained pathological image feature extraction sub-model, CT image feature extraction sub-model, clinical indicator feature extraction model, feature fusion sub-model, and effect prediction sub-model, a multimodal drug efficacy prediction model is obtained.

[0044] In this embodiment of the invention, the pathological image feature extraction sub-model, the CT image feature extraction sub-model, and the clinical indicator feature extraction sub-model are used to extract features from the corresponding modal data, respectively. The feature fusion sub-model is used to fuse the extracted features in a multimodal manner, and the efficacy prediction sub-model uses the fused multimodal features to predict the efficacy and output the predicted data. The pathological image feature extraction sub-model, the CT image feature extraction sub-model, the clinical indicator feature extraction sub-model, the feature fusion sub-model, and the efficacy prediction sub-model together constitute the initial multimodal drug efficacy prediction model.

[0045] Among them, historical subjects refer to past cases with clear efficacy results or actual clinical outcomes.

[0046] Among them, historical efficacy data includes glucocorticoid treatment regimens and follow-up outcomes of historical subjects, as well as their corresponding efficacy label categories.

[0047] In this embodiment of the invention, on the one hand, historical digital pathological scan images, historical CT scan data, and historical clinical data sets within a time window of 4 weeks before or after the start of glucocorticoid treatment for the historical subject should be acquired. Then, the historical digital pathological scan images, historical CT scan data, and historical clinical data sets are subjected to privacy desensitization, quality screening, and preprocessing to obtain historical pathological image data, historical CT image data, and historical clinical indicator data. These three types of data constitute the historical multimodal case data of the historical subject. Specifically, the methods for acquiring historical digital pathological scan images, historical CT scan data, and historical clinical data sets, as well as the methods for privacy desensitization, quality screening, and preprocessing of the three types of data, are the same as the methods for acquiring and processing the corresponding data of the target subject in this embodiment of the invention, and will not be repeated here.

[0048] On the other hand, it is also necessary to obtain historical raw efficacy data of the subjects after drug treatment. For example, historical raw efficacy data of the subjects can be retrospectively extracted from electronic medical record systems, clinical follow-up records, or previous research databases. Then, based on the preset glucocorticoid efficacy label classification criteria, the historical raw efficacy data is evaluated and labeled to obtain historical raw efficacy label data with efficacy labels. Specifically, when evaluation judgments are inconsistent, the final efficacy label can be determined by a higher-level physician through review or joint discussion. Subsequently, the historical raw efficacy label data undergoes privacy desensitization and quality screening to obtain historical efficacy data. Finally, the preprocessed historical multimodal case data and historical efficacy data are integrated according to a unique anonymous number to ensure that all data for each historical subject belongs to the same anonymous number, avoiding data leakage. Specifically, the method for privacy desensitization and quality screening of the historical raw efficacy label data is the same as the method for privacy desensitization and quality screening of the target subject data in the embodiments of this invention, and will not be repeated here.

[0049] In this embodiment of the invention, historical pathological image data is input into a pre-trained pathological image feature extraction sub-model. The pathological image feature extraction sub-model uses a pre-trained pathological image encoder to extract image patch features and generates historical pathological image features through an attention multi-instance aggregator. Historical CT image data is input into a pre-trained CT image feature extraction sub-model. The model uses a 3DTransformer to extract inter-slice correlation, spatial distribution of lesions, and overall lesion severity in continuous CT sequences to generate historical CT image features. Historical clinical indicator data is input into a pre-trained clinical indicator feature extraction sub-model. The model performs non-linear encoding on the historical clinical indicator data to extract historical clinical indicator features that can reflect the disease status of the historical subjects and the effect of glucocorticoid medication.

[0050] In this embodiment of the invention, before training begins, the parameters of the pre-trained pathological image feature extraction sub-model, CT image feature extraction sub-model, and clinical indicator feature extraction sub-model are frozen, so that they only participate in the calculation as fixed feature extractors during forward propagation and do not participate in gradient updates. Subsequently, the extracted historical pathological image features, historical CT image features, and historical clinical indicator features are unified in terms of feature dimensions. The unified features are input into the initial feature fusion sub-model, which further integrates cross-modal information through its internal fusion network to output a historical global multimodal representation. This representation is then input into the initial effect prediction sub-model to output the prediction probability of each preset drug efficacy label category. Next, the prediction probability and the real drug efficacy label are substituted into a preset loss function to calculate the fusion classification loss, and gradient updates are performed only on the parameters of the initial feature fusion sub-model and the initial effect prediction sub-model through backpropagation. The updates are iteratively updated until a preset stopping condition is met, and finally, the trained feature fusion sub-model and effect prediction sub-model are obtained.

[0051] In this embodiment of the invention, the trained feature fusion sub-model and effect prediction sub-model are cascaded and combined with the trained pathological image feature extraction sub-model, CT image feature extraction sub-model, and clinical indicator feature extraction sub-model to construct a complete end-to-end multimodal drug efficacy prediction model.

[0052] Thus, by adopting a phased supervised training strategy, that is, first using historical multimodal data to independently supervise the training of each single-modal feature extraction sub-model, and then freezing their parameters and only jointly training the fusion sub-model and the prediction sub-model, the goal is to avoid the convergence difficulties caused by gradient conflicts between modalities in the early stage of multimodal joint training, enable each modal encoder to fully learn its own discriminative features, and provide a stable and high-quality single-modal feature input foundation for subsequent multimodal collaborative fusion.

[0053] In some embodiments, based on historical pathological image features, historical CT image features, historical clinical indicator features, and drug efficacy label categories, an initial feature fusion sub-model and an initial effect prediction sub-model are trained to obtain trained feature fusion sub-models and effect prediction sub-models. This includes: constructing the initial feature fusion sub-model and the initial effect prediction sub-model; the initial feature fusion sub-model includes a linear adaptation unit, a routing control unit, multiple pathological feature processing expert sub-networks, a cross-modal interaction unit, and a case-level attention aggregation unit; inputting historical pathological image features, historical CT image features, and historical clinical indicator features into the linear adaptation unit respectively, so that the linear adaptation unit... The adaptation unit generates historical pathology adaptation features, historical CT adaptation features, and historical clinical adaptation features with consistent dimensions. The historical CT and clinical adaptation features are input into the routing control unit, which generates routing control features based on these features. The routing control unit then calculates dynamic routing weights for each of the multiple pathology feature processing expert sub-networks based on these routing control features. The historical pathology adaptation features are then input into each of the multiple pathology feature processing expert sub-networks to obtain multiple historical expert pathology output features. These historical expert pathology output features are then weighted and aggregated according to the dynamic routing weights to generate a historical route aggregated pathology feature. The historical routing aggregated pathological features, historical CT adaptation features, and historical clinical adaptation features are input into the cross-modal interaction unit. The cross-modal interaction unit performs at least element-wise multiplication and element-wise difference, and performs nonlinear mapping on the operation results after feature concatenation to generate historical cross-modal joint interaction features. The historical routing aggregated pathological features, historical CT adaptation features, historical clinical adaptation features, and historical cross-modal joint interaction features are then input into the case-level attention aggregation unit as historical features to be aggregated. The case-level attention aggregation unit calculates the case-level attention weights corresponding to each historical feature to be aggregated. Based on the case-level attention weights, each historical feature to be aggregated is... The features are weighted and summed to generate historical global multimodal representations, which are then input into the initial effect prediction sub-model to generate historical prediction probabilities corresponding to multiple preset drug efficacy label categories. A fusion classification loss is constructed based on the historical prediction probabilities and drug efficacy label categories, and a regularization loss is constructed based on the dynamic routing weights. A fusion prediction training loss is constructed based on the fusion classification loss and the regularization loss, and the initial feature fusion sub-model and the initial effect prediction sub-model are trained based on the fusion prediction training loss until the validation set loss meets the preset stopping condition, resulting in the trained feature fusion sub-model and effect prediction sub-model.

[0054] Figure 2 The diagram illustrates the structure of the feature fusion sub-model and the effect prediction sub-model in an embodiment of the present invention, as shown below. Figure 2As shown: On one hand, the linear adaptation unit is used to receive different modal features output by each feature extraction sub-model, and then unify the dimensions of each modal feature and complete feature mapping, providing a unified feature foundation for subsequent cross-modal fusion; for example, the linear adaptation unit can be a fully connected layer, a one-dimensional convolutional layer, or a global average pooling layer, etc. By setting up multiple pathological feature processing expert sub-networks to construct a collaborative expert routing network, the network utilizes its specific network structure or parameters to perform deep nonlinear extraction and transformation of pathological features from different perspectives (such as specific tissue textures, cell structures, or spatial patterns), thereby generating multiple historical expert pathological output features with specific representation capabilities; for example, the pathological feature processing expert sub-networks can be convolutional neural networks or Transformer encoders, etc. The routing control unit dynamically evaluates and calculates the dynamic routing weights of each expert subnetwork based on the input CT adaptation features and clinical adaptation features. It then uses these weights to weighted aggregate multiple historical expert pathology output features, generating historical route-aggregated pathology features. The cross-modal interaction unit performs deep information exchange and collaborative learning on the historical route-aggregated pathology features, CT adaptation features, and clinical adaptation features through specific mathematical operations or attention mechanisms, thereby generating historical cross-modal joint interaction features containing rich cross-modal correlation information. The case-level attention aggregation unit calculates the case-level attention weights of each feature to be aggregated through cross-modal attention pooling, and weights and aggregates each feature to be aggregated based on these weights to obtain a historical global multimodal representation, achieving cross-modal feature fusion. For example, the case-level attention aggregation unit can be a self-attention mechanism or a cross-modal cross-attention module. The linear adaptation unit, routing control unit, multiple pathology feature processing expert subnetworks, cross-modal interaction unit, and case-level attention aggregation unit together constitute the initial feature fusion sub-model. On the other hand, the initial effect prediction sub-model can be composed of a multi-layer perceptron (MLP) fully connected classification layer. The MLP fully connected classification layer performs a linear transformation on the historical global multimodal representation of the input to obtain the original score, and then normalizes the original score through the Softmax function to obtain a probability distribution with a sum of 1, which is the historical prediction probability corresponding to multiple preset drug efficacy label categories.

[0055] Among them, the routing control feature refers to an intermediate feature representation generated by the routing control unit after fusing and extracting historical CT adaptation features and historical clinical adaptation features. This feature can comprehensively reflect the current multimodal data status of historical objects and is specifically used to evaluate and determine the activation degree of each pathological feature processing expert sub-network.

[0056] In this embodiment of the invention, historical pathological image features, historical CT image features, and historical clinical indicator features are respectively input into a linear adaptation unit. The dimensions of the three types of features are unified, and feature mapping is completed to obtain historical pathological adaptation features, historical CT adaptation features, and historical clinical adaptation features with consistent dimensions, providing a unified feature dimension foundation for subsequent cross-modal fusion. For example, the linear adaptation unit can be a linear adaptation layer. Further, historical CT adaptation features and historical clinical adaptation features are input into a routing control unit. This unit first concatenates or adds the historical CT adaptation features and historical clinical adaptation features along the feature dimensions to fuse multimodal information and form a comprehensive joint feature representation. Then, this joint feature is input into a routing gating network to extract routing control features. Finally, the routing control features are normalized and transformed into a probability distribution with a sum of 1. This probability distribution is the dynamic routing weight corresponding to each pathological feature processing expert sub-network, reflecting the activation degree and feature contribution ratio of each expert sub-network under the current input sample. For example, a routing gating network may consist of one or more linear layers and activation functions, and the routing control features may be normalized using a Softmax function.

[0057] In this embodiment of the invention, multiple parallel pathological feature processing expert sub-networks are first set up to form a collaborative expert routing network. Then, historical pathological adaptation features are simultaneously input into each of the multiple pathological feature processing expert sub-networks. Each expert sub-network performs a deep nonlinear mapping and transformation on the input historical pathological adaptation features and outputs the corresponding historical expert pathological output features. Finally, based on the dynamic routing weights corresponding to each pathological feature processing expert sub-network, adaptive weighted aggregation of each historical expert pathological output feature is performed to obtain historical routing aggregated pathological features, thereby achieving collaborative modulation of the core pathological information by different modalities. Specifically, the process of adaptive weighted aggregation to obtain historical routing aggregated pathological features can be represented by the following formula:

[0058] In the formula, For the first The dynamic routing weights corresponding to the pathological feature processing expert subnetworks, and satisfying the following conditions: ; For the first The historical routing aggregates pathological features output by the expert sub-network for pathological feature processing; The historical pathological adaptation features of each pathological feature processing expert subnetwork are used as input.

[0059] In this embodiment of the invention, historical route aggregated pathological features, historical CT adaptation features, and historical clinical adaptation features are input into the cross-modal interaction unit. This unit performs element-wise multiplication and element-wise difference on the features, and performs nonlinear mapping on the operation results after feature splicing. This process is used to mine the correlation and complementary features between the three types of information: pathological microscopic, CT image macroscopic, and clinical function, and generate historical cross-modal joint interaction features to enrich the pharmacodynamic representation dimension of multimodal features.

[0060] In this embodiment of the invention, historical route aggregation pathological features, historical CT adaptation features, historical clinical adaptation features, and historical cross-modal joint interaction features are first input as historical features to be aggregated into a case-level attention aggregation unit. This unit calculates the case-level attention weights corresponding to each historical feature to be aggregated through attention pooling. Then, this unit performs a weighted summation of each historical feature to be aggregated based on the case-level attention weights to generate a historical global multimodal representation. Finally, the MLP fully connected classification layer in the initial effect prediction sub-model performs a linear transformation on the input historical global multimodal representation to obtain unnormalized raw scores corresponding to multiple preset drug efficacy label categories. The model normalizes these raw scores using a Softmax function to obtain a probability distribution with a sum of 1, which is the historical prediction probability corresponding to multiple preset drug efficacy label categories.

[0061] In this embodiment of the invention, the historical predicted probabilities and the corresponding true drug efficacy label categories of historical objects are first substituted into the multi-label classification loss function. After calculating the loss for each label, the average or summation is taken, and the resulting value is the fusion classification loss. For example, the multi-label classification loss function can be a binary cross-entropy loss function. Subsequently, a regularization loss is constructed based on the dynamic routing weights. For example, a numerical regularization loss can be obtained by calculating the L2 norm of the dynamic routing weight distribution, calculating its KL divergence with a uniform distribution, or applying a sparsity penalty.

[0062] In this embodiment of the invention, a fusion prediction training loss is first constructed based on the fusion classification loss and the regularization loss. The fusion prediction training loss can be expressed by the following formula:

[0063] In the formula, To integrate classification losses, For regularization loss, is the weight coefficient of the regularization term, and the preferred value is 0.001.

[0064] Then, based on the fusion prediction training loss, the Adam gradient optimization algorithm is used to train the initial feature fusion sub-model and the initial effect prediction sub-model until the validation set loss meets a preset stopping condition, resulting in the trained feature fusion sub-model and effect prediction sub-model. For example, the preset stopping condition can be that the validation set loss no longer decreases for 5 consecutive rounds.

[0065] Thus, by adopting a lightweight training strategy that only updates the fusion and prediction sub-models, and combining it with a collaborative expert routing mechanism and cross-modal attention pooling, we can significantly reduce the number of model parameters and the risk of overfitting, avoid information redundancy and noise accumulation caused by simple feature splicing, give full play to the complementary advantages of information between pathological microscopic and CT macroscopic and clinical states, and improve the efficiency of cross-modal collaborative utilization and the accuracy of drug efficacy prediction.

[0066] In some embodiments, historical pathological image data includes multiple historical image patches; the pathological image feature extraction sub-model is pre-trained as follows: an initial pathological image feature extraction sub-model is constructed, which includes a pre-trained encoder unit, a global representation branch unit, a detail-aware branch unit, and an attention multi-instance aggregation unit; each historical image patch is input into the pre-trained encoder unit to extract the historical depth feature vector corresponding to each historical image patch, and a historical image patch feature set is generated based on each historical depth feature vector; the historical image patch feature set is input into the global representation branch unit and the detail-aware branch unit respectively, so that the global representation branch unit outputs historical global representation features, and the detail-aware branch unit outputs historical local morphological features related to granuloma structure, inflammatory cell infiltration, fibrosis, or extracellular matrix changes; the historical local morphological features are mapped to the same feature space as the historical global representation features, and based on the mapped historical local morphological features and historical... The historical global representation features are used to construct a one-way distillation constraint loss. These features are then input into an attention multi-instance aggregation unit to calculate the attention weights corresponding to each historical image patch feature. The historical global representation features are then weighted and aggregated according to these attention weights to generate historical slice-level pathological image features. The historical image patch features are sorted from high to low according to their attention weights. The top K historical image patch features are selected to form a high-response set, and the bottom K historical image patch features are selected to form a low-response set. The historical image patch features in the high-response and low-response sets are weighted and aggregated separately to obtain high-response and low-response features. A high-low response contrast loss is then constructed based on these features. A classification loss is constructed based on the historical slice-level pathological image features and the drug efficacy label category. Finally, a total training loss is constructed based on the classification loss, the one-way distillation constraint loss, and the high-low response contrast loss. The initial pathological image feature extraction sub-model is then jointly supervised and trained based on this total training loss to obtain the pathological image feature extraction sub-model.

[0067] In this embodiment of the invention, the pre-trained encoder unit is used to extract the depth feature vectors corresponding to each image patch in the historical pathological image data, and output the feature set of the historical image patch corresponding to each historical pathological image data. The global representation branch unit and the detail-aware branch unit are used to extract the historical global representation features and historical local morphological features from the feature set of historical image patches, respectively, and use distillation constraints to make the historical global representation features learn the detail information contained in the historical local morphological features, so as to obtain the detail-enhanced historical global representation features. The attention multi-instance aggregation unit is used to receive the detail-enhanced historical global representation features and calculate the attention weights corresponding to each historical image patch feature, and perform weighted aggregation of the detail-enhanced historical global representation features according to the weights to generate historical slice-level pathological image features. The pre-trained encoder unit, the global representation branch unit, the detail-aware branch unit, and the attention multi-instance aggregation unit together constitute the initial pathological image feature extraction sub-model.

[0068] In this embodiment of the invention, historical image patches from historical pathological image data are input into a pre-trained encoder unit. This unit extracts the depth feature vectors corresponding to each image patch in the historical pathological image data and outputs a feature set of historical image patches corresponding to each historical pathological image data based on each historical depth feature vector. For example, the pre-trained encoder unit can be a pre-trained pathological image encoder, preferably a UNI-v2 network pre-trained on a large-scale pathological image dataset.

[0069] In this embodiment of the invention, the historical image patch feature set is first input into the global representation branch unit and the detail-aware branch unit, respectively. The detail-aware branch unit acts as the teacher branch, extracting subtle tissue morphological information such as granuloma structure, inflammatory cell infiltration, fibrosis, and extracellular matrix changes to obtain historical local morphological features. The global representation branch unit acts as the student branch, extracting the overall tissue morphological information of the image patch to obtain historical global representation features. Then, a feature dimension mapping function maps the historical local morphological features to the same feature space as the historical global representation features, unifying the two feature dimensions. Finally, a one-way distillation constraint loss is constructed using the mapped historical local morphological features and historical global representation features. This one-way distillation constraint is used to enable the global representation branch unit to learn the local morphological information contained in the detail-aware branch unit, outputting detailed-enhanced historical global representation features. Specifically, the one-way distillation constraint loss can be expressed as follows:

[0070] In the formula, n represents the number of image patches; Represents the historical global representation features of the i-th image patch; Represents the historical local morphological features of the i-th image patch; T represents the mapping function used to unify the feature dimensions of the two branch units; This represents the squared distance between two features. This loss reduces the distance between historical global representation features and historical local morphological features, enabling the global representation branch unit to learn the historical local morphological features extracted by the detail-aware branch unit.

[0071] In this embodiment of the invention, the enhanced historical global representation features are input into the attention multi-instance aggregation unit, the attention weights corresponding to each image patch are calculated, and weighted aggregation is performed according to these attention weights to obtain historical slice-level pathological image features. For example, the attention multi-instance aggregation unit can be an attention multi-instance aggregator.

[0072] In this embodiment of the invention, Figure 3 The diagram illustrates the construction process of the contrast loss. Historical image patch features are sorted from high to low according to their attention weights. The top K historical image patch features are selected to form a high-response set, and the bottom K historical image patch features are selected to form a low-response set. For example, K is preferably 10% of the total number of image patches, or fixed at 20. Further, the historical image patch features in the high-response and low-response sets are weighted and aggregated to obtain high-response features and low-response features, respectively. A high-low response contrast loss is then constructed based on these features to reduce the similarity between the two types of features. Specifically, the high-low response contrast loss can be expressed by the following formula:

[0073] In the formula, Indicates high response characteristics; Indicates a low response characteristic; represents the cosine similarity function; m represents the preset similarity interval, preferably ranging from 0.2 to 0.4. A constraint loss is generated when the similarity between a high-response feature and a low-response feature exceeds the similarity interval.

[0074] In this embodiment of the invention, historical slice-level pathological image features can be input into a fully connected layer, and after linear transformation, the original scores corresponding to the preset number of drug efficacy labels are output. Then, the Sigmoid activation function is used to convert them into independent prediction probabilities for each label category. The independent prediction probabilities and the real drug efficacy label categories are substituted into the binary cross-entropy loss function, and the loss of each label is calculated. The average or summation is then taken to obtain the classification loss.

[0075] In this embodiment of the invention, the total training loss is first constructed based on the classification loss, the one-way distillation constraint loss, and the high-low response contrast loss. The total training loss can be expressed by the following formula:

[0076] In the formula, For classification loss; This is due to the constraint loss during unidirectional distillation; For high and low response comparison loss; , These are the weighting coefficients for the corresponding loss terms.

[0077] Furthermore, the initial pathological image feature extraction sub-model is jointly supervised and trained based on the total training loss to obtain the pathological image feature extraction sub-model.

[0078] Thus, by employing a pre-trained encoder to extract basic features from image patches and introducing a detail-aware branch to enhance the representation of subtle tissue morphological changes such as granulomas and fibrosis, and combining this with an attention aggregator to weight and form slice-level features, while introducing a high-low response contrast constraint loss to widen the feature differences between high and low attention response regions, the sub-model can be made more focused on histological information closely related to the efficacy of glucocorticoids, enhance the differentiation of pathological features among different efficacy groups, and provide stable micro-histological support for drug efficacy prediction.

[0079] In some embodiments, the CT image feature extraction sub-model is pre-trained as follows: An initial CT image feature extraction sub-model is constructed, comprising a 3D voxel block embedding unit, a 3D position encoding unit, a 3D transformer encoding unit, a multi-scale feature fusion unit, a global pooling unit, and a CT-assisted classification unit; each historical CT image data is divided into multiple fixed-size historical 3D voxel blocks, and each historical 3D voxel block is input into the 3D voxel block embedding unit, which maps each historical 3D voxel block to a fixed-dimensional historical CT feature vector; the historical CT feature vector is input into the 3D position encoding unit, which adds 3D position encoding to each historical CT feature vector to generate a historical CT feature sequence; the historical CT feature sequence is input into the 3D transformer encoding unit, which uses 3D self-attention as a basis for the CT feature extraction sub-model. The force mechanism calculates the spatial correlation between different historical 3D voxel blocks and outputs historical CT coding features at multiple levels. These features include historical shallow CT image features, historical intermediate CT image features, and historical deep CT image features. The historical CT coding features at multiple levels are input into a multi-scale feature fusion unit to generate historical multi-scale CT representations. The historical multi-scale CT representations are then input into a global pooling unit to generate historical patient-level CT image features. These patient-level CT image features are input into a CT-assisted classification unit to generate historical CT prediction probabilities corresponding to multiple preset drug efficacy label categories. The CT classification loss is calculated based on the historical CT prediction probabilities and drug efficacy label categories. The initial CT image feature extraction sub-model is then trained under supervision based on the CT classification loss to obtain the CT image feature extraction sub-model.

[0080] In this embodiment of the invention, a 3D voxel block embedding unit is used to convert the input historical 3D voxel blocks into a fixed-dimensional historical CT feature vector; a 3D position encoding unit is used to add the 3D position encoding to the historical CT feature vector and output a historical CT feature sequence; a 3D transformer encoding unit utilizes the spatial correlation between different historical 3D voxel blocks to extract multiple levels of historical CT encoding features from the historical CT feature sequence; a multi-scale feature fusion unit is used to adjust the historical CT encoding features at multiple levels to a unified dimension and then fuse them to output a historical multi-scale CT representation; a global pooling unit is used to perform global pooling processing on the historical multi-scale CT representation to generate patient-level CT image features; and a CT-assisted classification unit is used to generate historical CT prediction probabilities corresponding to multiple preset drug efficacy label categories based on the patient-level CT image features. The 3D voxel block embedding unit, the 3D position encoding unit, the 3D transformer encoding unit, the multi-scale feature fusion unit, the global pooling unit, and the CT-assisted classification unit together constitute the initial CT image feature extraction sub-model.

[0081] In this embodiment of the invention, historical CT image data is first divided into multiple fixed-size historical three-dimensional voxel blocks. Each historical three-dimensional voxel block is then converted into a fixed-dimensional historical CT feature vector through a linear mapping layer in the three-dimensional voxel block embedding unit. Simultaneously, a three-dimensional position encoding unit adds three-dimensional position encoding to the historical CT feature vector, marking the scanning plane, intrapulmonary spatial location, and three-dimensional structural information corresponding to each historical three-dimensional voxel block, resulting in a historical CT feature sequence. This allows the model to retain the spatial distribution characteristics of the lesions during subsequent feature extraction.

[0082] In this embodiment of the invention, historical CT feature sequences are input into a 3D transformer encoding unit. The 3D transformer encoding unit calculates the spatial relationships between different historical 3D voxel blocks based on a 3D self-attention mechanism, extracts historical CT encoding features at different scales across multiple levels, and outputs historical CT encoding features at multiple levels. These multiple levels of historical CT encoding features include historical shallow CT image features, historical intermediate CT image features, and historical deep CT image features. Specifically, historical shallow CT image features are used to characterize fine-grained lesion morphology, historical intermediate CT image features are used to characterize the regional distribution characteristics of lesions within lobes and segments of the lung, and historical deep CT image features are used to characterize the overall affected area and severity of lesions in both lungs. Exemplarily, the 3D transformer encoding unit can be a 3D Transformer encoder. In particular, the 3D self-attention mechanism can be expressed by the following formula:

[0083] In the formula, This represents the query matrix obtained by mapping historical CT feature sequences; This represents the key matrix obtained by mapping historical CT feature sequences; This represents the value matrix obtained by mapping historical CT feature sequences; This represents the feature dimension of the key vector; It is a matrix operator.

[0084] In this embodiment of the invention, historical CT coding features at multiple levels are input into a multi-scale feature fusion unit. This unit adjusts historical CT coding features at different levels and spatial scales to a unified dimension through linear mapping and then fuses them to form a historical multi-scale CT representation that simultaneously contains local, regional, and global information.

[0085] In this embodiment of the invention, historical multi-scale CT representations are input into a global pooling unit, which generates patient-level CT image features by performing global pooling processing on the historical multi-scale CT representations.

[0086] In this embodiment of the invention, the CT-assisted classification unit can consist of one or more fully connected layers, and its output dimension is equal to the number of preset drug efficacy label categories. Historical patient-level CT image features are input into the CT-assisted classification unit to calculate the original score for each category. Subsequently, a Sigmoid activation function can be used to map each original score to a probability value between 0 and 1, thus obtaining the historical CT prediction probability for each preset drug efficacy label category.

[0087] In this embodiment of the invention, the historical CT prediction probabilities and corresponding drug efficacy label categories are substituted into the loss function to calculate the CT classification loss. For example, a binary cross-entropy loss function can be used to calculate the cross-entropy loss for each label separately, and then the average or summation of all labels is taken to obtain the final CT classification loss.

[0088] In this embodiment of the invention, the initial CT image feature extraction sub-model is trained under supervision based on the CT classification loss. The CT classification loss value can be used to measure the difference between the model prediction and the true label, and can guide the initial CT image feature extraction sub-model to perform end-to-end training through backpropagation to obtain the CT image feature extraction sub-model.

[0089] Thus, by employing a three-dimensional Transformer architecture and utilizing a three-dimensional self-attention mechanism to extract spatial relationships between different CT scan layers and intrapulmonary regions, and combining multi-scale feature fusion to extract multi-level image information such as local lesion morphology, spatial distribution, and overall lesion degree, this approach achieves the following effects: fully utilizing the inter-layer spatial structure information in continuous CT sequences, compensating for the shortcomings of two-dimensional processing methods that easily lose spatial relationships, enabling the extracted CT features to more completely reflect the overall lesion state of the patient's lungs, and improving the reference value of image features for predicting the efficacy of glucocorticoids.

[0090] In some embodiments, the clinical indicator feature extraction sub-model is pre-trained as follows: An initial clinical indicator feature extraction sub-model is constructed, comprising a clinical indicator projection unit, an anomaly degree encoding unit, a clinical prior combination unit, a clinical feature encoding unit, and a clinical auxiliary classification unit; historical clinical indicator data is input into the clinical indicator projection unit, which projects features onto the historical clinical indicator data to generate historical basic clinical features; historical clinical indicator data is input into the anomaly degree encoding unit, which calculates the deviation of continuous clinical indicators from the normal reference range based on a preset clinical reference range to generate historical indicator anomaly degree features; historical basic clinical features are input into the clinical prior combination unit, which projects features onto the historical clinical indicator data to generate historical basic clinical features. According to preset clinical prior rules, features corresponding to at least two clinical indicators are combined and calculated to generate historical multi-indicator joint representation features. Historical basic clinical features, historical indicator anomaly degree features, and historical multi-indicator joint representation features are concatenated to generate historical enhanced clinical features. These enhanced clinical features are then input into the clinical feature encoding unit, which extracts the correlation information between different clinical indicators through nonlinear transformation and outputs historical clinical indicator features. The historical clinical indicator features are then input into the clinical auxiliary classification unit to generate historical clinical prediction probabilities corresponding to multiple preset drug efficacy label categories. A clinical classification loss is constructed based on the historical clinical prediction probabilities and drug efficacy label categories. The initial clinical indicator feature extraction sub-model is then trained under supervision based on the clinical classification loss to obtain the clinical indicator feature extraction sub-model.

[0091] In this embodiment of the invention, a clinical indicator projection unit is used to perform preliminary feature projection on historical clinical indicator data to obtain historical basic clinical features; an anomaly degree encoding unit is used to calculate the deviation of clinical indicators from the normal reference range according to a preset clinical reference range, generating historical indicator anomaly degree features; a clinical prior combination unit is used to perform combination operations on the features corresponding to at least two clinical indicators according to preset clinical prior rules, generating historical multi-indicator joint representation features; the historical enhanced clinical features obtained by concatenating the historical indicator anomaly degree features and the historical multi-indicator joint representation features are input into the clinical feature encoding unit, which extracts the correlation information between different clinical indicators through nonlinear transformation and outputs historical clinical indicator features; a clinical auxiliary classification unit receives historical clinical indicator features and outputs the historical clinical prediction probabilities of multiple preset drug efficacy label categories. The clinical indicator projection unit, anomaly degree encoding unit, clinical prior combination unit, clinical feature encoding unit, and clinical auxiliary classification unit together constitute the initial clinical indicator feature extraction sub-model.

[0092] In this embodiment of the invention, historical clinical indicator data is first input into the clinical indicator projection unit. Preliminary feature projection is performed through the linear mapping layer within the unit to obtain historical basic clinical features of fixed dimensions, providing input for subsequent clinical semantic enhancement and nonlinear coding.

[0093] In this embodiment of the invention, historical clinical indicator data is input into an abnormality degree encoding unit. Based on a preset clinical reference range (such as the upper or lower limit of normal value), the unit calculates the deviation of the actual measured value of each clinical indicator from the normal range, or the deviation direction, and encodes the calculation result into a feature vector output, thereby obtaining the historical indicator abnormality degree feature that reflects the abnormal trend of the indicator.

[0094] In this embodiment of the invention, historical basic clinical features are input into a clinical prior combination unit. This unit performs combination operations and nonlinear mapping on the basic features corresponding to at least two clinical indicators according to preset clinical prior rules, thereby generating historical multi-indicator joint representation features that can reflect the synergistic clinical significance of multiple indicators. For example, the preset clinical prior rules can be ratios, products, or logical combination relationships of specific indicators.

[0095] In this embodiment of the invention, historical basic clinical features, historical abnormality features, and historical multi-indicator joint representation features are first vectorized and concatenated to obtain historical enhanced clinical features. These enhanced features are then input into a clinical feature encoding unit. This unit further extracts deep correlation information between different clinical indicators through multi-layer linear transformations and non-linear activation functions, ultimately outputting standardized historical clinical indicator features. The clinical feature encoding unit can, exemplarily, be a shallow fully connected encoder.

[0096] In this embodiment of the invention, the clinical auxiliary classification unit typically consists of one or more fully connected layers, with the output dimension corresponding to the number of preset pharmacological efficacy label categories. Historical clinical indicator features are input into the clinical auxiliary classification unit to obtain the raw scores corresponding to the preset pharmacological efficacy label categories; subsequently, a Sigmoid activation function is applied to the raw scores of each preset pharmacological efficacy label category to map them to between 0 and 1, thus obtaining the historical clinical prediction probability of each pharmacological efficacy label category.

[0097] In this embodiment of the invention, a clinical classification loss is constructed based on historical clinical prediction probabilities and actual drug efficacy label categories. Based on this loss value, end-to-end parameter optimization is performed on all trainable units in the initial clinical indicator feature extraction sub-model through backpropagation. Finally, the converged model is determined as the final clinical indicator feature extraction sub-model. For example, a binary cross-entropy loss function can be used to construct the clinical classification loss.

[0098] Thus, by adopting a data standardization process that standardizes continuous indicators, performs one-hot encoding and handles missing outliers for categorical indicators, extracts abnormality features by combining clinical normal reference ranges, constructs multi-indicator joint characterization features based on clinical prior rules, and then mines deep correlations between indicators through a shallow fully connected encoder, the system effectively eliminates noise interference caused by different clinical data value ranges and recording differences, enhances the stability and effective utilization of clinical features, and fully mines comprehensive information related to the efficacy of glucocorticoid medication from routine indicators.

[0099] In some embodiments, generating efficacy prediction data for a target object based on probability data corresponding to each preset efficacy label category includes: sorting multiple preset efficacy label categories according to the size of the probability data; determining the preset efficacy label category with the largest probability data as the target prediction category; generating first prediction data from the target prediction category and its corresponding probability data; generating second prediction data from each preset efficacy label category and its corresponding probability data; and generating and outputting efficacy prediction data for the target object based on the first prediction data and / or the second prediction data.

[0100] In this embodiment of the invention, after obtaining the probability data of all preset drug efficacy label categories, the probability data can be sorted in descending order of numerical value using a sorting function. Simultaneously, the order of the corresponding drug efficacy label categories is adjusted so that they are arranged in descending order of their predicted probabilities. Then, the drug efficacy label category that is first in the sorted sequence (i.e., with the highest probability value) is taken and converted into a specific drug efficacy label name using a preset drug efficacy label mapping dictionary or list. This label is then identified as the target prediction category, i.e., the single drug efficacy result that the prediction model considers most likely to exist. Further, the target prediction category and its corresponding highest probability value are encapsulated according to a predetermined output format to generate first prediction data, which is the best prediction data. Even further, all preset drug efficacy label categories are traversed, and each drug efficacy label name is paired with its corresponding probability data one by one. All categories and their probabilities are output in the form of a complete list or dictionary to generate second prediction data. This second prediction data is used to display the comprehensive probability distribution for detailed analysis or manual verification. Furthermore, based on the needs of the actual application scenario, by using conditional judgment or configuration options, it is possible to select to output only the first prediction data, only the second prediction data, or both simultaneously, and return the standardized encapsulated result data to the calling end, thereby completing the final generation and output of the drug efficacy prediction data for the target object.

[0101] In particular, the output predictive data can be applied to clinical decision-making and patient management. The output probability distribution data can be used as one of the reference information for medical staff when making independent judgments. It can select the drug efficacy category with the highest probability in the output data as a reference for medical staff to use as the initial medication. It can also identify high-risk individuals with uncertain drug efficacy based on the complete probability distribution and trigger manual review and warning. At the same time, it can also be combined with probability thresholds to stratify patients into high, medium and low response groups, thereby guiding the formulation of individualized follow-up cycles and the rational allocation of medical resources.

[0102] Thus, by employing a dual-output strategy that uses the highest probability label as the first predictive data and the complete probability distribution as the second predictive data, the system achieves the effect of quickly providing clinicians with the most likely efficacy category for immediate reference, while comprehensively displaying the confidence distribution of each efficacy label to support in-depth review. At the same time, by stratifying patients into high, medium, and low response groups based on probability intervals, the system assists in the formulation of individualized follow-up cycles and the rational allocation of medical resources, triggers manual review alerts to avoid the risk of misjudgment based on a single threshold, and realizes the value extension from single prediction to full-process clinical empowerment.

[0103] Example 2: Figure 4 This diagram illustrates the process of constructing a multimodal pharmacodynamic prediction model in an embodiment of the present invention. Figure 4 As shown: First, historical digital pathology scan images, historical CT scan data, historical clinical data sets, and historical original drug efficacy results data are acquired. After privacy desensitization, association matching, and quality screening, historical original pathology image data, historical original CT image data, historical original clinical indicator data, and historical drug efficacy results data are obtained. Drug efficacy labels are constructed based on the historical drug efficacy results data, and each modality of data is preprocessed separately: historical original pathology image data undergoes tissue region detection, image block segmentation, and Macenko staining normalization; historical original CT image data undergoes lung parenchyma segmentation, HU value truncation, and 3D resampling cropping; historical original clinical indicator data undergoes outlier processing, Z-score standardization, one-heat coding, and missing value identification. Subsequently, the preprocessed data was divided into training, validation, and test sets, and then proceeded to the independent feature extraction stage for each modality: the pathology modality was processed by a pre-trained encoder to extract image patch features, which were then aggregated through improved multi-instance learning; the CT modality was processed by 3D embedding and positional encoding, followed by multi-scale feature fusion and pooling through a 3D Transformer network; the clinical modality was encoded by a clinical encoder after feature projection, enhancement, and abnormality characterization. Each module was independently supervised and trained to output slice-level pathological features, patient-level CT image features, and historical clinical indicator features. Finally, the features from the three modalities were input into a collaborative expert routing multimodal fusion network, and after training and optimization, the optimal model was output, completing the construction of the multimodal drug efficacy prediction model.

[0104] It should be understood that Figure 4 The schematic diagrams shown are merely illustrative and not limiting, and are scalable; those skilled in the art can use them as a basis. Figure 4 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of the disclosure of the embodiments of this invention.

[0105] Example 3: This invention provides a device for predicting the efficacy of drugs for pulmonary granulomatous diseases, such as... Figure 5 As shown, the device may include: a data acquisition module 501, used to acquire multimodal case data of the target object; the multimodal case data includes at least the original lung tissue pathological images, the original chest CT images, and the original clinical indicator data.

[0106] The data processing module 502 is used to preprocess the original lung tissue pathological images, original chest CT images, and original clinical indicator data to obtain the corresponding target pathological image block dataset, target three-dimensional CT data, and target clinical data.

[0107] The model inference module 503 is used to input pathological image data, CT image data and clinical indicator data into a pre-trained multimodal drug efficacy prediction model to obtain probability data corresponding to multiple preset drug efficacy label categories.

[0108] The result generation module 504 is used to generate and output the efficacy prediction data of the target object based on the probability data corresponding to each preset efficacy label category.

[0109] The specific functions and examples of each module of the device in this embodiment of the invention can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0110] The pulmonary granulomatous disease drug efficacy prediction device in this embodiment of the invention utilizes a multimodal deep learning architecture that integrates pathological images, chest CT images, and clinical indicators. It leverages a collaborative expert routing mechanism to achieve cross-modal high-order interactive fusion, overcoming the limitations of traditional methods that rely on physician experience or single indicators. It automatically extracts microscopic tissue features, macroscopic imaging features, and clinical functional status, outputting objective glucocorticoid efficacy label categories and corresponding prediction probabilities, providing auxiliary basis for individualized clinical medication decisions. Simultaneously, through multimodal complementary collaboration, it effectively reduces the uncertainty of human judgment, improves the accuracy and generalization ability of drug efficacy prediction, thereby improving the decision support level for glucocorticoid treatment of pulmonary sarcoidosis.

[0111] Example 4: According to embodiments of the present invention, an electronic device is also provided.

[0112] Figure 6 A schematic block diagram of a device 600 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0113] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. The random access memory 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0114] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0115] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as a method for predicting the efficacy of drugs for pulmonary granulomatous diseases. For example, in some embodiments, the method for predicting the efficacy of drugs for pulmonary granulomatous diseases can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 600 via read-only memory 602 and / or communication unit 609. When the computer program is loaded into the random access memory 603 and executed by the computing unit 601, one or more steps of the pulmonary granulomatous disease efficacy prediction method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the pulmonary granulomatous disease efficacy prediction method by any other suitable means (e.g., by means of firmware).

[0116] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting the efficacy of drugs for pulmonary granulomatous diseases, characterized in that, The method includes: Acquire multimodal case data of the target object; the multimodal case data includes at least the original lung tissue pathological images, original chest CT images, and original clinical indicator data; The original lung tissue pathological images, the original chest CT images, and the original clinical indicator data are preprocessed to obtain the corresponding pathological image data, CT image data, and clinical indicator data. The pathological image data, the CT image data, and the clinical indicator data are input into a pre-trained multimodal drug efficacy prediction model to obtain probability data corresponding to multiple preset drug efficacy label categories; Based on the probability data corresponding to each of the preset efficacy label categories, the efficacy prediction data of the target object is generated and output.

2. The method according to claim 1, characterized in that, The preprocessing of the original lung tissue pathological images, the original chest CT images, and the original clinical indicator data to obtain corresponding pathological image data, CT image data, and clinical indicator data includes: Identify and preserve the effective tissue region in the original lung tissue pathological image, divide the effective tissue region into image blocks of fixed size at a preset magnification, and generate the pathological image data based on each image block; Extract the lung parenchyma region from the original chest CT image, truncate the CT image value of the lung parenchyma region to a preset range, resample it and adjust it to a preset input size to generate the CT image data; The original clinical indicator data is subjected to stratified cleaning to generate the clinical indicator data; the stratified cleaning includes at least one of outlier truncation, missing value imputation, dimensional difference elimination, and one-hot encoding.

3. The method according to claim 1, characterized in that, The multimodal drug efficacy prediction model was pre-trained in the following manner: An initial multimodal drug efficacy prediction model is constructed, which includes a pathological image feature extraction sub-model, a CT image feature extraction sub-model, a clinical indicator feature extraction sub-model, a feature fusion sub-model, and an efficacy prediction sub-model. Acquire historical multimodal case data and historical drug efficacy result data corresponding to multiple historical objects; the historical multimodal case data includes at least historical pathological image data, historical CT image data, and historical clinical indicator data; the historical drug efficacy result data includes drug efficacy label categories corresponding to the historical objects; The historical pathological image data is input into a pre-trained pathological image feature extraction sub-model to obtain historical pathological image features; The historical CT image data is input into a pre-trained CT image feature extraction sub-model to obtain historical CT image features. The historical clinical indicator data is input into a pre-trained clinical indicator feature extraction sub-model to obtain historical clinical indicator features. The model parameters of the pathological image feature extraction sub-model, the CT image feature extraction sub-model, and the clinical indicator feature extraction sub-model are frozen. Based on the historical pathological image features, the historical CT image features, the historical clinical indicator features, and the drug efficacy label category, the initial feature fusion sub-model and the initial effect prediction sub-model are trained to obtain the trained feature fusion sub-model and effect prediction sub-model. The multimodal drug efficacy prediction model is obtained based on the trained pathological image feature extraction sub-model, CT image feature extraction sub-model, clinical indicator feature extraction sub-model, feature fusion sub-model, and effect prediction sub-model.

4. The method according to claim 3, characterized in that, The initial feature fusion sub-model and initial effect prediction sub-model are trained based on the historical pathological image features, the historical CT image features, the historical clinical indicator features, and the drug efficacy label category to obtain the trained feature fusion sub-model and effect prediction sub-model, including: An initial feature fusion sub-model and an initial effect prediction sub-model are constructed. The initial feature fusion sub-model includes a linear adaptation unit, a routing control unit, multiple pathological feature processing expert sub-networks, a cross-modal interaction unit, and a case-level attention aggregation unit. The historical pathological image features, the historical CT image features, and the historical clinical indicator features are respectively input into the linear adaptation unit, so that the linear adaptation unit generates historical pathological adaptation features, historical CT adaptation features, and historical clinical adaptation features with consistent dimensions. The historical CT adaptation features and the historical clinical adaptation features are input into the routing control unit, so that the routing control unit generates routing control features based on the historical CT adaptation features and the historical clinical adaptation features, and calculates the dynamic routing weights corresponding to the multiple pathology feature processing expert sub-networks based on the routing control features. The historical pathological adaptation features are respectively input into multiple pathological feature processing expert sub-networks to obtain multiple historical expert pathological output features. The multiple historical expert pathological output features are then weighted and aggregated according to the dynamic routing weights to generate historical routing aggregated pathological features. The historical route aggregated pathological features, the historical CT adaptation features, and the historical clinical adaptation features are input into the cross-modal interaction unit, so that the cross-modal interaction unit performs at least element-wise multiplication and element-wise difference, and performs nonlinear mapping on the operation results after feature concatenation to generate historical cross-modal joint interaction features. The historical route aggregated pathological features, the historical CT adaptation features, the historical clinical adaptation features, and the historical cross-modal joint interaction features are input into the case-level attention aggregation unit as historical features to be aggregated, so that the case-level attention aggregation unit can calculate the case-level attention weight corresponding to each historical feature to be aggregated. The historical features to be aggregated are weighted and summed according to the case-level attention weights to generate a historical global multimodal representation. The historical global multimodal representation is then input into the initial effect prediction sub-model to generate historical prediction probabilities corresponding to multiple preset drug efficacy label categories. A fusion classification loss is constructed based on the historical predicted probabilities and the drug efficacy label categories, and a regularization loss is constructed based on the dynamic routing weights; The fusion prediction training loss is constructed based on the fusion classification loss and the regularization loss, and the initial feature fusion sub-model and the initial effect prediction sub-model are trained based on the fusion prediction training loss until the validation set loss meets the preset stopping condition, thus obtaining the trained feature fusion sub-model and effect prediction sub-model.

5. The method according to claim 3, characterized in that, The historical pathological image data includes multiple historical image blocks; The pathological image feature extraction sub-model was pre-trained in the following manner: An initial pathological image feature extraction sub-model is constructed, which includes a pre-trained encoder unit, a global representation branch unit, a detail-aware branch unit, and an attention multi-instance aggregation unit. Each of the historical image blocks is input into the pre-trained encoder unit, so that the pre-trained encoder unit extracts the historical depth feature vector corresponding to each historical image block, and generates a historical image block feature set based on each of the historical depth feature vectors. The set of historical image patch features is input into the global representation branch unit and the detail perception branch unit respectively, so that the global representation branch unit outputs historical global representation features, and the detail perception branch unit outputs historical local morphological features related to granuloma structure, inflammatory cell infiltration, fibrosis or extracellular matrix changes. The historical local morphological features are mapped to the same feature space as the historical global representation features, and a one-way distillation constraint loss is constructed based on the mapped historical local morphological features and the historical global representation features. The historical global representation features are input into the attention multi-instance aggregation unit, the attention weights corresponding to the features of each historical image block are calculated, and the historical global representation features are weighted and aggregated according to the attention weights to generate historical slice-level pathological image features. The historical image patch features are sorted from high to low according to the attention weights, and the top K historical image patch features are selected to form a high response set, and the bottom K historical image patch features are selected to form a low response set. The historical image patch features in the high response set and the low response set are weighted and aggregated respectively to obtain high response features and low response features, and a high-low response contrast loss is constructed based on the high response features and the low response features. A classification loss is constructed based on the historical slice-level pathological image features and the drug efficacy label categories; The total training loss is constructed based on the classification loss, the one-way distillation constraint loss, and the high-low response contrast loss. The initial pathological image feature extraction sub-model is then jointly supervised and trained based on the total training loss to obtain the pathological image feature extraction sub-model.

6. The method according to claim 3, characterized in that, The CT image feature extraction sub-model was pre-trained in the following manner: An initial CT image feature extraction sub-model is constructed, which includes a three-dimensional voxel block embedding unit, a three-dimensional position encoding unit, a three-dimensional transformer encoding unit, a multi-scale feature fusion unit, a global pooling unit, and a CT-assisted classification unit. Each historical CT image data is divided into multiple historical three-dimensional voxel blocks of fixed size, and each historical three-dimensional voxel block is input into the three-dimensional voxel block embedding unit so that the three-dimensional voxel block embedding unit maps each historical three-dimensional voxel block into a historical CT feature vector of fixed dimension. The historical CT feature vectors are input into the three-dimensional position encoding unit to add three-dimensional position encoding to each historical CT feature vector, thereby generating a historical CT feature sequence. The historical CT feature sequence is input into the 3D transformer encoding unit, which calculates the spatial correlation between different historical 3D voxel blocks based on the 3D self-attention mechanism and outputs historical CT encoding features at multiple levels. The historical CT encoding features at multiple levels include historical shallow CT image features, historical intermediate CT image features, and historical deep CT image features. The historical CT coding features of the multiple levels are input into the multi-scale feature fusion unit to generate historical multi-scale CT representations. The historical multiscale CT representation is input into the global pooling unit to generate historical patient-level CT image features. The historical patient-level CT image features are input into the CT-assisted classification unit to generate historical CT prediction probabilities corresponding to multiple preset drug efficacy label categories. CT classification loss is calculated based on the historical CT prediction probability and the drug efficacy label category; The initial CT image feature extraction sub-model is trained under supervision based on the CT classification loss to obtain the CT image feature extraction sub-model.

7. The method according to claim 3, characterized in that, The clinical indicator feature extraction sub-model was pre-trained in the following manner: An initial clinical indicator feature extraction sub-model is constructed, which includes a clinical indicator projection unit, an abnormality degree encoding unit, a clinical prior combination unit, a clinical feature encoding unit, and a clinical auxiliary classification unit. The historical clinical indicator data is input into the clinical indicator projection unit, so that the clinical indicator projection unit can perform feature projection on the historical clinical indicator data to generate historical basic clinical features. Historical clinical indicator data is input into the abnormality degree encoding unit, which calculates the degree of deviation of continuous clinical indicators from the normal reference range based on a preset clinical reference range, and generates historical indicator abnormality degree features. The historical basic clinical features are input into the clinical prior combination unit, so that the clinical prior combination unit performs combination operations on the features corresponding to at least two clinical indicators according to preset clinical prior rules to generate historical multi-indicator joint representation features. The historical basic clinical features, the historical abnormality features, and the historical multi-indicator joint characterization features are concatenated to generate historical enhanced clinical features. The historical enhanced clinical features are then input into the clinical feature encoding unit, which extracts the correlation information between different clinical indicators through nonlinear transformation and outputs historical clinical indicator features. The historical clinical indicator features are input into the clinical auxiliary classification unit to generate historical clinical prediction probabilities corresponding to multiple preset drug efficacy label categories. A clinical classification loss is constructed based on the historical clinical prediction probability and the drug efficacy label category, and the initial clinical indicator feature extraction sub-model is trained under supervision based on the clinical classification loss to obtain the clinical indicator feature extraction sub-model.

8. The method according to claim 1, characterized in that, The step of generating and outputting the efficacy prediction data of the target object based on the probability data corresponding to each of the preset efficacy label categories includes: Sort the multiple preset drug efficacy label categories according to the magnitude of the probability data; The preset efficacy label category with the highest probability data is determined as the target prediction category; Generate first prediction data by taking the target prediction category and its corresponding probability data; Generate second prediction data by generating each of the preset drug efficacy label categories and its corresponding probability data; Based on the first prediction data and / or the second prediction data, generate and output the drug efficacy prediction data for the target object.

9. A device for predicting the efficacy of drugs for pulmonary granulomatous diseases, characterized in that, include: The data acquisition module is used to acquire multimodal case data of the target object; the multimodal case data includes at least the original lung tissue pathological images, the original chest CT images, and the original clinical indicator data. The data processing module is used to preprocess the original lung tissue pathological images, the original chest CT images, and the original clinical indicator data respectively to obtain the corresponding target pathological image block dataset, target three-dimensional CT data, and target clinical data. The model inference module is used to input the pathological image data, the CT image data and the clinical indicator data into a pre-trained multimodal drug efficacy prediction model to obtain probability data corresponding to multiple preset drug efficacy label categories. The result generation module is used to generate and output the efficacy prediction data of the target object based on the probability data corresponding to each of the preset efficacy label categories.

10. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to at least one processor; wherein the memory stores instructions executable by at least one processor, the instructions being executed by at least one processor to enable at least one processor to perform the method of any one of claims 1-8.

Images