Multimodal deep learning method and device based on modality-specific weight learning

Abstract

A memory storing at least one instruction for multimodal deep learning based on modality-specific weight learning that receives image data and clinical pathology data according to an embodiment as input and predicts whether a target patient will relapse; and a processor that performs an operation according to said instruction, wherein the processor includes: an image feature extraction module that receives a plurality of image data and extracts multi-stage features; a non-image embedding module that receives a non-image variable including at least one of clinical, numerical, historical, and environmental information and generates a clinical embedding; a dimension transformation module that includes a reduction layer that compresses image features to a lower dimension and an expansion layer that expands non-image embeddings to a higher dimension to resolve the dimension imbalance between the image features and the non-image embeddings; an adaptive modality attention module (AdaMod) that calculates the contribution of each modality on a sample basis for the transformed image feature and non-image embedding vectors and generates a personalized fusion representation by applying normalized modality-specific weights; and a fusion and classification module that performs a final prediction by combining the feature vectors to which the attention weights are applied.

Description

Technology Field

[0001] The present invention relates to a multimodal deep learning method and apparatus based on modality-specific weight learning. Specifically, it relates to a multimodal deep learning method and apparatus that achieves personalized modality balance by dynamically assigning weights to heterogeneous multimodal features, maps features extracted from clinical and imaging data to the same representation space, and generates personalized fused representations by multiplying normalized attention weights for each sample (patient). Background Technology

[0002] With the recent advancement of deep learning-based data analysis technology, research is actively being conducted to improve prediction accuracy by fusing heterogeneous data of different natures, such as medical images, sensor data, text, and clinical information. Particularly in the medical field, since image-based diagnostic information and non-image data such as clinical, pathological, and biosignal data contain mutually complementary information, multimodal learning techniques that combine these two types of data are attracting attention as a major approach. For example, it has been reported that analyzing morphological features extracted from computed tomography (CT) images together with clinical variables such as pathological examinations, genetic mutations, and blood tests can improve the accuracy of predicting the risk of recurrence in patients with non-small cell lung cancer (NSCLC) to predict the preoperative prognosis.

[0003] However, many existing studies suggest that single-modality-based models utilize only one type of information—either imaging or clinical—and thus fail to reflect heterogeneous characteristics between data, resulting in unstable prediction results. For example, traditional clinical models like XGBoost cannot reflect subtle tumor morphological changes or pathological heterogeneity observed in images, while conversely, image-based CNN models cannot utilize non-imaging information such as clinical stage or blood markers.

[0004] Furthermore, multimodal models based on simple fusion methods (e.g., feature concatenation, addition) apply the same combination rules to all patients, and thus fail to correct for imbalances caused by patient-specific information quality, noise levels, or missing data. For instance, because patients with low image quality or missing specific clinical variables are synthesized with the same weighting ratio, the contribution by modality fails to reflect actual clinical importance.

[0005] Furthermore, existing multimodal studies have failed to adequately consider dimensional disparity between modalities. While image features are high-resolution vectors of hundreds of thousands of dimensions, clinical variables consist of low-dimensional scalars of fewer than tens; consequently, image features exert an excessively dominant influence when directly combined. Moreover, this imbalance leads to optimization problems biased toward specific modalities during the training process, thereby degrading the generalization performance of the fusion model.

[0006] To address this problem, modality attention-based weighted learning methods have recently been proposed; however, since most techniques assign weights using predefined rules or fixed parameters, it is difficult to reflect the varying information reliability or importance of each patient (or sample).

[0007] Meanwhile, lung cancer is one of the malignant tumors with a high mortality rate worldwide, and among them, non-small cell lung cancer (NSCLC) is known as a major histological subtype accounting for about 80–85% of all lung cancers.

[0008] Curative resection is the standard treatment for patients with early-stage non-small cell lung cancer, but the recurrence rate after surgery is reported to be as high as about 50%, so there are limitations in optimizing treatment strategies based solely on stage-based prognosis prediction.

[0009] Traditionally, the TNM staging system has been widely used for prognosis assessment and adjuvant therapy decisions. However, even among patients in the same stage, there is significant heterogeneity in actual clinical course and risk of recurrence, which acts as a factor making it difficult to accurately predict postoperative prognosis based on staging alone.

[0010] Due to these limitations, technologies capable of early identifying non-small cell lung cancer patients who have undergone surgery and are at high risk of recurrence to establish appropriate follow-up examinations and adjuvant treatment plans are emerging as a significant area of ​​interest. In particular, there is a growing need for technologies that can more precisely calculate the risk of recurrence for individual patients by integrating CT imaging with clinical and pathological information. Prior art literature

[0011] 1. Korean Patent Publication No. 10-2024-0005337 (Jan. 12, 2024) 2. Korean Patent Registration No. 10-2744981 (Dec. 17, 2024) The problem to be solved

[0012] The embodiments disclosed in this disclosure aim to provide a multimodal deep learning method and apparatus that go beyond simply combining heterogeneous multimodal data (e.g., images, clinical, biosignals, sensor data, etc.) and dynamically calculate optimal weights for each patient (or sample) based on the information quality, noise, and degree of missing data of each modality.

[0013] In addition, the embodiments disclosed in this disclosure aim to resolve the imbalance between image information, which possesses rich features of high dimensionality and high resolution, and clinical or non-image data, which possesses low dimensionality and standardized features, by normalizing the spatial representation differences between modalities and assigning attentive weights to each modality.

[0014] In addition, the embodiments disclosed in this disclosure aim to reflect different information reliability for each patient by calculating a normalized modality attention weight on a sample-by-sample basis.

[0015] In addition, the embodiments disclosed in this disclosure aim to compensate for heterogeneity within the weapon and improve prediction performance by mapping information of different modalities into the same representation space through the AdaMod (Adaptive Modality Attention) module and generating a fused representation based on dynamic weights.

[0016] In addition, the purpose of the embodiments disclosed in this disclosure is to generalize the modality structure and attention mechanism so that they are not limited to data of specific diseases (e.g., non-small cell lung cancer) but can be extended to other application fields including heterogeneous data (e.g., medical image analysis, process diagnostics, smart factories, wearable healthcare, etc.).

[0017] In addition, the embodiments disclosed in this disclosure aim to verify the clinical generalizability by demonstrating consistent predictive performance for multiple cohorts (CMC, VHSMC) collected in different institutional and instrument environments.

[0018] In addition, the embodiments disclosed in this disclosure aim to simultaneously improve prediction accuracy, reliability, and universality by providing a multimodal deep learning method and apparatus that generates an optimal fused representation by learningly adjusting modality-specific weights while correcting data heterogeneity and patient-specific information imbalance.

[0019] In addition, the embodiments disclosed in this disclosure aim to propose an Adaptive Modality Attention Network (AdaMod-Net) that adaptively balances and utilizes various data sources, such as image data, demographic information, and test results, to improve the performance of predicting recurrence after surgery.

[0020] In addition, the embodiments disclosed in this disclosure aim to achieve an optimized modality balance for each patient by introducing a new adaptive modality attention module called AdaMod and dynamically assigning personalized modality weights to heterogeneous multimodal features.

[0021] In addition, the embodiments disclosed in this disclosure aim to simulate the actual clinical decision-making process in which a clinician comprehensively interprets various information by dynamically adjusting attention between multiple modalities according to the situation.

[0022] In addition, the embodiments disclosed in this disclosure aim to comprehensively validate the performance of the proposed model through experiments conducted on two independent datasets.

[0023] Meanwhile, the technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0024] A memory storing at least one instruction for modality-specific weighted learning-based multimodal deep learning that receives image data and clinical pathology data as input to achieve the aforementioned technical problem according to the present disclosure and predicts whether a target patient will relapse; and a processor that performs an operation according to said instruction, wherein the processor may include: an image feature extraction module that receives a plurality of image data and extracts multi-stage features; a non-image embedding module that receives a non-image variable including at least one of clinical, numerical, historical, and environmental information and generates a clinical embedding; a dimension transformation module that includes a reduction layer that compresses image features to a lower dimension and an expansion layer that expands non-image embeddings to a higher dimension to resolve the dimension imbalance between the image features and the non-image embeddings; an adaptive modality attention module (AdaMod) that calculates the contribution of each modality on a sample basis for the transformed image feature and non-image embedding vectors and generates a personalized fusion representation by applying normalized modality-specific weights; and a fusion and classification module that performs a final prediction by combining the feature vectors to which the attention weights are applied.

[0025] In addition, the image feature extraction module processes image data and the corresponding mask data or Region-of-Interest into independent branches, and can combine the feature maps of each stage by applying a Tumor-Centric Attention (TCA) module.

[0026] In addition, the above Tumor Center Attention (TCA) module can independently calculate the activity of intra-region and peri-region regions based on the tumor mask, and then readjust the feature map based on the calculated weights.

[0027] In addition, the above non-image embedding module can perform univariate and multivariate analysis on input non-image variables to select statistically significant variables, and generate embedding vectors using a fully connected layer for the selected variables.

[0028] In addition, the reduction layer of the dimension transformation module described above can compress high-dimensional image features into low-dimensional vectors by including a 3×3 convolutional layer, an average pooling layer, and a fully connected layer.

[0029] In addition, the extension layer of the dimension transformation module includes a plurality of fully connected layers to extend non-image embedding vectors into a high-dimensional latent space.

[0030] In addition, the above-mentioned adaptive modality attention module (AdaMod) can generate a personalized fused representation by calculating modality-specific importance for transformed image feature vectors and non-image embedding vectors based on Softmax or normalization, and multiplying the modality-specific weights.

[0031] In addition, the fusion and classification module can generate a fusion representation by concatenating the transformed image vector and non-image vector, and classify the state of the target using a fully connected layer and a Softmax activation function.

[0032] Additionally, the processor performs data augmentation during the learning process, and the data augmentation may include one or more of horizontal inversion, rotation, translation, and scaling.

[0033] In addition, the processor can calculate and visualize the importance of input features using SHAP (SHapley Additive exPlanations) to quantitatively analyze the contribution of each modality.

[0034] A deep learning method by a multimodal deep learning device based on modality-specific weight learning according to another embodiment may include: a step of receiving a plurality of image data and extracting multi-stage features; a step of receiving a non-image variable including at least one of clinical, numerical, historical, and environmental information and generating a clinical embedding; a step of compressing the image features to a lower dimension and expanding the non-image embedding to a higher dimension to resolve the dimensionality imbalance between the image features and the non-image embedding; a step of generating a personalized fused representation by calculating the contribution of each modality on a sample basis for the transformed image feature and non-image embedding vectors and applying normalized modality-specific weights; and a step of performing a final prediction by combining the feature vectors to which the attention weights are applied. Effects of the invention

[0035] According to the aforementioned means for solving the problem of the present disclosure, a multimodal deep learning method and apparatus based on modality-specific weight learning automatically weights and adjusts different types of data (e.g., images, clinical data, biosignals, sensor data, etc.) according to patient-specific characteristics and fuses them, thereby preventing high-dimensional image features from overwhelming low-dimensional clinical information or low-quality clinical data from degrading overall performance, and providing the effect of generating a fused representation in which complementary information of all modalities is reflected in a balanced manner.

[0036] In addition, according to the aforementioned means for solving the problem of the present disclosure, unlike the prior art which applies the same combination rule to all patients, it provides the effect of calculating a normalized attention weight that reflects patient-specific data quality and information reliability.

[0037] Furthermore, according to the aforementioned means for solving the problem of the present disclosure, by adjusting the importance of image and clinical data differently for each patient, the effects of data omission, noise, and image quality degradation are minimized, and optimized prediction results are provided on a patient-by-patient basis.

[0038] In addition, according to the aforementioned means for solving the problem of the present disclosure, the effect of improving the average accuracy (AUC) by about 5 to 10% or more compared to a single modal in two independent cohorts of CMC and VHSMC is provided.

[0039] Furthermore, according to the aforementioned means for solving the problem of the present disclosure, for diseases with high pathological and imaging heterogeneity, such as non-small cell lung cancer (NSCLC), where recurrence rates vary significantly even within the same stage, the effect of finely reflecting the pathological condition of each patient through modality-specific attention weighting is provided.

[0040] Furthermore, according to the aforementioned means for solving the problem of the present disclosure, the effect of ensuring stable generalization performance even in multi-institution and multi-device environments is provided, without being specialized for a single hospital or equipment.

[0041] In addition, according to the means for solving the problem described above in the present disclosure, effects that can be utilized in various medical AI applications, such as recurrence prediction, treatment response prediction, and prognosis analysis, based on the fusion of clinical data and image data are provided.

[0042] Furthermore, according to the aforementioned means for solving the problem of the present disclosure, by clearly presenting on what information the model made predictions through the visualization of attention weights by modality (e.g., SHAP, Grad-CAM, etc.), it provides the effect of implementing a transparent model on which medical staff, operators, and users can trust the results.

[0043] The effects obtainable from the exemplary embodiments of the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by those skilled in the art to which the exemplary embodiments of the present disclosure belong from the description below. That is, unintended effects resulting from the implementation of the exemplary embodiments of the present disclosure can also be derived by those skilled in the art from the exemplary embodiments of the present disclosure. Brief explanation of the drawing

[0044] Figure 1 is a diagram showing the study cohort patient selection flowchart (1). Figure 2 is a table showing the patient characteristics of a study cohort (n=166) stratified according to recurrence. Figure 3 is a diagram showing the composition and patient characteristics of a dataset of 217 non-small cell lung cancer (NSCLC) patients who underwent curative resection surgery at the Central Veterans Hospital, according to an embodiment. FIG. 4 is a diagram illustrating the overall structure and data processing flow of a multimodal recurrence prediction device based on an AdaMod-Net (Adaptive Modality Attention Network) according to an embodiment of the present invention. Figure 5 is a diagram showing the results of univariate and multivariate logistic regression analysis on clinicopathological variables. Figure 6 illustrates an example (6) of visualization of deep image features when using the TCA Module. Figure 7 is a table (11) showing the results of a performance comparison between single-modal and multi-modal models based on the VHSMC dataset. Figure 8 is a diagram showing the results of a performance comparison (12) of a single-modal model based on the CMC dataset, a simple fusion model, and an AdaMod-Net model. Figure 9 is a diagram showing the weight distribution by modality calculated from the CMC and VHSMC datasets. FIG. 10 is a block diagram of a multimodal deep learning device based on modality-specific weight learning according to an embodiment. FIG. 11 is a diagram showing the configuration of an instruction set stored in memory according to an embodiment. FIG. 12 is a diagram illustrating a multimodal deep learning process based on modality-specific weight learning according to an embodiment. Specific details for implementing the invention

[0045] Hereinafter, various embodiments of the present disclosure are described in conjunction with the accompanying drawings. As various embodiments of the present disclosure may be subject to various modifications and may have various forms, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the various embodiments of the present disclosure to specific forms, and it should be understood that they include all modifications and / or equivalents and substitutions that fall within the spirit and scope of the various embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals have been used for similar components.

[0046] In various embodiments of the present disclosure, terms such as “comprising” or “having” are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0047] In various embodiments of the present disclosure, expressions such as “or” include any and all combinations of the words listed together. For example, “A or B” may include A, may include B, or may include both A and B.

[0048] Expressions such as "first," "second," "first," or "second" used in various embodiments of the present disclosure may modify various components of the various embodiments, but do not limit such components. For example, such expressions do not limit the order and / or importance of such components and may be used to distinguish one component from another.

[0049] When it is mentioned that a component is "connected" or "joined" to another component, it should be understood that the component may be directly connected or joined to the other component, but that a new component may also exist between the component and the other component.

[0050] In the embodiments of the present disclosure, terms such as "module," "unit," "part," etc. are used to refer to a component that performs at least one function or operation, and such component may be implemented in hardware or software, or in a combination of hardware and software. Additionally, a plurality of "modules," "units," "parts," etc. may be integrated into at least one module or chip and implemented as at least one processor, except where each needs to be implemented in specific individual hardware.

[0051] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the various embodiments of the present disclosure.

[0052] A multimodal deep learning device based on modality-specific weight learning according to one embodiment of the present invention can be applied in various implementation forms and, as a specific embodiment, may be configured to include an Adaptive Modality Attention Network (AdaMod-Net). The AdaMod-Net employs a structure that includes an Adaptive Modality Attention module (AdaMod Attention) which dynamically calculates the contribution of each modality and adaptively adjusts modality weights for each patient and sample.

[0053] Therefore, the multimodal deep learning device of the present invention can be implemented in various modified forms including the above model or technical concept, and this can be applied in the same way within the scope that does not deviate from the technical concept of 'modality-specific weight learning and personalized fusion expression generation,' which is the core concept of the present invention.

[0054] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0055] Accurately predicting recurrence in non-small cell lung cancer (NSCLC) patients who have undergone curative surgery is crucial for selecting appropriate adjuvant treatment and improving long-term survival rates. Therefore, the modality-specific weighted learning-based multimodal deep learning device and method according to the embodiment provides an end-to-end multimodal deep learning model for more accurately predicting recurrence within 2 years in NSCLC patients by integrating preoperative CT images and clinicopathological variables.

[0056] To this end, data from NSCLC patients who underwent curative resection at two hospitals were retrospectively analyzed, and the modality-specific weight learning-based multimodal deep learning device and method according to the embodiment provide a new multimodal fusion framework called AdaMod-Net.

[0057] Lung cancer is the leading cause of cancer-related deaths worldwide, accounting for approximately 18% of all cancer deaths. Among them, non-small cell lung cancer (NSCLC) is the most common histological subtype, accounting for about 85% of all lung cancers. For patients with early-stage NSCLC, complete surgical resection remains the standard local treatment method. However, the recurrence rate within five years of surgery remains high (30–55%), and the prognosis following recurrence is generally poor.

[0058] Therefore, accurately identifying patients at high risk of recurrence after surgery is essential for making appropriate adjuvant therapy decisions and improving long-term survival rates. Currently, the Tumor-Node-Metastasis staging system (TNM) and the National Comprehensive Cancer Network (NCCN) guidelines are utilized as primary criteria for treatment decision-making in clinical practice. However, there are significant differences in prognosis among patients even within the same stage, and a substantial level of prognostic heterogeneity exists.

[0059] With the recent advancement of artificial intelligence (AI), particularly machine learning and deep learning technologies, research is actively being conducted to predict the recurrence of NSCLC after surgery by utilizing heterogeneous data sources such as images and clinical records. Studies based on Electronic Health Records (EHR), which include patient history, clinical signs and symptoms, pathology reports, and treatment records, are demonstrating superior predictive performance compared to conventional statistical methods.

[0060] However, even when using the same dataset, results are inconsistent due to differences in the clinicopathological variables selected across studies, and there are limitations in that clinical variables alone are insufficient to adequately reflect tumor morphology. Preoperative computed tomography (CT) images are the standard imaging technique for lung cancer diagnosis and provide rich information regarding intra-tumor heterogeneity, which strongly influences treatment response and prognosis.

[0061] In particular, deep learning techniques such as Convolutional Neural Networks (CNNs) can automatically extract prognostic features from CT images, which often outperforms existing handcrafted feature approaches.

[0062] However, approaches based solely on image data have limitations in prediction accuracy because they cannot fully represent the patient's clinical condition. To overcome these limitations, multimodal deep learning frameworks are recently being applied in the field of NSCLC recurrence prediction.

[0063] One approach is to train separate models for each modality (e.g., imaging, clinical) and then combine the prediction results through an ensemble classifier. Another approach is to extract radiological features from CT images using a pretrained CNN, select statistically significant features, and concatenate them with clinicopathology variables for use in a single integrated classifier.

[0064] However, since most of these existing methods generate fixed modality-specific representations prior to fusion, end-to-end learning and optimization are limited.

[0065] Furthermore, due to the dimensional disparity between high-dimensional imaging embeddings and relatively low-dimensional clinicopathological vectors, excessive downsampling of image features becomes necessary, leading to the problem of losing important morphological information.

[0066] To overcome these limitations, the modality-specific weight learning-based multimodal deep learning method and apparatus according to the embodiment provides an end-to-end multimodal deep learning framework that utilizes preoperative CT images and clinical pathology variables simultaneously to perform joint learning and integrated fusion of features of the two modalities. The model proposed in the embodiment includes a dedicated subnetwork for extracting features from each modality.

[0067] Compact imaging embeddings that capture heterogeneity within and around the tumor (peri-tumoral) region from CT images, and clinicopathology variables that have a statistically high correlation with recurrence selected from clinical data are used as inputs.

[0068] The two subnetworks are trained simultaneously with a fusion module, and dynamic adaptation and cross-modality reinforcement occur during the training process. Additionally, to mitigate the dimensionality imbalance between high-resolution image features and low-dimensional clinical vectors, a learnable dimensionality transformation module is provided that projects all modalities into a balanced latent space prior to final classification. This allows for improved accuracy in recurrence prediction while maintaining representational balance between the two modalities.

[0069] The AdaMod Module provided in the embodiment is an adaptive attention-based fusion module that dynamically calculates the contribution of each modality on a sample basis for feature vectors input from multiple heterogeneous modalities, and readjusts modality-specific features based on the calculated weights. Specifically, the AdaMod Module first fuses the input image feature vectors and non-image embedding vectors through a Concatenation block, and then learns inter-modality interaction using multiple Conv2D + Thnh layers. Subsequently, the processing result is converted into modality-specific weights normalized by Softmax operation, and the corresponding weights are multiplied (Weighted Multiplication) to the feature vectors of each modality to generate personalized fusion features. As such, the AdaMod module is configured to suppress the dominance of a single modality and reflect differences in data quality and information volume among patients to derive a fair and balanced fusion representation between modalities in real time, thereby playing a key role in improving the predictive performance and generalization ability of the entire model.

[0070] In addition, as a retrospective study, data from Seoul St. Mary's Hospital (CMC) and Veterans Medical Center (VHSMC) were analyzed after obtaining approval from the Institutional Review Boards of each hospital.

[0071] Figure 1 is a diagram showing the study cohort patient selection flowchart (1).

[0072] As shown in Fig. 1, 825 non-small cell lung cancer (NSCLC) patients who underwent curative surgical resection at the Catholic University of Korea Seoul St. Mary's Hospital (CMC) from January 2017 to December 2020 were initially identified. All patients underwent contrast-enhanced CT before surgery. Patients meeting exclusion criteria were excluded from the analysis. In the examples, exclusion criteria may include cases where there is no lung tumor annotation in the CT images, cases where the tissue type is other than lung squamous cell carcinoma (LUSC) or lung adenocarcinoma (LUAD), cases where follow-up information for two years after surgery is not available, and cases where only diagnostic surgery, such as a biopsy, was performed and curative resection was not performed.

[0073] Since the number of patients who relapsed within two years was limited, all relapsed patients (n = 67) were included, and non-recurrence patients were randomly sampled, resulting in a final analysis cohort of 166 patients (67 relapsed, 99 non-recurrence). The demographic information and clinical characteristics of the patients are summarized in Table 1 of Fig. 2. Fig. 2 is a table showing the patient characteristics of a study cohort (n=166) stratified according to relapse status. Figs. 2 and 3 are diagrams showing the configuration of a dataset prepared to implement an Adaptive Modality Attention Network (AdaMod-Net) according to the embodiment, wherein the dataset consists of multimodal data used in both the training and testing phases.

[0074] A total of 16 clinicopathological variables were extracted from each patient's Electronic Health Record (EHR). These variables were items already proven to be associated with prognosis and specifically included smoking status, surgery type, EGFR and ALK mutation status, histological subtype and differentiation grade, pleural, lymphatic, vascular, and perineural invasion, T stage, N stage, hemoglobin, albumin, lactate dehydrogenase (LDH), and carcinoembryonic antigen (CEA). Contrast-enhanced CT images were acquired from four types of equipment, including SIEMENS SOMATOM Definition AS, GE Discovery CT750, SIEMENS SOMATOM Sensation 64, and TOSHIBA Aquilion PRIME, with an image matrix size of 512×512, an in-plane resolution of 0.535–0.855 mm, and a slice thickness of 1.0–5.0 mm.

[0075] The tumor region was manually annotated by a radiologist using ITK-SNAP software at the axial slice with the largest cross-sectional area, and the longest tumor diameter and its perpendicular line were displayed together.

[0076] Figure 3 is a diagram showing the composition and patient characteristics of a dataset of 217 non-small cell lung cancer (NSCLC) patients who underwent curative resection surgery at the Central Veterans Hospital, according to an embodiment.

[0077] In addition, Figure 3 summarizes the major clinical characteristics of patients included in the VHSMC dataset, which include the age distribution, gender ratio, histologic subtype, distribution of T stage and N stage, and two-year recurrence status. For example, the average age was 70.62 years (range: 56–87 years), with 212 men (98%) and 5 women (2%). Histological subtypes consisted of 89 adenocarcinoma patients (41%) and 128 squamous cell carcinoma patients (59%). T stage was T1 with 90 patients (41%), T2 with 113 patients (52%), and T3 with 14 patients (6%). N stage was N0 with 122 patients (56%), N1 with 56 patients (26%), and N2 with 39 patients (18%). 2-year recurrence status was non-recurrence with 103 patients (47%) and recurrence with 114 patients (53%).

[0078] Referring to FIG. 3, the dataset includes a total of 28 clinical variables from basic patient information, laboratory results, and pathology examinations, along with preoperative chest CT images. The image data consists of a total of 217 CT images, of which 206 are contrast-enhanced CTs and 11 are non-contrast CTs, and are composed of images acquired from CT scanners of three different manufacturers. The acquisition conditions of the CT image used in the embodiment of the present invention include parameters such as a matrix size of 512 × 512 pixels, a pixel spacing of 0.54 mm to 0.83 mm, a slice thickness of 1.0 mm to 7.5 mm, and a tube voltage of 99 to 129 kVp.

[0079] FIG. 4 is a diagram illustrating the overall structure and data processing flow of a multimodal recurrence prediction device based on an AdaMod-Net (Adaptive Modality Attention Network) according to an embodiment of the present invention.

[0080] Referring to FIG. 4, the adamode network according to the embodiment receives CT images on the left and corresponding tumor masks as inputs, and extracts features through a multi-stage neural network composed of Block A, Block B, Block C, Block D, and Block E, respectively. The output of each stage is input into a corresponding Tumor-Centric Attention (TCA) module, and a feature map corrected at each stage is generated. The outputs processed by TCA in multiple stages are merged to form Deep Image Features. The generated Deep Image Features are passed to the right and pass through a Condensation block. As illustrated in the figure, the Condensation block consists of 3×3 Convolution, Avg. Pooling, and Fully-Connected, and reduces the dimensionality of the image features.

[0081] Meanwhile, the Clinical variables illustrated in the upper right corner of Fig. 4 are transformed through an embedding path composed of two layers of Fully-Connected and then passed to the Expansion block. The Expansion block expands the clinical embeddings into a high-dimensional representation to match the dimensions with the image features. The reduced image features and the expanded clinical features are simultaneously input into the central AdaMod Attention module. The AdaMod Attention module consists of Concatenation, multiple Conv2D + Thnh blocks, and Softmax, and calculates the attention weights of the two modalities and performs weighting application for each modality.

[0082] The weighted feature vectors are combined in the Concat block on the right and then input into the Classifier module. The Classifier outputs a classification result, and as shown on the far right of the diagram, the conclusion is either Recurrence or Non-Recurrence.

[0083] In the embodiment, CT images and tumor masks can be input to the CT Feature Extraction Module through a dual-branch architecture. Each branch sequentially passes through ResNet-18-based blocks (Block A to Block E), and at each stage, the outputs of the two corresponding branches are combined through the Tumor-Centric Attention (TCA) module. The TCA module emphasizes spatial features of the intra-tumoral and peri-tumoral regions to more precisely reflect tumor morphology. The outputs of each block are hierarchically concatenated to generate deep image features.

[0084] The Clinical Variable Embedding Module is configured to generate clinical embeddings by receiving clinical variables extracted from a dataset (for example, including 16 CMCs and 28 VHSMCs) as input. In this embodiment, univariate and multivariate analyses can be utilized to select variables that are statistically significant with respect to recurrence. However, the variable selection step is not limited to essential components and can be implemented as an optionally applicable procedure in which all clinical variables may be used as is, or only the variables selected based on the analysis results may be used.

[0085] The variables selected in the example are converted into clinical embedding vectors through fully-connected layers. These clinical embeddings are then mapped to a high-dimensional space through an expansion layer to enable dimensional alignment with image features.

[0086] The Dimensionality Transformation Module compresses high-dimensional features into low dimensions through the Condensation Layer, 3×3 Convolution, Average Pooling, and Fully Connected Layer (FC Layer) as CT image features pass through the Condensation Layer. Conversely, clinical embedding vectors are expanded into high-dimensional representations through the Expansion Layer, passing through 2048 and 1024-dimensional fully connected layers.

[0087] Through these two transformation processes, image embeddings and clinical embeddings are balancedly aligned in a common latent space, resolving dimensional disparity between modalities.

[0088] Although the proposed architecture of the present invention does not necessarily include a statistical screening process for clinical variables, univariate and multivariate logistic regression analyses were performed on the CMC dataset used in the examples to examine the association with recurrence (based on a significance level of p<0.05).

[0089] As a result of the analysis of the examples, in the univariate analysis, a total of 7 variables were found to be statistically significant, including T stage (p=0.022), N stage (p<0.001), histologic differentiation (p<0.001), pleural infiltration (p=0.002), lymphatic infiltration (p<0.001), vascular infiltration (p<0.001), and LDH levels (p=0.001).

[0090] In addition, as a result of multivariate analysis in the example, N stage (p=0.003), vascular infiltration (p=0.006), and LDH (p<0.001) were identified as risk factors independently associated with recurrence within 2 years. In the example, a clinicopathology vector was constructed using these three variables, which is shown in the table in Figure 5.

[0091] In addition, FIG. 5 exemplarily illustrates the results of univariate and multivariate logistic regression analysis performed on the dataset used in the embodiment, and values ​​in bold in the figure indicate variables that satisfy statistical significance (p<0.05). The analysis process and variable selection described herein are intended to explain a specific embodiment, and it is made clear that the architecture of the present invention is an optional configuration in which all clinical variables may be used as is, or only some variables selected through analysis may be used.

[0092] For example, in the CT image processing process, pixel values ​​were normalized to the range [0,255] to match the lung window settings (WW: 1500, WL: -600), and all CT images were resampled to an isotropic resolution of 1×1×1 mm³. Subsequently, an axial slice containing the maximum cross-section of the tumor was selected from each patient's image and cropped into a 224×224 patch (tumor-centered crop) to center the tumor, and a bounding box-based tumor mask was constructed for that location. In addition, deep learning-based image features were extracted using at least one of a dual-branch ResNet-18 backbone, ResNet-34, and ResNet-50.

[0093] One input is the original CT patch, and the other is the tumor mask; at each stage, the feature maps of the two branches are fused through the Tumor-Centric Attention (TCA) module. The feature maps obtained from all TCA stages are concatenated to generate a compact representation that emphasizes tumor morphology. Additionally, this network fuses modality-specific embeddings at the intermediate level. However, since deep learning-based image features are high-dimensional while clinicopathology vectors are relatively low-dimensional, a problem of feature dimension disparity arises. To resolve this and reduce bias between modalities, the present invention designs a learnable dimensionality transformation module.

[0094] The learnable dimensionality transformation module consists of a condensation layer and an expansion layer. The condensation layer compresses high-dimensional image embeddings into low-dimensional vectors using convolution, average pooling, and a fully connected layer. The expansion layer expands the clinical pathology vector into a high-dimensional space through two fully connected layers, richly representing the interactions between clinical variables.

[0095] The AdaMod module (Adaptive Modality Attention Module) according to the present invention is an adaptive attention-based fusion module configured to calculate the relative importance of each modality on a sample basis for multiple modality inputs having different characteristics (e.g., image feature vectors and non-image embedding vectors), and to dynamically readjust the features of each modality according to the calculated weights. Specifically, the AdaMod module learns the inter-modality interaction between input modality vectors and adjusts weights so that modalities that are more meaningful to a specific patient have a greater influence, thereby resolving information imbalance between modalities and generating a personalized fusion representation.

[0096] The vectors of the two transformed modalities are aligned to the same dimension and concatenated into a single fused vector, and the likelihood of recurrence is predicted through a final classifier (composed of two fully connected layers).

[0097] The number of image patches was increased by approximately three times, including horizontal flip, rotation, translation, and scaling. In the comparative experiment, models were constructed including a single-modality baseline, an XGBoost model using only clinical data, a TCA-Net model using only image data, a multimodal fusion model, a simple fusion model including all clinical variables, a modified model with the dimensionality transformation module removed, and an IC-TCA-Net model using only statistically selected variables.

[0098] All models were implemented using Python 3.7 and the PyTorch framework. Accuracy, sensitivity, specificity, and Area Under the ROC curve (AUROC) were used as performance metrics.

[0099] In addition, using only the variables selected through univariate and multivariate analysis showed better performance than using all variables. Furthermore, Grad-CAM (Gradient-weighted Class Activation Mapping) was utilized to visualize how the TCA module captures contextual information in the intratumoral and peritumoral regions.

[0100] Figure 6 illustrates an example (6) of visualization of deep image features when using the TCA Module.

[0101] Figure 6(a) shows a CT image with the tumor region highlighted, Figure 6(b) shows a feature map of the TCA module with the dimension transformation module excluded, and Figure 6(c) shows a feature map of the proposed TCA module.

[0102] Referring to Figure 6, the visualization results confirmed that the TCA module learns complex morphological features effective for predicting recurrence by focusing on both the tumor and surrounding tissues.

[0103] In addition, CT images provide rich high-dimensional morphological information, and the dual-branch architecture and TCA module according to the embodiment effectively captured features of the tumor interior and surrounding regions to generate a refined image representation.

[0104] Referring again to FIG. 4, the multimodal recurrence prediction device based on an AdaMod-Net (Adaptive Modality Attention Network) according to the embodiment can perform the process of feature extraction from CT images, clinical variable embedding, dynamically adjusting the weights of the two modalities, and predicting the final recurrence through a classifier. In the embodiment, for image feature extraction, the CT images are processed through a multi-stage convolutional neural network structure composed of ResNet-18-based Block A to Block E.

[0105] In the Fusion and Classification Module, two modality vectors that have undergone dimension transformation are combined in a concatenation layer indicated by the symbol in Fig. 4. The fused vector is input into a classifier, which ultimately outputs either "Recurrence" or "Non-Recurrence." The classifier consists of fully connected layers (FC Layers) and produces a probability-based binary classification result through a Softmax activation function.

[0106] As described above, the present invention includes a condensation block and an expansion block in the dimension transformation module (123) to solve the problem of dimensional imbalance between modalities (image features and non-image embeddings) having different dimensional structures.

[0107] In particular, the Condensation Block is a structure designed to effectively compress high-dimensional image feature vectors and convert them into low-dimensional latent representations, and is formed by sequentially combining a 3x3 convolution layer, an average pooling layer, and a fully-connected layer. Through this, it is possible to map to a representation space at the same level as non-image embeddings, such as clinical and environmental, while preserving core information of image features. The Expansion Block is a structure designed to expand relatively low-dimensional non-image embedding vectors into a high-dimensional latent space, and includes multiple fully connected layers (e.g., 2048 dimensions, 1024 dimensions, etc.) to enhance non-linear representation capabilities.

[0108] This enables richer modeling of correlations and interactions between clinical variables.

[0109] Therefore, the Condensation Block and Expansion Block proposed in this invention can be considered key components that enable effective integrated fusion based on Adaptive Modality Attention (AdaMod) performed in subsequent steps by converting the dimensions of image data and non-image data, respectively, into mutually scale-matched representations.

[0110] By including these two blocks, the present invention structurally improves the modality dominance phenomenon that occurred in existing simple fusion methods and realizes multimodal personalized prediction modeling in which information of different scales is reflected in a balanced manner.

[0111] The image feature vector and the clinical embedding vector are input into AdaMod Attention. The AdaMod Attention consists of a concatenation layer, multiple 2D convolutions, an activation function (Thmh) block, and a softmax-based weight generator, and dynamically calculates the relative contribution of each modality (image, clinical) and applies personalized modality weighting for each sample (patient).

[0112] In the embodiment, the vectors of two modalities weighted and adjusted in AdaMod Attention are combined into a single fused vector through concatenation, and this combined vector is input into a classifier to output postoperative recurrence or non-recurrence through softmax-based binary classification.

[0113] Unlike conventional technology that simply combined image and clinical data, the AdaMod-Net provided in the embodiment can provide multimodal personalized recurrence prediction performance by dynamically adjusting modality-specific weights to reflect the data quality and information volume of each patient.

[0114] In one embodiment of the present invention, the following experimental environment was configured to evaluate the performance of the proposed multimodal deep learning model. In this experiment, 5-fold cross-validation was applied to ensure the generalization performance of the data.

[0115] Data partitioning was performed based on stratified sampling to prevent class imbalance, and the partitioning of each dataset is as follows.

[0116] For the CMC dataset, the training set consists of 104 to 106 people, the validation set consists of 27 to 28 people, and the test set consists of 33 to 34 people.

[0117] For the VHSMC dataset, the training set consists of 137 to 140 people, the validation set consists of 36 people, and the test set consists of 41 to 44 people.

[0118] In addition, the embodiment of the present invention used ResNet-18 with ImageNet pre-trained weights for image feature extraction in an image backbone network environment. Furthermore, the concatenation method was used as the basic fusion strategy for the fusion of multimodal vectors. Moreover, the Stochastic Gradient Descent (SGD) optimization technique was used for model training.

[0119] In addition, in the examples, at least one model among a clinical-based model, an image-based model, and a simple fusion model was evaluated together to verify the superior performance of the model according to the present invention.

[0120] In the examples, the clinical model is an XGBoost classifier that takes clinical variables selected through statistical significance analysis as input, and the image model

[0121] It is a ResNet-18 unimodal model with ImageNet pre-trained weights applied. The simple fusion model is structured to perform a final prediction by independently training a clinical model with statistically selected clinical variables as input and an imaging model based on CT images, and then simply merging the feature vectors produced by the two models using concatenation or addition methods.

[0122] The AdaMod-Net according to the embodiment is a fusion method of clinical models, imaging models, and modality weights based on AdaMod attention.

[0123] In the examples, model performance evaluation was performed using four standard metrics: Accuracy, Sensitivity, Specificity, and AUC (Area Under the ROC Curve). Each metric was reported by averaging the results of 5-fold cross-validation.

[0124] Figure 7 is a table showing the results of a performance comparison between single-modal and multi-modal models based on the VHSMC dataset.

[0125] Referring to FIG. 7, this is a table presenting a quantitative comparison of the prediction performance between the AdaMod-Net model, which has a multimodal deep learning device applied according to the embodiment, and the single-modal model and simple fusion model as comparative examples. Based on the VHSMC (Veterans Health Service Medical Center) dataset (n=217) used as the experimental group, this table compares and evaluates performance by calculating accuracy, sensitivity, specificity, and area under the curve (AUC) for each methodology.

[0126] According to Figure 7, the single-modality-based clinical model (XGBoost) and the single-modality-based imaging model (ResNet-18) showed limited prediction performance with accuracy levels of 61.74% and 60.93%, respectively, confirming the limitations of a method that relies on a single piece of information.

[0127] The Simple Fusion model is a structure that performs prediction by combining all clinical features or statistically selected clinical features with image features. Although some performance improvement compared to a single modality was confirmed with an accuracy of 62.68% to 69.14% despite simply concatenating the two modalities, it appears that the improvement in prediction performance is limited because it does not sufficiently reflect the dimensional imbalance between modalities and the heterogeneity between information quality.

[0128] AdaMod-Net has a multimodal structure that includes an AdaMod module that calculates adaptive attention weights for each modality, and dynamically adjusts the interaction between clinical variables and CT image features. As shown in Fig. 11, AdaMod-Net achieves an accuracy of 66.84% when using all clinical variables and 67.75% when using only selected clinical variables, which demonstrates superior performance compared to other comparative models. In particular, the AUC (0.68–0.70) is higher than that of a single-modal model or a simple fusion model, confirming that the multimodal weight learning method according to the present invention significantly improves recurrence prediction performance.

[0129] Figure 8 is a diagram showing the results of a performance comparison between a single-modal model, a simple fusion model, and an AdaMod-Net model based on the CMC dataset.

[0130] Figure 8 is a table showing the results of a comparative experiment conducted on a clinical and imaging dataset (n=166) built at Seoul St. Mary's Hospital (CMC) to verify the performance of the modality-specific weighted learning-based multimodal deep learning device (AdaMod-Net) of the present invention. This table presents a comparison of the prediction performance between the single-modal method, the simple fusion method, and the AdaMod-Net method of the present invention based on accuracy, sensitivity, specificity, and AUC (Area Under ROC Curve).

[0131] According to the performance comparison results based on the CMC dataset shown in Fig. 8, the performance of a single-modal model is presented first as a comparative example. The XGBoost model, using only clinical variables, showed performance with an accuracy of 81.90% ± 4.35, sensitivity of 70.11% ± 12.91, specificity of 89.89% ± 8.37, and AUC of 0.87 ± 0.04. While it secured relatively high accuracy and specificity, it was confirmed that there is a limitation in that the sensitivity for detecting recurrence cases is low due to the absence of image-based information. On the other hand, the ResNet-18-based model, which is an image single-modal model, showed the lowest performance among all modalities with an accuracy of 69.82% ± 12.05 and AUC of 0.64 ± 0.13, demonstrating that it is difficult to sufficiently reflect the complex biological characteristics of the risk of recurrence using only single image information.

[0132] Next, the Simple Fusion model corresponding to the comparative example performs prediction by simply combining clinical variables and image features. When all clinical variables are used, it shows an accuracy of 83.71% ± 3.72 and an AUC of 0.88 ± 0.03, and when statistically selected clinical variables are used, it shows an accuracy of 88.56% ± 2.25 and an AUC of 0.88 ± 0.02, confirming some performance improvement compared to the single-modal model. However, due to dimensional imbalance between modalities and information heterogeneity, there is a problem where a specific modality is excessively dominated during the fusion process, and this structural limitation is expressed by a lower AUC indicator compared to AdaMod-Net according to the present invention.

[0133] On the other hand, AdaMod-Net according to one embodiment of the present invention includes a fusion structure that assigns adaptive weights to clinical and imaging modalities, and when all clinical variables are used, it exhibits performance of 87.35% ± 1.26 accuracy, 80.66% ± 7.41 sensitivity, 91.89% ± 6.82 specificity, and 0.90 ± 0.01 AUC. Furthermore, even when using statistically selected clinical variables, it achieves 87.35% ± 2.29 accuracy, 80.55% ± 8.33 sensitivity, 91.84% ± 6.19 specificity, and 0.91 ± 0.03 AUC, confirming that it stably provides superior performance compared to the comparative example across all evaluation indicators. These results empirically support the technical effect that AdaMod-Net of the present invention effectively improves recurrence prediction accuracy and generalization performance by dynamically adjusting the contribution of each modality at the patient level.

[0134] As shown in Fig. 8, the AdaMod-Net of the present invention demonstrates superior performance in all evaluation metrics compared to single-modal and simple fusion models. In particular, the AUC (0.90–0.91) recorded the highest value compared to the comparison model, proving that the adaptive modality weight learning (AdaMod Attention) of the present invention substantially improves the accuracy and stability of recurrence prediction by dynamically adjusting the weight between clinical and imaging at the patient level.

[0135] Figure 9 is a diagram showing the distribution of modality-specific weights calculated from the CMC and VHSMC datasets. Referring to Figure 9, the distribution of the relative contribution (weight) to clinical features calculated for each patient is presented in the form of a histogram, allowing verification of the operational characteristics of the adaptive modality weight learning structure according to the present invention. The left diagram of Figure 13 shows the distribution of clinical modality weights for the CMC dataset. It is observed that for most patients, clinical weights are concentrated in the range of approximately 0.45 to 0.55, while for some patients, low or high weights in the 0.3 or 0.6 range are observed. This reflects that the importance of clinical variables varies for each individual patient and suggests that the AdaMod module of the present invention performs personalized fusion, which cannot be implemented in a fixed fusion method, by automatically adjusting the weight of modalities at the patient level. The diagram on the right side of Fig. 13 shows the clinical modality weights for the VHSMC dataset. Similar to CMC, the clinical weights are widely distributed in the range of 0.45 to 0.55, but show a wider range of variation depending on the combination of patient group characteristics and clinical variables within the dataset. In particular, extreme values ​​skewed to 0.4 or lower or 0.6 or higher are observed more frequently in VHSMC than in CMC, which demonstrates that the modality weight learning method of the present invention exhibits adaptive behavior according to data characteristics, even for heterogeneous data collected from different institutions.

[0136] From the results shown in Fig. 9, the contribution between clinical and imaging modalities is not fixed equally for all patients, but changes dynamically depending on the individual patient's data quality, significance of clinical variables, and imaging signal characteristics, and a consistent adaptation pattern is confirmed even in the different data environments of the two institutions (CMC, VHSMC).

[0137] A multimodal deep learning device according to one embodiment of the present invention includes an Adaptive Modality Attention Module (AdaMod) that dynamically calculates modality-specific weights to reflect different information quality and importance on a patient-by-patient basis for heterogeneous modality-specific input features. Unlike conventional technology that combines the contributions of each modality at a fixed ratio, the AdaMod module enables multimodal personalized feature fusion and information balancing by automatically adjusting the weight of each modality in response to the feature distribution of each input patient.

[0138] Furthermore, the present invention conducted extensive experiments on two real-world clinical datasets (CMC dataset and VHSMC dataset) collected independently. As a result, the AdaMod-based model demonstrated consistent performance across both datasets, empirically confirming that the structure of the present invention possesses robustness against data variations and high generalizability regarding inter-agency heterogeneity. In particular, as presented in the patented embodiments, while the simple concatenation-based fusion method showed slightly superior performance compared to the addition method, the fact that stable predictive performance was maintained in both methods supports the idea that the multimodal structure of the present invention operates stably without heavily relying on the combination method.

[0139] Furthermore, the AdaMod-Net of the present invention demonstrated consistent performance superiority over existing single-modal models and simple fusion models even when clinical variable selection was not performed. This implies that the modality weight calculation structure of the present invention provides high structural stability by automatically suppressing unnecessary clinical variables and utilizing only valid information, even when multiple clinical variables or noise exist. Therefore, the multimodal deep learning device according to the present invention provides advantageous technical effects for improving disease prediction performance by integrating heterogeneous medical data such as images and clinical data, and possesses excellent technical differentiation capable of fundamentally improving the problem of information imbalance by modality at the patient level, which existing fusion-based models have failed to resolve.

[0140] FIG. 10 is a block diagram of a multimodal deep learning device based on weight learning by modality according to an embodiment. The configuration of the multimodal deep learning device (100) shown in FIG. 10 is merely a simplified example.

[0141] The communication module (110) can be configured regardless of the mode of communication, such as wired or wireless, and can be configured with various communication networks, such as a Personal Area Network (PAN) or a Wide Area Network (WAN). Additionally, the communication module (110) can operate based on the known World Wide Web (WWW) and may utilize wireless transmission technologies used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. For example, the communication module (110) may be responsible for transmitting and receiving data necessary to perform a technique according to one embodiment of the present disclosure.

[0142] Memory (120) may refer to any storage medium. For example, memory (120) may include at least one storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. Such memory (120) may also constitute the database shown in FIG. 1.

[0143] The memory (120) can store at least one instruction that can be executed by the processor (130). Additionally, the memory (120) can store any form of information generated or determined by the processor (130) and any form of information received by the server (200). Additionally, the memory (120) stores various types of modules, instruction sets, or models.

[0144] The processor (130) can perform technical features according to embodiments of the present disclosure to be described below by executing at least one instruction stored in memory (120). In one embodiment, the processor (130) may be composed of at least one core and may include a processor for data analysis and / or processing, such as a central processing unit (CPU) of a computer device, a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).

[0145] This processor (130) can train a neural network or model designed in a machine learning or deep learning manner. To this end, the processor (130) can perform calculations for training the neural network, such as processing input data for training, extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. Additionally, the processor (130) can perform inference for a specific purpose using a model implemented in an artificial neural network manner.

[0146] FIG. 11 is a diagram showing the configuration of an instruction set stored in memory according to an embodiment.

[0147] Referring to FIG. 11, the instruction set according to the embodiment may be configured to include an image feature extraction module (121), a non-image embedding module (122), a dimension transformation module (123), an adaptive modality attention module (124), and a fusion and classification module (125). The term "module" as used herein should be interpreted to include software, hardware, or a combination thereof, depending on the context in which the term is used. For example, software may be machine language, firmware, embedded code, and application software. As another example, hardware may be a circuit, processor, computer, integrated circuit, integrated circuit core, sensor, MEMS (Micro-Electro-Mechanical System), passive device, or a combination thereof.

[0148] The image feature extraction module (121) receives multiple image data and extracts multi-stage features. The image feature extraction module (121) is configured to extract hierarchical features using a multi-stage convolutional neural network for multiple input image data (e.g., medical images, sensor images, general images, etc.). Specifically, the image feature extraction module (121) may include multiple convolutional blocks that sequentially produce features of different levels of abstraction, ranging from low-level features (edges, textures, etc.) of the input image to high-level semantic features (morphological patterns, structural features, etc.).

[0149] In this embodiment, the image feature extraction module (121) includes a ResNet-based backbone network or a similar multi-stage neural network structure, and each block is configured to progressively refine and compress spatial and structural information of the image by including a convolutional layer, a normalization layer, and an activation function. Additionally, the intermediate feature map generated at each stage may be stored for combination with subsequent modules or further enhanced by combining with a specific attention mechanism. In this embodiment, the image feature extraction module (121) generates a multi-layered feature representation including morphological characteristics of a target object, region-based patterns, and surrounding environment information from a plurality of input image data, and the generated features are provided to be utilized in a dimension transformation module and an adaptive modality weight module, thereby improving the prediction accuracy and information utilization of the multimodal deep learning model.

[0150] The non-image embedding module (122) receives a non-image variable containing at least one of clinical, numerical, historical, and environmental information and generates a clinical embedding. In an embodiment, the non-image embedding module (122) receives a non-image variable containing at least one of clinical information, numerical data, historical information, or environmental information as input, and generates a low-dimensional latent embedding that can be processed by a prediction model from the received non-image variable.

[0151] In this embodiment, considering that the data types and scales of the non-image variables differ from one another, the non-image embedding module (122) first performs vector transformation using one-hot or embedding tables on categorical variables, and transforms continuous variables into a common numerical range through normalization or standardization. The set of non-image variables thus preprocessed is input into one or more fully connected layers, and the fully connected layers are combined with non-linear activation functions to produce clinical embeddings that model potential interactions and dependencies between non-image variables.

[0152] Additionally, in a specific embodiment, the non-image embedding module (122) may be configured to automatically select statistically significant variables by performing univariate and multivariate analyses, taking into account cases where the input non-image variables contain high dimensions or noise. The selected non-image variables allow weights to be reflected more effectively during the subsequent embedding generation process, thereby improving prediction accuracy and the generalization performance of the model.

[0153] The non-image embedding module (122) according to the embodiment maps different types of non-image data into a common latent space, thereby enabling fusion with image-based features in the subsequent dimension transformation module and adaptive modality attention module, and thus improves the information integration performance of the multimodal prediction device.

[0154] The dimension transformation module (123) includes a reduction layer that compresses image features to a lower dimension and an expansion layer that expands non-image embeddings to a higher dimension in order to resolve the dimensional disparity between the image features and non-image embeddings. The dimension transformation module (123) is configured to resolve the dimensional disparity between the image feature vector and the non-image embedding vector, thereby enabling balanced fusion of the two modalities. Generally, image-based deep learning features have high-dimensional representations (hundreds to thousands of dimensions), whereas non-image variables composed of clinical, numerical, and historical data are limited to relatively low-dimensional vector forms; therefore, when the two representations are directly combined, a problem arises in which a specific modality becomes excessively dominant.

[0155] In this embodiment, the dimension transformation module (123) includes a condensation layer and an expansion layer to solve this problem. The condensation layer sequentially applies a plurality of convolution operations (e.g., a 3×3 convolution layer), an average pooling operation, and a fully connected layer to compress high-dimensional image features into a low-dimensional latent space. Through this, the dimensions of the image features are normalized to a level that can be combined with clinical embeddings while maintaining the main structural and morphological information of the image features.

[0156] On the other hand, the extension layer includes multiple fully connected layers to project non-image embedding vectors into a high-dimensional latent space, and each layer includes activation functions and normalization to enable the learning of non-linear interactions between non-image variables and high-dimensional representations. As a result, the non-image embeddings acquire a level of representation capacity similar to that of image features.

[0157] The representations of the two modalities transformed through the reduction and expansion layers are mutually dimensionally compatible and can be reflected in a balanced manner in the subsequent adaptive modality attention module and feature fusion module. Thus, the dimension transformation module (123) eliminates the difference in representation between image and non-image, thereby supporting the multimodal deep learning model to effectively utilize the complementary characteristics of both sets of information without being biased toward a specific modality.

[0158] The Adaptive Modality Attention Module (AdaMod, 124) calculates the contribution of each modality on a sample basis for the transformed image feature and non-image embedding vectors and generates a personalized fusion representation by applying normalized modality-specific weights. The Adaptive Modality Attention Module (124) (AdaMod, Adaptive Modality Attention Module) is configured to dynamically calculate the relative contribution of each modality on a sample basis for the image feature vector and non-image embedding vector transformed into a mutually compatible representation space through the dimension transformation module (123), and to generate a multimodal personalized fusion representation based on the calculated weights.

[0159] Specifically, the module calculates an independent raw importance score for each of the transformed image feature vector and the non-image embedding vector, and derives a modality-specific attention weight that is normalized such that the sum of the weights of the two modalities is 1 by applying at least one of a normalization function (e.g., Softmax normalization, min-max normalization, or L1-normalization) to the importance score. At this time, since the derived weight is calculated differently for each input individual sample, a patient-adaptive (personalized) fusion structure is provided that can reflect patient-specific data deviation, noise, information quality, etc.

[0160] Normalized modality-specific weights are multiplied by each feature vector to generate weighted feature vectors that reflect the importance of each modality, and subsequently, the two weighted feature vectors are combined (e.g., concatenation, weighted sum, etc.) to produce a final multi-modal fused representation. Unlike conventional fusion methods that simply combine two modalities at a fixed ratio, this process can automatically adjust weights by mechanically determining whether image information or non-image information has higher reliability in each sample; thus, it provides stable prediction performance even in environments with data heterogeneity or unbalanced modality importance.

[0161] Additionally, the fusion and classification module (125) performs a final prediction by combining the feature vectors to which the attention weights are applied. In an embodiment, the fusion and classification module (125) receives an image feature vector and a non-image embedding vector to which modality-specific importance weights are applied as input through the adaptive modality attention module (124), combines the two vectors into a single fusion representation, and produces a final prediction result based on the combined fusion representation.

[0162] Specifically, the fusion and classification module (125) first generates a multimodal integrated feature vector by combining a weighted image feature vector and a weighted non-image embedding vector (e.g., concatenation, weighted sum, Hadamard product, etc.). While the combination method may be applied selectively depending on the implementation environment, this embodiment may include concatenation as a basic structure, which is a combination operation advantageous for preserving different semantic information.

[0163] The generated integrated feature vector then learns the interactions between high-dimensional features through one or more fully connected layers and non-linear activation functions (e.g., ReLU, GELU, etc.), and in the final output layer, a probability-based activation function such as Softmax or Sigmoid is applied to produce a final prediction result that classifies the state of the target (e.g., recurrence / non-recurrence, normal / abnormal, event occurrence / non-occurrence, etc.).

[0164] In addition, in this embodiment, the fusion and classification module (125) may optionally include regularization techniques such as dropout and batch normalization to prevent overfitting and improve generalization performance, and the prediction results may be transmitted to a subsequent analysis module or user interface. In this embodiment, the fusion and classification module (125) learns an integrated representation that reflects the information of each modality in a balanced manner by combining features extracted from different modalities based on weights, and enables stable final prediction based thereon.

[0165] Additionally, the image feature extraction module (121) processes the image data and the corresponding mask data or region of interest into independent branches, and can combine the feature maps of each stage by applying a tumor-centric attention (TCA) module. In the embodiment, the image feature extraction module (121) processes the input image data and the mask data or region of interest (ROI) corresponding to the image data into independent input branches, thereby enabling the extraction of key structural and morphological information within the image in multiple stages.

[0166] Specifically, in this embodiment, the image feature extraction module (121) includes a dual-branch neural network structure, wherein the first branch processes original image data and the second branch processes mask data or ROI information corresponding to the image to produce feature representations of different perspectives in parallel.

[0167] Each branch includes multiple stages consisting of a convolution layer, a normalization layer, an activation function, and a downsampling operation, and the feature maps generated at each stage are combined through a Tumor-Centric Attention (TCA) module at the same layer location.

[0168] The above TCA module calculates the spatial importance of the intra-region and peri-region based on the central location of the tumor or region of interest, and generates an integrated feature that simultaneously reflects the morphological characteristics of the tumor center and the semantic contextual information of the surrounding tissue by weighted fusion of the feature maps of the two branches according to the importance.

[0169] In this way, TCA-based stepwise combination prevents local information loss prone to occur in single-branch neural networks and enables balanced utilization of complementary information from the original image and mask-based features.

[0170] In the embodiment, the image feature extraction module (121) processes image data and ROI-based information in parallel in independent paths and combines feature maps by applying a TCA module at each step, thereby generating a sophisticated high-level feature representation that simultaneously reflects the morphological structure of the input image, the lesion center pattern, and the contextual information of the surrounding tissue.

[0171] In addition, the Tumor Center Attention (TCA) module independently calculates the activity of the intra-region and peri-region regions based on the tumor mask, and then readjusts the feature map based on the calculated weights.

[0172] The Tumor-Centric Attention (TCA) module is configured to differentiate intra-region and peri-region regions of the image based on the tumor mask and to progressively enhance key structural features centered on the lesion and contextual information of surrounding tissues by independently calculating the activation response in each region.

[0173] Specifically, the TCA module first sets the center coordinates of the lesion or target of interest, calculated from image or mask data, as an anchor point. Subsequently, based on the said center, it classifies pixel information within the image or location information on the feature map using distance-based or region-segmentation methods to distinguish between the intra-region and the peri-region of the lesion.

[0174] Since each region contains different morphological and contextual information, the TCA module calculates region-specific importance scores by applying independent activity functions to each region. For example, the internal region is suitable for reflecting lesion morphological distortions, textural changes, and tumor boundary characteristics, while the peripheral region may include external features that contribute to recurrence prediction, such as the tumor microenvironment, inflammatory signals, and structural alterations.

[0175] The calculated internal and peripheral activities undergo normalization operations (Softmax normalization, L2-normalization, etc.) to be converted into region-specific attention weights, and these weights are multiplied by the positional feature values ​​of the input feature map to generate a new feature map in which the features of the lesion's internal and peripheral regions are semantically re-weighted.

[0176] Thus, by emphasizing lesion-centered high-resolution structural information and appropriately supplementing the semantic associations of surrounding tissues, the model becomes able to effectively enhance tumor-centered valid information that is difficult for simple convolutional neural networks to capture.

[0177] In the embodiment, the TCA module independently calculates the importance of internal and surrounding regions based on the lesion center coordinates and readjusts the feature map with the calculated weights, thereby providing a technical effect in which prediction-related information centered on the lesion is emphasized in the image-based model, which contributes to improving the multimodal fusion accuracy in subsequent modules.

[0178] In addition, the non-image embedding module (122) performs univariate analysis and multivariate analysis on the input non-image variables to select statistically significant variables, and generates an embedding vector using a fully connected layer for the selected variables.

[0179] In an embodiment, the non-image embedding module (122) is configured to perform feature selection based on statistical significance for an input non-image variable and to generate an embedding vector that converts the selected variable into a low-dimensional latent space.

[0180] Specifically, in this embodiment, the non-imaging embedding module (122) first receives a non-imaging variable as input that includes at least one of clinical, numerical, historical, and environmental information, and performs a univariate analysis on each of the variables to test for an independent association with a dependent variable (e.g., whether or not there is a recurrence).

[0181] At this time, for univariate analysis, p-values ​​for each variable are calculated using, for example, logistic regression, chi-square test, t-test, or a statistical test method suitable for the data type, and variables satisfying significance below a certain threshold (p < 0.05, etc.) are selected as the primary candidate group.

[0182] Subsequently, by performing additional multivariate analysis on the selected candidate variables, the independent contribution between the correlated variables is evaluated, and a final set of significant variables with statistically independent contributions to the dependent variable is derived. For example, a multivariate logistic regression model can be used for multivariate analysis, and redundant variables and stability can be ensured by applying the Variance Inflation Factor (VIF) or regularization techniques.

[0183] Finally, the selected non-image variables undergo preprocessing procedures based on data type, such as normalization or one-hot encoding, and are then input into one or more fully connected layers to be transformed into unique latent embeddings combined with non-linear activation functions (such as ReLU).

[0184] The generated embedding vector is provided as input to be combined with image features in a subsequent dimension transformation module or multimodal fusion module.

[0185] The non-image embedding module (122) can improve the performance and generalization ability of the multimodal prediction model by removing noise or unnecessary clinical variables through statistical significance testing and generating a stable embedding vector using only selected key variables.

[0186] In addition, the reduction layer of the dimension transformation module includes a 3×3 convolution layer, an average pooling layer, and a fully-connected layer to compress high-dimensional image features into low-dimensional vectors. Specifically, the reduction layer includes a 3×3 convolution layer, an average pooling layer, and a fully-connected layer to perform multiple stages of compression operations.

[0187] First, a 3×3 convolutional layer is designed to extract key patterns while preserving spatial adjacency on the feature map, and to mitigate unnecessary noise and local variation. Subsequently, the mean pooling layer spatially reduces the convolutional results to derive global summary information of the features, thereby effectively reducing the size of the feature map.

[0188] Subsequently, the pooled vectors are input into a fully connected layer to transform multidimensional features into low-dimensional latent vectors. This process may include non-linear activation functions (such as ReLU), which enables the generation of a compressed representation that can be combined with subsequent modules while preserving important information about high-dimensional image features.

[0189] In the embodiment, the reduction layer compresses high-dimensional image features into low-dimensional vectors stepwise, thereby preventing the problem of a specific modality (especially image) being excessively dominated during the multimodal fusion process and maintaining a balance of representation between different modalities, such as image and clinical.

[0190] In addition, the extension layer of the dimension transformation module (123) includes a plurality of fully connected layers to extend the non-image embedding vector into a high-dimensional latent space.

[0191] In the embodiment, the extension layer of the dimension transformation module (123) is configured to project a low-dimensional non-image embedding vector generated through the non-image embedding module (122) into a high-dimensional latent space to resolve the difference in expressiveness with image-based features and enable balanced multimodal fusion.

[0192] Specifically, the expansion layer includes a plurality of fully-connected layers, and each fully-connected layer is designed to progressively expand the dimension of an embedding vector by applying weight matrix operations and non-linear activation functions (ReLU, GELU, etc.) to an input embedding vector.

[0193] In this case, the extension layer may include one or more intermediate hidden layers, and each hidden layer enhances the interaction between variables and high-dimensional representation capacity through non-linear transformation.

[0194] The high-dimensional embedding vectors transformed by the extension layer are adjusted into latent representations on a scale similar to the dimensions of the image features; this high-dimensional projection enables sufficient expressive power through non-linear transformation, even when the information content of non-image variables is low or some items are missing.

[0195] In addition, the extension layer can optionally include regularization techniques such as normalization and dropout to prevent overfitting and improve the generalization performance of the model. In the embodiment, the extension layer of the dimension transformation module (123) extends low-dimensional non-image embedding vectors into a high-dimensional latent space, thereby achieving a balance of representation with image-based high-dimensional features and supporting the multimodal deep learning model to stably fuse mutually complementary information between the two modalities.

[0196] In addition, the above-mentioned adaptive modality attention module (AdaMod) calculates modality-specific importance for transformed image feature vectors and non-image embedding vectors based on Softmax or normalization, and generates a personalized fused representation by multiplying modality-specific weights.

[0197] In an embodiment, the adaptive modality attention module (AdaMod) is configured to generate a multimodal personalized fusion representation by calculating modality importance for each image feature vector and non-image embedding vector converted into a mutually compatible representation space through a dimension transformation module (123), and applying modality weights based on the calculated importance.

[0198] In this embodiment, the adaptive modality attention module first calculates a raw importance score for each of the image feature vector and the non-image embedding vector. The importance score may be calculated based on the statistical distribution of the vector, activity values, or preprocessed linear / non-linear transformation results.

[0199] Subsequently, the adaptive modality attention module applies at least one normalization technique, such as Softmax normalization, min-max normalization, and L1-normalization, based on the calculated importance score to derive a normalized attention weight for each modality such that the sum of the weights of the two modalities is 1.

[0200] In this case, since weights are dynamically determined based on the data characteristics of individual samples, personalized weight adjustments are possible to reflect differences in data quality by patient or environment. The normalized modality-specific weights are multiplied by the corresponding modality's feature vector to generate a weighted feature vector.

[0201] For example, if higher weights are assigned to image features, the model performs predictions by placing greater emphasis on image-based morphological information; conversely, if the reliability of non-image variables is deemed high, clinical, numerical, and environmental information have a greater influence on the model's final judgment.

[0202] The two weighted feature vectors generated thereafter are combined in subsequent fusion and classification modules to form a final fused representation. Thus, the Adaptive Modality Attention Module (AdaMod) reflects the differences in information quality and importance between image and non-image modalities in real time, preventing excessive dominance of a specific modality or loss of information, and enables personalized multimodal fusion at the sample level, thereby improving the performance and generalization ability of the entire deep learning prediction model.

[0203] Additionally, the fusion and classification module (125) concatenates the transformed image vector and non-image vector to generate a fused representation and classifies the state of the target using a fully connected layer and a softmax activation function. The fusion and classification module (125) receives the transformed image feature vector and non-image embedding vector calculated through the dimension transformation module (123) and the adaptive modality attention module (124) as input, generates a fused representation that integrates complementary information between the two modalities, and is configured to classify the state of the target based on the fused representation.

[0204] Specifically, the fusion and classification module (125) integrates the transformed image vector and non-image vector into a single high-dimensional vector through a concatenation operation. The concatenation operation has the advantage of being able to integrate different types of feature information into a single consistent latent representation while preserving them independently. Additionally, while combination methods such as weighted sum and Hadamard product may be selectively applied as needed, this embodiment uses a concatenation method that minimizes information loss as the basic structure.

[0205] Subsequently, the combined fusion vector passes through one or more fully-connected layers, where a non-linear activation function (ReLU, GELU, etc.) is applied to learn the interaction between multimodal inputs and the combination of high-dimensional features. In the final output layer, a softmax activation function is applied to classify the state to which the target belongs based on the probability distribution represented by the combined fusion vector.

[0206] For example, when the prediction target is one of discrete states (classes) such as “recurrence / non-recurrence”, “normal / abnormal”, or “event occurrence / non-occurrence”, the Softmax activation function calculates the posterior probability for each state and provides it as the final prediction result.

[0207] In addition, the fusion and classification module (125) of the present embodiment may optionally include regularization techniques such as batch normalization and dropout to prevent overfitting and secure generalization performance, and the weights within the module are automatically adjusted through backpropagation based on prediction error during the learning process.

[0208] Accordingly, the fusion and classification module (125) integrates feature information of the transformed image vector and non-image vector in a balanced manner to generate an advanced integrated representation that reflects mutually complementary information between the two modalities, and enables stable and accurate state prediction based thereon.

[0209] Additionally, in the embodiment, the processor (130) performs data augmentation during the learning process, and the data augmentation includes one or more of horizontal inversion, rotation, translation, and scaling.

[0210] The processor (130) is configured to perform a data augmentation procedure to ensure diversity of training image data and prevent overfitting during the learning process.

[0211] Specifically, the processor (130) automatically generates a new type of learning sample by applying a geometric transformation including one or more of horizontal flip, rotation, translation, and scaling to the input original image data.

[0212] In this context, horizontal inversion ensures that the model does not rely on specific orientations through left-right symmetrical deformation of the image, while rotational transformation enhances the model's robustness against directional changes of tumors and anatomical structures by rotating the image within a certain angular range. Additionally, translation reduces positional bias by simulating positional variations in the image, and scaling enables the learning of structural patterns of various sizes by performing enlargement or reduction.

[0213] The above-described data augmentation process can be applied randomly at every epoch during model training, thereby generating multiple modified training samples from the same original image, which has the effect of expanding the effective size of the entire dataset. Through this data augmentation function, the processor (130) can perform stable training even with limited medical image data, and has the technical effect of improving generalization performance regarding various changes in shooting conditions occurring in the actual environment.

[0214] Additionally, the processor (130) calculates and visualizes the importance of input features using SHAP (SHapley Additive exPlanations) to quantitatively analyze the contribution of each modality.

[0215] The processor (130) is configured to perform an importance analysis procedure based on SHAP (SHapley Additive exPlanations) to quantitatively evaluate how the learned multimodal prediction model is contributed from each input modality.

[0216] Specifically, the processor (130) calculates the degree to which each input image feature vector and non-image embedding vector contributes to the change in model output as a SHAP value calculated on a sample basis. The SHAP value is calculated using a game theory-based shuffling value (Shapley value) to represent the average contribution of a specific input feature to the final prediction result of the model.

[0217] At this time, the processor (130) calculates SHAP values ​​by distinguishing between image feature sets and non-image clinical feature sets in order to independently evaluate the feature contribution of each modality, and aggregates the contribution of all features belonging to each modality to generate a modality-level importance score.

[0218] Additionally, the processor (130) transmits the calculated SHAP value to the analysis engine to generate visualization results such as a bar graph, a dot plot, and a list of importance rankings, thereby allowing the influence of each input characteristic on the model prediction to be intuitively confirmed.

[0219] This visualization can be used to evaluate whether the model relies excessively on a specific modality (e.g., CT images) or a specific variable (e.g., LDH, N-stage). According to this configuration, the processor (130) can quantitatively determine the contribution by modality and by variable, thereby significantly improving the explainability of the model and providing technical effects that can be utilized to optimize the model structure or improve variable selection by identifying unnecessary or noisy variables.

[0220] Hereinafter, we will look at FIG. 12. The modality-specific weight learning-based multimodal deep learning method illustrated in FIG. 12 can be performed by a modality-specific weight learning-based multimodal deep learning device (100) including a processor (130).

[0221] Meanwhile, FIG. 12 is merely illustrative, and the concept of the present invention is not to be interpreted as being limited to that shown in FIG. 12. For example, each step may be configured in a different order than that shown in FIG. 12, at least one of the steps shown in FIG. 12 may not be performed, or one or more steps not shown in FIG. 12 may be additionally performed.

[0222] Below, a multimodal deep learning method based on modality-specific weight learning is described in turn. Since the operation (function) of the multimodal deep learning method based on modality-specific weight learning according to the embodiment is essentially the same as the function of the multimodal deep learning device based on modality-specific weight learning, descriptions that overlap with FIGS. 1 to 11 will be omitted.

[0223] FIG. 12 is a diagram illustrating a multimodal deep learning process based on weight learning by modality according to an embodiment.

[0224] Referring to Fig. 12, in step S110, image data is input from the image feature extraction module, image features are extracted through a multi-stage convolutional neural network (ResNet-18), and a Tumor-Centric Attention (TCA) module is applied to the output of each step to emphasize morphological features of the tumor interior and surrounding regions.

[0225] In step S120, the clinical variable embedding module receives clinical pathology data and performs univariate and multivariate analyses. Based on the analysis results, clinical variables recognized as significant for recurrence are selectively identified, or the selection process may be omitted if necessary. Additionally, a low-dimensional clinical embedding vector is generated through a fully connected layer.

[0226] In step S130, to resolve the dimensional imbalance between the image features and the clinical embedding vector in the dimension transformation module, a condensation layer that compresses the image features to a lower dimension and an expansion layer that expands the clinical embedding vector to a higher dimension are provided.

[0227] In step S140, weighted feature vectors are generated by applying modality-specific weights calculated through the AdaMod module in the classification module to the transformed image feature vector and clinical embedding vector, respectively, and the weighted feature vectors are concatenated to generate an integrated fused representation, and the patient's recurrence status is classified using a fully connected layer and a Softmax activation function.

[0228] Meanwhile, the methods according to the various embodiments of the present invention described above can be implemented in the form of an application or software program that can be installed on an existing electronic device.

[0229] In addition, the whole or part of the method may be composed of multiple software function modules and implemented on an operating system (OS). Alternatively, each step may be composed of a single software function module, or each step may be combined to form a single software function module and implemented on an operating system. Therefore, even if all of the embodiments of the present disclosure are not implemented as a single software function module, if multiple software function modules implement each step of the present disclosure and multiple software function modules are implemented on a single operating system, it can be understood that the method of the present disclosure has been implemented.

[0230] In addition, the methods according to the various embodiments of the present invention described above can be implemented solely through software upgrades or hardware upgrades of existing electronic devices. Furthermore, the various embodiments of the present invention described above can also be performed through an embedded server equipped in an electronic device or an external server of the electronic device.

[0231] Meanwhile, according to one embodiment of the present invention, the various embodiments described above may be implemented as software comprising instructions stored on a computer-readable recording medium using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented as the processor itself. According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0232] Meanwhile, a computer or a similar device may include a device according to the disclosed embodiments, which is capable of calling instructions stored from a storage medium and operating according to the called instructions. When said instructions are executed by a processor, the processor may perform a function corresponding to said instructions directly or by using other components under the control of said processor. The instructions may include code generated or executed by a compiler or an interpreter.

[0233] A computer-readable recording medium may be provided in the form of a non-transitory computer-readable recording medium. Here, "non-transitory" simply means that the storage medium does not contain a signal and is tangible, without distinguishing whether data is stored semi-permanently or temporarily on the storage medium. In this context, a non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short moment, such as registers, caches, or memory. Specific examples of non-transitory computer-readable media may include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0234] As described above, exemplary embodiments have been disclosed in the drawings and specification. Although specific terms have been used to describe the embodiments in this specification, they are used only for the purpose of explaining the technical concept of this disclosure and are not intended to limit the meaning or the scope of this disclosure as defined in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of this disclosure should be determined by the technical concept of the appended claims. Explanation of the symbols

[0235] 100: Multimodal deep learning device based on modality-specific weight learning

Claims

Claim 1 A memory storing at least one instruction for modality-specific weighted learning-based multimodal deep learning that receives image data and clinical pathology data as input to predict whether a target patient will relapse; The processor includes a processor that performs operations according to the above command, wherein the processor comprises: an image feature extraction module that receives multiple image data and extracts multi-stage features to generate image features; a non-image embedding module that receives a non-image variable including at least one of clinical, numerical, historical, and environmental information and generates non-image embeddings; a dimension transformation module that includes a reduction layer that compresses the image features to a lower dimension and an expansion layer that expands the non-image embeddings to a higher dimension to resolve the dimension imbalance between the image features and the non-image embeddings; and for an image feature vector in which the image features are compressed and transformed to a lower dimension through the reduction layer and a non-image embedding vector in which the non-image embeddings are expanded and transformed to a higher dimension through the expansion layer, the contribution of each modality is dynamically calculated on a sample basis, and normalized modality-specific weights are calculated by sequentially performing a concatenation block, a plurality of Conv2D+Tanh layers, and Softmax operations, and the normalized modality-specific weights are multiplied by each of the image feature vector and the non-image embedding vector to obtain weighted feature vectors An apparatus comprising: an adaptive modality attention module (AdaMod) that generates a personalized fused expression by producing; and a fusion and classification module that combines the weighted feature vectors to generate a final fused expression and performs a prediction based on the final fused expression. Claim 2 In claim 1, the device wherein the image feature extraction module processes image data and corresponding mask data or Region-of-Interest into independent branches, and combines the feature maps of each stage by applying a Tumor-Centric Attention (TCA) module. Claim 3 In claim 2, the tumor center attention (TCA) module independently calculates the activity of intra-region and peri-region regions based on a tumor mask, and then readjusts the feature map based on the calculated weights. Claim 4 In claim 1, the device comprises a non-image embedding module that performs univariate analysis and multivariate analysis on input non-image variables to select statistically significant variables, and generates the non-image embedding using a fully connected layer for the variables. Claim 5 In claim 1, the reduction layer of the dimension transformation module comprises a 3×3 convolutional layer, an average pooling layer, and a fully connected layer to compress and transform the image features into the lower-dimensional image feature vector. Claim 6 In claim 1, the extension layer of the dimension transformation module comprises a plurality of fully connected layers to extend and transform the non-image embedding into the high-dimensional non-image embedding vector. Claim 7 delete Claim 8 In claim 1, the device wherein the fusion and classification module classifies the state of an object from the final integrated representation using a fully connected layer and a Softmax activation function. Claim 9 In claim 1, the device, wherein the processor performs data augmentation during the learning process, and the data augmentation includes one or more of horizontal inversion, rotation, translation, and scaling. Claim 10 In claim 1, the processor is a device that calculates and visualizes the importance of input characteristics using SHAP (SHapley Additive exPlanations) to quantitatively analyze the contribution of each modality. Claim 11 A deep learning method using a multimodal deep learning device based on modality-specific weight learning comprises: a step of receiving multiple image data and extracting multi-stage features to generate image features; a step of receiving a non-image variable including at least one of clinical, numerical, historical, and environmental information and generating non-image embeddings; a step of generating an image feature vector by compressing the image features to a lower dimension and generating a non-image embedding vector by expanding the non-image embeddings to a higher dimension in order to resolve the dimensionality imbalance between the image features and the non-image embeddings; a step of dynamically calculating the contribution of each modality on a sample basis for the image feature vector and the non-image embedding vector, and sequentially performing a concatenation block, a plurality of Conv2D+Tanh layers, and a Softmax operation to calculate normalized modality-specific weights, and multiplying the normalized modality-specific weights by each of the image feature vector and the non-image embedding vector to calculate weighted feature vectors, thereby generating a personalized fused representation. A method comprising the step of combining the weighted feature vectors to generate a final integrated representation and performing a prediction based on the final integrated representation. Claim 12 In claim 11, the step of generating the image features comprises processing the image data and the corresponding mask data or Region-of-Interest into independent branches, and combining the feature maps of each step by applying a Tumor-Centric Attention (TCA) module. Claim 13 In claim 12, the tumor center attention (TCA) module independently calculates the activity of intra-region and peri-region regions based on the center coordinates within the image, and then resizes the feature map based on the calculated weights. Claim 14 In claim 11, the step of generating the non-image embedding comprises performing univariate and multivariate analysis on input non-image variables to select statistically significant variables, and generating the non-image embedding using a fully connected layer for the variables. Claim 15 In claim 11, the step of generating the image features comprises a 3×3 convolutional layer, an average pooling layer, and a fully connected layer, wherein the image features are generated by compressing and transforming them into the lower-dimensional image feature vector. Claim 16 In claim 11, the step of generating the non-image embedding vector comprises a plurality of fully connected layers, wherein the non-image embedding is generated by expanding and transforming the non-image embedding into the high-dimensional non-image embedding vector. Claim 17 delete Claim 18 In claim 11, the step of performing the prediction is a method of classifying the state of a target from the final integrated representation using a fully connected layer and a Softmax activation function. Claim 19 In claim 11, the method further comprises the step of performing data augmentation during the learning process, wherein the data augmentation includes one or more of horizontal inversion, rotation, translation, and scaling. Claim 20 In claim 11, the step of performing the above prediction is a method of calculating and visualizing the importance of input features using SHAP (SHapley Additive exPlanations) to quantitatively analyze the contribution of each modality.

Images