Method, device, medium, program product and terminal for multi-task prediction of medical lesion segmentation and prognosis based on causal perception

Abstract

The application provides a multi-task prediction method and device based on causal perception for medical lesion segmentation and prognosis, a medium, a program product and a terminal. By acquiring medical image data and clinical text information, an image encoder and a text encoder are used to extract preliminary features. Subsequently, a structural causal model purifies image features using text features, removes confounding factors, generates more accurate image representations, and obtains lesion segmentation results through an image decoder. The preliminary features are also combined by a dynamic cross-modal fusion module to form a joint representation vector, and the prognosis-related factors and confounding interference factors are decoupled. Based on the prognosis-related factors, a variational inference model is constructed to generate accurate prognosis prediction information. Through the deep interaction network and the double-task collaborative learning framework based on causal perception, the lesion quantification and prognosis prediction are mutually enhanced, effectively solving the problems of insufficient deep interaction, task fragmentation and confounding interference in existing multi-modal fusion.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to a multi-task prediction method, device, medium, program product and terminal for medical lesion segmentation and prognosis based on causal perception. Background Technology

[0002] Imaging assessment of acute ischemic stroke (AIS) mainly relies on non-contrast computed tomography (NCCT) scans, which are the preferred method in emergency departments due to their high availability and fast scanning speed. However, NCCT has low sensitivity in detecting early ischemic lesions, especially in the hyperacute phase and posterior circulation infarction, where subtle low-density shadows are easily overlooked, and manual interpretation is susceptible to artifacts.

[0003] In recent years, while deep learning-based automatic lesion segmentation methods have made some progress, most rely solely on imaging data, failing to fully integrate clinical information (such as blood glucose and onset time) and neglecting causal relationships between variables. A causal relationship refers to the direct influence of one factor on another, while a spurious association is a superficial correlation without direct causation. For example, age may be associated with prognosis, but the poor prognosis is not directly caused by age itself, but by underlying diseases or delayed treatment. If the model cannot distinguish between these two, the predicted results are prone to deviating from clinical reality.

[0004] Currently, there are three main types of multimodal fusion methods: simple image stitching with clinical features, staged independent modeling (segmentation followed by prediction), and attention-based fusion. However, the first two methods suffer from insufficient data fusion, leading to easy error accumulation. While attention-based methods can capture complex correlations between multimodal data, they still struggle to distinguish between genuine causal relationships and spurious associations. Furthermore, due to significant confounding factors in AIS data, such as hypertension affecting both infarct size and treatment decisions, models lacking causal inference struggle to accurately identify these genuine relationships, resulting in inconsistent performance across different hospitals or patient groups and limiting clinical application. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a multi-task prediction method, device, medium, program product and terminal for medical lesion segmentation and prognosis based on causal perception, in order to solve the problem that the prior art has obvious deficiencies in multimodal fusion depth, task collaborative optimization and causal relationship modeling, which affect the accuracy and clinical interpretability of AIS artificial intelligence-assisted diagnosis and treatment.

[0006] To achieve the above and other related objectives, a first aspect of this application provides a multi-task prediction method for medical lesion segmentation and prognosis based on causal perception, comprising: acquiring medical image data and clinical text information; inputting the medical image data into an image encoder to generate a first image feature; inputting the clinical text information into a text encoder to generate a first text feature; inputting the first image feature and the first text feature into a structural causal model, and purifying the first image feature through the first text feature to generate a second image feature; inputting the second image feature into an image decoder to generate a lesion segmentation result; inputting the first image feature and the first text feature into a dynamic cross-modal fusion module to generate a joint representation vector; performing a decoupling operation on the joint representation vector to generate prognostic factors and confounding interference factors; constructing a variational inference model based on the prognostic factors, and generating prognostic prediction information based on the variational inference model.

[0007] In some embodiments of the first aspect of this application, the process of inputting the first image feature and the first text feature into a structural causal model to cleanse the first image feature using the first text feature to generate a second image feature includes: inputting the first image feature into an activation function to generate a first mask and a second mask; performing a product operation between the first image feature and the first mask to generate a first foreground feature; performing a product operation between the first image feature and the second mask to generate a first background feature; calculating the similarity between the first text feature and the first foreground feature to generate a first weight; calculating the similarity between the first text feature and the first background feature to generate a second weight; performing an orthogonal decoupling operation on the first text feature based on the first weight and the second weight to generate a second foreground feature and a second background feature; performing a clutter removal operation on the first image feature according to the second background feature to generate a cleaned image feature; and performing a feature enhancement operation on the cleaned image feature according to the second foreground feature to generate a second image feature.

[0008] In some embodiments of the first aspect of this application, the process of inputting the first image features and the first text features into a dynamic cross-modal fusion module to generate a joint representation vector includes: inputting the first image features and the first text features into a cross-modal key-value gating to generate a first attention output and a second attention output; and inputting the first attention output and the second attention output into a normalization fusion operation to generate a joint representation vector.

[0009] In some embodiments of the first aspect of this application, the method includes: constructing a first loss function based on the lesion segmentation result; constructing a second loss function based on the prognostic prediction information; constructing a composite loss function based on the first loss function and the second loss function; and updating one or more execution parameters of an image encoder, an image decoder, and a text encoder through the composite loss function.

[0010] In some embodiments of the first aspect of this application, the process of constructing a composite loss function based on the first loss function and the second loss function includes: adaptively weighting the first loss function and the second loss function according to an uncertainty weighting mechanism to construct a composite loss function; the first loss function includes one or more of cross-entropy loss, Dice loss, and orthogonal constraint loss; the second loss function includes decoupling ELBO loss; wherein the decoupling ELBO loss includes reconstruction loss, regularization loss, and causal independence loss.

[0011] In some embodiments of the first aspect of this application, the process of constructing a variational inference model based on the prognostic correlation factors to generate prognostic prediction information includes: obtaining prognostic labels; establishing a prior distribution based on the prognostic correlation factors through a parameterized network to generate prior distribution parameters; establishing a posterior distribution based on the prognostic correlation factors and the prognostic labels through a second decoder to generate posterior distribution parameters; sampling training samples from the posterior distribution and inputting the training samples into the second decoder to generate predicted values ​​corresponding to the training samples; and performing a parameter update operation on the second decoder based on the predicted values ​​and the second loss function to output prognostic prediction information optimized by variational inference.

[0012] To achieve the above and other related objectives, a second aspect of this application provides a multi-task prediction device for medical lesion segmentation and prognosis based on causal perception, comprising: a data acquisition module for acquiring medical image data and clinical text information; a feature extraction module for inputting the medical image data into an image encoder to generate first image features; inputting the clinical text information into a text encoder to generate first text features; a lesion segmentation module for inputting the first image features and the first text features into a structural causal model, purifying the first image features through the first text features to generate second image features; inputting the second image features into an image decoder to generate lesion segmentation results; and a prognosis prediction module for inputting the first image features and the first text features into a dynamic cross-modal fusion module to generate a joint representation vector; performing a decoupling operation on the joint representation vector to generate prognostic factors and confounding interference factors; and constructing a variational inference model based on the prognostic factors to generate prognostic prediction information based on the variational inference model.

[0013] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception.

[0014] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code that, when executed on a computer, enables the computer to implement the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception.

[0015] To achieve the above and other related objectives, a fifth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception.

[0016] As described above, the multi-task prediction method, device, medium, program product, and terminal for medical lesion segmentation and prognosis based on causal perception of this application have the following beneficial effects: By acquiring medical image data and clinical text information, preliminary features are extracted using an image encoder and a text encoder, respectively. Subsequently, a structural causal model uses text features to purify image features, remove confounding factors, and generate a more accurate image representation. The lesion segmentation result is then obtained through an image decoder. The preliminary features are further combined by a dynamic cross-modal fusion module to form a joint representation vector, and prognostic-related factors and confounding interference factors are decoupled. Based on the prognostic-related factors, a variational inference model is constructed to generate accurate prognostic prediction information. This method, through a causal perception-based deep interactive network and a dual-task collaborative learning framework, promotes the mutual enhancement of lesion quantification and prognostic prediction, effectively solving the problems of insufficient deep interaction, task fragmentation, and confounding interference in existing multimodal fusion methods. Attached Figure Description

[0017] Figure 1 The diagram shows a flowchart of an embodiment of the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception of this application.

[0018] Figure 2 This illustration shows a flowchart of another embodiment of the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception of this application.

[0019] Figure 3 This image shows a text-to-text conversion comparison diagram of clinical tabular data in one embodiment of the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception of this application.

[0020] Figure 4 The image shows two-dimensional and three-dimensional results of lesion quantification in a public dataset in one embodiment of the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception of this application.

[0021] Figure 5 The diagram shows the ROC curves in public and private datasets in one embodiment of the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception of this application.

[0022] Figure 6 This paper presents a comparison of feature contributions before and after causal implementation in an embodiment of the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception of this application.

[0023] Figure 7 The training curves for two tasks are shown in one embodiment of the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception of this application.

[0024] Figure 8This paper shows a schematic diagram of an embodiment of the multi-task prediction device for medical lesion segmentation and prognosis based on causal perception according to this application.

[0025] Figure 9 This paper shows a schematic diagram of the structure of a multi-task prediction terminal for medical lesion segmentation and prognosis based on causal perception, according to an embodiment of this application. Detailed Implementation

[0026] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0027] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0028] <1> nnUNet: An automatically configured medical image segmentation framework based on the U-Net structure, which can automatically adjust the network architecture and training strategy according to the data to achieve efficient and accurate segmentation.

[0029] <2> DICOM anonymization tools: Specialized software or modules used to remove patient identity and privacy information from medical image DICOM files, ensuring data security and facilitating sharing and research.

[0030] <3> Image codec: A neural network component in which the encoder extracts high-level features of an image, and the decoder reconstructs the image or generates outputs for tasks such as segmentation and detection based on the features.

[0031] <4> Text encoder: Converts raw text into a continuous low-dimensional vector representation using a deep learning model, which facilitates fusion with other modal data and downstream task processing.

[0032] <5> Adapter: A lightweight parameter module that inserts a pre-trained model to adapt it to a new task, enabling efficient fine-tuning and reducing computational resource consumption.

[0033] <6> Orthogonal decoupling operation: By decomposing joint features into mutually orthogonal subspaces, different information factors, such as task relevance and confounding interference, are separated to achieve a purer feature representation.

[0034] <7> Foreground / Background Features: In image processing, foreground features refer to the features of the region of interest (such as lesions), while background features refer to the features of other regions besides the foreground.

[0035] <8> Linear projection function: Using matrix multiplication and bias vector, a high-dimensional input is linearly mapped to another feature space, achieving dimensionality reduction or transformation.

[0036] <9> Based on the Pearl backdoor criterion: Based on Pearl's causal inference theory, it is used to identify and cut off the influence of confounding variables on causal relationships, ensuring the accuracy of effect estimation.

[0037] <10> Cross-modal key-value gating: By combining different modal inputs, information flow is dynamically adjusted through key-value attention mechanisms and gating strategies to achieve effective modal fusion.

[0038] <11> ELBO (Evidence Lower Bound): The optimization objective in variational inference is to improve the performance of latent variable models by maximizing the lower bound to approximate the log-likelihood of the true data.

[0039] <12> KL divergence (Kullback-Leibler Divergence): An asymmetric metric that measures the difference between two probability distributions, often used for regularization or probabilistic model training.

[0040] <13> HSIC (Hilbert-Schmidt Independence Criterion): A metric based on kernel techniques to measure the degree of statistical dependence between two variables, used to detect and separate correlations between features.

[0041] To facilitate understanding of the embodiments of this application, firstly, in conjunction with Figure 1 and Figure 2 A detailed explanation will be provided. Figure 1 This document illustrates a flowchart of a multi-task prediction method for medical lesion segmentation and prognosis based on causal perception, as described in an embodiment of the present invention. The multi-task prediction method for medical lesion segmentation and prognosis based on causal perception in this embodiment mainly includes the following steps:

[0042] Step S11: Obtain medical imaging data and clinical text information.

[0043] In one embodiment of this application, the training phase employs multi-task validation using multiple datasets. Specifically, the publicly available dataset ISLES2024 is used for dual-task validation of stroke lesions (lesion segmentation and prognostic prediction), the publicly available AISD dataset is used for additional validation of the segmentation task (lesion segmentation), and the private GLP-AIS dataset is used for validation in the prognostic prediction stage (prognostic prediction), thereby determining the model parameters. The ISLES2024 dataset refers to multimodal stroke medical images and their corresponding manually annotated dataset, while the AIS dataset refers to a dataset of acute ischemic stroke patients with images and rich clinical information.

[0044] In one embodiment of this application, the medical image data used is non-contrast computed tomography (NCCT). The processing of the medical image data includes: first, inputting the raw image data into a MATLAB tool for skull dissection preprocessing, and then adjusting the window width and window level of the raw image data. Specifically, the window width or window level is first set to [0, 100] Henle units to enhance the contrast of brain tissue. Henle units represent the density units of CT (computed tomography) images in medical imaging, used to measure the degree of X-ray attenuation in different tissues. Subsequently, the medical image data after the window width and window level adjustment operation is preprocessed using the nnUNet normalization process. The preprocessing process includes background cropping and resampling to unify the input size of each dataset. For example, the size of ISLES2024 is adjusted to [40, 224, 192], the size of AISD data is adjusted to [32, 256, 224], and the size of GLP-AIS data is adjusted to [80, 192, 160] to ensure that the data augmentation strategy is consistent with the default framework of nnUNet, thereby improving the stability and generalization ability of model training.

[0045] In one embodiment of this application, the following preprocessing steps are performed on the acquired clinical text information: the tabular clinical variables of ISLES2024 and the private dataset are transformed into text description paragraphs with clinical semantics. Specifically, the content of the clinical text information is organized and classified into five categories: "basic information" (including age, gender, etc.), "medical history" (such as hypertension, diabetes, etc.), "laboratory tests" (blood glucose, international normalized ratio INR, etc.), "stroke characteristics" (such as post-awareness stroke), and "clinical scores" (such as consciousness score upon admission, etc.).

[0046] Furthermore, for the prognostic prediction stage, the 90-day modified Rankin Scale (mRS-90) and the National Institutes of Health Stroke Scale (NIHSS) score measured 24 hours after stroke onset were used as the prognostic evaluation criteria. Labeling for the prognostic prediction stage included: the mRS-90 score ranged from 0 to 6, where 0 represented asymptomatic and 6 represented death. According to clinical guidelines, the mRS score was divided into good prognosis (mRS ≤ 2) and poor prognosis (mRS > 2), reflecting the recovery of the patient's daily living functions. In the GLP-AIS dataset, the NIHSS score at 24 hours after treatment ranged from 0 to 42, with higher scores indicating more severe neurological impairment. A critical threshold of 5 points, widely accepted in thrombolysis trials, was used for binary classification (0 indicating NIHSS ≤ 5, 1 indicating NIHSS > 5) to determine clinically significant treatment response. In addition, the entire data processing employed DICOM anonymization to strictly protect patient privacy and ensure compliance with medical ethics.

[0047] Step S12: Input the medical image data into the image encoder to generate the first image feature; input the clinical text information into the text encoder to generate the first text feature.

[0048] Figure 2 A schematic diagram of the model structure in one embodiment of this application is shown, as follows: Figure 2 As shown, the feature extraction step includes encoding medical image data and clinical text information separately. The medical image data uses an nnUNet-based image encoder, whose parameters are updated during training to ensure that the image features closely match the specific task requirements. The clinical text information uses a pre-trained language model encoder with an adapter module. During training, the main parameters of the text encoder remain frozen, and only the adapter parameters are updated.

[0049] It should be noted that this structural design is adopted because the pre-trained language model encoder has already learned rich linguistic knowledge from a large-scale corpus. Freezing its main parameters can avoid performance degradation caused by overfitting and insufficient training data. Meanwhile, small-scale parameter updates of the adaptor module can effectively fine-tune the specific clinical text semantics in this application, improving the adaptability of the text representation. At the same time, the full update of the image encoder parameters ensures the full extraction of medical image features and task matching.

[0050] Furthermore, based on the multimodal features extracted by the encoder, this application further adopts a language model embedding technique with a Transformer architecture to effectively retain and utilize the inherent correlation of features that are easily ignored by the traditional one-hot encoding method, so as to achieve the natural fusion of clinical text and medical image features and significantly improve the expressive power of multimodal information.

[0051] In one embodiment of this application, to address the problem of missing values ​​in clinical data, this application designs a missing value completion method based on an attention mechanism. This method includes the following steps: First, the known parts of the input data are encoded. Then, an attention mechanism is used for dynamic weight allocation to intelligently capture the correlation between different features and infer the possible values ​​of the missing values. Specifically, the model adaptively completes the missing data by calculating the dependency relationship between known variables and missing variables, combined with historical data and multimodal information. Compared with traditional matrix interpolation or simple imputation, this method not only improves the accuracy of completion but also enhances the stability and robustness of the model when facing incomplete data. For example, when key clinical indicators such as blood pressure and blood glucose are missing, the model can comprehensively consider other relevant variables and historical trends to ensure the integrity and consistency of multimodal features.

[0052] To address the lack of certain clinical metadata in the AISD dataset, this application proposes a unified method for generating synthetic text descriptions. The method includes the following steps: First, key information is extracted from the imaging diagnostic report; second, this information is standardized according to a preset template to generate structured text such as "CT scan shows a low-density lesion in the left hemisphere, with an average density of 20.1 HU, an estimated volume of 29.23 cm³, no obvious mass effect, and imaging features suggesting ischemic changes"; finally, the generated standard text is used as a clinical description to supplement the multimodal input. This approach not only enriches the expression of clinical data but also ensures the consistency and integrity of multimodal data, thereby improving the model's adaptability and predictive accuracy. Figure 3 This document illustrates a text conversion comparison diagram of clinical table data in one embodiment of this application. Here, Category represents the category of clinical information, Field represents the specific data field under that category, Description Rule represents the description rule when the field value is 1, and Example represents an example of the converted clinical text description.

[0053] Step S13: Input the first image feature and the first text feature into the structural causal model, and purify the first image feature through the first text feature to generate the second image feature; input the second image feature into the image decoder to generate the lesion segmentation result.

[0054] In one embodiment of this application, the process of inputting the first image feature and the first text feature into a structural causal model to cleanse the first image feature using the first text feature to generate a second image feature includes: inputting the first image feature into an activation function to generate a first mask and a second mask; performing a product operation between the first image feature and the first mask to generate a first foreground feature; performing a product operation between the first image feature and the second mask to generate a first background feature; calculating the similarity between the first text feature and the first foreground feature to generate a first weight; calculating the similarity between the first text feature and the first background feature to generate a second weight; performing an orthogonal decoupling operation on the first text feature based on the first weight and the second weight to generate a second foreground feature and a second background feature; performing a clutter removal operation on the first image feature according to the second background feature to generate a cleaned image feature; and performing a feature enhancement operation on the cleaned image feature according to the second foreground feature to generate a second image feature.

[0055] In this embodiment, the first image feature refers to the dimension obtained by encoding the NCCT image using an nnUNet-based image encoder. The feature representation; the first text feature refers to encoding the clinical text through a text encoder with frozen subject parameters, resulting in a dimension of The structural causal model is represented by its features. Figure 2 As shown in the COP (Causal Ortho Purifier) ​​module, the COP module is used to perform text-guided orthogonal decomposition and foreground enhancement processing of multimodal features.

[0056] Furthermore, the COP module first uses the first image features Apply a Sigmoid-activated convolution operation to generate a lesion attention mask. Where B represents the batch size, i.e., the number of samples processed at one time; 1 indicates that the mask has a single channel; D′, H′, and W′ represent the spatial dimensions of the mask in the depth, height, and width directions, respectively. The numerical range of this mask is [0,1], used to characterize the model's attention weight to different spatial regions, helping to highlight the feature information of the lesion site. Subsequently, the lesion attention mask is used... Image features Pixel-level weighted decomposition is performed, and the feature is decoupled into first foreground features related to the lesion using Equations 1 and 2. First background features related to confounding factors , where the symbol This represents the element-wise multiplication of pixels at corresponding locations. This operation allows the COP model to clearly distinguish and enhance features related to the lesion area, while suppressing interfering information in the background.

[0057] (Formula 1)

[0058] (Formula 2)

[0059] At the same time, the first foreground features and the first background features were analyzed separately. Perform normalization to generate ; for the calculated and Perform average pooling to generate image features. and In this process, average pooling is a downsampling operation that averages a local region to reduce the feature map size and extract smooth features; subsequently, formulas 3 and 4 are used to calculate... and , and text features Similarity between and weight .

[0060] (Formula 3)

[0061] (Formula 4)

[0062] Using the first and second weights, the text features are decoupled and decomposed along different directions according to Formulas 5 and 6, respectively, to distinguish multi-dimensional semantics in the text, thereby obtaining the second foreground feature and the second background feature. The i-th column represents the weight. This represents the feature orientation related to confounding factors (background). This captures the causal (foreground) feature direction. Thus, by using background text similarity weights, components related to confounding factors are identified and located in the feature space, effectively capturing background interference.

[0063] (Formula 5)

[0064] (Formula 6)

[0065] Furthermore, the process of performing a clutter removal operation on the first image feature to generate a clean image feature based on the second background feature includes: to eliminate clutter effects, using an orthogonal projection (OP) operation to project the first image feature... Orthogonal projection is performed along the identified hybrid directions using Equation 7 to generate a projected representation that includes the hybrid components. , where the symbol This represents the dot product operation. This represents a specific linear projection function.

[0066] (Formula 7)

[0067] Finally, to further amplify the foreground signal related to causality, the foreground feature components are enhanced. Based on the foreground features from the second stage, the purified image features are enhanced to generate the second-stage image features. This process is shown in Equation 8, where the parameters... It is used to balance the removal of confounding factors and the enhancement of foreground features. Overall, the COP module effectively suppresses confounding effects through text-guided orthogonal projection and foreground enhancement, thus more accurately approximating the distribution after causal intervention. .

[0068] (Formula 8)

[0069] It should be noted that, compared to the traditional Structure Causal Model (SCM) in existing technologies, this application not only draws on the modeling approach of SCM for multimodal causal relationships, but also achieves explicit identification and proactive intervention of confounding factors in multimodal medical scenarios through the systematic application of the Pearl backdoor criterion. In SCM, by constructing a directed acyclic graph (DAG), NCCT images can be characterized (…). ), clinical texts ( ), confounding factors ( ), and the predicted target ( Including lesion segmentation The causal relationship between them is unclear. However, in actual stroke diagnosis and treatment scenarios, due to confounding factors... These factors are often not directly observable; they may be detected through backdoor paths (such as...). This indirectly affects the prediction results, causing the model output to rely on spurious (non-causal) correlations, thus affecting generalization ability and interpretability.

[0070] Figure 4 The illustration shows two-dimensional and three-dimensional results of lesion quantification in a public dataset in one embodiment of this application. Figure 5The present invention illustrates receiver operating characteristic (ROC) curves on public and private datasets for an embodiment of this application. The vertical axis of the ROC curve represents sensitivity (True Positive Rate, TPR), and the horizontal axis represents specificity (False Positive Rate, FPR). Figure 5 Several existing lesion segmentation methods and their corresponding Area Under Curve (AUC) values ​​were used. For the public dataset on the left, the results include: Logistic Regression (LR) (AUC=0.7024), Random Forest (RF) (AUC=0.7440), Support Vector Machine (SVM) (AUC=0.7560), eXtreme Gradient Boosting (XGB) (AUC=0.6161), IB-OP (AUC=0.7619), U-ICH-PP (AUC=0.7778), FuseM (AUC=0.8154), MFMF (AUC=0.8291), MMMT (AUC=0.8571), ICHPro (AUC=0.8843), ICH-PR (AUC=0.8673), and the method of this application (AUC=0.9083). For the private dataset on the right, the results include: Logistic Regression (LR) (AUC=0.7083), Random Forest (RF) (AUC=0.7604), Support Vector Machine (SVM) (AUC=0.7458), eXtreme Gradient Boosting (XGB) (AUC=0.6771), IB-OP (AUC=0.7649), U-ICH-PP (AUC=0.7308), FuseM (AUC=0.8304), MFMF (AUC=0.8549), MMMT (AUC=0.8784), ICHPrO (AUC=0.9059), ICH-PR (AUC=0.8824), and the method of this application (AUC=0.9176).

[0071] This application addresses the problem of causal effect estimation bias caused by confounding factors in existing technologies by strictly adhering to Pearl's backdoor criterion for causal intervention. Specifically, it first identifies and models the latent representations of confounding factors in a multimodal feature space, and then conditions these latent confounding variables to effectively block the backdoor paths formed by confounding factors, ensuring that the model accurately reflects causal effects. An unbiased estimate. Even with confounding variables. Since they cannot be directly observed, this application also uses latent variables in the feature space (such as latent variables obtained through variational decoupling, or surrogate variables based on feature orthogonalization) as... The proxy strictly adheres to the core idea of ​​the backdoor principle, which is that as long as all influences are exposed during model training... With results By completing the control (conditioning) of the common parent node, confounding effects can be effectively eliminated and causal relationships can be accurately inferred.

[0072] Specifically, in the prognostic prediction task, this application explicitly separates confounding factors in SCM through a variational decoupling method, achieving proactive identification and processing of potential confounding factors, which is an explicit causal intervention operation. In the lesion segmentation and quantification task, a text-guided orthogonal projection (COP) module is used to perform causal purification and foreground enhancement of lesion features, completing an implicit intervention at the feature level. These two operations correspond to the explicit and implicit causal processing mechanisms in SCM theory, respectively, and their synergistic effect significantly improves the multimodal model's resistance to confounding factors and the accuracy of causal effect estimation. Through proactive confounding separation, orthogonalization, and conditionalization, this application truly realizes interventional causal modeling based on the Pearl backdoor criterion, improving the model's robustness and causal validity.

[0073] This application addresses the limitations of existing technologies in handling confounding factors by proposing an intervention method based on the Pearl backdoor criterion. Specifically, it first actively seeks and models representations of confounding factors in a multimodal feature space, then conditions these potential confounding variables to effectively block backdoor paths caused by these confounding factors, ensuring the model can handle causal effects. Perform unbiased estimation. It is worth noting that even with confounding variables... Even though they cannot be directly observed, this application can also use latent variables learned in the feature space (e.g., obtained using feature orthogonalization methods) as... This allows for the use of proxies to ensure strict adherence to backdoor guidelines.

[0074] Building upon this foundation, this application further combines text-guided feature orthogonalization and foreground enhancement operations to effectively filter out confounding signals from multimodal inputs. This proactive identification, orthogonalization, and conditionalization mechanism not only theoretically ensures the scientific validity of the model's causal estimation but also significantly improves the model's robustness to confounding factors in practice. In contrast, traditional structural causal models (SCMs) rely more on predetermined DAG structures and passive modeling for handling confounding factors. The method in this application, through proactive intervention using a backdoor criterion, significantly enhances the model's causal validity and application robustness.

[0075] Figure 6 This paper presents a comparison chart of feature contributions before and after causal implementation in one embodiment of this application. For example... Figure 6 As shown in the figure above, the situation before implementation is illustrated. It can be seen that the SHAP values ​​of various features (such as Feature 510, Feature 170, etc.) are relatively dispersed, with the horizontal axis ranging from approximately -0.05 to 0.15. High and low feature values ​​contribute positively and negatively, respectively. The figure below depicts the feature contributions after implementation. At this point, the SHAP values ​​of all features are significantly concentrated between -0.05 and 0.05. This significant convergence indicates that causal intervention successfully reduced the impact of each feature on the model output, making the model prediction more stable and robust, reducing over-reliance on specific features, and thus achieving the expected optimization effect.

[0076] Step S14: Input the first image features and the first text features into the dynamic cross-modal fusion module to generate a joint representation vector; perform a decoupling operation on the joint representation vector to generate prognostic correlation factors and confounding interference factors; construct a variational inference model based on the prognostic correlation factors to generate prognostic prediction information based on the variational inference model.

[0077] For the dynamic cross-modal fusion module corresponding to step S14 of this application, the core objective of the prognostic prediction branch is to block confounding factors. Backdoor path to accurately estimate intervention effect , Indicates by analyzing the image and text Intervention The obtained clinical prediction results The probability. If the traditional multimodal fusion method is used to directly optimize... This approach makes it easy to learn spurious correlations in the data, leading to a decline in the model's generalization performance. As mentioned earlier, backdoor tuning requires a sufficient set of confounding variables as surrogates. While conditional processing is performed, in actual clinical scenarios, there are often problems such as missing observations or noise interference.

[0078] To address the aforementioned issues, this invention first maps images and text into a latent space using a multimodal fusion module (MoMA). This achieves efficient modality fusion while preserving causal characteristics. The latent representation is then decoupled into two orthogonal components. and ,in Characteristics indicating causal relationship with clinical prognosis (equivalent to) Characteristics after intervention). Then capture the confounding factors The resulting variation (as) Potential proxies). By forcibly implementing and The statistical independence makes Established, so that, through the image and text Intervention The obtained clinical prediction results The probability is approximately equal to the representation of a given confounding factor. Under these conditions, clinical prediction results The probability of . Among them Eliminated This design makes it dependent. It can effectively replace unobserved confounding variables Without relying on explicit Even under measurement conditions, the backdoor criterion is still met, thus ensuring the causal reliability of prognostic predictions.

[0079] In one embodiment of this application, the process of inputting the first image features and the first text features into a dynamic cross-modal fusion module to generate a joint representation vector includes: inputting the first image features and the first text features into a cross-modal key-value gating to generate a first attention output and a second attention output; and inputting the first attention output and the second attention output into a normalization fusion operation to generate a joint representation vector.

[0080] In this embodiment, in cross-modal key-value gating, each modal pair is gated using a cross-modal key-value gating mechanism. and These are mapped to query (Q), key (K), and value (V), respectively. Then, using a gating module, for each modality, guided by Q, the top-k most relevant feature pairs are selected from the corresponding KV pairs. For example, in the image modality, the gating module, based on the text query, retains the k most relevant image feature regions and their corresponding key values ​​V using Equation 9, thereby achieving effective information filtering and fusion across modalities.

[0081] (Formula 9)

[0082] It should be noted that the MoMA block model structure in this embodiment is innovatively configured with cross-modal adaptation, inspired by Mixture of Block Attention (MoBA), that is, performing bidirectional cross-attention in the attention structures of two modalities. Specifically, the text-to-image (T2I) flow focuses on lesion regions that correspond to clinical descriptions, while the image-to-text (I2T) flow reinforces textual concepts corresponding to image manifestations. Thus, the cross-modal structure simulates the diagnostic thinking of radiologists through bidirectional comparison between "images and clinical presentation." Finally, the first and second attention outputs are fused through layer normalization, as shown in Equation 10, to generate a redundant joint representation vector. .

[0083] (Formula 10)

[0084] It is worth noting that, compared with traditional cross-attention or simple splicing and fusion, the sparse Top-k gating mechanism of MoMA has the following advantages: First, the MoMA structure proposed in this application only handles 20%-30% of cross-modal correlations, which greatly improves computational efficiency; second, this application can automatically filter out noisy interactions that are irrelevant to the task, thereby completing feature selection; more importantly, this application effectively suppresses feature responses dominated by confounding factors through such a MoMA structure, naturally blocking the backdoor path of SCM.

[0085] In one embodiment of this application, the process of constructing a variational inference model based on the prognostic factors to generate prognostic prediction information includes: obtaining prognostic labels; establishing a prior distribution based on the prognostic factors through a parameterized network to generate prior distribution parameters; establishing a posterior distribution based on the prognostic factors and the prognostic labels through a second decoder to generate posterior distribution parameters; sampling training samples from the posterior distribution and inputting the training samples into the second decoder to generate predicted values ​​corresponding to the training samples; and performing a parameter update operation on the second decoder based on the predicted values ​​and the second loss function to output prognostic prediction information optimized by variational inference.

[0086] In this embodiment, the fused multimodal feature vector is obtained. Subsequently, variational reasoning is used to achieve decoupling modeling. To achieve this... To estimate causality, this invention constructs an improved Evidence Lower Bound (ELBO), which explicitly decouples prognostic-related causal factors through variational inference. Related interference factors Assume latent space vectors It can be decomposed into generating prognostic targets. of and coding hybrid variations Its marginal likelihood is shown in Equation 11.

[0087] (Formula 11)

[0088] In one embodiment of this application, the process of constructing a composite loss function based on the first loss function and the second loss function includes: adaptively weighting the first loss function and the second loss function according to an uncertainty weighting mechanism to construct a composite loss function; the first loss function includes one or more of cross-entropy loss, Dice loss, and orthogonal constraint loss; the second loss function includes decoupling ELBO loss; wherein the decoupling ELBO loss includes reconstruction loss, regularization loss, and causal independence loss.

[0089] In this embodiment, since the true posterior distribution is difficult to solve directly, a variational posterior is introduced. An approximation is made, and the standard ELBO lower bound including the reconstruction term and KL divergence is derived as shown in Equation 12:

[0090] (Formula 12)

[0091] To strictly meet Required Independence constraints, i.e., the heterogeneous feature representations obtained by decoupling from the original data. Must be characterized by non-confounding (or causal) features Maintaining statistical independence is due to only when and When strictly independent, Only through control or elimination can confounding effects be purely isolated and removed, thereby achieving an unbiased estimation of the target causal effect. This application innovatively introduces a penalty term based on the Hilbert-Schmidt Independence Criterion (HSIC) into ELBO to form a triple-constrained optimization objective as shown in Equation 13, which includes reconstruction accuracy, posterior stability, and causal independence.

[0092] , (Formula 13)

[0093] Where, coefficient This improved ELBO framework not only modulates the latent representation capacity but also works synergistically with the HSIC term. The KL term ensures the robustness of the posterior distribution, while the HSIC term enforces statistical independence between causal and confounding features. This improved ELBO framework has dual theoretical advantages: firstly, it maintains the discriminative performance of prognostic prediction through variational inference; secondly, it ensures... Satisfying the independence requirement of backdoor adjustment provides a theoretical guarantee for multimodal causal representation learning that goes beyond heuristic decoupling methods. The original ELBO lower bound included a reconstruction term and a regularization term (KL constraint). In SCM, to achieve explicit removal of confounding factors, an independence term was added, which uses HSIC to ensure... Independent of That is, to make the prediction results dependent on causal features. Furthermore, it aims to make causal features as independent as possible from confounding features. This is because the HSIC value is 0 if and only if two random variables are statistically independent (under certain regularization conditions). Therefore, minimizing HSIC as a loss term effectively promotes model learning that allows for... and The statistical dependence between them tends to a minimum, thereby achieving and They are mutually independent.

[0094] In one embodiment of this application, the method includes: constructing a first loss function based on the lesion segmentation result; constructing a second loss function based on the prognostic prediction information; constructing a composite loss function based on the first loss function and the second loss function; and updating one or more execution parameters of the image encoder, image decoder, and text encoder through the composite loss function.

[0095] In this embodiment, to achieve joint optimization of lesion segmentation and prognosis prediction, and to address the imbalance problem between tasks, this invention proposes a multi-task loss function based on causal regularization, as shown in Equation 14. Denotes the first loss function. This represents the second loss function. This represents the composite loss function.

[0096] (Formula 14)

[0097] The innovative loss function incorporates the Kendall uncertainty weighting mechanism into the causal learning framework. This mechanism learns the uncertainty of each task and automatically adjusts the weights of the losses for each task in multi-task learning, enabling the model to dynamically balance the importance of tasks and improve overall performance and stability. The composite loss function comprises three key components: a first loss function representing the lesion segmentation loss; and a third loss function representing the lesion segmentation loss. It combines cross-entropy, Dice loss, and orthogonality constraints; the second loss function represents the prognostic prediction loss. This includes decoupling ELBO loss; learnable task uncertainty parameters. and .pass The weighted form allows the model to dynamically adjust the contribution of different tasks. For example, when a task has high predictive uncertainty (e.g., cases with ambiguous prognoses), its loss weight is automatically reduced, while the constant term... This is used to prevent the weights from increasing indefinitely. This design retains the advantages of traditional multi-task learning while ensuring key characteristics through causal constraints: maintaining [the desired balance] while adaptively balancing the tasks. The decontamination properties of the feature are unaffected.

[0098] In one embodiment of this application, the model framework of the present invention is implemented using the PyTorch 1.12 deep learning framework, based on the CUDA 12.2 parallel computing architecture, and trained and tested on an Ubuntu 20.04 workstation equipped with an NVIDIA GeForce RTX 3080 Ti graphics card (12GB VRAM). The network training adopts a three-dimensional data processing mode, with all datasets maintaining a batch size of 2. Model convergence was confirmed after 500 training epochs. Parameter initialization uses the Kaiming method, with the optimizer selected as stochastic gradient descent (SGD) with a momentum (0.99) and weight decay (3e-5). The initial learning rate is set to 1e-4 and a multinomial decay strategy (decay exponent 0.9) is adopted. Specifically, the Kaiming initialization method randomly initializes the weights to a mean of 0 and a variance of based on the number of fan-in connections of the layer. The distribution is adjusted to accommodate the ReLU activation function and maintain stable signal transmission.

[0099] In one embodiment of this application, to ensure the reliability of the evaluation, the experiment adopted stratified five-fold cross-validation: the ISLES2024 dataset (129 cases) was divided into training and validation sets at a ratio of 103:26, the AISD dataset (395 cases) was divided at a ratio of 316:79, and the private GLP-AIS dataset (232 cases) was divided at a ratio of 185:47. All partitions maintained class balance based on mRS prognostic grouping and maintained the consistency of center distribution in multi-center data. Clinical text encoding adopted an adapter-enhanced Transformer architecture. Under the premise of freezing the parameters of the pre-trained language model, a trainable adapter layer was inserted, which not only preserved the language understanding ability of the pre-trained model, but also achieved efficient transfer learning. The performance of four encoder variants (including PubMedBERT, Bio_ClinicalBERT, CLIP, etc.) and three adaptation strategies (internal adaptation, external adaptation, and low-rank LoRA adaptation) was studied.

[0100] Figures 4 to 7This paper presents the results of the quantitative assessment and prognostic prediction tasks in this application, as well as the impact of the causal structure model on the network and the network training process. Lesion segmentation was evaluated using metrics such as Dice similarity coefficient (Dice Score, DSC), precision (Precision, PRE), recall (Recall, REC), and 95th percentile Hausdorff distance (HD95). Prognostic prediction was assessed using accuracy (ACC), mean intersection-over-union ratio (mIoU), recall (REC), and F1 score (F1). Except for HD95, higher values ​​for all metrics indicated better performance. Segmentation metrics were calculated based on the spatial overlap between the predicted results for each sample and the gold standard. Statistical significance was tested using p-value analysis (threshold p < 0.05). For the prognostic task, receiver operating characteristic (ROC) curves were additionally plotted to analyze the performance of the classification threshold.

[0101] Figure 7 This paper illustrates the training curves of a causal perception-based multi-task prediction method for medical lesion segmentation and prognosis in one embodiment of this application, performing well in both lesion segmentation and prognosis prediction tasks. The left subplot depicts the change in accuracy (ACC) for the lesion segmentation task with training epochs, where "Img_Cli_Model" significantly outperforms both "Img_Model" and "Cli_Model," indicating that the multi-task model fusing image and clinical information achieves the best results in the segmentation task. The right subplot presents the Dice similarity coefficient (DSC) training curve for the prognosis prediction task, again showing that "Img_Cli_Model" has a higher DSC value than "Img_Model," further confirming the effectiveness of integrating multimodal data in improving the model's segmentation performance in the prognosis prediction task. Overall, this method, through multi-task learning and information fusion, demonstrates superior comprehensive performance in the accurate segmentation and prognosis assessment of medical lesions.

[0102] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, "first image feature" and "second image feature" are used only to distinguish different image features and do not limit their order. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that terms such as "first" and "second" do not necessarily imply that they are different.

[0103] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0104] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0105] Figure 8 This is a schematic block diagram of a multi-task prediction device 800 for medical lesion segmentation and prognosis based on causal perception, provided in an embodiment of this application. Figure 8 As shown, the device includes a data acquisition module 801, a feature extraction module 802, a lesion segmentation module 803, and a prognosis prediction module 804.

[0106] Data acquisition module 801: Used to acquire medical image data and clinical text information.

[0107] Feature extraction module 802: used to input the medical image data into the image encoder to generate first image features; and to input the clinical text information into the text encoder to generate first text features.

[0108] The lesion segmentation module 803 is used to input the first image features and the first text features into the structural causal model, purify the first image features through the first text features to generate second image features, and input the second image features into the image decoder to generate lesion segmentation results.

[0109] The prognostic prediction module 804 is used to input the first image features and the first text features into the dynamic cross-modal fusion module to generate a joint representation vector; perform a decoupling operation on the joint representation vector to generate prognostic correlation factors and confounding interference factors; construct a variational inference model based on the prognostic correlation factors, and generate prognostic prediction information based on the variational inference model.

[0110] It should be understood that the specific process of each module performing the above-mentioned steps has been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0111] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0112] Figure 9 This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 9 As shown, the electronic terminal includes at least one processor 901, a memory 902, at least one network interface 903, and a user interface 905. The various components in the device are coupled together via a bus system 904. It is understood that the bus system 904 is used to implement communication between these components. In addition to a data bus, the bus system 904 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 9 The general will label all buses as bus systems.

[0113] The user interface 905 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0114] It is understood that memory 902 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0115] In this embodiment of the invention, the memory 902 is used to store various types of data to support the operation of the electronic terminal 900. Examples of this data include: any executable program for operation on the electronic terminal 900, such as the operating system 9021 and application programs 9022; the operating system 9021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 9022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The multi-task prediction method for medical lesion segmentation and prognosis based on causal perception provided in this embodiment of the invention can be included in the application program 9022.

[0116] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 901. Processor 901 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 901 or by instructions in software form. The processor 901 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 901 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 901 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0117] In an exemplary embodiment, the electronic terminal 900 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0118] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception as described in any of the embodiments above.

[0119] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to execute the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception as described in any of the embodiments above.

[0120] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0121] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0122] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0124] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0126] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0127] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0129] In summary, this application provides a multi-task prediction method, device, medium, program product, and terminal for medical lesion segmentation and prognosis based on causal perception. It acquires medical image data and clinical text information, extracting preliminary features using an image encoder and a text encoder, respectively. Subsequently, a structural causal model uses text features to purify image features, removing confounding factors and generating a more accurate image representation. This image representation is then processed by an image decoder to obtain the lesion segmentation result. The preliminary features are further combined by a dynamic cross-modal fusion module to form a joint representation vector, decoupling prognostic factors and confounding interference factors. Based on the prognostic factors, a variational inference model is constructed to generate accurate prognostic prediction information. This method, through a causal perception-based deep interactive network and a dual-task collaborative learning framework, promotes mutual enhancement between lesion quantification and prognostic prediction, effectively solving problems such as insufficient deep interaction, task fragmentation, and confounding interference in existing multimodal fusion methods. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0130] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A multi-task prediction method for medical lesion segmentation and prognosis based on causal perception, characterized in that, include: Acquire medical imaging data and clinical text information; The medical image data is input into an image encoder to generate the first image feature; The clinical text information is input into the text encoder to generate the first text feature; The first image features and the first text features are input into the structural causal model, and the first image features are purified by the first text features to generate the second image features. The second image feature is input into the image decoder to generate lesion segmentation results; The first image features and the first text features are input into the dynamic cross-modal fusion module to generate a joint representation vector; a decoupling operation is performed on the joint representation vector to generate prognostic correlation factors and confounding interference factors; a variational inference model is constructed based on the prognostic correlation factors to generate prognostic prediction information based on the variational inference model.

2. The multi-task prediction method for medical lesion segmentation and prognosis based on causal perception according to claim 1, characterized in that, The process of inputting the first image features and the first text features into a structural causal model to cleanse the first image features using the first text features to generate the second image features includes: The first image features are input into the activation function to generate a first mask and a second mask; The first image features are multiplied with the first mask to generate a first foreground feature; the first image features are multiplied with the second mask to generate a first background feature. Calculate the similarity between the first text feature and the first foreground feature to generate a first weight; calculate the similarity between the first text feature and the first background feature to generate a second weight; Based on the first weight and the second weight, an orthogonal decoupling operation is performed on the first text features to generate a second foreground feature and a second background feature; Based on the second background feature, a clutter removal operation is performed on the first image feature to generate a clean image feature; Based on the second foreground feature, a feature enhancement operation is performed on the cleaned image feature to generate a second image feature.

3. The multi-task prediction method for medical lesion segmentation and prognosis based on causal perception according to claim 1, characterized in that, The process of inputting the first image features and the first text features into the dynamic cross-modal fusion module to generate a joint representation vector includes: The first image features and the first text features are input into a cross-modal key-value gating system to generate a first attention output and a second attention output; The first attention output and the second attention output are input to perform a normalization fusion operation to generate a joint representation vector.

4. The multi-task prediction method for medical lesion segmentation and prognosis based on causal perception according to claim 1, characterized in that, The method includes: A first loss function is constructed based on the lesion segmentation results; a second loss function is constructed based on the prognostic prediction information. Construct a composite loss function based on the first loss function and the second loss function; The composite loss function is used to update one or more execution parameters of the image encoder, image decoder, and text encoder.

5. The multi-task prediction method for medical lesion segmentation and prognosis based on causal perception according to claim 4, characterized in that, The process of constructing a composite loss function based on the first loss function and the second loss function includes: The first loss function and the second loss function are adaptively weighted according to the uncertainty weighting mechanism to construct a composite loss function; The first loss function includes one or more of cross-entropy loss, Dice loss, and orthogonal constraint loss; The second loss function includes the decoupling ELBO loss; wherein the decoupling ELBO loss includes the reconstruction loss, the regularization loss, and the causal independence loss.

6. The multi-task prediction method for medical lesion segmentation and prognosis based on causal perception according to claim 5, characterized in that, The process of constructing a variational inference model based on the prognostic factors, and generating prognostic prediction information based on the variational inference model, includes: Obtain prognostic tags; Based on the aforementioned prognostic factors, a prior distribution is established through a parameterized network to generate prior distribution parameters; Based on the prognostic factors and the prognostic labels, a posterior distribution is established through a second decoder to generate posterior distribution parameters; Training samples are obtained by sampling from the posterior distribution, and the training samples are input into the second decoder to generate the predicted values ​​corresponding to the training samples; Based on the predicted value and the second loss function, a parameter update operation is performed on the second decoder to output the post-prediction information optimized by variational inference.

7. A multi-task prediction device for medical lesion segmentation and prognosis based on causal perception, characterized in that, include: Data acquisition module: used to acquire medical imaging data and clinical text information; Feature extraction module: used to input the medical image data into the image encoder to generate first image features; The clinical text information is input into the text encoder to generate the first text feature; Lesion segmentation module: used to input the first image features and the first text features into the structural causal model, and to purify the first image features through the first text features to generate the second image features; The second image feature is input into the image decoder to generate lesion segmentation results; The prognostic prediction module is used to input the first image features and the first text features into the dynamic cross-modal fusion module to generate a joint representation vector; perform a decoupling operation on the joint representation vector to generate prognostic correlation factors and confounding interference factors; construct a variational inference model based on the prognostic correlation factors, and generate prognostic prediction information based on the variational inference model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes computer program code, which, when run on a computer, enables the computer to implement the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception as described in any one of claims 1 to 6.

10. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the multi-task prediction method for medical lesion segmentation and prognosis based on causal perception as described in any one of claims 1 to 6.

Images