Method for predicting treatment response of colorectal cancer hepatic metastasis by using Dual-input ViT model

Abstract

The invention provides a method for applying a Dual-input ViT model to colorectal cancer hepatic metastatic focus treatment response prediction, and relates to the field of image processing. The method comprises the following steps: S1) constructing a structured image tag data set, sketching a liver metastasis focus ROI on a double-time-point enhanced CT image of a colorectal cancer liver metastasis patient, and carrying out data processing on the double-time-point enhanced CT image and the liver metastasis focus ROI; step S2), constructing a Dual-input ViT model based on a DDFM (Double Data Frequency Modulation); s3) constructing a joint loss function to carry out constraint optimization on the Dual-input ViT model based on the DDFM, carrying out supervised training based on a curative effect response label, and then verifying the Dual-input ViT model through an ablation experiment; and S4) reasoning by using the trained dual-input ViT model based on the DDFM, and predicting the treatment effectiveness of the colorectal cancer liver metastases. According to the application, the change forms of the lesions before and after the treatment can be dynamically captured, and the sensitivity of curative effect prediction is improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method for predicting the treatment response of colorectal cancer liver metastases using a Dual-input ViT model. Background Technology

[0002] After systemic treatment for colorectal liver metastases (CRLM), efficacy assessment is usually performed using imaging diagnostics to classify the response as complete remission (CR), partial remission (PR), stable disease (SD), and progressive disease (PD). CR and PR indicate that the treatment is effective, while SD and PD indicate that the treatment is ineffective or that drug resistance has occurred.

[0003] Currently, some studies have attempted to use artificial intelligence methods to model treatment responses by classifying or risk-stratifying them using single-time-point radiomics features or enhanced CT image data. These methods have improved the objectivity of efficacy assessment to some extent. However, the essence of tumor response to treatment is a dynamic process, reflected in the multidimensional changes in lesion morphology, boundaries, and density at the imaging level. Previous methods have ignored this dynamic process, relying solely on the patient's baseline features at a single time point and failing to consider the differences between enhanced CT images before and after treatment. This can easily lead to insufficient understanding of the boundary information between "stable" and "partial remission" of lesions, resulting in low generalization ability of the model to predict individualized efficacy outcomes.

[0004] In summary, considering that the treatment response in CRLM patients is inherently a dynamic process, and that traditional methods only use single-time-point information for modeling, failing to capture subtle changes in lesions before and after treatment on enhanced CT scans, the models' ability to comprehensively understand and predict treatment efficacy is limited. Therefore, there is an urgent need for a method capable of predicting treatment response, providing a reliable approach for preoperative treatment assessment and individualized decision-making. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a method for predicting the treatment response of colorectal cancer liver metastases using a dual-input ViT model, which can be used to predict the treatment response and provide a reliable method for preoperative treatment assessment and individualized decision-making.

[0006] To achieve the above and other related objectives, the first aspect of this application provides a Dual-input ViT model for predicting the treatment response of colorectal cancer liver metastases, comprising: Step S1) Construct a structured image label dataset, delineate the ROI of liver metastases on dual-time-contrast CT images of patients with colorectal cancer liver metastases, and perform data processing on dual-time-contrast CT images and liver metastases ROI. Step S2) Construct a Dual-input ViT model based on DDFM. The Dual-input ViT model adopts a dual-time-point input ViT network architecture. The two ViT networks extract differential features from the feature maps of the baseline enhanced CT and the enhanced CT after the first treatment, respectively. A DDFM is formed between the Transformer Encoders in the two ViT networks. The DDFM is used to assign contribution values ​​to the feature maps at different levels and dynamically capture key treatment response changes under hierarchical differential semantics. The outputs of the two ViT networks are passed through a fully connected layer and used as the prediction result of the Dual-input ViT model. Step S3) Construct a joint loss function to constrain and optimize the DDFM-based Dual-input ViT model, perform supervised training based on efficacy response labels, and then validate the Dual-input ViT model through ablation experiments. Step S4) Use the trained DDFM-based Dual-input ViT model to perform inference and predict the treatment effectiveness of colorectal cancer liver metastases.

[0007] In some embodiments of the first aspect of this application, step S1) further includes: Step S1a) Construct a lesion consistency checker. The lesion consistency checker determines whether the lesion has been eliminated based on dual-time-point enhanced CT images. If the lesion has not been eliminated, data processing is performed on the dual-time-point enhanced CT images and the ROI of liver metastases. Step S1b) In the grayscale remapping stage, the window width and window level of the dual-time point CT images are set, the grayscale range of the dual-time point enhanced CT images is scaled, and then the grayscale range is normalized. In step S1c), during the voxel resampling stage, the target voxel size is set, and the dual-time point CT images and ROI labels are standardized to a uniform spatial scale.

[0008] Step S1d) Lesion area standardization extraction stage: Extract the minimum three-dimensional volume region of the lesion area in the dual-time-point enhanced CT image and reconstruct it.

[0009] In some embodiments of the first aspect of this application, step S2) further includes: In step S2a), the feature maps of different Transformer Encoder layers in the ViT network are fused using a difference weighting method to capture the changes in the fine structure, density, and morphology of metastatic lesions in baseline enhanced CT and enhanced CT after the first treatment at different semantic levels, and to capture the differences between the microstructure and pixel fine granularity of the tumor region through a weighting strategy. In step S2b), the semantic information between different levels in DDFM is fused by concatenation and compression. Concatenation preserves the feature information between different levels, while compression uses non-linear compression to enhance the coupling relationship between layer features and obtain fused response features of the same dimension, which are aligned with the output features of the Dual-input ViT model.

[0010] In some embodiments of the first aspect of this application, step S2a) further includes: Feature maps from different Transformer Encoder layers are fused using a weighted interpolation method, as shown in Expression 1:

[0011] in, The characteristics after fusion For features at different levels, For baseline contrast-enhanced CT in Features at different levels Enhanced CT scan after initial treatment Features at different levels for The feature differences that can be automatically updated through backpropagation within the hierarchy, i.e. The weight of the hierarchy.

[0012] In some embodiments of the first aspect of this application, step S2b) further includes: fusing semantic information between different levels through a method of splicing and compression, as shown in Expression 2:

[0013] in, The resulting difference features are those that have been integrated. These are low-level weighted differential semantic features. These are intermediate-weighted differential semantic features. For advanced weighted differential semantic features, for Real vectors, For the concatenation operation of low-level weighted differential semantic features, mid-level weighted differential semantic features, and high-level weighted differential semantic features in the dimensional direction, MLP uses a multi-layer perceptual layer for compression.

[0014] In some embodiments of the first aspect of this application, the compression operation further includes: Will Features are obtained through matrix multiplication. from Dimension mapping to Dimension and output ; Using nonlinear activation layers The processing is as shown in expression three: ; Using CNN from Dimension mapping to Dimension and output ; in, for Features in dimensionality for Features in dimension for Features in dimensionality , GELU is a nonlinear activation layer, and erf is the error function used to describe... Cumulative probability under a Gaussian distribution.

[0015] In some embodiments of the first aspect of this application, in step S3), a joint loss function is used to measure the difference between the predicted response and the actual response of the Dual-input ViT model, and the joint loss function is specifically shown in Expression 4: ; in, For the joint loss function, To predict the cross-entropy loss for the treatment response, For the two-time-point weighted difference balance loss, Weighted regularization loss for hierarchical features; The treatment response prediction cross-entropy loss Supervised learning for the treatment response of liver metastases is used, as shown in Expression 5: ; in, , To truly respond to treatment labels, , The probability of treatment response predicted by the Dual-input ViT model is as follows: the closer to 1, the more likely the Dual-input ViT model predicts that the treatment is effective for the patient; conversely, the closer to 0, the more likely the Dual-input ViT model predicts that the treatment is ineffective or that the patient exhibits drug resistance. The dual-time-point input weighted difference balance loss This is a weighted balance term used to dynamically adjust the importance of the two time-point inputs, as shown in expression six: ; in, This is the paradigm of the L2 loss function. Indicates hierarchy, For the first Weighting factors of enhanced CT feature maps after the first treatment in the hierarchy. For the first The weighting factor of the baseline enhanced CT feature map in the slice, if The larger the value, the more the Dual-input ViT model focuses on the impact of post-treatment contrast-enhanced CT in predicting the patient's treatment response; conversely, if... The larger the value, the more the Dual-input ViT model pays attention to the impact of baseline enhanced CT in predicting the patient's treatment response; The hierarchical feature weighted regularization loss This is used to control the magnitude of semantic feature importance between different levels, as shown in Expression 7: ; in, This is the paradigm of the L2 loss function. For the first Trainable dynamic weighting factors in the hierarchy.

[0016] To achieve the above and other related objectives, a second aspect of this application provides a system for predicting the treatment response of colorectal cancer liver metastases using a DDFM-based Dual-input ViT model, comprising: The dataset module is used to construct a structured image label dataset, delineate the ROI of liver metastases on dual-time-contrast CT images of patients with colorectal cancer liver metastases, and perform data processing on dual-time-contrast CT images and liver metastases ROI. The model training module is used to construct a Dual-input ViT model based on DDFM. The Dual-input ViT model adopts a dual-time-point input ViT network architecture. The two ViT networks extract differential features from the feature maps of the baseline enhanced CT and the enhanced CT after the first treatment, respectively. A DDFM is formed between the Transformer Encoders in the two ViT networks to assign contribution values ​​to the feature maps at different levels and dynamically capture key treatment response changes under hierarchical differential semantics. The outputs of the two ViT networks are passed through a fully connected layer and used as the prediction result of the Dual-input ViT model. The model supervision module is used to construct a joint loss function to constrain and optimize the DDFM-based Dual-input ViT model, perform supervised training based on efficacy response labels, and then validate the Dual-input ViT model through ablation experiments. The model inference module is used to perform inference using a trained DDFM-based Dual-input ViT model to predict the treatment effectiveness of colorectal cancer liver metastases.

[0017] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the Dual-input ViT model described in the first aspect of this application for predicting the treatment response of colorectal cancer liver metastases.

[0018] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code that, when executed on a computer, causes the computer to implement the Dual-input ViT model described in the first aspect of this application for predicting the treatment response of colorectal cancer liver metastases.

[0019] To achieve the above and other related objectives, a fifth aspect of this application provides a computer device / equipment / system / electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the Dual-input ViT model described in the first aspect of this application for predicting the treatment response of colorectal cancer liver metastases.

[0020] As described above, the method for predicting the treatment response of colorectal cancer liver metastases using the Dual-input ViT model provided in this application has the following beneficial effects: 1) This application uses a Dual-input ViT model based on DDFM to process dual-time-point enhanced CT images, dynamically capture the morphological changes of lesions before and after treatment, improve the sensitivity of efficacy prediction, overcome the inability of traditional single-time-point deep learning and radiomics methods to dynamically perceive, and better reflect the continuity of lesion changes in actual clinical practice.

[0021] 2) This application introduces a dual-time-point multi-level semantic fusion structure, which effectively integrates the expression of local information of metastatic lesions with high-level semantic features, and improves the model's adaptability to the heterogeneity of complex metastatic lesion morphology.

[0022] 3) This application introduces the DDFM mechanism to dynamically weight the differences in characteristics at different time points and levels, dynamically adjust the focus of the model in liver metastases of different patients, and enhance the robustness of individualized efficacy prediction.

[0023] 4) This application introduces a dual regularization loss term to control the model’s over-reliance on a single time point or a single level and optimize feature utilization. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the process of using the Dual-input ViT model for predicting the treatment response of colorectal cancer liver metastases in one embodiment of the present invention.

[0025] Figure 2 This is a flowchart illustrating the sub-steps of step S1 in this invention.

[0026] Figure 3 This is a flowchart illustrating the sub-step of step S2 in this invention.

[0027] Figure 4 This is a diagram illustrating the architecture of the Dual-input ViT model used for predicting the treatment response of colorectal cancer liver metastases in an embodiment of the present invention.

[0028] Figure 5 This is an architecture diagram of DDFM, Transformer Encoder, and MLP Block in this invention.

[0029] Figure 6 This is the verification data used in this invention.

[0030] Figure 7 This is a system diagram of a Dual-input ViT model used for predicting the treatment response of colorectal cancer liver metastases in one embodiment of the present invention. Detailed Implementation

[0031] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0032] Before providing a further detailed description of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows: <1> DDFM (Dynamic Difference Feature Module): This module adaptively fuses multi-scale local features by introducing a dynamic mechanism that incorporates global information. Specifically, the dynamic mechanism includes: Dynamic channel selection: Assigning corresponding weights to channels based on global features to enhance the response of key feature channels, suppress low-quality, redundant, and irrelevant features, and improve feature utilization. Dynamic spatial selection: Combining global spatial feature information, weighting the fused feature map to highlight key spatial regions and increase the model's focus on the target region.

[0033] To address the aforementioned technical problems, this application constructs a method for predicting the treatment response of colorectal cancer liver metastases using a Dual-input ViT model. It should be understood that the method provided in this application, besides being applied to CT images of colorectal cancer liver metastases, can also be applied to CT images of other symptoms. Furthermore, it can be applied to other types of medical images, such as X-ray images, MRI (magnetic resonance imaging) images, SPECT (single-photon emission computed tomography) and PET (positron emission tomography) images, ultrasound images, pathological slide images, and endoscopic images. In addition to its application in the medical field, it can also be used in industrial inspection, video surveillance, military defense, aerospace, autonomous driving, and other fields; this application does not limit its application in these areas.

[0034] Figure 1 and Figure 4 This document illustrates a flowchart and framework diagram of a method for predicting the treatment response of colorectal cancer liver metastases using a Dual-input ViT model, as described in an embodiment of this application. The method in this embodiment mainly includes the following steps: Step S1: Construct a structured image label dataset, delineate the ROI of liver metastases on dual-time-point enhanced CT images of colorectal cancer liver metastases, and perform data processing on the dual-time-point enhanced CT images and the ROI of liver metastases.

[0035] For illustration, when including dual-timepoint enhanced CT images (baseline enhanced CT and enhanced CT after first treatment) of patients with pathologically and imaging-confirmed CRLM (colorectal cancer liver metastases) as the training set, all patients had received systemic chemotherapy, radiotherapy, neoadjuvant therapy or combination therapy, and had completed standardized enhanced CT examinations before and after treatment.

[0036] The inclusion criteria for the training set include: (1) patients have undergone enhanced CT scans before treatment; (2) the enhanced CT images of the first follow-up examination after treatment are complete, and the scan time should be between 4 and 12 weeks after the end of treatment; (3) the enhanced CT images are free of serious artifacts and blurring, and there are no problems of low contrast or low resolution (such as slice thickness > 5 mm), and the images contain the complete liver area; (4) patients can clearly obtain treatment response assessment results and can be classified as effective (CR or PR), ineffective or resistant (SD or PD) according to RECIST 1.1 criteria; (5) liver metastases are clearly visible in CT images.

[0037] As a supplementary note, all images are saved in the Medical Digital Imaging and Communications (.Dicom) format. In addition to dual-timepoint enhanced CT images, portal venous phase enhanced scan images before systemic treatment and portal venous phase scan images obtained from the first follow-up examination after the initial systemic treatment can also be collected.

[0038] As an explanation, in the process of ROI annotation, two radiologists with more than 5 years of experience used the open-source software ITK-SNAP to delineate the three-dimensional ROI of the lesion area. The lesion delineation was done manually throughout the process, without any semi-automatic or fully automatic delineation. The delineation standards are as follows: (1) The liver window setting during ROI annotation was set by the doctor's personal habits and no uniform fixed value was set. The purpose was to improve the recognition of the lesion boundary; (2) For clearly visible multiple / single metastatic lesions in the liver, the boundary of the lesion area was delineated layer by layer to ensure that the ROI area covered the entire lesion volume; (3) The delineated lesions were ensured to correspond one-to-one with the baseline and after the first treatment; (4) If the Dice coefficient of the ROI after annotation by two doctors was <0.85, then a third senior radiologist would delineate it; (5) The ROI labels were all saved as parsable nii.gz compressed files.

[0039] Step S1a: Construct a lesion consistency verifier. The lesion consistency verifier determines whether the lesion has been eliminated based on dual-time-point enhanced CT images. If the lesion has not been eliminated, it enters the Dual-input ViT model for prediction.

[0040] Understandably, when the lesion consistency checker makes its judgment, if only one lesion exists and the other does not, it means that the lesion has been eliminated after the first treatment, and there is no need to enter the Dual-input ViT model for prediction. If both lesion consistency checker and lesion consistency checker exist, then the lesion continues to enter the Dual-input ViT model for prediction.

[0041] Step S1b: Gray-scale remapping stage. The window width and window level of the dual-timepoint CT images are set, the gray-scale range of the dual-timepoint enhanced CT images is scaled, and then normalized. It is understandable that the gray-scale remapping stage can improve the contrast between liver metastases and liver parenchyma. In CT images, the window width determines the gray-scale range displayed in the image, while the window level determines the center position of this range. By adjusting the window width and window level, details of different tissue structures can be highlighted.

[0042] Specifically, the window width for the dual-timepoint CT effect is set to 200 HU, and the window level is set to 40 HU. Secondly, to eliminate outlier effects, a linear truncation method is used to scale the grayscale values ​​to [-60, 140]. Finally, pixels within this range are normalized to [0, 1], with those above this range set to 1 and those below to 0. The expression for the linear truncation is shown below:

[0043] Where WL represents the window level and WW represents the window width.

[0044] Step S1c: Voxel resampling stage. The target voxel size is set, and the dual-timepoint CT images and ROI labels are standardized to a uniform spatial scale. It can be understood that the voxel resampling stage can unify the spatial resolution of the patient images at both time points.

[0045] Specifically, the target voxel size is set to 1.0*1.0*1.0 mm. 3 Subsequently, linear interpolation was used to resample the two-time-point CT images. Nearest neighbor interpolation was used for ROI labels to preserve binary label consistency, ensuring spatial structure consistency in subsequent modeling. The expression used for linear interpolation to resample the two-time-point CT images is shown below:

[0046] in, The resampled CT image value is calculated by weighting the 8 nearest neighbor pixels of the original CT image; To ensure that pixel values ​​are integers during the round-down operation; The weights are calculated based on the distance; the closer the pixel is to the sampled pixel, the greater the weight. i, j, k are the pixel values ​​of the original pixel, and m, n, o are the indices of the eight possible adjacent voxel positions, controlling the X, Y, and Z directions respectively.

[0047] When using the nearest neighbor interpolation method to preserve the consistency of binary labels for ROI labels, the expression used is as follows:

[0048] in, This is the resampled ROI label file. Let i, j, k be the pixel values ​​in the original image, where i, j, k are the pixel values ​​of the original pixels.

[0049] Step S1d: Lesion region standardization extraction stage. The minimum three-dimensional volume region of the lesion region in the dual-time-point enhanced CT images is extracted and reconstructed. Specifically, based on the standardized lesion region, the minimum three-dimensional volume region (Bounding Box) of the lesion region in the original dual-time-point enhanced CT images is extracted. Subsequently, the Bounding Box region is resized according to the model input size, and the center is aligned to a size of (16, 224, 224, 3). The first dimension is the number of full-volume layers of the lesion, the second and third dimensions are the length and width of the lesion region in each layer, and the fourth dimension is the number of channels, to ensure the uniformity of the sample input shape during modeling.

[0050] Step S2: Construct a Dual-input ViT model based on DDFM. The Dual-input ViT model adopts a dual-time-point input ViT network architecture. The two ViT networks extract differential features from the feature maps of the baseline enhanced CT and the enhanced CT after the first treatment, respectively. A DDFM is formed between the Transformer Encoders in the two ViT networks. The DDFM is used to assign contribution values ​​to the feature maps at different levels and dynamically capture key treatment response changes under the semantic differences at different levels. The outputs of the two ViT networks are passed through a fully connected layer and used as the prediction result of the Dual-input ViT model.

[0051] DDFM stands for Dynamic Difference Feature Module. In a typical ViT network, it includes Patch Embedding, Position Encoding, and Transformer Encoder. A typical Transformer Encoder consists of three core modules: multi-head self-attention mechanism, feedforward neural network, and residual connection.

[0052] In this application, the input to a ViT network is a baseline enhanced CT scan, denoted as: Another input to the ViT network is the enhanced CT scan after the initial treatment, represented as... .

[0053] In the embodiments of this application, the specific implementation process of step S2 can be further divided into: Figure 3 The following steps are shown: Step S2a: In the DDFM, the feature maps of different Transformer Encoder layers in the ViT network are fused using a difference-weighted method to capture the changes in the fine structure, density, and morphology of metastatic lesions in baseline enhanced CT and enhanced CT after the first treatment at different semantic levels, and to capture the differences between the microstructure and pixel fine granularity of the tumor region through a weighted strategy.

[0054] Specifically, both ViT networks in the Dual-input ViT model include 12 Transformer Encoder layers. Low-level semantic features of local structures in lesions are output through layers 1 and 2 of the Transformer Encoder; mid-level semantic features expressing structural combinations in lesions are output through layers 5 and 6 of the Transformer Encoder; and high-level semantic features of global response patterns in lesions are output through layers 11 and 12 of the Transformer Encoder.

[0055] As an explanation, low-level semantic features focus on and Changes in local structures within the lesion: such as subtle variations in density and margins. Intermediate semantic features focus on... and Changes in the structural combination between metastases and liver parenchyma, such as the boundary enhancement zone between metastases and liver parenchyma, changes in adjacent blood vessels, and morphological homogeneity. High-level semantic features focus on changes in the global response model, such as overall changes in metastases, alterations in enhancement patterns, and changes in lesion boundary morphology before and after treatment.

[0056] Feature maps from different Transformer Encoder layers are fused using a difference-weighted method to capture the subtle structural, density, and morphological changes of metastatic lesions between baseline images and post-treatment images at different semantic levels. A weighted strategy is also used to capture the differences between the microstructure and pixel-level fine-grained features of the tumor region, as shown in Expression 1. ; in, The characteristics after fusion For features at different levels, For baseline contrast-enhanced CT in Features at different levels Enhanced CT scan after initial treatment Features at different levels for The feature differences that can be automatically updated through backpropagation within the hierarchy, i.e. Hierarchical weights. Because the contribution of different feature channels and dimensions to predicting efficacy varies, a weighting mechanism is introduced. As weights.

[0057] Step S2b: In the DDFM, semantic information between different levels is fused through concatenation and compression. Concatenation preserves the feature information between different levels, while compression uses non-linear compression to enhance the coupling relationship between layer features, obtain fused response features of the same dimension, and align with the output features of the Dual-input ViT model.

[0058] Specifically, when semantic information between different levels is fused through concatenation and compression, the expression used is shown in Expression 2: ; in, The resulting difference features are those that have been integrated. These are low-level weighted differential semantic features. These are intermediate-weighted differential semantic features. For advanced weighted differential semantic features, for Real vectors, For the concatenation operation of low-level weighted differential semantic features, mid-level weighted differential semantic features, and high-level weighted differential semantic features in the dimensional direction, MLP uses a multi-layer perceptual layer for compression.

[0059] Furthermore, the compression operation also includes: 1) Features are obtained through matrix multiplication. from Dimension mapping to Dimension and output ; 2) Employing a nonlinear activation layer The processing is as shown in expression three:

[0060] Dropout can also be used to prevent overfitting.

[0061] 3) Using CNN from Dimension mapping to Dimension and output .

[0062] in, for Features in dimensionality for Features in dimension for Features in dimensionality , GELU is a nonlinear activation layer, and erf is the error function used to describe... Cumulative probability under a Gaussian distribution.

[0063] For illustration, the final output vector The resulting difference feature will be fused with the two feature outputs of the final two ViT networks.

[0064] Step S3: Construct a joint loss function to constrain and optimize the DDFM-based Dual-input ViT model, perform supervised training based on efficacy response labels, and then validate the Dual-input ViT model through ablation experiments.

[0065] The joint loss function is used to measure the difference between the predicted response and the actual response of the Dual-input ViT model. The joint loss function is shown in Expression 4: ; in, For the joint loss function, To predict the cross-entropy loss for the treatment response, For the two-time-point weighted difference balance loss, Weighted regularization loss for hierarchical features; The treatment response prediction cross-entropy loss Supervised learning for the treatment response of liver metastases is used, as shown in Expression 5: ; in, , To truly respond to treatment labels, , The probability of treatment response predicted by the Dual-input ViT model is as follows: the closer to 1, the more likely the Dual-input ViT model predicts that the treatment is effective for the patient; conversely, the closer to 0, the more likely the Dual-input ViT model predicts that the treatment is ineffective or that the patient exhibits drug resistance. The dual-time-point input weighted difference balance loss This is a weighted balance term used to dynamically adjust the importance of inputs at two different time points. Specifically, This method constrains the difference between feature weighting factors at two time points, preventing the model from over-relying on feature representations at a single time point during training, thus more effectively capturing the dynamic changes in image features before and after treatment. The dual-time-point input weighted difference balance loss... As shown in expression six: ; in, This is the paradigm of the L2 loss function. Indicates hierarchy, For the first Weighting factors of enhanced CT feature maps after the first treatment in the hierarchy. For the first The weighting factor of the baseline enhanced CT feature map in the slice, if The larger the value, the more the Dual-input ViT model focuses on the impact of post-treatment contrast-enhanced CT in predicting the patient's treatment response; conversely, if... The larger the value, the more the Dual-input ViT model pays attention to the impact of baseline enhanced CT in predicting the patient's treatment response; The hierarchical feature weighted regularization loss This is used to control the magnitude of semantic feature importance across different levels. Specifically, semantic features across different levels are prone to redundancy in representation, causing the prediction model to over-rely on information from a certain level, thus ignoring the dynamic changes of global and local features. By controlling the magnitude of semantic features through weighting factors mapped to different levels, the model's extreme dependence on a particular level during training is suppressed, resulting in a more balanced distribution of contributions among low-level, mid-level, and high-level semantic feature representations. The hierarchical feature weighted regularization loss... As shown in expression seven: ; in, This is the paradigm of the L2 loss function. For the first Trainable dynamic weighting factors in the hierarchy.

[0066] Step S4: Use the trained DDFM-based Dual-input ViT model for inference to predict the treatment effectiveness of colorectal cancer liver metastases.

[0067] Example 1

[0068] The original format of the dual-time-point enhanced CT images included was Dicom's medical image storage standard data format. This was to ensure the consistency of data format in data processing, ROI delineation, feature engineering, and modeling.

[0069] Furthermore, after outlining, the doctor delineates the ROI based on the baseline and the corresponding lesions in the CT images after the first treatment. At the same time, the original CT images are desensitized, retaining only the image information and removing sensitive information such as the patient's name, ID number, and contact information.

[0070] Simultaneous resampling of NIfTI format CT images and ROI tag files was performed using the sitkLinear and Nearest Neighbor Interpolation functions from the SimpleITK scientific database.

[0071] The Dual-input ViT network architecture uses traditional ViT network structures in both branches, loading their pre-trained weights for transfer learning, enabling it to converge quickly on small medical samples and achieve high generalization performance.

[0072] Furthermore, in the ablation experiments, the same dataset partitioning method and training batches were used to train the baseline and post-treatment performance of the single-path traditional ViT network, as well as the baseline and post-treatment performance of the dual-input ViT network. The training epochs were set to 500, the batch size to 1024, the learning rate to 0.002, and the Adam optimizer to be used. During training, the validation loss (val_loss) and the accuracy (val_acc) on the validation set were observed during cross-validation. Early stopping and Dropout methods were implemented during training to prevent overfitting. Specifically, if the val_loss did not decrease and the val_acc did not increase within 10 epochs, training was stopped, and 10% of the neurons were randomly deactivated in Dropout.

[0073] All steps in this application were completed using Python (3.8.20), and the training and ablation experiments were completed using the PyTorch (2.4.1 GPU version) open-source deep learning framework.

[0074] After the lesion consistency checker, it is determined whether the lesion exists in the (baseline enhanced CT) and (enhanced CT after the first treatment). If there is no gold standard drawn by the doctor in the corresponding image, the effective result after treatment of the lesion is directly output. If both exist, the image and ROI region feature engineering steps are entered.

[0075] First, the input size of is set to (16, 224, 224, 3). After entering the Dual-input ViT network, it first passes through the Patch Embedding part of the ViT network, where 3D convolution is used to extract patches and simultaneously perform feature embedding. After passing through patches with a convolution kernel size of (4, 32, 32, 768), the output size of is (1, 4, 7, 7, 768). Then, the Fallten operation flattens it into a feature of size (1, 196, 768). Its function is to divide the input into multiple patches and map them to a high-dimensional space for preprocessing operations in the subsequent Transformer Encoder.

[0076] Secondly, in the Position Encoding section of the Vit network, a learnable Class Token is added to the Patches. This Class Token, with a feature shape of (1, 1, 768), is concatenated with the Patches to output a feature of size (1, 197, 768). Its purpose is to use the Class Token to guide the final classification task. Next, in the Positional Embedding section, a learnable positional encoding is added. This provides the Transformer with effective positional information, allowing the model to focus on changes in the lesion area rather than surrounding tissue. This process does not change the feature size; the input and output remain consistent at (1, 197, 768). Following this, a Dropout regularization is applied, randomly deactivating 10% of the features to improve generalization performance while preventing overfitting.

[0077] The Dynamic Difference Feature Module (DDFM) consists of 12 Transformer Encoders per ViT network, for a total of 24 Transformer Encoders. Each Transformer Encoder encodes the features output by the previous Transformer Encoder or Position Encoding.

[0078] In the Transformer Encoder, the input features are first normalized using Layer Normalization to stabilize the training process between encoding steps. Then, they undergo Multi-Head Self-Attention with 12 heads; this process does not alter the shape of the input features. Next, Layer Normalization normalizes the output features of Multi-Head Self-Attention before finally entering MLPBlock.

[0079] In the MLP Block, the feature first passes through a Linear layer with 3072 neurons. After this Linear layer, the feature shape changes to (768, 3072). Then, the GELU activation function is used to reverse the feature's dimensions, resulting in a feature of size (3072, 768). Next, another Linear layer restores the feature shape, reducing it to (1, 197, 768). After restoration, a Dropout regularization is applied to capture global and local dependencies between different features. Finally, a Layer Normalization process is used for normalization. During this process, after the feature is reversed by the GELU activation function, another Dropout regularization can be applied.

[0080] Furthermore, the features obtained from the MLP Block are processed by an Extract Class Token to reduce the dimensionality of the feature shape, resulting in a two-dimensional feature of (1, 768), which is used to represent the features required for the Classification Head and classification tasks.

[0081] Furthermore, in the Classification Head, the features obtained from the previous module are first processed by MLP Block, and after passing through Linear, a two-dimensional feature of (768, 768) is output. After passing through GELU and another Linear, a two-dimensional feature of (768, 2) is obtained. Finally, it enters a 128 Fully Connected layer, and concatenates the features to obtain a two-dimensional feature of (768, 2). After a Dropout operation, a feature vector of (1, 2) dimension is obtained through Fully Connected. The result of feature extraction is mapped to the final classification result and the classification treatment effect (valid is represented by 1, invalid is represented by 0).

[0082] Specifically, in the DDFM module, after feature subtraction and fusion of low-level, mid-level and high-level semantic features respectively, the features of the three levels are finally weighted and fused to obtain a fused feature map, which is then added to the two branches respectively.

[0083] Furthermore, for low-level semantic features, the difference is performed in layers 1 and 2 of the Transformer Encoder in the DDFM module to highlight the changes in tumor edges, textures, and gray values ​​before and after treatment in the low-level semantic features, in order to capture the complementary information between the two; then weighting is performed so that the model can dynamically learn the relative importance between different features.

[0084] Furthermore, for intermediate semantic features, the process is consistent with that for low-level semantic features. However, this process focuses on the changes in the shape and size of the tumor in the enhanced CT images before and after treatment. This process is carried out in layers 5 and 6 of the Transformer Encoder in the DDFL module, and the intermediate dynamic difference features are generated after fusion.

[0085] Furthermore, for high-level semantic features, the operation method is consistent with that for primary and intermediate features. This part is performed in layers 11 and 12 of the Transformer Encoder in the DDFL module. This process is used to capture changes in high-level semantic features of tumor heterogeneity and blood vessels, and finally generate high-level dynamic differential features.

[0086] Furthermore, the low-level, mid-level, and high-level dynamic differential features are weighted and fused, and then mapped to a shape of (1, 197, 768) through an MLP layer. The fused differential features are then fused with the features passed through the 12th layer Transformer Encoder by feature concatenation, and then the output is fed into the subsequent LayerNormalization operation.

[0087] The ratio of training set, validation set and test set was 72% (82 cases): 8% (8 cases): 20% (23 cases).

[0088] By using the single-input traditional ViT model, the dual-input ViT without DDFM, and the dual-input ViT DDDFM model with DDFM, the impact of the model on the overall performance under different conditions of single-input, dual-input, and dual-input conditions is systematically evaluated to gain a deeper understanding of the design choices in this application and their contribution to the results.

[0089] The ViT model using only data had an AUROC of 0.674, Accuracy of 0.655, Sensitivity of 0.821, Specificity of 0.334, PPV of 0.701, NPV of 0.506, and F1 Score of 0.756. The ViT model using only data had an AUROC of 0.707, Accuracy of 0.750, Sensitivity of 0.867, Specificity of 0.531, PPV of 0.776, NPV of 0.680, and F1 Score of 0.819. Compared to the ViT model using only data, the ViT model showed improved performance, indicating that post-treatment imaging data provided added value.

[0090] The Dual-input ViT model using data further improved the prediction performance, with AUROC of 0.739, Accuracy of 0.721, Sensitivity of 0.810, Specificity of 0.555, PPV of 0.773, NPV of 0.609, and F1 Score of 0.791.

[0091] The Dual-input ViT model, which incorporates a dynamic difference feature module, achieved the best overall performance with an AUROC of 0.816, an Accuracy of 0.799, a Sensitivity of 0.854, a Specificity of 0.695, a PPV of 0.840, an NPV of 0.718, and an F1 Score of 0.847.

[0092] Therefore, the DDFM-based dual-input ViT model improves the predictive ability of treatment effects, outperforming the single-input model. Compared to the dual-input ViT model without DDFM, the AUROC increases by 10.42%. This indicates that the DDFM-based dual-input ViT model can effectively capture feature changes before and after treatment.

[0093] In addition, the DDFM-based Dual-input ViT model improved specificity by 25.23%, PPV by 8.67%, reduced the false positive rate, and improved sensitivity by 5.43%, although it was still 1.50% lower than the single-input ViT model. NPV improved by 17.90%, and F1 Score by 7.08%, reflecting the improved negative prediction ability and balanced performance of the dual-input ViT model with dual time points.

[0094] For details regarding the specific calculations of a Dual-input ViT model for predicting the treatment response of colorectal cancer liver metastases, please refer to the descriptions above of the data collection and ROI annotation, image and ROI region feature engineering, Dual-input ViT network architecture design, dynamic differential feature module construction, and model training and validation in the invention method; these will not be repeated here. All or part of the above-mentioned method for predicting the treatment response of colorectal cancer liver metastases using the Dual-input ViT model can be implemented in software to facilitate the calculation of the corresponding operations in each step.

[0095] The process embodiments described above are merely illustrative. The ROI labeling, resampling, differential feature extraction methods, or dual-input ViT network structure mentioned therein can also be other different structures. The loss function, data partitioning ratio, and validation method during training may also differ from those proposed herein. Parts of these methods can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand the various embodiments and how the proposed methods are implemented using different programming languages ​​and deep learning frameworks. Based on this understanding, the essence of the above technical solutions, or the contribution to the technology, can be presented in a software or Web full-stack manner. This model can be embedded in software or deployed in a Web database, such as MySQL, SQL Server, Oracle, DB2, etc., and includes several instructions that cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0098] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the technical principles of this application, and these improvements and substitutions should also be considered within the scope of protection of this application.

Claims

1. A method for predicting the treatment response of colorectal cancer liver metastases using a dual-input ViT model, characterized in that, It includes the following steps: Step S1) Construct a structured image label dataset, delineate the ROI of liver metastases on dual-time-contrast CT images of patients with colorectal cancer liver metastases, and perform data processing on dual-time-contrast CT images and liver metastases ROI. Step S2) Construct a Dual-input ViT model based on DDFM. The Dual-input ViT model adopts a dual-time-point input ViT network architecture. The two ViT networks extract differential features from the feature maps of the baseline enhanced CT and the enhanced CT after the first treatment, respectively. A DDFM is formed between the Transformer Encoders in the two ViT networks. The DDFM is used to assign contribution values ​​to the feature maps at different levels and dynamically capture key treatment response changes under hierarchical differential semantics. The outputs of the two ViT networks are passed through a fully connected layer and used as the prediction result of the Dual-input ViT model. Step S3) Construct a joint loss function to constrain and optimize the DDFM-based Dual-input ViT model, perform supervised training based on efficacy response labels, and then validate the Dual-input ViT model through ablation experiments. Step S4) Use the trained DDFM-based Dual-input ViT model to perform inference and predict the treatment effectiveness of colorectal cancer liver metastases.

2. The method for predicting the treatment response of colorectal cancer liver metastases using the Dual-input ViT model according to claim 1, characterized in that, The feature is that step 1) further includes: Step S1a) Construct a lesion consistency checker. The lesion consistency checker determines whether the lesion has been eliminated based on dual-time-point enhanced CT images. If the lesion has not been eliminated, data processing is performed on the dual-time-point enhanced CT images and the ROI of liver metastases. Step S1b) In the grayscale remapping stage, the window width and window level of the dual-time point CT images are set, the grayscale range of the dual-time point enhanced CT images is scaled, and then the grayscale range is normalized. Step S1c) Voxel resampling stage: Set the target voxel size and standardize the dual-time point CT images and ROI labels to a uniform spatial scale; Step S1d) Lesion area standardization extraction stage: Extract the minimum three-dimensional volume region of the lesion area in the dual-time-point enhanced CT image and reconstruct it.

3. The method for predicting the treatment response of colorectal cancer liver metastases using the Dual-input ViT model according to claim 1, further comprising step S2): In step S2a), the feature maps of different Transformer Encoder layers in the ViT network are fused using a difference weighting method to capture the changes in the fine structure, density, and morphology of metastatic lesions in baseline enhanced CT and enhanced CT after the first treatment at different semantic levels, and to capture the differences between the microstructure and pixel fine granularity of the tumor region through a weighting strategy. In step S2b), the semantic information between different levels in DDFM is fused by concatenation and compression. Concatenation preserves the feature information between different levels, while compression uses non-linear compression to enhance the coupling relationship between layer features and obtain fused response features of the same dimension, which are aligned with the output features of the Dual-input ViT model.

4. The method for predicting the treatment response of colorectal cancer liver metastases using the Dual-input ViT model according to claim 3, characterized in that, Step S2a) also includes: fusing the feature maps of different Transformer Encoder layers using a difference-weighted method, as shown in Expression 1: ； in, The characteristics after fusion For features at different levels, For baseline contrast-enhanced CT in Features at different levels Enhanced CT scan after initial treatment Features at different levels for The feature differences that can be automatically updated through backpropagation within the hierarchy, i.e. The weight of the hierarchy.

5. The method for predicting the treatment response of colorectal cancer liver metastases using the Dual-input ViT model according to claim 3, characterized in that, Step S2b) also includes: the semantic information between different levels is fused through concatenation and compression, as shown in expression two: ； in, The resulting difference features are those that have been integrated. These are low-level weighted differential semantic features. These are intermediate-weighted differential semantic features. For advanced weighted differential semantic features, for Real vectors, For the concatenation operation of low-level weighted differential semantic features, mid-level weighted differential semantic features, and high-level weighted differential semantic features in the dimensional direction, MLP uses a multi-layer perceptual layer for compression.

6. The method for predicting the treatment response of colorectal cancer liver metastases using the Dual-input ViT model according to claim 5, characterized in that, The compression operation also includes: Will Features are obtained through matrix multiplication. from Dimension mapping to Dimension and output ; Using nonlinear activation layers The processing is as shown in expression three: ； Using CNN from Dimension mapping to Dimension and output ; in, for Features in dimensionality for Features in dimension for Features in dimensionality , GELU is a nonlinear activation layer, and erf is the error function used to describe... Cumulative probability under a Gaussian distribution.

7. The method for predicting the treatment response of colorectal cancer liver metastases using the Dual-input ViT model according to claim 1, characterized in that, In step S3), a joint loss function is used to measure the difference between the predicted response and the actual response of the Dual-input ViT model. The joint loss function is specifically shown in expression four: ； in, For the joint loss function, To predict the cross-entropy loss for the treatment response, For the two-time-point weighted difference balance loss, Weighted regularization loss for hierarchical features; The treatment response prediction cross-entropy loss Supervised learning for the treatment response of liver metastases is used, as shown in Expression 5: ； in, , To truly respond to treatment labels, , The probability of treatment response predicted by the Dual-input ViT model is as follows: the closer to 1, the more likely the Dual-input ViT model predicts that the treatment is effective for the patient; conversely, the closer to 0, the more likely the Dual-input ViT model predicts that the treatment is ineffective or that the patient exhibits drug resistance. The dual-time-point input weighted difference balance loss This is a weighted balance term used to dynamically adjust the importance of the two time-point inputs, as shown in expression six: ； in, This is the paradigm of the L2 loss function. Indicates hierarchy, For the first Weighting factors of enhanced CT feature maps after the first treatment in the hierarchy. For the first The weighting factor of the baseline enhanced CT feature map in the slice, if The larger the value, the more the Dual-input ViT model focuses on the impact of post-treatment contrast-enhanced CT in predicting the patient's treatment response; conversely, if... The larger the value, the more the Dual-input ViT model pays attention to the impact of baseline enhanced CT in predicting the patient's treatment response; The hierarchical feature weighted regularization loss This is used to control the magnitude of semantic feature importance between different levels, as shown in Expression 7: ； in, This is the paradigm of the L2 loss function. For the first Trainable dynamic weighting factors in the hierarchy.

8. A dual-input ViT model for predicting the treatment response of colorectal cancer liver metastases, characterized in that, include: The dataset module is used to construct a structured image label dataset, delineate the ROI of liver metastases on dual-time-contrast CT images of patients with colorectal cancer liver metastases, and perform data processing on dual-time-contrast CT images and liver metastases ROI. The model training module is used to construct a Dual-input ViT model based on DDFM. The Dual-input ViT model adopts a dual-time-point input ViT network architecture. The two ViT networks extract differential features from the feature maps of the baseline enhanced CT and the enhanced CT after the first treatment, respectively. A DDFM is formed between the Transformer Encoders in the two ViT networks to assign contribution values ​​to the feature maps at different levels and dynamically capture key treatment response changes under hierarchical differential semantics. The outputs of the two ViT networks are passed through a fully connected layer and used as the prediction result of the Dual-input ViT model. The model supervision module is used to construct a joint loss function to constrain and optimize the DDFM-based Dual-input ViT model, perform supervised training based on efficacy response labels, and then validate the Dual-input ViT model through ablation experiments. The model inference module is used to perform inference using a trained DDFM-based Dual-input ViT model to predict the treatment effectiveness of colorectal cancer liver metastases.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the Dual-input ViT model as described in any one of claims 1 to 7 for predicting the treatment response of colorectal cancer liver metastases.

10. A computer program product, characterized in that, The computer program product includes computer program code, which, when run on a computer, enables the computer to implement the Dual-input ViT model as described in any one of claims 1 to 7 for predicting the treatment response of colorectal cancer liver metastases.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the Dual-inputViT model-based prediction of treatment response for colorectal cancer liver metastases as described in any one of claims 1 to 7.

Images