Cancer prognosis prediction method and system combining medical large model assistance and multimodal image fusion

Abstract

The present invention discloses a cancer prognosis prediction system and method that combines medical large model assistance and multimodal image fusion. The system uses the medical large model MedSAM for assisted segmentation. Compared with existing manual segmentation, it reduces the workload of doctors in determining the lesion area and optimizes the availability of tumor slice data. On this basis, resampling, minimum envelope rectangle selection, and rectangular expansion are sequentially performed to obtain PET tumor slices and CT tumor slices, and the PET tumor slices and CT tumor slices are fused to obtain fused slices. The slices obtained in this way can ensure the accuracy and reliability of subsequent analysis. The present invention uses a cancer prognosis prediction model based on multimodal image fusion to effectively integrate multimodal slices and then perform cancer prognosis prediction, thereby improving the accuracy of cancer prognosis prediction results. At the same time, the cancer prognosis prediction results can be used to guide doctors in selecting cancer treatment plans.

Description

Technical Field

[0001] The present invention belongs to the technical field of cancer prognosis prediction, and specifically relates to a cancer prognosis prediction method and system combining medical large model assistance and multimodal image fusion. Background Art

[0002] Malignant tumors are a common and diverse class of diseases, accounting for the vast majority of all cancer cases. Malignant tumors are highly heterogeneous, with significant differences between different patients in terms of tumor biology, genetic mutations, and response to treatment. This results in poor treatment outcomes for malignant tumors, and accurate prediction of treatment outcomes for malignant tumors is also extremely complex and difficult. For patients with advanced malignant tumors, even after receiving chemotherapy and other treatments, their median survival is still short, and existing treatments have limited effectiveness in improving the prognosis of advanced patients. Although there are currently a variety of treatment options available, such as targeted therapy, immunotherapy, and antibody-drug conjugate therapy, how to correctly select a treatment strategy remains an urgent issue.

[0003] In recent years, medical imaging technology, particularly PET / CT imaging, has played a vital role in the diagnosis, staging, and prognosis of malignant tumors. PET (positron emission tomography) provides information on tumor metabolic activity, while CT (computed tomography) provides precise anatomical structure. Simply overlaying these two images fails to fully utilize their respective strengths. How to effectively fuse the information from PET and CT images to improve the accuracy and reliability of malignant tumor prognosis prediction and to inform treatment selection is a key research question.

[0004] Reference 1 (Samuel H. Hawkins, John N. Korecki. Predicting outcomes of non-small cell lung cancer using CT image features. IEEE Access, 2, 1418-1426, 2014, doi: 10.1109 / ACCESS.2014.2373335) discloses a method for predicting the prognosis of malignant tumors based on deep learning. Specifically, malignant tumor CT image data is obtained by manual segmentation combined with a region growing algorithm, and treatment effect prediction is achieved with the help of deep learning. This method relies solely on CT image data and does not deeply fuse PET / CT image information, making it impossible to accurately predict the prognosis. Reference 2 (Wei Mu, Matthew B. Schabath. Non-invasive decision support for NSCLC treatment using PET / CT radiomics, Nature communications, 11, 5228, 2020, doi: 10.1038 / s41467-020-19116-x) discloses a method for predicting the prognosis of malignant tumors based on a deep learning SResCNN model. Specifically, tumor slices are obtained by manual segmentation, and three-channel super-images and α-fused PCT / CT images are constructed. The SResCNN model is input, and the EGFR mutation status (positive or negative), that is, the deep learning score (EGFR-DLS), is used as the classification result to represent the probability of EGFR mutation positivity. By averaging the EGFR-DLS of the valid slices, the patient's EGFR mutation positivity probability is obtained for malignant tumor prognosis prediction. Although this method performs deep fusion of PET / CT based on a deep learning model, the acquisition of tumor slice images relies on doctors to manually segment the lesion area, which is a huge workload. Summary of the Invention

[0005] The purpose of the present invention is to provide a cancer prognosis prediction method and system that combines medical large-scale model assistance and multimodal image fusion to solve the technical problems of the huge workload of manually segmenting the lesion area and the lack of deep fusion of PET / CT, which makes it impossible to accurately predict the prognosis effect.

[0006] To achieve the above-mentioned purpose of the invention, the embodiment provides a cancer prognosis prediction method and system that combines medical big model assistance and multimodal image fusion, including an auxiliary processing unit based on the medical big model MedSAM and a cancer prognosis prediction model based on multimodal image fusion.

[0007] In one embodiment, the auxiliary processing unit based on the medical large model MedSAM is used to perform auxiliary segmentation, resampling, minimum envelope rectangle selection, and rectangle expansion based on the medical large model MedSAM on the initial PET slice and the initial CT slice respectively to obtain PET tumor slices and CT tumor slices, and to fuse the PET tumor slices and the CT tumor slices to obtain fused slices.

[0008] In one embodiment, the cancer prognosis prediction model based on multimodal image fusion is used to extract features from three modalities: PET tumor slices, CT tumor slices, and fusion slices; perform feature fusion on the extracted features to obtain fusion features; and predict the fusion features to obtain cancer prognosis prediction results.

[0009] In one embodiment, the auxiliary segmentation based on the medical large model MedSAM includes: the medical large model MedSAM assists in outlining a mask of the tumor boundary within the area initially marked by the doctor in the input slice, thereby achieving auxiliary segmentation and providing reliable basic data for subsequent processing;

[0010] The resampling method includes: using cubic interpolation resampling to resample the area initially marked by the physician with the tumor boundary mask after auxiliary segmentation, unifying the pixel size to 1 cubic millimeter. Resampling is used to standardize the resolution of different samples, eliminating differences caused by different equipment and scanning parameters, thereby ensuring data consistency and comparability;

[0011] The selection based on the minimum envelope rectangle includes: selecting the minimum rectangle that can completely envelop the tumor boundary mask according to the resampled tumor boundary mask, so as to remove unnecessary information and focus on the tumor area;

[0012] The rectangular expansion method includes expanding the width and height of the mask after selecting the minimum enveloping rectangle by 20 mm, and then adjusting the size of the expanded mask to 64×64. This method is used to include tissue information of the tumor surrounding area while retaining the tumor area, which helps the cancer prognosis prediction model more comprehensively learn the characteristics of the tissue information of the tumor and the surrounding area, thereby improving the accuracy of the cancer prognosis prediction results.

[0013] The fusion-based fusing of the PET tumor slice and the CT tumor slice to obtain the fused slice includes: fusing in an alpha fusion manner:

[0014] fusion_img=ct×α+pet×(1-α)

[0015] Among them, α represents the fusion factor, ct represents the CT tumor slice, pet represents the PET tumor slice, and fusion_img represents the fusion slice;

[0016] Through α fusion, the two modal information of PET / CT tumor slices are initially fused together, utilizing the advantages of each modality to provide more comprehensive and detailed fused slice information, thereby enhancing the model's ability to recognize tumor slice features.

[0017] In one embodiment, the cancer prognosis prediction model includes: an initial feature extraction module, a primary feature fusion module, a secondary feature extraction module, a secondary feature fusion module, and a prediction module;

[0018] In one embodiment, the initial feature extraction module is used to perform multi-scale feature extraction on PET tumor slices, CT tumor slices, and fusion slices to obtain initial features for each type of slice;

[0019] In one embodiment, the first-level feature fusion module is used to fuse the initial features of the three types of slices using a compressed multi-head attention mechanism, specifically including: performing a compressed self-attention mechanism on the initial features of the PET tumor slices and the CT tumor slices to obtain the first-level self-attention features of the two types of tumor slices, and then performing a compressed cross-attention mechanism on the first-level self-attention features of the two types of tumor slices and the initial features of the fused slice to obtain the first-level cross-attention features of the two types of tumor slices, and finally splicing the first-level cross-attention features of the two types of tumor slices with the initial features of the fused slice and then performing convolution compression with channel reduction to obtain the first-level fusion features of the fused slice.

[0020] The secondary feature extraction module is used to extract the self-attention features of the two types of tumor slices and the primary fusion features of the fusion slices to obtain the secondary features of each type of slice;

[0021] The secondary feature fusion module is used to use a compressed multi-head attention mechanism to fuse the secondary features of the three types of slices, specifically including: performing a compressed self-attention mechanism on the secondary features of the PET tumor slices and the CT tumor slices to obtain the secondary self-attention features of the two types of tumor slices respectively, and then performing a compressed cross-attention mechanism on the secondary self-attention features of the two types of tumor slices and the secondary features of the fused slices to obtain the secondary cross-attention features of the two types of tumor slices respectively, and finally splicing the secondary cross-attention features of the two types of tumor slices with the secondary features of the fused slice and then performing channel-reduced convolution compression to obtain the secondary fusion features of the fused slice.

[0022] The prediction module performs cancer prognosis prediction based on the secondary fusion features to obtain a cancer prognosis prediction result.

[0023] In one embodiment, in the compressed self-attention mechanism, the input features of the tumor slice are used as query features and key features respectively, and the spatial or channel dimensions of the query features are compressed before the self-attention mechanism calculation is performed.

[0024] In one embodiment, in the compressed cross-attention mechanism, the input self-attention features of the tumor slice are used as query features, and the input features of the fused slice are used as key features. The cross-attention mechanism calculation is performed after the spatial or channel dimension of the query feature is compressed.

[0025] In one embodiment, in the compressed self-attention mechanism and the compressed cross-attention mechanism, the space of the query feature is compressed through the average pooling layer, and the number of channels of the query feature is compressed through the convolution layer.

[0026] In one embodiment, the initial feature extraction module and the secondary feature extraction module both use Resnet50 for feature extraction.

[0027] In one embodiment, an auxiliary processing unit based on the medical large model MedSAM is used to sequentially perform auxiliary segmentation, resampling, minimum envelope rectangle selection, and rectangle expansion based on the medical large model MedSAM on the initial PET slice and the initial CT slice to obtain a PET tumor slice and a CT tumor slice, and the PET tumor slice and the CT tumor slice are fused to obtain a fused slice.

[0028] A cancer prognosis prediction model based on multimodal image fusion is used to extract features from the three modalities of PET tumor slices, CT tumor slices, and fusion slices. The extracted features are fused to obtain fusion features, and the fusion features are predicted to obtain cancer prognosis prediction results.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention uses the medical large model MedSAM for assisted segmentation. Compared with existing manual segmentation, it reduces the workload of doctors in determining the lesion area and optimizes the availability of tumor slice data. On this basis, resampling, minimum envelope rectangle selection, and rectangular expansion are sequentially performed to obtain PET tumor slices and CT tumor slices, and the PET tumor slices and CT tumor slices are fused to obtain fused slices. The slices obtained in this way can ensure the accuracy and reliability of subsequent analysis; the present invention uses a cancer prognosis prediction model based on multimodal image fusion to effectively integrate multimodal slices and then perform cancer prognosis prediction, thereby improving the accuracy of cancer prognosis prediction results. At the same time, the cancer prognosis prediction results can be used to guide doctors in selecting cancer treatment plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0032] Figure 1 This is a schematic diagram of the structure of the cancer prognosis prediction system;

[0033] Figure 2 It is a process flow chart of the auxiliary processing unit of MedSAM;

[0034] Figure 3 This is a schematic diagram of the structure of the cancer prognosis prediction model PCFNet;

[0035] Figure 4 This is a schematic diagram of the principle of SMHA;

[0036] Figure 5 is a slice image of a malignant tumor; Figure 5 A in the figure is a CT image; Figure 5 B in the figure is a PET image; Figure 5 The red area in the figure is the segmented and annotated image processed by its corresponding MedSAM auxiliary processing unit;

[0037] Figure 6 It is a flow chart of the cancer prognosis prediction method. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0039] To achieve accurate prediction of cancer prognosis, the embodiment provides a cancer prognosis prediction method and system that combines an auxiliary processing unit of the medical large model MedSAM and multimodal image fusion. The system can be used to determine the cancer lesion area, fuse the multimodal information of tumor slices, and predict the cancer prognosis of patients. Figure 1 As shown, the cancer prognosis prediction system provided in the embodiment includes: an auxiliary processing unit based on the medical big model MedSAM, and a cancer prognosis prediction model based on multimodal image fusion.

[0040] In the embodiment, the auxiliary processing unit based on the medical large model MedSAM is used to solve the problem of excessive workload of doctors in manually segmenting the lesion area, ensure the consistency and quality of input data, and reduce the impact of different equipment and reconstruction parameters on model performance. Figure 2As shown, the auxiliary processing unit of MedSAM provided in the embodiment mainly includes five steps: MedSAM auxiliary segmentation, resampling, selecting the minimum envelope rectangle, rectangle expansion, and fusion.

[0041] In the embodiment, MedSAM-assisted segmentation is used to accurately segment the imaging slices (initial PET slices and initial CT slices), providing reliable basic data for subsequent processing. Optionally, MedSAM is used to assist in outlining the tumor boundary mask within the doctor's initial marked area in the input slice to achieve assisted segmentation.

[0042] In this embodiment, resampling is used to standardize the resolution of different samples, eliminating variations caused by different equipment and scanning parameters, thereby ensuring data consistency and comparability. Optionally, a cubic interpolation resampling method is used to resample the doctor's initial marked area with the tumor boundary mask after auxiliary segmentation to a uniform pixel size. For example, the pixels with the tumor boundary mask after resampling are unified to 1 cubic millimeter.

[0043] In this embodiment, the minimum enveloping rectangle is selected to remove unnecessary information, focus on the tumor region, and improve data processing efficiency and model training effectiveness. Optionally, based on the tumor boundary mask after cubic interpolation resampling, a minimum enveloping rectangle that can completely envelop the tumor boundary mask is selected.

[0044] In the embodiment, rectangular expansion is used to include a certain range of surrounding tissue information while retaining the tumor area, which helps the model to more comprehensively learn the characteristics of the tumor and its surrounding environment, thereby improving the accuracy of cancer prognosis prediction results; optionally, the width and height of the selected minimum enveloping rectangle are expanded by 20 mm, and then the size of the expanded minimum enveloping rectangle is adjusted to 64×64.

[0045] In the embodiment, fusion is used to fuse image information, provide more comprehensive and detailed image information, and enhance the model's ability to identify tumor features. Optionally, fusion of PET tumor slices and CT tumor slices to obtain fused slices can be performed using alpha fusion:

[0046] fusion_img=ct×α+pet×(1-α)(1)

[0047] Among them, α represents the fusion factor, ct represents the CT tumor slice, pet represents the PET tumor slice, and fusion_img represents the fusion slice;

[0048] Through α fusion, the two modal information of PET / CT tumor slices are initially fused together, utilizing the advantages of each modality to provide more comprehensive and detailed fused slice information, thereby enhancing the model's ability to recognize tumor slice features.

[0049] In the embodiment, a cancer prognosis prediction model based on multimodal image fusion is used to capture metabolic information in PET images and anatomical structure information in CT images; the image information of the two is fused to improve the accuracy of cancer prognosis prediction results. Figure 3 As shown, the PCFNet provided by the embodiment includes: an initial feature extraction module, a primary feature fusion module, a secondary feature extraction module, a secondary feature fusion module, and a prediction module.

[0050] In this embodiment, the initial feature extraction module is used to perform multi-scale feature extraction on PET tumor slices, CT tumor slices, and fusion slices to obtain initial features of each type of slice, which are represented by f pet 、f ct , and f fuse . Optionally, the initial feature extraction module can use Resnet50 for feature extraction, specifically using three independent Resnet50s to perform feature extraction on PET tumor slices, CT tumor slices, and fusion slices respectively. Resnet50 is a deep residual network that solves the gradient vanishing and representation bottleneck problems in deep neural networks by introducing residual connections, which allows the network to be trained at a deeper level, thereby extracting richer and more complex features. All three Resnet50s use weights pre-trained on the ImageNet dataset. ImageNet is a large-scale visual database that contains 1,000 categories and more than 14 million images. Using pre-trained weights can help the model achieve better performance in the initial stage, and can also speed up the training process of the model.

[0051] Resnet50 is used to extract the morphological features of the tumor from all input slices. These features include information such as the size, shape, and edge of the tumor. The deep network structure of Resnet50 enables it to learn the complex patterns of tumors and capture subtle changes in tumor growth. Specifically, Resnet50 has a total of 5 scale transformations, and the number of channels for the 5 transformations is 64, 256, 512, 1024, and 2048 respectively. In order to deeply fuse the features of PET tumor slices, CT tumor slices, and fused slices, and to fuse features of different scales, in the initial feature extraction module, the first 4 scale transformations are first performed to obtain the initial features of the PET tumor slices, CT tumor slices, and fused slices. The initial features of the three slices are input into the first-level feature fusion module for fusion.

[0052] In an embodiment, a primary feature fusion module is used to fuse the initial features of PET tumor slices, CT tumor slices, and fused slices to fully utilize the complementary information between them. In the field of medical imaging, PET tumor slices and CT tumor slices are two commonly used medical imaging modalities, each providing different information. PET tumor slices can provide information about the metabolic activity of tissues or lesions, while CT tumor slices provide detailed information about anatomical structures. At the same time, fused slices combine information from PET and CT. Therefore, by comprehensively utilizing the information from these three types of tumor slices, the cancer prognosis prediction model can make predictions based on the encoded features of multiple tumor slices, which can improve the model's understanding of the patient's disease status and the accuracy of cancer prognosis prediction.

[0053] The embodiment adopts the designed compressed multi-head attention mechanism (SMHA) to perform feature fusion on the initial features of the three types of slices. The SMHA can be a plug-and-play feature interaction block, which undertakes the functions of information exchange and feature enhancement within the fusion process. The SMHA is obtained by improving the multi-head attention mechanism (MHA), wherein the MHA abandons the traditional convolutional neural network (CNN) method of extracting features from 3D tensors using convolution kernels. By calculating the 2D query tensor f of length dk x and the key tensor f y Then, the similarity result is compared with f y Multiply the values ​​in as follows:

[0054]

[0055] Among them, W Q ∈R dk×dk , W K ∈R dk×dk and W V ∈R dk×dk It is the projection matrix of query, key and value.

[0056] MHA exchanges information between CNN features and reshapes CNN features from 3D to 2D to meet the input form of MHA. However, flattening features greatly increases the computational complexity of the network. Therefore, SMHA is designed to improve MHA by compressing the spatial or channel dimension of the query feature as follows:

[0057] Spatial SMHA: query features and key-value features are f q ∈R C×H×W and f kv ∈R C×H×W , C represents the number of channels. In spatial SMHA, f q Converted to R through an average pooling layer1×C , f kv Reshaped into R (H×W)×C , and then the pooled f q and the reshaped f kv Perform MHA calculation, where dk in MHA is equal to the number of channels. Spatial SMHA can be described as:

[0058] Spatial-SMHA(f q , f kv )=MHA(AvgPool(f q ), f kv ) (3)

[0059] Among them, AvgPool(f q ), f kv Indicates f q Pooling results in MHA(AvgPool(f q ), f kv ) represents the pooled f q and the reshaped f kv Perform MHA calculations.

[0060] Channel SMHA: Similarly, f q After a convolutional layer, the number of channels is reduced to a single channel R 1×H×W , f kv Reshaped into R c×(H×W) , and then the convolution compressed f q and the reshaped f kv Perform MHA calculation, where dk in MHA is equal to H × W. The channel SMHA can be described as:

[0061] Channel-SMHA(f q , f kv )=MHA(Conv(f q ), f kv ) (4)

[0062] Among them, Conv(f q ), f kv Indicates f q Convolution compression obtains the result after convolution compression, MHA(Conv(f q ), f kv ) represents the convolution compressed f q and the reshaped f kv Perform MHA calculations.

[0063] Based on the above SMHA, in the first-level feature fusion module, the compressed multi-head attention mechanism is used to fuse the initial features of the three types of slices. Specifically, the compressed self-attention mechanism (self-SMHA) is performed on the initial features of PET tumor slices and CT tumor slices to obtain the first-level self-attention features of the two types of tumor slices, which are represented as f pet o and f ct o , and then the first-level self-attention features (f pet o and f ct o ) are respectively combined with the initial features f of the fused slice fuse The compressed cross-SMHA mechanism is used to obtain the first-level cross-attention features f of the two types of tumor slices. pet ′ and f ct ′, and finally the first-level cross-attention features (f pet ′ and f ct ′) and the initial feature f of the fused slice fuse Perform splicing to obtain the initial splicing feature f concat , initial splicing feature f concat After a channel-reduced convolution compression, the first-level fusion feature f is obtained fuse o , as shown below:

[0064] f pet o =self--SMHA(f pet , f pet ) (5)

[0065] f ct o =self--SMHA(f ct , f ct ) (6)

[0066] f pet ′cross-SMHA(f pet o , f fuse ) (7)

[0067] f ct ′=cross-SMHA(f ct o , f fuse ) (8)

[0068] f concat =Concat(f pet ′, fct ′, f fuse ) (9)

[0069] f fuse o =Conv(f concat ) (10)

[0070] In the first-level feature fusion module, the compressed self-attention mechanism is used to take the input features of the tumor slice (i.e., the initial features of the PET tumor slice f pet Or the initial features of CT tumor slices f ct ) are respectively used as query features f q and key-value features f kv , compress the space or channel dimension of the query feature and perform MHA calculation, that is, execute formula (3) or formula (4) to obtain the respective first-level self-attention features (f pet o or f ct o ).

[0071] In the first-level feature fusion module, the compressed cross-attention mechanism is used to take the input features of the tumor slice (i.e., the first-level self-attention features of the PET tumor slice f pet o Or the first-level self-attention feature f of CT tumor slice ct o ) are respectively used as query features f q , to fuse the input features of the slice (that is, the initial features of the fused slice f fuse ) as the key-value feature f kv , and then perform MHA calculation after compressing the spatial or channel dimension of the query feature, that is, execute formula (3) or formula (4) to obtain the respective first-level cross attention features (f pet ′ or f ct ′).

[0072] In this embodiment, the secondary feature extraction module is used to extract the primary self-attention features of the two types of tumor slices and the primary fusion features of the fusion slices to obtain the secondary features of each type of slice, which are represented as f pet ″、f ct ″、and f fuse Optionally, in the secondary feature extraction module, Resnet50 is also used for feature extraction. In this case, the feature extraction adopts the fifth scale transformation with 2048 channels in Resnet50 to obtain the secondary features of each type of slice. The secondary features of each type of slice are input into the secondary feature fusion module for fusion.

[0073] In this embodiment, the secondary feature fusion module is used to perform feature fusion on the secondary features of the PET tumor slices, CT tumor slices, and fusion slices respectively using the compressed multi-head attention mechanism, specifically including: performing the compressed self-attention mechanism on the secondary features of the PET tumor slices and the CT tumor slices to obtain the secondary self-attention features of the two types of tumor slices, respectively expressed as f pet ″′ and f ct ″′, and then the secondary self-attention features of the two types of tumor slices are combined with the secondary features f fuse ′ performs a compressed cross self-attention mechanism to obtain the secondary cross attention features f of the two types of tumor slices. pet ″″ and f ct Finally, the secondary cross-attention features of the two types of tumor slices are spliced ​​with the secondary features of the fusion slice to obtain the secondary splicing feature f concat ′, secondary splicing feature f concat 'After a channel-reduced convolution compression, the secondary fusion feature f is obtained fuse ″, as shown below:

[0074] f pet ″′=self-SMHA(f pet ″, f pet ″) (11)

[0075] f ct ″′=self-SMHA(f ct ″, f ct ″) (12)

[0076] f pet ″″cross-SMHA(f pet ″′,f fuse ′) (13)

[0077] f ct ″″cross-SMHA(f ct ″′,f fuse ′) (14)

[0078] f concat ′=Concat(f pet ″″,f ct ″″,f fuse ′) (15)

[0079] f fuse =Conv(f concat ′)(16)

[0080] In the secondary feature fusion module, the compressed self-attention mechanism is used to take the input features of the tumor slice (i.e., the secondary features f of the PET tumor slice) as the input features of the tumor slice. pet ″ or secondary features of CT tumor sections f ct ″) are respectively used as query features f q and key-value features f kv , compress the space or channel dimension of the query feature and then perform MHA calculation, that is, execute formula (3) or formula (4) to obtain the respective secondary self-attention features (f pet ″′ or f ct ″′).

[0081] In the secondary feature fusion module, a compressed cross attention mechanism is used to take the input features of the tumor slice (i.e., the secondary self-attention features f of the PET tumor slice) as the input features of the tumor slice. pet ″′ or the secondary self-attention feature f of CT tumor slices ct ″′) are respectively used as query features f q , to fuse the input features of the slice (i.e., the secondary features f of the fused slice fuse ′) as the key feature f kv , compress the spatial or channel dimension of the query feature and perform MHA calculation, that is, execute formula (3) or formula (4) to obtain the respective secondary cross attention features (i.e., f pet ″″ or f ct ″″).

[0082] After the initial feature extraction module, the first-level feature fusion module, the second-level feature extraction module, and the second-level feature fusion module, information interaction and feature enhancement are completed. Through feature extraction and fusion, the tumor metabolic activity information provided by PET and the anatomical structure information provided by CT can be deeply integrated.

[0083] In the embodiment, the prediction module is used to predict the cancer prognosis and obtain the cancer prognosis prediction result. fuse Input a prediction module, where the prediction module can use a fully connected layer to obtain a final probability prediction output for cancer prognosis prediction.

[0084] In the above cancer prognosis prediction model PCFNet, in order to capture and fuse the key features of various slices, a feature fusion module is introduced. The design of this feature fusion module is to enhance the CT tumor slice feature f ct and PET tumor slice features f pet , and extract the fusion slice feature f fuse For feature fusion. First, use the feature fusion module to pet and f ct This will make f pet and fct Capture key information more specifically and increase their importance in feature fusion. Next, extract the initial feature f from the α fusion image fuse This can extract the most representative and discriminative features from the α-fused image, providing important input for the subsequent fusion process. Then, the feature fusion module combines the first-level cross-attention features f ct ′, first-level cross-attention feature f of PET tumor slice pet ′, and the initial features f of the fused slice fuse Here, f ct ′ and f pet ′ is f pet and f ct The fusion result of the compressed cross attention mechanism is further processed to meet the needs of fusion. By splicing, the initial feature f of the fused slice is fuse With the processed f ct ′ and f pet ′ are fused to form a more comprehensive and rich feature representation. Finally, the concatenated feature f concat Further fusion is performed through a channel-reduced convolutional layer to reduce the channel dimension of the features to reduce computational complexity and extract more compact feature representations.

[0085] By designing a feature fusion module, PCFNet is able to capture and fuse key features from PET and CT tumor slices. This fully utilizes the complementary information in PET and CT tumor slices and provides a more comprehensive and accurate feature representation, thus providing a better foundation for subsequent tasks and decision-making.

[0086] In the embodiment, a method for predicting cancer prognosis by combining the auxiliary processing unit of the medical large model MedSAM and multimodal image fusion is also provided. The method adopts the auxiliary processing unit of the medical large model MedSAM and the cancer prognosis prediction system of multimodal image fusion, such as Figure 6 As shown, the following steps are included:

[0087] S1. Using an auxiliary processing unit based on the medical large model MedSAM, respectively, perform auxiliary segmentation, resampling, minimum envelope rectangle selection, and rectangle expansion based on the medical large model MedSAM on the initial PET slice and the initial CT slice to obtain a PET tumor slice and a CT tumor slice, and then fuse the PET tumor slice and the CT tumor slice to obtain a fused slice;

[0088] S2. Use the cancer prognosis prediction model based on multimodal image fusion to extract features from the three modalities of PET tumor slices, CT tumor slices, and fusion slices, fuse the extracted features to obtain fusion features, and predict the fusion features to obtain cancer prognosis prediction results.

[0089] Compared to existing manual segmentation, the methods and systems provided in this embodiment reduce the workload for doctors in determining lesion areas and optimize the usability of tumor slice data. On this basis, PET and CT tumor slices are obtained through sequential resampling, minimum envelope rectangle selection, and rectangular expansion. These slices are then fused to obtain fused slices. The resulting three-modality slices ensure the accuracy and reliability of subsequent analysis. The present invention utilizes a cancer prognosis prediction model based on multimodal image fusion to effectively integrate multimodal slices for cancer prognosis prediction, thereby improving the accuracy of cancer prognosis prediction results. Furthermore, the cancer prognosis prediction results can be used to guide doctors in selecting cancer treatment options.

[0090] To verify the effectiveness of the method and system of the present invention, a series of empirical experiments were conducted. The data for this project came from a hospital, with a total of 424 patients with non-small cell carcinoma. Each case included CT tumor slice images, CT tumor slice segmentation annotations, PET tumor slice images, and PET tumor slice segmentation annotations. Figure 5 CT tumor slice images, CT tumor slice segmentation annotations, PET tumor slice images, and PET tumor slice segmentation annotations are displayed.

[0091] After being processed by the auxiliary processing unit of the MedSAM medical model, a processed three-channel image was obtained. Different screening strategies were applied to the images, removing some samples with too small tumor blocks in the slices. Two screening strategies were used to restrict the data: 1) the area of ​​the rectangle before expansion accounted for more than 30% of the area after expansion; 2) the area of ​​the rectangle before expansion was greater than 50 pixels. Based on these screening strategies, after processing the data based on an area ratio greater than 30%, there were 4960 cases in the training set and 1012 cases in the test set; for an area greater than 50 pixels, there were 8522 cases in the training set and 1806 cases in the test set. Furthermore, without the screening strategy restrictions, there were 12467 cases in the training set and 2604 cases in the test set.

[0092] Table 1 Experimental data set for the verification experiment of the present invention

[0093] training set Test set 30% lesion area 4960 1012 50 pixels 8522 1806 No filter 12467 2604

[0094] For cancer prognostic efficacy, RECIST (Response Evaluation Criteria In Solid Tumors) is used as a standard for evaluating treatment efficacy. RECIST is a commonly used standard for evaluating the efficacy of solid tumors. It assesses tumor response by measuring the maximum diameter of the tumor. Terms such as CR, PR, SD, and PD are used to describe a tumor's response to treatment.

[0095] Table 2 Labels for cancer prognostic effects

[0096]

[0097] In a two-class prediction, CR and PR are labeled 1 and are considered positive samples; SD and PD are labeled 0 and are considered negative samples. In a four-class prediction, CR, PR, SD, and PD are labeled 0, 1, 2, and 3, respectively, representing one class.

[0098] Table 3 Classification labels

[0099] RECIST Binary classification labels Four classification labels CR 1 0 PR 1 1 SD 0 2 PD 0 3

[0100] After the aforementioned data preprocessing, four sets of experiments were designed for the experimental validation phase, each undergoing 5-fold cross-validation. These experiments included three comparison models: SResCNN, ResNet50, and PCFNet. Alternatively, 5-fold cross-validation can be used as a model evaluation method. This method divides the dataset into five parts, using four of the parts as training sets and the remaining part as testing sets. This allows for a more comprehensive assessment of model performance.

[0101] Table 430% lesion area screening results

[0102]

[0103] The table above shows the experimental results of a 30% lesion area screening strategy, using AUC and F1 score as evaluation metrics. For AUC, PCFNet achieved an average AUC of 0.984 across all folds, exceeding the average AUC of SResCNN (0.944) and ResNet50 (0.954). This indicates that PCFNet demonstrated superior prediction accuracy in this experiment, more accurately distinguishing between positive and negative examples. For F1 score, PCFNet achieved an average F1 score of 0.956 across all folds, exceeding the average F1 score of SResCNN (0.868) and ResNet50 (0.892). This indicates that PCFNet demonstrated better balance in this experiment, better balancing the model's precision and recall. Overall, based on the results presented, PCFNet demonstrated higher prediction accuracy and balance in this experiment, achieving better overall performance compared to the SResCNN and ResNet50 models.

[0104] Table 550 pixel screening results

[0105]

[0106] The table above shows the experimental results for a 50-pixel filter. PCFNet achieved an average AUC of 0.970, higher than SResCNN's 0.944 and ResNet50's 0.942. This indicates that PCFNet achieved superior prediction accuracy in this experiment. PCFNet also achieved an average F1 Score of 0.986, higher than SResCNN's 0.870 and ResNet50's 0.876. This indicates that PCFNet achieved superior precision and recall in this experiment.

[0107] Table 6 No screening test results

[0108]

[0109]

[0110] The above table shows the AUC and F1 Score of three different models on unfiltered data. In terms of the AUC indicator, SResCNN achieved the highest AUC value (0.94) in the fourth fold, but its performance in other folds was relatively low, with an average AUC of 0.904. Resnet50 achieved the highest AUC value (0.92) in the fifth fold, but its overall performance was slightly lower than SResCNN, with an average AUC of 0.892. PCFNet achieved the highest AUC (0.95) in the second fold and performed slightly better than the other two models overall, with an average AUC of 0.910. In terms of the F1 Score indicator, SResCNN achieved the highest F1 Score (0.89) in the third fold, with an average F1 Score of 0.812. Resnet50 achieved the highest F1 Score (0.83) in the third and fifth folds, with an average F1 Score of 0.802, slightly lower than SResCNN. The PCFNet model achieved the highest F1 score (0.88) in the second fold and slightly outperformed the other two models overall, with an average F1 score of 0.824. Based on these results, we can see that the PCFNet model performed best on this task, followed by SResCNN and Resnet50.

[0111] Table 7 Four-category task results

[0112]

[0113] The table above shows the AUC and F1 score of the three models in the four-class classification task. These results were obtained using a 30% lesion area screening strategy for the four-class classification task. The following is an analysis of these results: Regarding AUC, PCFNet achieved an average AUC of 0.988, surpassing SResCNN's 0.876 and ResNet50's 0.898. This demonstrates that PCFNet maintains high prediction accuracy even in more complex classification tasks. Regarding F1 score, PCFNet achieved an average F1 score of 0.846, surpassing SResCNN's 0.608 and ResNet50's 0.802. This demonstrates that PCFNet exhibits a certain advantage in balancing precision and recall when handling more complex classification tasks. Furthermore, in the four-class classification task, the AUC and F1 scores of SResCNN and ResNet50 both showed a significant decline, while PCFNet remained stable, achieving results that are generally consistent with those for the two-class classification task. This shows that PCFNet can maintain high prediction accuracy in both simple two-classification tasks and complex four-classification tasks.

[0114] Table 8 Comparison of training effects of auxiliary processing units of doctors’ annotation and MedSAM

[0115]

[0116] The table above shows the AUC and F1 score comparisons between physician annotation and MedSAM's auxiliary processing unit training under a 30% lesion area screening strategy. The average AUC for physician annotation was 0.984 and the average F1 score was 0.956, achieving near-perfect performance. This demonstrates that the model is able to very accurately predict prognosis when physicians annotate lesion areas. In contrast, MedSAM's auxiliary processing unit achieved an average AUC of 0.936 and an average F1 score of 0.862. While slightly lower than physician annotation, these results are still relatively high. This demonstrates that MedSAM's auxiliary processing unit has strong capabilities in predicting cancer prognosis. While there is some margin of error, it still performs well overall.

[0117] By comparing with existing methods, it was found that the auxiliary processing unit constructed by the present invention in combination with the medical large model MedSAM greatly reduced the workload of lesion area division and optimized the availability of tumor slice data; the cancer prognosis prediction model based on multimodal image fusion effectively integrated multimodal slices to perform cancer prognosis prediction, thereby improving the accuracy of cancer prognosis prediction results. At the same time, the cancer prognosis prediction results can be used to guide doctors in choosing cancer treatment plans.

[0118] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cancer prognosis prediction system that combines medical large model assistance and multimodal image fusion, characterized by: It includes an auxiliary processing unit based on the medical large model MedSAM and a cancer prognosis prediction model based on multimodal image fusion; The auxiliary processing unit based on the medical large model MedSAM is used to perform auxiliary segmentation, resampling, minimum envelope rectangle selection, and rectangle expansion based on the medical large model MedSAM on the initial PET slice and the initial CT slice respectively to obtain PET tumor slices and CT tumor slices, and fuse the PET tumor slices and the CT tumor slices to obtain a fused slice; The cancer prognosis prediction model based on multimodal image fusion is used to extract features from three modalities: PET tumor slices, CT tumor slices, and fused slices; perform feature fusion on the extracted features to obtain fused features; and predict the fused features to obtain cancer prognosis prediction results. The cancer prognosis prediction model includes: an initial feature extraction module, a primary feature fusion module, a secondary feature extraction module, a secondary feature fusion module, and a prediction module. The initial feature extraction module is used to perform multi-scale feature extraction on PET tumor slices, CT tumor slices, and fusion slices to obtain initial features of each type of slice; The first-level feature fusion module is used to fuse the initial features of the three types of slices using a compressed multi-head attention mechanism, specifically comprising: performing a compressed self-attention mechanism on the initial features of the PET tumor slices and the CT tumor slices to obtain the first-level self-attention features of the two types of tumor slices respectively, then performing a compressed cross-attention mechanism on the first-level self-attention features of the two types of tumor slices and the initial features of the fused slice respectively to obtain the first-level cross-attention features of the two types of tumor slices respectively, and finally concatenating the first-level cross-attention features of the two types of tumor slices with the initial features of the fused slice, and then performing convolution compression with channel reduction to obtain the first-level fusion features of the fused slice; The secondary feature extraction module is used to extract the primary self-attention features of the two types of tumor slices and the primary fusion features of the fusion slices to obtain the secondary features of each type of slice: The secondary feature fusion module is used to fuse the secondary features of the three types of slices using a compressed multi-head attention mechanism, specifically comprising: performing a compressed self-attention mechanism on the secondary features of the PET tumor slices and the CT tumor slices to obtain the secondary self-attention features of the two types of tumor slices respectively, then performing a compressed cross-attention mechanism on the secondary self-attention features of the two types of tumor slices and the secondary features of the fused slice respectively to obtain the secondary cross-attention features of the two types of tumor slices respectively, and finally splicing the secondary cross-attention features of the two types of tumor slices with the secondary features of the fused slice and then performing convolution compression with channel reduction to obtain the secondary fusion features of the fused slice; The prediction module performs cancer prognosis prediction based on the secondary fusion features to obtain a cancer prognosis prediction result.

2. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 1 is characterized in that: The auxiliary segmentation based on the medical large model MedSAM includes: the medical large model MedSAM assists in outlining the mask of the tumor boundary in the area initially marked by the doctor in the input slice to achieve auxiliary segmentation.

3. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 1 is characterized in that: The resampling adopts a cubic interpolation resampling method, including performing cubic interpolation resampling on the auxiliary segmentation result to achieve uniform pixel size.

4. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 1, characterized in that: The fusion of the PET tumor slice and the CT tumor slice to obtain the fused slice comprises: α Fusion method: ， in, represents the fusion factor, represents CT tumor slices, represents a PET tumor slice, Indicates fused slices.

5. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 1, characterized in that: In the compressed self-attention mechanism, the input features of the tumor slice are used as query features and key features respectively, and the space or channel dimension of the query feature is compressed before the self-attention mechanism calculation is performed.

6. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 1, characterized in that: In the compressed cross-attention mechanism, the input self-attention features of the tumor slice are used as query features, and the input features of the fused slice are used as key features. The cross-attention mechanism calculation is performed after the spatial or channel dimensions of the query features are compressed.

7. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 5 or 6, characterized in that: In the compressed self-attention mechanism and the compressed cross-attention mechanism, the space of the query feature is compressed through the average pooling layer, and the number of channels of the query feature is compressed through the convolution layer.

8. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 1, characterized in that: The initial feature extraction module and the secondary feature extraction module both use Resnet50 for feature extraction.

9. A cancer prognosis prediction method combining medical large model assistance and multimodal image fusion, characterized in that: The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to any one of claims 1 to 8 comprises the following steps: Using an auxiliary processing unit based on the MedSAM medical model, the initial PET slices and initial CT slices are sequentially subjected to auxiliary segmentation, resampling, minimum envelope rectangle selection, and rectangle expansion based on the MedSAM medical model to obtain PET tumor slices and CT tumor slices, which are then fused to obtain fused slices. A cancer prognosis prediction model based on multimodal image fusion is used to extract features from the three modalities of PET tumor slices, CT tumor slices, and fusion slices. The extracted features are fused to obtain fusion features, and the fusion features are predicted to obtain cancer prognosis prediction results.

Images