Cancer prognosis prediction method and system combining medical large model assistance and multi-modal image fusion

Through the medical big model MedSAM assisted processing unit and multimodal image fusion prediction model, the problem of manual segmentation and un-integrated image information in the prognosis prediction of malignant tumors is solved, and more accurate cancer prognosis prediction and treatment plan selection are achieved.

CN119067943AActive Publication Date: 2024-12-03ZHEJIANG UNIV

Patent Information

Application Number
CN202411164396.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-12-03
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

In the prior art, the prognosis prediction of malignant tumors relies on manual segmentation to obtain lesion areas, which has a huge workload and has failed to fully integrate PET/CT imaging information, resulting in inaccurate prognosis effect.

Method used

The MedSAM auxiliary processing unit of medical big model is used to segment, resample, select the minimum envelope rectangle and expand the rectangle to generate fusion slices, and feature extraction and fusion are performed through multimodal image fusion prediction model to realize PET/ Deep fusion of CT images.

Benefits of technology

It reduces the workload of manual segmentation by doctors, improves the availability of tumor section data and the accuracy of prognosis analysis, improves the accuracy of cancer prognosis prediction, and can guide the selection of treatment plans.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067943A_ABST
    Figure CN119067943A_ABST
Patent Text Reader

Abstract

The invention discloses a cancer prognosis prediction system and a cancer prognosis prediction method in combination with medical large model assistance and multi-modal image fusion, and adopts medical large model MedSAM assisted segmentation, compared with the existing manual segmentation, the workload of a doctor for determining a focus area is reduced, the availability of tumor slice data is optimized, and on the basis, the cancer prognosis prediction system and the cancer prognosis prediction method have the advantages that the medical large model MedSAM assisted segmentation is adopted; according to the method, a PET tumor section and a CT tumor section are obtained through re-sampling, minimum enveloping rectangle selection and rectangle expansion in sequence, and the PET tumor section and the CT tumor section are fused to obtain a fused section, so that the obtained three-modal sections can ensure the accuracy and reliability of subsequent analysis; the cancer prognosis prediction model based on multi-modal image fusion is used for effectively integrating the multi-modal slices and then performing cancer prognosis prediction, so that the accuracy of the cancer prognosis prediction result is improved, and meanwhile, the cancer prognosis prediction result can be used for guiding a doctor to select a cancer treatment scheme.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of cancer prognosis prediction, and specifically relates to a cancer prognosis prediction method and system combining large medical model assistance and multimodal image fusion. Background Art

[0002] Malignant tumors are a common and diverse class of diseases, accounting for the vast majority of all cancer cases. Malignant tumors are highly heterogeneous, and different patients have significant differences in tumor biology, gene mutations, and response to treatment, which makes the treatment of malignant tumors poor. At the same time, the accurate prediction of the treatment effect of malignant tumors is also extremely complex and difficult. For patients with advanced malignant tumors, even after receiving chemotherapy and other treatments, their median survival is still short, and existing treatments have limited effect in improving the prognosis of advanced patients. Although there are currently a variety of treatment options, such as targeted therapy, immunotherapy, antibody-drug conjugate therapy, etc., how to correctly choose a treatment strategy remains an urgent problem to be solved.

[0003] In recent years, medical imaging technology, especially PET / CT imaging, has played an important role in the diagnosis, staging and prognosis assessment of malignant tumors. PET imaging (positron emission tomography) can provide information on the metabolic activity of tumors, while CT imaging (computed tomography) provides precise anatomical information. Simply superimposing the two images cannot fully utilize their respective advantages. How to effectively fuse the information in PET and CT images to improve the accuracy and reliability of prognosis prediction for malignant tumors and to select treatment options is an important research issue.

[0004] Reference 1 (Samuel H. Hawkins, John N. Korecki. Predicting outcomes of non-small cell lung cancer using CT image features. IEEE Access, 2, 1418-1426, 2014, doi: 10.1109 / ACCESS.2014.2373335) discloses a method for predicting the prognosis of malignant tumors based on deep learning. Specifically, malignant tumor CT image data is obtained by manual segmentation combined with a region growing algorithm, and treatment effect prediction is achieved with the help of deep learning. This method relies only on CT image data, without deep fusion of PET / CT image information, and cannot accurately predict the prognosis effect; Reference 2 (Wei Mu, Matthew B. Schabath. Non-invasive decision support for NSCLC treatment using PET / CT radiomics, Nature communications, 11, 5228, 2020, doi: 10.1038 / s41467-020-19116-x) discloses a method for predicting the prognosis of malignant tumors based on a deep learning SResCNN model. Specifically, tumor slices are obtained by manual segmentation, and three-channel super images and α-fused PCT / CT images are constructed and input into the SResCNN model. The EGFR mutation status (positive or negative) is the deep learning score (EGFR-DLS) as the classification result, indicating the probability of EGFR mutation positivity. By averaging the EGFR-DLS of the effective slices, the patient's EGFR mutation positivity probability is obtained to predict the prognosis of malignant tumors. Although this method performs deep fusion of PET / CT based on a deep learning model, the acquisition of tumor slice images relies on doctors to manually segment the lesion area, which is a huge workload. Summary of the invention

[0005] The purpose of the present invention is to provide a cancer prognosis prediction method and system that combines medical large model assistance and multimodal image fusion to solve the technical problems of huge workload of manually segmenting the lesion area and the failure to deeply fuse PET / CT, making it impossible to accurately predict the prognosis effect.

[0006] To achieve the above-mentioned purpose of the invention, an embodiment provides a cancer prognosis prediction method and system combining medical big model assistance and multimodal image fusion, including an auxiliary processing unit based on the medical big model MedSAM and a cancer prognosis prediction model based on multimodal image fusion.

[0007] In one embodiment, the auxiliary processing unit based on the medical large model MedSAM is used to perform auxiliary segmentation, resampling, minimum envelope rectangle selection, and rectangle expansion based on the medical large model MedSAM on the initial PET slices and the initial CT slices respectively to obtain PET tumor slices and CT tumor slices, and to fuse the PET tumor slices and the CT tumor slices to obtain fused slices.

[0008] In one embodiment, the cancer prognosis prediction model based on multimodal image fusion is used to extract features from three modalities: PET tumor slices, CT tumor slices, and fused slices, fuse the extracted features to obtain fused features, and predict the fused features to obtain cancer prognosis prediction results.

[0009] In one embodiment, the auxiliary segmentation based on the medical big model MedSAM includes: the medical big model MedSAM assists in outlining the mask of the tumor boundary in the area initially marked by the doctor in the input slice, so as to realize auxiliary segmentation and provide reliable basic data for subsequent processing;

[0010] The resampling method includes: using a cubic interpolation resampling method to perform cubic interpolation resampling on the doctor's initial marked area with a tumor boundary mask after auxiliary segmentation, and unifying the pixel size to 1 cubic millimeter. Resampling is used to standardize the resolution of different samples and eliminate the differences caused by different equipment and scanning parameters, thereby ensuring the consistency and comparability of the data;

[0011] The selection based on the minimum envelope rectangle includes: selecting the minimum rectangle that can completely envelop the mask with the tumor boundary according to the resampled mask with the tumor boundary, so as to remove unnecessary information and focus on the tumor area;

[0012] The rectangular expansion includes: the rectangular expansion expands the width and height of the mask after the minimum envelope rectangle is selected by 20mm, and then adjusts the size of the expanded mask to 64×64. It is used to include the tissue information of the tumor surrounding area on the basis of retaining the tumor area, which helps the cancer prognosis prediction model to more comprehensively learn the characteristics of the tissue information of the tumor and the tumor surrounding area, thereby improving the accuracy of the cancer prognosis prediction result;

[0013] The method of fusing the PET tumor slice and the CT tumor slice based on fusion to obtain a fused slice includes: fusing in an alpha fusion mode:

[0014] fusion_img=ct×α+pet×(1-α)

[0015] Among them, α represents the fusion factor, ct represents the CT tumor slice, pet represents the PET tumor slice, and fusion_img represents the fusion slice;

[0016] Through the α-fusion method, the two modal information of PET / CT tumor slices are initially fused together, utilizing the advantages of each modality to provide more comprehensive and detailed fused slice information, thereby enhancing the model's ability to recognize tumor slice features.

[0017] In one embodiment, the cancer prognosis prediction model includes: an initial feature extraction module, a primary feature fusion module, a secondary feature extraction module, a secondary feature fusion module, and a prediction module;

[0018] In one embodiment, the initial feature extraction module is used to perform multi-scale feature extraction on PET tumor slices, CT tumor slices, and fusion slices to obtain initial features of each type of slice;

[0019] In one embodiment, the first-level feature fusion module is used to perform feature fusion on the initial features of the three types of slices using a compressed multi-head attention mechanism, specifically including: performing a compressed self-attention mechanism on the initial features of the PET tumor slices and the CT tumor slices to obtain the first-level self-attention features of the two types of tumor slices, and then performing a compressed cross-attention mechanism on the first-level self-attention features of the two types of tumor slices and the initial features of the fused slices to obtain the first-level cross-attention features of the two types of tumor slices, and finally splicing the first-level cross-attention features of the two types of tumor slices with the initial features of the fused slices and then performing convolution compression with channel reduction to obtain the first-level fusion features of the fused slices;

[0020] The secondary feature extraction module is used to extract the self-attention features of the two types of tumor slices and the primary fusion features of the fusion slices to obtain the secondary features of each type of slice;

[0021] The secondary feature fusion module is used to use a compressed multi-head attention mechanism to fuse the secondary features of the three types of slices, specifically including: performing a compressed self-attention mechanism on the secondary features of the PET tumor slices and the CT tumor slices to obtain the secondary self-attention features of the two types of tumor slices respectively, and then performing a compressed cross-attention mechanism on the secondary self-attention features of the two types of tumor slices and the secondary features of the fused slices to obtain the secondary cross-attention features of the two types of tumor slices respectively, and finally splicing the secondary cross-attention features of the two types of tumor slices with the secondary features of the fused slices and then performing convolution compression with channel reduction to obtain the secondary fusion features of the fused slices.

[0022] The prediction module performs cancer prognosis prediction based on the secondary fusion features to obtain a cancer prognosis prediction result.

[0023] In one embodiment, in the compressed self-attention mechanism, the input features of the tumor slice are used as query features and key-value features respectively, and the space or channel dimension of the query feature is compressed before the self-attention mechanism calculation is performed.

[0024] In one embodiment, in the compressed cross-attention mechanism, the input self-attention features of the tumor slice are used as query features, and the input features of the fused slice are used as key features. The cross-attention mechanism calculation is performed after the spatial or channel dimension of the query feature is compressed.

[0025] In one embodiment, in the compressed self-attention mechanism and the compressed cross-attention mechanism, the space of the query feature is compressed through the average pooling layer, and the number of channels of the query feature is compressed through the convolution layer.

[0026] In one embodiment, the initial feature extraction module and the secondary feature extraction module both use Resnet50 for feature extraction.

[0027] In one embodiment, an auxiliary processing unit based on the medical large model MedSAM is used to perform auxiliary segmentation, resampling, minimum envelope rectangle selection, and rectangle expansion based on the medical large model MedSAM on the initial PET slice and the initial CT slice to obtain a PET tumor slice and a CT tumor slice, and the PET tumor slice and the CT tumor slice are fused to obtain a fused slice;

[0028] A cancer prognosis prediction model based on multimodal image fusion is used to extract features from the three modalities of PET tumor slices, CT tumor slices, and fusion slices. The extracted features are fused to obtain fusion features, and the fusion features are predicted to obtain cancer prognosis prediction results.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention adopts the auxiliary segmentation of the medical large model MedSAM. Compared with the existing manual segmentation, it reduces the workload of doctors in determining the lesion area and optimizes the availability of tumor slice data. On this basis, PET tumor slices and CT tumor slices are obtained by resampling, minimum envelope rectangle selection, and rectangular expansion in sequence, and the PET tumor slices and CT tumor slices are fused to obtain fused slices. The slices of the three modalities obtained in this way can ensure the accuracy and reliability of subsequent analysis; the present invention uses a cancer prognosis prediction model based on multimodal image fusion to effectively integrate multimodal slices and then perform cancer prognosis prediction to improve the accuracy of cancer prognosis prediction results. At the same time, the cancer prognosis prediction results can be used to guide doctors in choosing cancer treatment plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0032] Figure 1 It is a schematic diagram of the structure of the cancer prognosis prediction system;

[0033] Figure 2 It is a processing flow chart of the auxiliary processing unit of MedSAM;

[0034] Figure 3 This is a schematic diagram of the structure of the cancer prognosis prediction model PCFNet;

[0035] Figure 4 It is a schematic diagram of the principle of SMHA;

[0036] Figure 5 is a malignant tumor slice image; Figure 5 A in the figure is a CT image; Figure 5 B in the figure is a PET image; Figure 5 The red area in the figure is the segmented and annotated image processed by its corresponding MedSAM auxiliary processing unit;

[0037] Figure 6 It is a flow chart of the cancer prognosis prediction method. DETAILED DESCRIPTION

[0038] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0039] In order to achieve the accuracy of cancer prognosis prediction, the embodiment provides a cancer prognosis prediction method and system that combines the auxiliary processing unit of the medical large model MedSAM and multimodal image fusion. The system can be used to determine the cancer lesion area, fuse the multimodal information of tumor slices, and predict the cancer prognosis of patients. Figure 1 As shown, the cancer prognosis prediction system provided by the embodiment includes: an auxiliary processing unit based on the medical big model MedSAM, and a cancer prognosis prediction model based on multimodal image fusion.

[0040] In the embodiment, the auxiliary processing unit based on the medical large model MedSAM is used to solve the problem of excessive workload of doctors in manually segmenting the lesion area, ensure the consistency and quality of input data, and reduce the impact of different equipment and reconstruction parameters on model performance. Figure 2As shown, the auxiliary processing unit of MedSAM provided in the embodiment mainly includes five steps: MedSAM auxiliary segmentation, resampling, selecting the minimum envelope rectangle, rectangle expansion, and fusion.

[0041] In the embodiment, MedSAM-assisted segmentation is used to accurately segment the imaging slices (initial PET slices and initial CT slices) to provide reliable basic data for subsequent processing. Optionally, MedSAM is used to assist in outlining the mask of the tumor boundary in the area initially marked by the doctor in the input slice to achieve assisted segmentation.

[0042] In the embodiment, resampling is used to standardize the resolution of different samples and eliminate the differences caused by different equipment and scanning parameters, thereby ensuring the consistency and comparability of the data. Optionally, a cubic interpolation resampling method is used to perform cubic interpolation resampling on the doctor's initial marked area with the tumor boundary mask after auxiliary segmentation to unify the pixel size. For example, the pixels with the tumor boundary mask after resampling are unified to 1 cubic millimeter.

[0043] In the embodiment, the minimum envelope rectangle is selected to remove unnecessary information, focus on the tumor area, and improve the efficiency of data processing and the effect of model training. Optionally, based on the tumor boundary mask after cubic interpolation resampling, the minimum envelope rectangle that can completely envelop the tumor boundary mask is selected.

[0044] In the embodiment, the rectangular expansion is used to include a certain range of surrounding tissue information while retaining the tumor area, which helps the model to learn the characteristics of the tumor and its surrounding environment more comprehensively, thereby improving the accuracy of cancer prognosis prediction results; optionally, the width and height of the selected minimum enveloping rectangle are expanded by 20mm, and then the size of the expanded minimum enveloping rectangle is adjusted to 64×64.

[0045] In the embodiment, fusion is used to fuse image information, provide more comprehensive and detailed image information, and enhance the model's ability to identify tumor features. Optionally, fusion of PET tumor slices and CT tumor slices to obtain fused slices can be performed using α fusion:

[0046] fusion_img=ct×α+pet×(1-α)(1)

[0047] Among them, α represents the fusion factor, ct represents the CT tumor slice, pet represents the PET tumor slice, and fusion_img represents the fusion slice;

[0048] Through the α-fusion method, the two modal information of PET / CT tumor slices are initially fused together, utilizing the advantages of each modality to provide more comprehensive and detailed fused slice information, thereby enhancing the model's ability to recognize tumor slice features.

[0049] In the embodiment, the cancer prognosis prediction model based on multimodal image fusion is used to capture the metabolic information in the PET image and the anatomical structure information in the CT image; the image information of the two is fused to improve the accuracy of the cancer prognosis prediction result. Figure 3 As shown, the PCFNet provided in the embodiment includes: an initial feature extraction module, a primary feature fusion module, a secondary feature extraction module, a secondary feature fusion module, and a prediction module.

[0050] In the embodiment, the initial feature extraction module is used to perform multi-scale feature extraction on the PET tumor slices, CT tumor slices, and fusion slices to obtain the initial features of each type of slice, which are represented by f pet 、f ct , and f fuse . Optionally, the initial feature extraction module can use Resnet50 for feature extraction, specifically using three independent Resnet50s to extract features from PET tumor slices, CT tumor slices, and fusion slices, respectively. Resnet50 is a deep residual network that solves the gradient vanishing and representation bottleneck problems in deep neural networks by introducing residual connections, which allows the network to be trained at a deeper level to extract richer and more complex features. All three Resnet50s use weights pre-trained on the ImageNet dataset. ImageNet is a large-scale visual database that contains 1,000 categories and more than 14 million images. Using pre-trained weights can help the model achieve better performance in the initial stage, and can also speed up the training process of the model.

[0051] Resnet50 is used to extract the morphological features of the tumor from all the input slices. These features include information such as the size, shape and edge of the tumor. The deep network structure of Resnet50 enables it to learn the complex patterns of tumors and capture subtle changes in tumor growth. Specifically, Resnet50 has 5 scale transformations, and the number of channels for the 5 transformations is 64, 256, 512, 1024, and 2048 respectively. In order to deeply fuse the features of PET tumor slices, CT tumor slices, and fused slices, and to fuse features of different scales, in the initial feature extraction module, the first 4 scale transformations are performed first to obtain the initial features of the PET tumor slices, CT tumor slices, and fused slices. The initial features of the three slices are input into the first-level feature fusion module for fusion.

[0052] In an embodiment, a primary feature fusion module is used to perform feature fusion on the initial features of PET tumor slices, CT tumor slices, and fused slices to make full use of the complementary information between them. In the field of medical imaging, PET tumor slices and CT tumor slices are two commonly used medical imaging modalities, which provide different information respectively, among which PET tumor slices can provide information about the metabolic activity of tissues or lesions, while CT tumor slices provide detailed information about anatomical structures. At the same time, fused slices combine information from PET and CT. Therefore, by comprehensively utilizing the information of these three types of tumor slices, the cancer prognosis prediction model can be predicted based on the coded features of multiple tumor slices, which can improve the model's understanding of the patient's disease status and the accuracy of cancer prognosis prediction.

[0053] The embodiment adopts the designed compressed multi-head attention mechanism (SMHA) to perform feature fusion on the initial features of the three types of slices. The SMHA can be a plug-and-play feature interaction block, which undertakes the functions of information exchange and feature enhancement within the fusion process. The SMHA is obtained by improving the multi-head attention mechanism (MHA), wherein the MHA abandons the traditional convolutional neural network (CNN) method of extracting features from a 3D tensor using a convolution kernel. By calculating the 2D query tensor f with a length of dk x and the key tensor f y Then, the similarity result is compared with f y Multiply the values ​​in as follows:

[0054]

[0055] Among them, W Q ∈R dk×dk , W K ∈R dk×dk and W V ∈R dk×dk It is the projection matrix of query, key and value.

[0056] By exchanging information between CNN features through MHA, CNN features are reshaped from 3D to 2D to meet the input form of MHA. However, flattening features greatly increases the computational complexity of the network. Therefore, SMHA is designed to improve MHA by compressing the spatial or channel dimension of the query feature as follows:

[0057] Spatial SMHA: The query feature and key-value feature are f q ∈R C×H×W and f kv ∈R C×H×W , C represents the number of channels. In spatial SMHA, f q Transformed to R through an average pooling layer1×C , f kv Reshaped as R (H×W)×C , and then the pooled f q and the reshaped f kv Perform MHA calculation, where dk in MHA is equal to the number of channels. The spatial SMHA can be described as:

[0058] Spatial-SMHA(f q , f kv )=MHA(AvgPool(f q ), f kv ) (3)

[0059] Among them, AvgPool(f q ), f kv Indicates f q Pooling obtains the pooling result, MHA(AvgPool(f q ), f kv ) represents the pooled f q and the reshaped f kv Perform MHA calculations.

[0060] Channel SMHA: Similarly, f q After a convolutional layer, the number of channels is reduced to a single channel R 1×H×W , f kv Reshaped as R c×(H×W) , and then the convolution compressed f q and the reshaped f kv Perform MHA calculation, where dk in MHA is equal to H × W. The channel SMHA can be described as:

[0061] Channel-SMHA(f q , f kv )=MHA(Conv(f q ), f kv ) (4)

[0062] Among them, Conv(f q ), f kv Indicates f q Convolution compression obtains the result after convolution compression, MHA(Conv(f q ), f kv ) represents the convolution compressed f q and the reshaped f kv Perform MHA calculations.

[0063] Based on the above SMHA, in the first-level feature fusion module, the compressed multi-head attention mechanism is used to fuse the initial features of the three types of slices, specifically including: the compressed self-attention mechanism (self-SMHA) is performed on the initial features of the PET tumor slices and CT tumor slices to obtain the first-level self-attention features of the two types of tumor slices, which are represented as f pet o and f ct o , and then the first-level self-attention features (f pet o and f ct o ) are respectively combined with the initial features f of the fused slice fuse The compressed cross-SMHA mechanism is used to obtain the first-level cross-attention features f of the two types of tumor slices. pet ′ and f ct ′, and finally the first-level cross-attention features (f pet ′ and f ct ′) and the initial feature f of the fused slice fuse Perform splicing to obtain the initial splicing feature f concat , initial splicing feature f concat After a channel-reduced convolution compression, the first-level fusion feature f is obtained fuse o , as shown below:

[0064] f pet o =self--SMHA(f pet , f pet ) (5)

[0065] f ct o =self--SMHA(f ct , f ct ) (6)

[0066] f pet ′cross-SMHA(f pet o , f fuse ) (7)

[0067] f ct ′=cross-SMHA(f ct o , f fuse ) (8)

[0068] f concat =Concat(f pet ′,fct ′,f fuse ) (9)

[0069] f fuse o =Conv(f concat ) (10)

[0070] In the first-level feature fusion module, a compressed self-attention mechanism is used to take the input features of the tumor slice (i.e., the initial features of the PET tumor slice f pet Or the initial features of CT tumor slices f ct ) are used as query features f q and key-value feature f kv , after compressing the spatial or channel dimension of the query feature, perform MHA calculation, that is, execute formula (3) or formula (4) to obtain the respective first-level self-attention features (f pet o or ct o ).

[0071] In the first-level feature fusion module, a compressed cross-attention mechanism is used to take the input features of the tumor slice (i.e., the first-level self-attention features of the PET tumor slice f pet o Or the first-level self-attention feature f of CT tumor slice ct o ) are used as query features f q , to fuse the input features of the slice (i.e., the initial features of the fused slice f fuse ) as the key feature f kv , after compressing the spatial or channel dimension of the query feature, MHA calculation is performed, that is, formula (3) or formula (4) is executed to obtain the respective first-level cross-attention features (f pet ′ or f ct ′).

[0072] In the embodiment, the secondary feature extraction module is used to extract the primary self-attention features of the two types of tumor slices and the primary fusion features of the fusion slices to obtain the secondary features of each type of slice, which are represented as f pet ″、f ct ″, and f fuse '. Optionally, in the secondary feature extraction module, Resnet50 is also used for feature extraction. At this time, the feature extraction adopts the fifth scale transformation with a channel number of 2048 in Resnet50 to obtain the secondary features of each type of slice, and the secondary features of each type of slice are input into the secondary feature fusion module for fusion.

[0073] In the embodiment, the secondary feature fusion module is used to perform feature fusion on the secondary features of the PET tumor slices, the CT tumor slices, and the fusion slices by using the compressed multi-head attention mechanism, specifically including: performing the compressed self-attention mechanism on the secondary features of the PET tumor slices and the CT tumor slices to obtain the secondary self-attention features of the two types of tumor slices, respectively, represented as f pet ″′ and f ct ″′, and then the secondary self-attention features of the two types of tumor slices are respectively combined with the secondary features f of the fusion slice fuse ′ performs a compressed cross self-attention mechanism to obtain the secondary cross attention features f of the two types of tumor slices. pet ″″ and f ct Finally, the secondary cross-attention features of the two types of tumor slices are concatenated with the secondary features of the fusion slices to obtain the secondary concatenated features f concat ′, secondary splicing feature f concat ′After a channel-reduced convolution compression, the second-level fusion feature f is obtained fuse ″, as shown below:

[0074] f pet ″′=self-SMHA(f pet ″, f pet ″) (11)

[0075] f ct ″′=self-SMHA(f ct ″, f ct ″) (12)

[0076] f pet ″″cross-SMHA(f pet ″′,f fuse ′) (13)

[0077] f ct ″″cross-SMHA(f ct ″′,f fuse ′) (14)

[0078] f concat ′=Concat(f pet ″″,f ct ″″,f fuse ′) (15)

[0079] f fuse =Conv(f concat ′)(16)

[0080] In the secondary feature fusion module, a compressed self-attention mechanism is used to take the input features of the tumor slice (i.e., the secondary features f of the PET tumor slice) as input features. pet ″ or secondary features of CT tumor sections ct ″) are used as query features f q and key-value feature f kv , compress the space or channel dimension of the query feature and perform MHA calculation, that is, execute formula (3) or formula (4) to obtain the respective secondary self-attention features (f pet ″′ or f ct ″′).

[0081] In the secondary feature fusion module, a compressed cross-attention mechanism is used to take the input features of the tumor slice (i.e., the secondary self-attention features f of the PET tumor slice) as input features of the tumor slice. pet ″′ or the secondary self-attention feature f of CT tumor slices ct ″′) are respectively used as query features f q , to fuse the input features of the slice (i.e., the secondary features of the fused slice f fuse ′) as the key feature f kv , after compressing the spatial or channel dimension of the query feature, perform MHA calculation, that is, execute formula (3) or formula (4) to obtain the respective secondary cross attention features (i.e., f pet ″″ or f ct ″″).

[0082] After the initial feature extraction module, the first-level feature fusion module, the second-level feature extraction module, and the second-level feature fusion module, the information interaction and feature enhancement are completed. Through feature extraction and fusion, the tumor metabolic activity information provided by PET and the anatomical structure information provided by CT can be deeply integrated.

[0083] In the embodiment, the prediction module is used to predict the cancer prognosis to obtain the cancer prognosis prediction result. fuse "Input a prediction module, where the prediction module can use a fully connected layer to obtain a final probability prediction output for cancer prognosis prediction.

[0084] In the above cancer prognosis prediction model PCFNet, in order to capture and fuse the key features in various slices, a feature fusion module is introduced. The feature fusion module is designed to enhance the CT tumor slice feature f ct and PET tumor slice features pet , and extract the fusion slice feature f fuse For feature fusion. First, use the feature fusion module to pet and f ct This can make f pet and fct Capture key information more specifically and increase their importance in feature fusion. Next, extract the initial feature f from the α-fused image. fuse This can extract the most representative and distinguishing features from the α-fused image, providing important input for the subsequent fusion process. Then, the feature fusion module combines the first-level cross-attention features f ct ′, first-level cross-attention feature of PET tumor slice f pet ′, and the initial features of the fused slice f fuse Here, f ct ′ and f pet ′ is f pet and f ct The fusion is obtained through the compressed cross-attention mechanism and is further processed to meet the needs of fusion. fuse With the processed f ct ′ and f pet ′ to form a more comprehensive and rich feature representation. Finally, the concatenated feature f concat Further fusion is performed through a channel-reduced convolutional layer to reduce the channel dimension of the features to reduce the computational complexity and extract a more compact feature representation.

[0085] By designing a feature fusion module, PCFNet is able to capture and fuse key features in PET tumor slices and CT tumor slices. Doing so can fully utilize the complementary information in PET tumor slices and CT tumor slices and provide a more comprehensive and accurate feature representation, thus providing a better basis for subsequent tasks and decisions.

[0086] In the embodiment, a cancer prognosis prediction method combining the auxiliary processing unit of the medical large model MedSAM and multimodal image fusion is also provided. The method adopts the above-mentioned cancer prognosis prediction system of the auxiliary processing unit of the medical large model MedSAM and multimodal image fusion, such as Figure 6 As shown, the following steps are included:

[0087] S1. Using an auxiliary processing unit based on the medical large model MedSAM, respectively, perform auxiliary segmentation, resampling, minimum envelope rectangle selection, and rectangle expansion based on the medical large model MedSAM on the initial PET slice and the initial CT slice to obtain a PET tumor slice and a CT tumor slice, and fuse the PET tumor slice and the CT tumor slice to obtain a fused slice;

[0088] S2. Use the cancer prognosis prediction model based on multimodal image fusion to extract features from the three modalities of PET tumor slices, CT tumor slices, and fused slices, fuse the extracted features to obtain fused features, and predict the fused features to obtain cancer prognosis prediction results.

[0089] Compared with the existing manual segmentation, the method and system provided in the embodiment reduce the workload of doctors in determining the lesion area and optimize the availability of tumor slice data. On this basis, resampling, minimum envelope rectangle selection, and rectangular expansion are performed in sequence to obtain PET tumor slices and CT tumor slices, and the PET tumor slices and CT tumor slices are fused to obtain fused slices, so that the slices of the three modalities can ensure the accuracy and reliability of subsequent analysis; the present invention uses a cancer prognosis prediction model based on multimodal image fusion to effectively integrate multimodal slices for cancer prognosis prediction, so as to improve the accuracy of cancer prognosis prediction results, and at the same time, the cancer prognosis prediction results can be used to guide doctors in the selection of cancer treatment plans.

[0090] In order to verify the effectiveness of the method and system of the present invention, a series of empirical experiments are conducted below. The data of this project comes from a hospital, with a total of 424 patients with non-small cell carcinoma, each of which includes CT tumor slice images, CT tumor slice segmentation annotations, PET tumor slice images, and PET tumor slice segmentation annotations. Figure 5 CT tumor slice images, CT tumor slice segmentation annotations, PET tumor slice images, and PET tumor slice segmentation annotations are displayed.

[0091] After the auxiliary processing unit of the medical large model MedSAM, the processed three-channel image can be obtained. Different screening strategies were applied to the images, and samples with too small tumor blocks in some slices were deleted, and two screening strategies were used to restrict them respectively: 1) The area of ​​the rectangle before expansion accounts for more than 30% of the area after expansion. 2) The area of ​​the rectangle before expansion is greater than 50 pixels. Based on the screening strategy, after the data is processed according to the area ratio greater than 30%, there are 4960 training sets and 1012 test sets; according to the area greater than 50 pixels, there are 8522 training sets and 1806 test sets. In addition, without the restriction of the screening strategy, there are 12467 training sets and 2604 test sets.

[0092] Table 1 Experimental data set for the verification experiment of the present invention

[0093] Training set Test Set 30% lesion area 4960 1012 50 pixels 8522 1806 No filter 12467 2604

[0094] For the label of cancer prognosis effect, RECIST (Response Evaluation Criteria In Solid Tumors) is used as the evaluation standard of treatment effect. RECIST is a commonly used solid tumor efficacy evaluation standard, which evaluates the tumor response by measuring the maximum diameter of the tumor. Among them, CR, PR, SD, PD, etc. are all terms used to describe the tumor response to treatment.

[0095] Table 2 Labels for cancer prognostic effects

[0096]

[0097] In the two-class prediction, the labels of CR and PR are 1 and are considered positive samples; the labels of SD and PD are 0 and are considered negative samples. In the four-class prediction, the labels of CR, PR, SD, and PD are 0, 1, 2, and 3, respectively, representing one class.

[0098] Table 3 Classification labels

[0099] RECIST Binary classification labels Four classification labels CR 1 0 PR 1 1 SD 0 2 PD 0 3

[0100] After the above data pre-processing, four groups of experiments were designed in the experimental verification stage, and each group of experiments was subjected to five-fold cross-validation. These experiments include three comparative models, namely SResCNN, Resnet50, and PCFNet. Optionally, five-fold cross-validation is used as the model evaluation method, and the data set is divided into five parts, four of which are used as training sets each time, and the remaining one is used as a test set, so that the performance of the model can be more comprehensively evaluated.

[0101] Table 430% lesion area screening results

[0102]

[0103] In the table above, the experimental results of the 30% lesion area screening strategy are shown, using AUC and F1 Score as evaluation indicators. For AUC, the average AUC of PCFNet in all folds is 0.984, which is higher than the average AUC of SResCNN (0.944) and the average AUC of ResNet50 (0.954). This shows that PCFNet showed better prediction accuracy in this experiment and can more accurately distinguish positive and negative examples. For F1 Score, the average F1 Score of PCFNet in all folds is 0.956, which is higher than the average F1 Score of SResCNN (0.868) and the average F1 Score of ResNet50 (0.892). This shows that PCFNet showed better balance in this experiment and can better comprehensively consider the precision and recall of the model. In summary, according to the results given, PCFNet showed higher prediction accuracy and balance in this experiment, and had better overall performance than the SResCNN and ResNet50 models.

[0104] Table 550 pixel screening results

[0105]

[0106] In the table above, the experimental results are screened by 50 pixels. For AUC, the average AUC of PCFNet is 0.970, which is higher than 0.944 of SResCNN and 0.942 of ResNet50. This shows that PCFNet has better prediction accuracy in this experiment. For F1 Score, the average F1Score of PCFNet is 0.986, which is higher than 0.870 of SResCNN and 0.876 of ResNet50. This shows that PCFNet has better precision and recall in this experiment.

[0107] Table 6 No screening experiment results

[0108]

[0109]

[0110] The above table shows the AUC and F1 Score of three different models in unfiltered data. In terms of AUC, SResCNN achieved the highest AUC value (0.94) in the fourth fold, but its performance in other folds was relatively low, with an average AUC of 0.904. Resnet50 achieved the highest AUC value (0.92) in the fifth fold, but its overall performance was slightly inferior to SResCNN, with an average AUC of 0.892. PCFNet achieved the highest AUC (0.95) in the second fold, and its overall performance was slightly higher than the other two models, with an average AUC of 0.910. In terms of F1 Score, SResCNN achieved the highest F1 Score (0.89) in the third fold, with an average F1 Score of 0.812. Resnet50 achieved the highest F1 Score (0.83) in the third and fifth folds, with an average F1 Score of 0.802, slightly lower than SResCNN. The PCFNet model achieved the highest F1 Score (0.88) in the second fold, and slightly outperformed the other two models overall, with an average F1 Score of 0.824. Based on these results, it can be seen that the PCFNet model performs best on this task, followed by SResCNN and Resnet50.

[0111] Table 7 Four classification task results

[0112]

[0113] The above table shows the AUC and F1 Score of the three models in the four-classification task. These results are obtained by performing the four-classification task under the screening strategy of 30% lesion area. The following is an analysis of these results: For AUC, the average AUC of PCFNet is 0.988, which is higher than 0.876 of SResCNN and 0.898 of ResNet50. This shows that PCFNet can still maintain a high prediction accuracy in more complex classification tasks. For F1 Score, the average F1Score of PCFNet is 0.846, which is higher than 0.608 of SResCNN and 0.802 of ResNet50. This shows that when dealing with more complex classification tasks, PCFNet has a certain advantage in the balance between precision and recall. At the same time, in the four-classification task, the AUC and F1 Score of SResCNN and ResNet50 have declined significantly, while PCFNet remains stable, which is basically consistent with the results of the two-classification task. This shows that PCFNet can maintain high prediction accuracy in both simple two-classification tasks and complex four-classification tasks.

[0114] Table 8 Comparison of training effects of auxiliary processing units of doctors’ annotation and MedSAM

[0115]

[0116] The above table shows the AUC and F1 Score of the training effect of the doctor's annotation and MedSAM's auxiliary processing unit under the screening strategy of 30% lesion area. The average AUC of the doctor's annotation is 0.984, and the average F1 Score is 0.956, which is almost perfect. This shows that when the doctor annotates the lesion area, the model can predict the prognosis very accurately. In contrast, the average AUC of MedSAM's auxiliary processing unit is 0.936 and the average F1 Score is 0.862. Although slightly lower than the doctor's annotation, it is still at a high level. This shows that MedSAM's auxiliary processing unit has a strong ability in cancer prognosis prediction; although there are certain errors, it can still predict cancer prognosis well overall.

[0117] By comparing with the existing methods, it is found that the auxiliary processing unit combined with the medical large model MedSAM constructed in the present invention greatly reduces the workload of lesion area division and optimizes the availability of tumor slice data; the cancer prognosis prediction model based on multimodal image fusion effectively integrates multimodal slices to perform cancer prognosis prediction, thereby improving the accuracy of cancer prognosis prediction results. At the same time, the cancer prognosis prediction results can be used to guide doctors in choosing cancer treatment plans.

[0118] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cancer prognosis prediction system combining medical large model assistance and multimodal image fusion, characterized in that: Including an auxiliary processing unit based on the medical big model MedSAM and a cancer prognosis prediction model based on multimodal image fusion; The auxiliary processing unit based on the medical large model MedSAM is used to perform auxiliary segmentation, resampling, minimum envelope rectangle selection, and rectangle expansion based on the medical large model MedSAM on the initial PET slice and the initial CT slice respectively to obtain a PET tumor slice and a CT tumor slice, and fuse the PET tumor slice and the CT tumor slice to obtain a fused slice; The cancer prognosis prediction model based on multimodal image fusion is used to extract features from three modalities: PET tumor slices, CT tumor slices, and fused slices, fuse the extracted features to obtain fused features, and predict the fused features to obtain cancer prognosis prediction results.

2. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 1, characterized in that: The auxiliary segmentation based on the medical large model MedSAM includes: the medical large model MedSAM assists in outlining the mask of the tumor boundary in the area initially marked by the doctor in the input slice to achieve auxiliary segmentation.

3. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 1, characterized in that: The resampling adopts a cubic interpolation resampling method, including performing cubic interpolation resampling on the result after auxiliary segmentation to achieve uniform pixel size.

4. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 1, characterized in that: The fusion of the PET tumor slice and the CT tumor slice to obtain the fused slice includes: fusing in an alpha fusion mode: fusion_img=ct×α+pet×(1-α) Among them, α represents the fusion factor, ct represents CT tumor slice, pet represents PET tumor slice, and fusion_img represents fusion slice.

5. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 1, characterized in that: The cancer prognosis prediction model comprises: an initial feature extraction module, a primary feature fusion module, a secondary feature extraction module, a secondary feature fusion module, and a prediction module; The initial feature extraction module is used to perform multi-scale feature extraction on PET tumor slices, CT tumor slices, and fusion slices to obtain initial features of each type of slice; The first-level feature fusion module is used to perform feature fusion on the initial features of the three types of slices using a compressed multi-head attention mechanism, specifically including: performing a compressed self-attention mechanism on the initial features of the PET tumor slices and the CT tumor slices to obtain the first-level self-attention features of the two types of tumor slices, and then performing a compressed cross-attention mechanism on the first-level self-attention features of the two types of tumor slices and the initial features of the fused slices to obtain the first-level cross-attention features of the two types of tumor slices, and finally splicing the first-level cross-attention features of the two types of tumor slices with the initial features of the fused slice, and then performing convolution compression with channel reduction to obtain the first-level fusion features of the fused slice. The secondary feature extraction module is used to extract the primary self-attention features of the two types of tumor slices and the primary fusion features of the fusion slices to obtain the secondary features of each type of slice: The secondary feature fusion module is used to fuse the secondary features of the three types of slices using a compressed multi-head attention mechanism, specifically including: performing a compressed self-attention mechanism on the secondary features of the PET tumor slices and the CT tumor slices to obtain the secondary self-attention features of the two types of tumor slices, and then performing a compressed cross-attention mechanism on the secondary self-attention features of the two types of tumor slices and the secondary features of the fused slices to obtain the secondary cross-attention features of the two types of tumor slices, and finally splicing the secondary cross-attention features of the two types of tumor slices with the secondary features of the fused slices, and then performing convolution compression with channel reduction to obtain the secondary fusion features of the fused slices; The prediction module performs cancer prognosis prediction based on the secondary fusion features to obtain a cancer prognosis prediction result.

6. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 5, characterized in that: In the compressed self-attention mechanism, the input features of the tumor slice are used as query features and key-value features respectively, and the space or channel dimension of the query feature is compressed before the self-attention mechanism calculation is performed.

7. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 5, characterized in that: In the compressed cross-attention mechanism, the input self-attention features of the tumor slices are used as query features, and the input features of the fused slices are used as key features. The cross-attention mechanism calculation is performed after the spatial or channel dimensions of the query features are compressed.

8. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 6 or 7, characterized in that: In the compressed self-attention mechanism and the compressed cross-attention mechanism, the space of the query feature is compressed through the average pooling layer, and the number of channels of the query feature is compressed through the convolution layer.

9. The cancer prognosis prediction system combining medical large model assistance and multimodal image fusion according to claim 5, characterized in that: The initial feature extraction module and the secondary feature extraction module both use Resnet50 for feature extraction.

10. A cancer prognosis prediction method combining medical large model assistance and multimodal image fusion, characterized in that: The following steps are involved: Using an auxiliary processing unit based on the medical large model MedSAM, the initial PET slices and the initial CT slices are respectively subjected to auxiliary segmentation, resampling, minimum envelope rectangle selection, and rectangle expansion based on the medical large model MedSAM to obtain PET tumor slices and CT tumor slices, and the PET tumor slices and the CT tumor slices are fused to obtain fused slices; A cancer prognosis prediction model based on multimodal image fusion is used to extract features from the three modalities of PET tumor slices, CT tumor slices, and fusion slices. The extracted features are fused to obtain fusion features, and the fusion features are predicted to obtain cancer prognosis prediction results.

Citation Information

Patent Citations

  • Cross-modal attention tumor segmentation method and system based on PET-CT

    CN114862881A

  • Tumor classification method based on multiplexing self-prediction segmentation result

    CN117274714A

  • Cross-modal tumor automatic segmentation method based on PET-CT medical image and storage medium

    CN118172372A

Cited By

  • Method, system and device for constructing diffusion converter model and storage medium

    CN118536571A

  • A method, system, device and storage medium for constructing a diffusion transformer model

    CN118536571B

  • Dynamic lung cancer curative effect prediction method and system based on CT image change

    CN120376155A

  • Dynamic prediction method and system for lung cancer treatment efficacy based on CT image changes

    CN120376155B

  • PET-CT (positron emission tomography-computed tomography)-based non-small cell lung cancer neoadjuvant immunotherapy curative effect multi-mode intelligent prediction method and model

    CN120565021A