Automatic auxiliary diagnosis method for pancreatic cancer and autoimmune pancreatitis based on pet-ct

Abstract

The application discloses a PET-CT-based automatic auxiliary diagnosis method for pancreatic cancer and autoimmune pancreatitis, which comprises the following steps: image preprocessing is performed on PET images and CT images respectively; based on a Faster-Rcnn algorithm, an attention mechanism module is introduced into a residual network of Resnet50, an FPN network is added to form a feature pyramid network, and an ROIAlign is used to replace an ROIPooling layer, an improved detection network is constructed, target detection is performed on the image-preprocessed PET images and CT images respectively, and a lesion detection result is output 18 F-FDG PET / CT images; a hybrid classification model is constructed, imageomics feature extraction and deep learning feature extraction are performed on the F-FDG PET / CT images, multi-modal feature fusion, multi-domain feature fusion and feature correlation analysis are performed, and a lesion classification result is output. 18 F-FDG PET / CT images; a hybrid classification model is constructed, imageomics feature extraction and deep learning feature extraction are performed on the F-FDG PET / CT images, multi-modal feature fusion, multi-domain feature fusion and feature correlation analysis are performed, and a lesion classification result is output.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and more specifically, to an automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT, for the automated auxiliary diagnosis of pancreatic cancer and autoimmune pancreatitis. Background Technology

[0002] Differentiating between autoimmune pancreatitis (AIP) and pancreatic ductal adenocarcinoma (PDAC) is a diagnostic challenge and can lead to misdiagnosis. Currently, imaging is the most common diagnostic method in clinical practice. Among clinical manifestations, CT, PET, MR imaging, and endoscopic ultrasound (EUS) can detect subtle pancreatic abnormalities and are often used as tools for detection and staging.

[0003] With the continuous development of artificial intelligence, its application in clinical medical diagnosis has demonstrated immense value. Linning et al. established a radiomics classification model based on CT images using a random forest algorithm. Marya et al. used convolutional neural networks to differentiate PDAC from AIP in endoscopic ultrasound (EUS) images, achieving a sensitivity of 0.88 and a specificity of 0.88. Zhang et al. proposed a prognostic model based on risk scores using CT images of PDAC patients, fusing deep transfer learning and radiomics features. This model showed a significant improvement in the prognosis of resectable PDAC patients, with an AUC (Area Under Curve) of 0.84. In studies using PET / CT images to differentiate PDAC from AIP, Zhang et al. compared the maximum standardized uptake value (SUVmax) and FDG (Fluorodeoxyglucose) uptake lesion-related results using PET / CT images, finding that PET / CT results may help distinguish between PDAC and AIP. Cheng et al. first proposed using radiomics methods based on PET / CT images to differentiate between PDAC and AIP. They extracted texture features from PET images in 3D space and combined them with SUVmax, the number of extrapancreatic lesions, and the lesion's glucose analogue uptake performance to construct a logistic regression model, achieving an AUC of 0.95. Research on the fusion of radiomics and deep learning includes: Fu et al., who combined CNN features with handcrafted features to train a classifier and used support vector machines to achieve the final lung nodule classification; Zhu et al., who used a deep learning radiomics (DLR) model based on conventional magnetic resonance imaging for non-invasive differentiation of meningioma grading; and Wanga et al., who proposed a method combining radiomics and deep learning (RDL) to predict adenocarcinoma (ADC), with the fused model showing significantly better performance than radiomics alone. All these results indicate that the fusion of radiomics and deep learning features can effectively improve the accuracy of classification diagnosis and prognosis.

[0004] However, the traditional Faster R-CNN algorithm performs poorly in small object detection. Although some algorithms use multi-scale feature fusion, they generally use the fused features for prediction. In common two-level detection frameworks (such as Fast R-CNN, Faster R-CNN, and R-FCN), ROI Pooling is used to pool the corresponding regions in the feature map into fixed-size feature maps based on the position coordinates of the pre-selected boxes, so as to perform subsequent classification and bounding box regression operations. The ROI Pooling operation involves two quantization processes. After two quantizations, the candidate boxes at this point have a certain deviation from the positions initially regressed. This deviation will affect the accuracy of detection or segmentation.

[0005] Therefore, how to construct a diagnostic auxiliary model that integrates radiomics and deep learning to further improve the accuracy of distinguishing between PDAC and AIP is of practical research significance. Summary of the Invention

[0006] To address the shortcomings of the aforementioned technologies, this invention provides an automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT. By designing an improved detection network and multimodal feature fusion, it effectively combines the advantages of high contrast in PET images and high spatial resolution and sufficient structural information in CT images. This improves upon the problems of blurred tumor edges in single-modal PET images and low contrast between lesions and surrounding tissues in CT images. It can effectively mine the spatial and textural features of lesions, achieving fully automated lesion detection and disease classification without the need for doctor intervention, and also has high detection accuracy.

[0007] To achieve these objectives and other advantages according to the present invention, the present invention is implemented through the following technical solutions:

[0008] This invention provides an automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT, which includes the following steps:

[0009] Image preprocessing was performed on PET and CT images respectively;

[0010] Based on the Faster-R-CNN algorithm, an improved detection network is constructed by introducing an attention mechanism module into the residual network of ResNet50, adding an FPN network to form a feature pyramid network, and replacing the ROI Pooling layer with ROI Align. This network performs target detection on the pre-processed PET and CT images, outputting the lesion detection results. 18 F-FDG PET / CT images;

[0011] Construct a hybrid classification model for the aforementioned 18 F-FDG PET / CT images are subjected to radiomics feature extraction and deep learning feature extraction. The radiomics features and deep learning features are then subjected to multimodal feature fusion, multidomain feature fusion, and feature correlation analysis, respectively, to output lesion classification results.

[0012] Preferably, the image preprocessing of PET images and CT images respectively includes the following steps:

[0013] The PET image was resampled using bilinear interpolation;

[0014] The pixel values ​​of the CT image are converted into HU values, and the pixel values ​​of the resampled PET image are converted into SUV values.

[0015] The HU value threshold of the CT image is adjusted to -10≤HU≤100.

[0016] Preferably, the attention mechanism module includes a channel attention module and a spatial attention module connected in sequence;

[0017] The attention module mainly includes a global average pooling layer, a first fully connected layer, a first activation function layer, a second fully connected layer, and a second activation function layer connected in sequence. The global average pooling layer obtains global information, and then obtains a new weight by sequentially connecting the first fully connected layer, activating the first activation function layer, connecting the second fully connected layer, and activating the second activation function layer.

[0018] The spatial attention module mainly includes an average pooling layer, a max pooling layer, a shearing layer, a convolutional layer, and a third activation function layer; the average pooling layer and max pooling layer are respectively connected to the second activation function layer, and the output is sequentially processed by shearing and splicing of the shearing layer, convolution of the convolutional layer, and activation of the third activation function layer.

[0019] Preferably, the construction of the hybrid classification model includes the following steps:

[0020] We used the Pyradiomics open-source code in Python to extract radiomics features; we used the VGG11 network to extract deep learning features, built a dual-branch network model, and trained the network on preprocessed PET and CT images respectively.

[0021] The radiomics features and PET image features are fused at a fully connected layer; the deep learning features and CT image features and PET image features are fused at multiple scales; and the radiomics features and deep learning features are fused at a fully connected layer across multiple domains to form... 18 Multimodal characteristics of F-FDG PET / CT;

[0022] A first classification prediction model, a second classification prediction model, and a third classification prediction model are set up. The first classification prediction model is used to classify CT image features and PET image features extracted from the radiomics features. The second classification prediction model extracts high-level semantic features of CT images and PET images through a VGG11 network for classification. The third classification prediction model is used to classify the high-level semantic features of CT images and PET images extracted from the radiomics features. 18 F-FDG PET / CT multi-domain features are used for classification.

[0023] Preferably, the initial framework of the VGG11 network includes 5 convolutional modules, and the convolutional layers in each convolutional module use 3*3 convolutional kernels.

[0024] Preferably, the image omics features and the deep learning features are fused using multimodal features, including the following steps:

[0025] Feature fusion is performed on CT image features and PET image features from radiomics features;

[0026] Multi-scale feature fusion of deep learning CT and PET image features: When training the network separately on the preprocessed PET and CT images, feature maps generated by each convolutional module in the VGG11 network are extracted; the PET feature maps and CT feature maps are superimposed and fed into the convolutional layers at the corresponding positions. The image features at different positions are weighted to form a spatially varying PET / CT fusion map, resulting in a multi-scale integrated hybrid deep learning feature.

[0027] Preferably, the extracted image omics features and the deep learning features are fused using a multi-domain feature fusion method, including the following steps:

[0028] The 18 The radiomics features extracted from F-FDG PET / CT images are fused with deep learning features in a fully connected layer to form multi-domain features for PET / CT. The resulting multi-domain features of PET / CT are then input into linear blocks for classification.

[0029] Preferably, the feature correlation analysis includes the following steps:

[0030] The image omics features are classified according to statistical features and then combined with the deep learning features to obtain a preliminary mixed feature set including preliminary mixed features;

[0031] The results of classifying the radiomics features according to the statistical features are compared with the preliminary mixed feature set. Based on the comparison results, the weights of the contributions of the features in the preliminary mixed feature set to the mixed classification model are determined, thereby obtaining the adjusted mixed feature set including the final mixed features.

[0032] The statistical features include texture features, histogram features, and morphological features.

[0033] The present invention has at least the following beneficial effects:

[0034] 1. The automatic auxiliary diagnosis method for pancreatic cancer and autoimmune pancreatitis based on PET-CT provided by this invention introduces an attention mechanism module into the residual network of ResNet50 based on the Faster-Rcnn algorithm, adds an FPN network to form a feature pyramid network, and replaces the ROI Pooling layer with ROI Align to design an improved detection network. It also designs multimodal feature fusion, which effectively combines the advantages of high contrast of PET images and high spatial resolution and sufficient structural information of CT images. It improves the problems of blurred tumor edges in single-modal PET images and low contrast between lesions and surrounding tissues in CT images. It can effectively mine the spatial and texture features of lesions, realize the full automation of lesion detection and disease classification without the need for doctor intervention, and has high detection accuracy.

[0035] 2. This invention performs image preprocessing on PET and CT images respectively, including a process of resampling PET images using bilinear interpolation, a pixel value conversion process of converting the pixel values ​​of the CT image into HU values ​​and converting the pixel values ​​of the resampled PET image into SUV values, and a threshold adjustment process of adjusting the HU value threshold of the CT image to -10≤HU≤100. Resampling is used because the difference in spatial resolution between PET and CT images is not conducive to lesion localization and extraction of local features. The step of resampling PET images using bilinear interpolation in step S11 makes its spatial resolution consistent with that of the CT image. After pixel value conversion, digital images can be further correlated with clinical indicators. Threshold adjustment filters artifacts in CT images, thereby reducing the interference of fat, bone tissue and other factors on texture features.

[0036] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0037] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 A schematic flowchart illustrating the automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT provided by this invention;

[0039] Figure 2This is a schematic diagram illustrating the image preprocessing process for PET and CT images provided by the present invention.

[0040] Figure 3 A schematic diagram illustrating the composition of the attention mechanism module provided by this invention;

[0041] Figure 4 A schematic diagram illustrating the process of constructing a hybrid classification model provided by this invention;

[0042] Figure 5 This is a schematic diagram of the multimodal feature fusion method provided by the present invention, which involves fusing image omics features and deep learning features separately.

[0043] Figure 6 This is a schematic diagram of the feature correlation analysis method provided by the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] The terms such as “having,” “comprising,” and “including” used in the different embodiments of the present invention described below do not exclude the presence or addition of one or more other elements or combinations thereof; the technical features involved may be combined with each other as long as they do not conflict with each other.

[0046] like Figure 1 As shown, this invention provides an automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT, which includes steps S10, S20 and S30.

[0047] S10, Data Preprocessing;

[0048] S20, Target Detection;

[0049] S30, Feature Extraction and Classification.

[0050] Specifically, the data preprocessing in step S10 includes image preprocessing of the PET image and the CT image respectively.

[0051] Target detection in step S20 includes steps S21 and S22:

[0052] S21, based on the Faster-Rcnn algorithm, improves the detection network by introducing an attention mechanism module into the residual network of ResNet50, adding an FPN network to form a feature pyramid network, and replacing the ROI Pooling layer with ROI Align.

[0053] S22 performs target detection on the preprocessed PET and CT images respectively, and outputs the lesion detection results. 18 F-FDG PET / CT image.

[0054] Step S30, feature extraction and classification, includes the following steps: constructing a hybrid classification model, for... 18 F-FDG PET / CT images are subjected to radiomics feature extraction and deep learning feature extraction. The radiomics features and deep learning features are then subjected to multimodal feature fusion, multidomain feature fusion, and feature correlation analysis, respectively, to output lesion classification results.

[0055] In this embodiment, the data preprocessing in step S10 is necessary because different modalities of medical images have different imaging principles, and their resolution and signal-to-noise ratio also differ accordingly. Therefore, image preprocessing is required first. By performing image preprocessing on PET and CT images respectively, the detectability of useful information in the images can be enhanced, and factors that are detrimental to subsequent tasks can be eliminated.

[0056] In step S21, the FPN network is added to form a feature pyramid network because pancreatic cancer has many small lesions. The traditional Faster R-CNN algorithm performs poorly in small target detection. Most target detection algorithms only use top-level features for prediction, resulting in a coarse target location. In addition, although some algorithms use multi-scale feature fusion, they generally use the fused features for prediction. The difference with FPN is that prediction is performed independently at different feature layers. Multi-scale fusion improves the accuracy of small target detection.

[0057] In step S21, ROI Align is used instead of ROI Pooling because, in common two-level detection frameworks (such as Fast R-CNN, Faster R-CNN, R-FCN), ROI Pooling pools the corresponding regions in the feature map into fixed-size feature maps based on the position coordinates of the pre-selected boxes, so as to perform subsequent classification and bounding box regression operations. This ROI Pooling operation involves two quantization processes. After two quantizations, the candidate boxes at this point have a certain deviation from the positions initially regressed. This deviation will affect the accuracy of detection or segmentation. ROI Align cancels the quantization operation and uses bilinear interpolation to obtain the image values ​​of the pixels with floating-point coordinates, thereby transforming the entire feature aggregation process into a continuous operation, which greatly avoids the occurrence of errors.

[0058] In step S21, an improved detection network is constructed because the image imaging principles are significantly different, and different modal images have complex relationships. Effectively learning the complementary information between different modalities is crucial. This improved detection network is based on the Faster-R-CNN algorithm, introduces an attention mechanism module into the ResNet50 residual network, adds an FPN network to form a feature pyramid network, and replaces the ROI Pooling layer with ROI Align, which is more conducive to achieving effective feature fusion.

[0059] In step S30, for 18 F-FDG PET / CT images are subjected to radiomics feature extraction and deep learning feature extraction. Multimodal and multi-domain feature fusions are performed on both radiomics and deep learning features, effectively combining the advantages of high contrast in PET images and high spatial resolution and abundant structural information in CT images. This improves upon the problems of blurred tumor edges in single-modal PET images and low contrast between lesions and surrounding tissues in CT images, effectively uncovering the spatial and textural features of lesions. The improved detection network provided in this invention demonstrates relatively accurate localization in these samples, with correspondingly high detection precision, and a detection time of 0.02 seconds per image, significantly shorter than the diagnostic time of radiologists. This proves that it can effectively assist doctors in making a good diagnosis of lesions, even small ones, improving diagnostic efficiency while mitigating the differences in doctors' experience to some extent. This also verifies the significance of the multimodal PET / CT pancreatic lesion detection algorithm.

[0060] In summary, the present invention provides an automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT. On the one hand, by improving the classic Faster-R-CNN dual-feature extraction network and introducing attention, feature pyramid networks, and ROI Align methods, an improved detection network is formed, which achieves automatic lesion detection while improving the accuracy of lesion detection. On the other hand, by designing multimodal feature fusion, the advantages of high contrast in PET images and high spatial resolution and sufficient structural information in CT images are well combined, improving the problems of blurred tumor edges in single-modal PET images and low contrast between lesions and surrounding tissues in CT images. It can effectively mine the spatial and textural features of lesions, complete the disease classification of lesions, and form an end-to-end automated auxiliary diagnostic function. From lesion detection to final classification, automation is achieved without doctor intervention, and the automatic detection accuracy is also high.

[0061] As a further preferred embodiment of the above implementation, image preprocessing is performed on the PET images and CT images respectively, such as... Figure 2 As shown, the process includes steps S11, S12, and S13. S11 is resampling; S12 is pixel value conversion; and S13 is threshold adjustment.

[0062] In step S11, resampling specifically includes the step of resampling the PET image using bilinear interpolation. Resampling is necessary because the difference in spatial resolution between PET and CT images is not conducive to lesion localization and extraction of local features. By resampling the PET image using bilinear interpolation in step S11, the spatial resolution of the PET image is made consistent with that of the CT image.

[0063] Step S12 specifically includes the following steps: converting the pixel values ​​of the CT image into HU values, and converting the pixel values ​​of the resampled PET image into SUV values. After pixel value conversion, the digital image can be further correlated with clinical indicators.

[0064] In step S13, the threshold adjustment specifically includes the following steps: adjusting the HU value threshold of the CT image to -10≤HU≤100. This step filters out artifacts in the CT image, thereby reducing the interference of fat, bone tissue, and other factors on texture features.

[0065] As a further preferred embodiment of the above implementation, in order to further ensure the effectiveness of multimodal feature information fusion, specifically, the attention mechanism module includes a channel attention module and a spatial attention module connected in sequence, such as... Figure 3As shown, the channel attention module mainly consists of a globally average pooling layer, a first fully connected layer (FC), a first activation function layer (ReLU), a second fully connected layer (FC), and a second activation function layer (Sigmoid), connected sequentially. The globally average pooling layer first acquires global information, and then, through the sequential connections of the first fully connected layer, activation of the first activation function layer, connections of the second fully connected layer, and activation of the second activation function layer, a new weight is obtained. This new weight uses global information to selectively select desired information features and suppress less desirable features. The spatial attention module mainly consists of an average pooling layer (Avg Pooling), a max pooling layer (Max Pooling), a pruning layer (Cat), a convolutional layer (Conv), and a third activation function layer (Sigmoid), connected to the average pooling layer and max pooling layer of the second activation function layer, respectively. The output is then processed through pruning and concatenation by the pruning layer, convolution by the convolutional layer, and activation by the third activation function layer. Building upon the channel attention module, the spatial attention module finds the location with the most key information based on the channel direction, thus complementing the channel attention module. This implementation method sets up an attention mechanism module, which includes two feature extraction networks connected in sequence: a channel attention module and a spatial attention module. The two feature extraction networks extract features from PET and CT images respectively, and then stitch them together to achieve effective fusion of multimodal feature information, which is then input into the subsequent network to complete lesion detection.

[0066] As a further preferred embodiment of the above implementation, the high-level semantic information extracted by deep learning and the statistical features obtained by radiomics methods are fused to form a complementary relationship, which can provide beneficial assistance for the diagnosis and prognosis of related diseases. The feature extraction and classification prediction process is as follows: first, manual features of the image are extracted; then, deep learning extracts multi-scale spatial features; a fusion strategy is used to construct a new feature set, which is then input into a classifier to complete the classification task. Therefore, in order to better complete the classification detection after multimodal feature fusion, such as... Figure 4 As shown, constructing a hybrid classification model preferably includes the following steps:

[0067] S31, Feature Extraction;

[0068] S32, Feature Fusion;

[0069] S33, Feature classification prediction.

[0070] The feature extraction in step S31 specifically includes the following steps: extracting radiomics features using the Pyradiomics open-source code in Python; extracting deep learning features using the VGG11 network, building a dual-branch network model, and simultaneously training the network on the preprocessed PET and CT images respectively.

[0071] Step S32, feature fusion, specifically includes the following steps: fusing CT image features and PET image features from radiomics features in a fully connected layer; fusing CT image features and PET image features from deep learning features in a multi-scale manner; and fusing radiomics features and deep learning features in a multi-domain manner in a fully connected layer, forming... 18 Multimodal characteristics of F-FDG PET / CT;

[0072] Step S33, feature classification prediction, specifically includes the following steps: setting up a first classification prediction model, a second classification prediction model, and a third classification prediction model. The first classification prediction model is used to classify CT image features and PET image features extracted from radiomics features. The second classification prediction model extracts high-level semantic features of CT images and PET images through a VGG11 network for classification. The third classification prediction model is used to classify... 18 F-FDG PET / CT multi-domain features are used for classification.

[0073] In this embodiment, 18 Multimodal features of F-FDG PET / CT images include two sets of features: radiomics features and deep learning features. These are extracted in step S31: Radiomics features are extracted using the Pyradiomics open-source code in Python, based on PET images, CT images, and ROI masks, with 102 statistical features extracted from each of the original images; deep learning features are extracted using the VGG11 network, building a dual-branch network model, and training the network on both preprocessed PET and CT images. Further optimization involves fine-tuning the VGG11 network, setting its initial framework to include 5 convolutional modules, each with a 3x3 convolutional kernel. Feature extraction is performed using the pre-trained, fine-tuned VGG11 network, extracting 4096 one-dimensional features from both CT and PET images.

[0074] As a further preferred embodiment of the above implementation, in step S32, the image omics features and deep learning features are respectively subjected to multimodal feature fusion, such as... Figure 5 As shown, it also includes the following steps:

[0075] S321, which integrates CT image features and PET image features from radiomics features;

[0076] S322, multi-scale feature fusion of CT image features and PET image features in deep learning features: when training the network separately on the preprocessed PET image and CT image, the feature map generated by each convolutional module in the VGG11 network is extracted; the PET feature map and CT feature map are superimposed and fed into the convolutional layer at the corresponding position to form a spatially varying PET / CT fusion map, and a multi-scale integrated hybrid deep learning feature is obtained.

[0077] This implementation primarily provides a deep learning multi-scale feature fusion method because feature images at different spatial scales represent the differences in expression between different modalities. Fully utilizing this difference information is expected to improve the analytical performance of classification models. CT images provide more discriminative anatomical information, reflecting the contour information between the lesion area and surrounding blood vessels and organs, while PET images reflect the metabolic level of the lesion. The multi-spatial feature fusion network combines the advantages of both CT and PET images, fusing features from images at different spatial scales. This network will effectively learn the spatial variation information between different modalities, and by integrating multi-dimensional information from CT and PET, it is beneficial to improve the accuracy and generalization ability of the model classification, potentially providing new reference value for disease classification. Therefore, a multi-spatial feature fusion strategy is adopted to achieve the fusion and connection of PET and CT images from different spatial layers. Specifically, the deep learning multi-spatial feature fusion method extracts feature maps generated by each convolutional module in the VGG11 network when training the network separately on preprocessed PET and CT images. The PET feature maps and CT feature maps are then superimposed and fed into the convolutional layers at the corresponding positions. The image features at different positions are weighted to form a spatially varying PET / CT fusion map, resulting in a hybrid deep learning feature map that integrates multiple scales.

[0078] As a further preferred embodiment of the above implementation, step S32, which involves multi-domain feature fusion of the extracted image omics features and deep learning features, also includes the following steps:

[0079] Will 18 The radiomics features extracted from F-FDG PET / CT images are fused with deep learning features in a fully connected layer to form multi-domain features for PET / CT. The resulting multi-domain features of PET / CT are then input into linear blocks for classification.

[0080] As a further preferred embodiment of the above implementation, in step S32, feature correlation analysis, such as... Figure 6 As shown, it also includes the following steps:

[0081] S323, after classifying the radiomics features according to statistical features, they are combined with deep learning features to obtain a preliminary mixed feature set including preliminary mixed features;

[0082] S324, compare the results of classifying radiomics features according to statistical features with the preliminary mixed feature set, and determine the weights of the contribution of the features in the preliminary mixed feature set to the mixed classification model based on the comparison results, so as to obtain the adjusted mixed feature set including the final mixed features; among which, statistical features include texture features, histogram features, and morphological features.

[0083] This implementation method categorizes radiomics features and then fuses them with deep learning to form comparative experiments. Based on the differences in experimental results, the weights of each feature are assigned to create a new hybrid feature set that demonstrates the best classification performance. Meanwhile, the features extracted by deep learning are abstract and lack intuitiveness and interpretability; however, the information attributes of deep learning features can be inferred by utilizing the differences in combinations. If the abstract features extracted by Faster-RCNN contain classification-related information, the overall fusion will result in redundant information or negative correlations, leading to a decrease in results. However, fusion with deep learning features improves the model's performance, indicating a complementary trend, possibly representing information missing from the abstract features. Thus, some characteristics of deep learning features can be obtained. Specifically, statistical features include texture features, histogram features, and morphological features. Texture features include five subcategories. These features are categorized and combined with deep learning features to form a feature set (including preliminary hybrid features) that fuses multiple radiomics features with deep learning features, and the differences in classification results are analyzed. Weights are assigned based on the contribution of different features to the model, ultimately resulting in an adjusted hybrid feature set (including the final hybrid features). Furthermore, based on the characteristic performance of different combinations, we can analyze the correlation between radiomics and deep learning features, infer the attributes of information in deep learning, and increase the interpretability of abstract high-level semantic features.

[0084] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT, characterized in that, It includes the following steps: Image preprocessing was performed on PET and CT images respectively; Based on the Faster-R-CNN algorithm, an improved detection network is constructed by introducing an attention mechanism module into the residual network of ResNet50, adding an FPN network to form a feature pyramid network, and replacing the ROI Pooling layer with ROI Align. This network performs target detection on the pre-processed PET and CT images, outputting the lesion detection results. 18 F-FDG PET / CT images; Construct a hybrid classification model for the aforementioned 18 F-FDG PET / CT images are subjected to radiomics feature extraction and deep learning feature extraction. The radiomics features and deep learning features are then subjected to multimodal feature fusion, multidomain feature fusion, and feature correlation analysis, respectively, to output lesion classification results. The construction of the hybrid classification model includes the following steps: We used the Pyradiomics open-source code in Python to extract radiomics features; we used the VGG11 network to extract deep learning features, built a dual-branch network model, and trained the network on preprocessed PET and CT images respectively. The radiomics features and PET image features are fused at a fully connected layer; the deep learning features and CT image features and PET image features are fused at multiple scales; and the radiomics features and deep learning features are fused at a fully connected layer across multiple domains to form... 18 Multimodal characteristics of F-FDG PET / CT; A first classification prediction model, a second classification prediction model, and a third classification prediction model are set up. The first classification prediction model is used to classify CT image features and PET image features extracted from the radiomics features. The second classification prediction model extracts high-level semantic features of CT images and PET images through a VGG11 network for classification. The third classification prediction model is used to classify the high-level semantic features of CT images and PET images extracted from the radiomics features. 18 F-FDG PET / CT multi-domain features are used for classification.

2. The automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT as described in claim 1, characterized in that, The image preprocessing of PET and CT images respectively includes the following steps: The PET image was resampled using bilinear interpolation; The pixel values ​​of the CT image are converted into HU values, and the pixel values ​​of the resampled PET image are converted into SUV values. The HU value threshold of the CT image is adjusted to -10≤HU≤100.

3. The automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT as described in claim 1, characterized in that, The attention mechanism module includes a channel attention module and a spatial attention module connected in sequence; The channel attention module mainly includes a global average pooling layer, a first fully connected layer, a first activation function layer, a second fully connected layer, and a second activation function layer connected in sequence. The global average pooling layer obtains global information, and then obtains a new weight by sequentially connecting the first fully connected layer, activating the first activation function layer, connecting the second fully connected layer, and activating the second activation function layer. The spatial attention module mainly includes an average pooling layer, a max pooling layer, a shearing layer, a convolutional layer, and a third activation function layer; the average pooling layer and max pooling layer are respectively connected to the second activation function layer, and the output is sequentially processed by shearing and splicing of the shearing layer, convolution of the convolutional layer, and activation of the third activation function layer.

4. The automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT as described in claim 1, characterized in that, The initial framework of the VGG11 network consists of 5 convolutional modules, each with a 3x3 convolutional kernel in its convolutional layer.

5. The automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT as described in claim 4, characterized in that, The image omics features and the deep learning features are fused using multimodal features, including the following steps: The CT image features and PET image features in the radiomics features are fused together; The CT image features and PET image features in the deep learning features are fused at multiple scales: When training the network on the preprocessed PET image and CT image respectively, the feature map generated by each convolutional module in the VGG11 network is extracted; the PET feature map and CT feature map are superimposed and fed into the convolutional layer at the corresponding position. The image features at different positions are weighted to form a spatially varying PET / CT fusion map, resulting in a hybrid deep learning feature that integrates multiple scales.

6. The automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT as described in claim 4, characterized in that, The extracted image omics features and the deep learning features are fused using multi-domain features, including the following steps: The 18 The radiomics features extracted from F-FDG PET / CT images are fused with deep learning features in a fully connected layer to form multi-domain features for PET / CT. The resulting multi-domain features of PET / CT are then input into linear blocks for classification.

7. The automated auxiliary diagnostic method for pancreatic cancer and autoimmune pancreatitis based on PET-CT as described in claim 4, characterized in that, The feature correlation analysis includes the following steps: The image omics features are classified according to statistical features and then combined with the deep learning features to obtain a preliminary mixed feature set including preliminary mixed features; The results of classifying the radiomics features according to the statistical features are compared with the preliminary mixed feature set. Based on the comparison results, the weights of the contributions of the features in the preliminary mixed feature set to the mixed classification model are determined, thereby obtaining the adjusted mixed feature set including the final mixed features. The statistical features include texture features, histogram features, and morphological features.

Images