Pancreatic ductal cancer prognosis survival analysis method based on double-branch adaptive network

Abstract

The invention discloses a pancreatic ductal cancer prognosis survival analysis method based on a double-branch adaptive network, and the method comprises the steps: firstly, employing a multi-center clinical abdominal CT image as a data set, carrying out the preprocessing of an abdominal CT, carrying out the coarse segmentation, and achieving the positioning of a pancreas and a pancreatic tumor; the method comprises the following steps: constructing pancreas-pancreatic tumor parallel branches by using a segmentation network to obtain anatomical characteristics of the pancreas and the pancreatic tumor, then performing feature extraction on the double branches by using an encoder, deeply extracting anatomical characteristics of the pancreas and the pancreatic tumor by using a dual-stage pre-training-fine tuning adaptive training method in a feature extraction stage, and extracting the anatomical characteristics of the pancreas and the pancreatic tumor. Finally, the spatial position relationship between the pancreas and the pancreatic tumor plays a crucial role in prognosis analysis. A fusion module under branch guidance is used for comprehensively capturing the complex and diversified spatial position relative relation of pancreas and pancreatic tumors, the output is input into a prognosis network, and a final pancreatic ductal cancer prognosis survival analysis result is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, relates to the prognostic survival analysis of pancreatic ductal carcinoma, and particularly relates to a dual-branch adaptive network for assisting in the prognostic survival analysis of patients with pancreatic ductal carcinoma for diagnosis. Background Art

[0002] The pancreas is one of the most important organs in the digestive system. It can secrete hormones and digestive enzymes commonly used in the digestive process and regulate the blood sugar level of the human body, thus assisting the entire digestive process. Pancreatic ductal carcinoma is a malignant tumor originating from pancreatic duct epithelial cells and is considered the most malignant type of pancreatic cancer in clinical practice. It has characteristics such as insidious onset, rapid progression, and extremely fatal prognosis, and is known as the "king of cancers" in the field of oncology. In order to better assist medical staff in dynamically grasping the disease course development of patients with pancreatic ductal carcinoma and minimizing the pain brought by pancreatic ductal carcinoma to patients, the prognostic survival analysis of pancreatic ductal carcinoma plays an important role.

[0003] Abdominal CT is one of the commonly used diagnostic tools for pancreatic ductal carcinoma. As a non-invasive, fast, and widely available imaging modality, it plays an important role in the examination of pancreatic ductal carcinoma. The CT image contains various anatomical characteristics such as the gray scale, texture, and shape of the pancreas and pancreatic tumor, which make the CT image have great potential in the prognostic analysis of pancreatic ductal carcinoma. Based on the clinical characteristics of the pancreas and pancreatic tumor in the CT image for prognostic analysis, patients are divided into high-risk groups and low-risk groups. Medical staff can adopt personalized treatment means for patients of different risk types based on this, thereby reducing the pain brought by pancreatic ductal carcinoma to patients to a certain extent. Currently, manually extracting the clinical anatomical characteristics of pancreatic ductal carcinoma in CT images is the main prognostic method, but this method highly depends on the experience and subjective judgment of clinicians, with low efficiency. At the same time, some characteristics of the pancreas and pancreatic tumor are relatively hidden, and their spatial relationship is complex, posing challenges to the clinical diagnosis of pancreatic ductal carcinoma.

[0004] To address the challenges brought by the self-anatomical characteristics of the pancreas and pancreatic tumor, deep learning-based prognostic survival analysis algorithms for pancreatic ductal carcinoma are constantly evolving. Relevant introductions are disclosed in the prior art, and a model for predicting the survival period and tumor subtype of PDAC is established by analyzing the preoperative imaging data of PDAC patients.

[0005] Parameters such as the maximum standardized uptake value, mean standardized uptake value, metabolic tumor volume, and lesion glycolysis amount of tumors in PDAC patients are studied to achieve prognostic analysis. However, the characteristics of pancreatic tumors, such as their growth direction, growth rate, and internal texture in CT images, are relatively hidden. The anatomical characteristics that affect the prognostic analysis of pancreatic ductal carcinoma in different patients are complex and diverse, and the relative spatial position relationship between the pancreas and pancreatic tumors is complex and variable, which limits the performance of the prognostic effect. Therefore, it is crucial to further improve the prognostic survival analysis of pancreatic ductal carcinoma to meet the clinical needs for the prognosis of pancreatic ductal carcinoma. Summary of the Invention

[0006] Object of the Invention: The present invention aims to provide a prognostic survival analysis method for pancreatic ductal carcinoma based on a dual-branch adaptive network. First, rough segmentation is used to locate the pancreas and pancreatic tumors. On this basis, a parallel branch structure of the pancreas-pancreatic tumor is constructed using a segmentation network. Based on the segmentation results and segmentation features, the target region and its neighborhood information are fully utilized to obtain the anatomical characteristics of the pancreas and pancreatic tumors. Then, encoders are respectively used to extract features from the two branches. In the feature extraction stage, a two-stage pre-training-fine-tuning adaptive training method is used to enhance the learning ability of the prognostic model for complex anatomical characteristics. Finally, a fusion module guided by the branches comprehensively captures the complex and diverse relative spatial position relationships between the pancreas and pancreatic tumors, and the output is input into the prognostic network to obtain the final prognostic survival analysis result of pancreatic ductal carcinoma.

[0007] To achieve the above object, the present invention provides a prognostic survival analysis method for pancreatic ductal carcinoma based on a dual-branch adaptive network, including the following steps:

[0008] S1: Preprocessing: Uniformly preprocess the multi-center clinical abdominal CT dataset, including: window width and window level adjustment, histogram equalization, Sobel operator, data augmentation;

[0009] S2: Roughly segment the preprocessed CT image slices in step S1 to obtain the approximate position information of the pancreas and pancreatic tumors;

[0010] S3: According to the rough segmentation result in step S2, crop the CT image, and use a segmentation network to construct a parallel branch structure of the pancreas-pancreatic tumor, including: pancreas region image, pancreas region features, pancreatic tumor region image, and pancreatic tumor region features, so as to fully utilize the target region and its neighborhood information of the pancreas and pancreatic tumors;

[0011] S4: Extract features according to the dual-branch input in step S3, including: pancreas encoder and pancreatic tumor encoder;

[0012] S5: Apply an adaptive training method of two-stage pre-training and fine-tuning to the dual-branch features in step S4. The first stage is the pre-training stage, and the second stage is the fine-tuning and adaptation stage. The two stages are trained serially to deeply capture the anatomical features of the pancreas and pancreatic tumors, and enhance the learning ability of the prognosis model for complex anatomical characteristics;

[0013] S6: Fuse the dual-branch features in step S5, and use the fusion module guided by branches to comprehensively capture the complex and diverse relative spatial position relationships of the pancreas and pancreatic tumors from the multi-scale feature extraction module and the multi-view information fusion module;

[0014] S7: Input the dual-branch fusion features in step S6 into the prognosis network to obtain the final prognosis survival analysis result of pancreatic ductal carcinoma.

[0015] As an improvement of the present invention, perform unified preprocessing on the multi-center clinical abdominal CT dataset, including: (1) Window width and window level adjustment: According to the clinical prior information of the pancreas and pancreatic tumors, set the window width of the abdominal CT image to 200 HU and the window level to 100 HU, and normalize the image to constrain the range of voxel values to [0, 1]. (2) Histogram equalization: Change the gray levels of each pixel in the image by changing the histogram of the image to improve the contrast of pancreatic tumors in the CT image. (3) Sobel operator: Detect the edges of the pancreas and pancreatic tumors to make the contrast between the edges and adipose tissue more obvious. (4) Data augmentation: Rotate the CT image at 90°, 180°, and 270° respectively, and perform image scaling to achieve data augmentation.

[0016] As an improvement of the present invention, the rough segmentation process in step S2 includes the following steps: Based on the position information of pancreatic tumors in the clinical prior, crop the CT slices. The corresponding coordinates of the cropped area are length: 100 - 400, width: 120 - 480 (both in pixels). Use the segmentation network to segment the image. The encoder and decoder of the network contain four convolutional layers, which are composed of 2D convolutional layers of 3X3, normalization layers of Batch Normalization, and activation layers with the activation function Relu. The encoder adds a max-pooling layer at the end of each layer, and the decoder adds an upsampling layer at the end of each layer to obtain the rough segmentation result. By performing rough segmentation on the CT image, the influence brought by the redundant spatial environment to the prognosis analysis can be effectively alleviated.

[0017] As an improvement of the present invention, a parallel branch structure of pancreas-pancreatic tumor is constructed based on a segmentation network. First, the CT image is cropped in the region centered on the tumor according to the rough segmentation result in step S2. The cropping range is: length: -72 to 72, width: -56 to 56, height: -88 to 88 (in voxels), and the image of the pancreas and its surrounding tissues is obtained by cropping. , and then we use the segmentation network to segment the image. The encoder and decoder of the network contain four convolutional layers, which are composed of 3D convolutional layers of 5X5, normalization layers of Batch Normalization, and activation layers with the activation function Relu. The encoder adds a max pooling layer at the end of each layer, and the decoder adds an upsampling layer at the end of each layer to obtain the segmentation results of the pancreas and pancreatic tumor. and . Based on this, we construct the target regions of the pancreas and pancreatic tumor. We use the segmentation result to obtain the CT region image containing only the pancreas and pancreatic tumor. and . In order to obtain the neighborhood information containing the pancreas and pancreatic tumor, we use the features of the third-layer decoder in the segmentation network as another input. and . Therefore, in the pancreas branch, we construct the pancreatic CT region image and the pancreatic domain features as inputs. In the pancreatic tumor branch, we construct the pancreatic tumor CT region image and the pancreatic domain features as inputs. This step can effectively integrate the information in the spatial microenvironment of the pancreas and pancreatic tumor, thereby improving the performance of the prognosis network to a certain extent.

[0018] As an improvement of the present invention, feature extraction is performed on the parallel branch of pancreas-pancreatic tumor. First, the dual inputs of the pancreas and pancreatic tumor are combined, and the specific implementation formula is as follows:

[0019]

[0020] where and are the final inputs of the combination of the CT image and features of the pancreas and pancreatic tumor, and C represents the concatenation operation. Then, feature extraction is performed on and respectively. The feature extractor consists of an encoding-decoding structure, which contains four convolutional layers, which are composed of 3D convolutional layers of 5X5, normalization layers of Batch Normalization, and activation layers with the activation function Relu. Max pooling layers are added at the end of each layer during encoding, and upsampling layers are added at the end of each layer during decoding. Finally, the dual-branch features and It provides rich semantic information for subsequent information fusion of the dual branches.

[0021] As an improvement of the present invention, we adopt a pre-training - fine-tuning adaptive training method for the dual-branch feature extraction stage. The first stage is the pre-training stage, and the second stage is the fine-tuning adaptive stage. In the first stage, we pre-train the two branches of the pancreas and pancreatic tumor respectively, and use the model parameters obtained from the pre-training and to initialize the model parameters of the dual-branch feature extraction stage. In the second stage, the model parameters of the feature extraction are updated as the model training progresses on the basis of the pre-training weights. The specific weight change formula is as follows:

[0022]

[0023] where is the weight matrix for fine-tuning the pre-training parameters. By adding it to the pre-training weight matrix, the final weight matrix and is obtained. In order to enable the prognosis network to better process the complex and diverse features of the pancreas and pancreatic tumor, we perform an adaptive decomposition on the fine-tuning weight matrix, decomposing it into two low-rank matrices A and B, thus simplifying the process of parameter fine-tuning. The specific implementation formula is as follows:

[0024]

[0025] where , , , d is the rank of the updated parameter matrix. A and B are the low-rank matrices after decomposition. n and m are the number of rows and columns of the parameter matrix respectively. d is the rank of the parameter matrix. R represents the set of real numbers. and respectively represent the weight matrices of the pancreas and pancreatic tumor after the pre-training update. In the process of decomposing the fine-tuning weight matrix into and two matrices, the matrix is initialized with a random Gaussian distribution, providing a more balanced starting point for the training of the network. This step more accurately extracts the anatomical features of the pancreas and pancreatic tumor from the CT images and enhances the learning ability of the prognosis model for complex anatomical characteristics. Through pre-training, the model can achieve a preliminary localization of the target area in the CT images and features. Through the fine-tuning adaptive stage, the further refinement of the detailed features of the pancreas and pancreatic tumor is realized.

[0026] As an improvement of the present invention, the dual-branch features are fused. The fusion module guided by the branches includes two parts: a multi-scale feature extraction module and a multi-view information fusion module; the dual-branch features are fused. The fusion module guided by the branches includes two parts: a multi-scale feature extraction module and a multi-view information fusion module. Multi-scale feature extraction module: This module respectively performs multi-scale extraction on the features of the dual branches, uses dilated convolutions with different dilation rates to construct four parallel feature groups, so as to obtain feature information under different receptive fields. First, four parallel feature groups are constructed, and the specific implementation formula is as follows: and are subjected to multi-scale extraction, and dilated convolutions with different dilation rates are used to construct four parallel feature groups, so as to obtain feature information under different receptive fields. First, four parallel feature groups are constructed, and the specific implementation formula is as follows:

[0027]

[0028] where and represent the convolutional features of the pancreas and the tumor at a dilation rate of , and represent the features obtained after weighting the dilated convolutional features of the pancreas and the tumor by the Sigmoid function. Among them, The dilation rate can take values of 1, 2, 3, and 5. Therefore, represents the four parallel feature groups of the pancreas, represents the four parallel feature groups of the tumor, represents the Sigmoid function. Next, the multi-scale features will be aggregated, and the specific implementation formula is as follows:

[0029]

[0030] where represents the concatenation operation, and are the output results of the multi-scale feature extraction module.

[0031] Multi-view information fusion module: Through the above multi-scale feature extraction module, the multi-scale features of the pancreas and pancreatic tumor are respectively obtained and . Based on this, multi-view information fusion is performed. First, the multi-scale features of the same layer from the pancreas and pancreatic tumor are aggregated together, and based on this, the dual-branch features are fused. Then, through max-pooling and average-pooling, the key information in the feature channels can be captured from multiple perspectives. Specifically, a multi-layer perception mechanism is used together with average-pooling and max-pooling, and the Sigmoid function is applied to calculate the channel attention coefficient K. Then, the attention coefficient is multiplied by the features to obtain a reweighted feature vector. Finally, these reweighted features are residually connected with the multi-scale pancreas and tumor features to obtain the final fused features , and the specific implementation formula is as follows:

[0032]

[0033]

[0034]

[0035] Where represents the multi-layer perception mechanism, represents the max pooling layer, represents the average pooling layer, represents the Sigmoid function, represents the concatenation operation, represents the feature after concatenating the multi-scale features of the pancreas and pancreatic tumors together. The multi-view information fusion module further fuses the pancreas and pancreatic tumors in the multi-scale features containing spatial information, so that the relative spatial position relationship is further characterized.

[0036] As an improvement of the present invention, the features after double-branch fusion are input into the prognosis network for prognosis survival analysis. The prognosis network includes three fully connected layers, and DropOut is added after the first two layers to obtain the final prognosis result.

[0037] Through the above mechanism, we obtain the prognosis survival analysis result of pancreatic ductal carcinoma.

[0038] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0039] 1. The present invention makes full use of the anatomical characteristics and spatial relationships of the pancreas and pancreatic tumors, and proposes a method for prognosis survival analysis of pancreatic ductal carcinoma based on a double-branch adaptive network;

[0040] 2. The present invention fully captures the anatomical characteristics and neighborhood information of the pancreas and pancreatic tumors, reduces the interference of redundant information, and proposes a parallel branch structure of the pancreas-pancreatic tumor based on a segmentation network;

[0041] 3. The present invention uses a pre-training-fine-tuning adaptive training method, where the first stage is the pre-training stage and the second stage is the fine-tuning adaptive stage, so that the prognosis network can better capture the complex and diverse anatomical characteristics of the pancreas and pancreatic tumors in different patients;

[0042] 4. Aiming at the complex relative spatial position relationship between the pancreas and pancreatic tumors, the present invention adopts a fusion module guided by branches, and effectively enhances the topological space representation ability of the prognosis model by constructing two parts: a multi-scale feature extraction module and a multi-view information fusion module. Brief Description of the Drawings

[0043] Figure 1 Schematic diagram of the topological structure of the pancreatic ductal adenocarcinoma prognosis survival analysis method based on a dual-branch adaptive network provided by the present invention

[0044] Figure 2 Schematic diagram of the pre-training - fine-tuning adaptive training method provided by the present invention

[0045] Figure 3 Schematic diagram of the topological structure of a fusion module under branch guidance provided by the present invention

[0046] Figure 4 Schematic diagram of the flow of a pancreatic ductal adenocarcinoma prognosis survival analysis method based on a dual-branch adaptive network provided by the present invention

[0047] Figure 5 Schematic diagram of the Kaplan-Meier survival curve of the present invention in this method and other prognosis survival analysis methods

[0048] Figure 6 Schematic diagram of the present invention on Grad-CAM images in this method and other prognosis survival analysis methods

[0049] Figure 7 Schematic diagram of the Kaplan-Meier survival curve of the prognosis survival analysis method of the ablation experiment of the present invention Detailed implementation manners

[0050] The present invention will be further clarified below with reference to the accompanying drawings and specific embodiments

[0051] Example: In the process of prognostic survival analysis for pancreatic ductal carcinoma, due to the important influence of the anatomical characteristics of the pancreas and pancreatic tumors and their complex spatial positional relationships on prognostic analysis, a prognostic survival analysis method for pancreatic ductal carcinoma based on a dual-branch adaptive network was designed. This method constructs a pancreas-pancreatic tumor parallel-branch structure based on a segmentation network. By introducing target region and neighborhood information, it reduces the interference of redundant information on prognostic performance and also provides a solid guarantee for subsequent information extraction. Secondly, considering the impact of the complex and diverse anatomical characteristics of the pancreas and pancreatic tumors on the prognostic network, we designed a pre-training-fine-tuning adaptive training method, which consists of two stages. The first stage is the pre-training stage, where we initialize the model feature extractor using pre-trained parameters to achieve a preliminary localization of the target region in CT images and features. The second stage is the fine-tuning adaptive stage, in which parameters are adaptively adjusted to further refine the detailed features of the pancreas and pancreatic tumors. Finally, to comprehensively capture the spatial relationship between the pancreas and pancreatic tumors, we designed a fusion module guided by branches. The fusion module guided by branches consists of two parts: one is the multi-scale feature extraction module, which performs multi-scale feature fusion on the semantic features of the dual branches; the other is the multi-view information fusion module, which fuses the dual-branch features from multiple different attention perspectives.

[0052] As Figure 4 shown, it is a schematic flowchart of the present invention. A prognostic survival analysis method for pancreatic ductal carcinoma based on a dual-branch adaptive network includes the following steps:

[0053] S1: Uniformly preprocess the multi-center clinical abdominal CT dataset, including: (1) Window width and window level adjustment: According to the clinical prior information of the pancreas and pancreatic tumors, set the window width of the abdominal CT image to 200 HU and the window level to 100 HU, and normalize the image to constrain the range of voxel values to [0, 1]. (2) Histogram equalization: Change the grayscale of each pixel in the image by changing the histogram of the image to improve the contrast of pancreatic tumors in the CT image. (3) Sobel operator: Perform edge detection on the pancreas and pancreatic tumors to make the contrast between the edges and adipose tissue more obvious. (4) Data augmentation: Rotate the CT image at 90°, 180°, and 270° respectively and perform image scaling to achieve data augmentation.

[0054] S2: The coarse segmentation process includes the following steps: the location information of pancreatic tumors in clinical priors, cropping of CT slices, the corresponding coordinates of the cropped area are length: 100~400, width: 120~480 (both in pixels), and the image is segmented using a segmentation network. The encoder and decoder of the network contain four convolutional layers, which are 3X3 2D convolutional layers, BatchNormalization normalization layers, and activation layers with Relu activation function. The encoder adds a maximum pooling layer at the end of each layer, and the decoder adds an upsampling layer at the end of each layer to obtain the coarse segmentation result. .

[0055] S3: Construct a pancreas-pancreatic tumor parallel branch structure based on the segmentation network. First, the CT image is cropped with the tumor center area according to the coarse segmentation result in step S2. The cropping range is: length: -72~72, width: -56~56, height: -88~88 (in voxels). The cropping results are the pancreas and surrounding tissue images. Then we use the segmentation network to segment the image. The encoder and decoder of the network contain four convolutional layers, which are 5X5 3D convolutional layers, BatchNormalization normalization layers, and activation layers with Relu activation function. The encoder adds a maximum pooling layer at the end of each layer, and the decoder adds an upsampling layer at the end of each layer to obtain the segmentation results of the pancreas and pancreatic tumors. and Based on this, the target area of ​​the pancreas and pancreatic tumors is constructed, and the segmentation results are used to obtain a CT regional image containing only the pancreas and pancreatic tumors. and To obtain the neighborhood information containing the pancreas and pancreatic tumors, we use the features of the third layer decoder in the segmentation network as another input and Therefore, in the pancreatic branches, we constructed a pancreatic CT regional image and pancreatic domain characteristics As input, in the pancreatic tumor branch, a pancreatic tumor CT region image is constructed. and pancreatic domain characteristics as input.

[0056] S4: Feature extraction for the pancreas-pancreatic tumor parallel branch. First, the dual inputs of the pancreas and pancreatic tumor are combined. The specific implementation formula is as follows:

[0057]

[0058] in and is the final input of the CT image and feature combination of the pancreas and pancreatic tumor, and C represents the serial operation. Then, and Feature extraction is performed. The feature extractor consists of an encoding-decoding structure, including four convolutional layers, which are 3D convolutional layers with a size of 5X5, normalization layers of Batch Normalization, and activation layers with the activation function Relu. Max pooling layers are added at the end of each layer during encoding, and upsampling layers are added at the end of each layer during decoding. Finally, dual-branch features are obtained and .

[0059] S5: An adaptive training method of pre-training - fine-tuning is adopted for the dual-branch feature extraction stage, as Figure 2 shown, where the first stage is the pre-training stage and the second stage is the fine-tuning and adaptive stage. In the first stage, pre-training is performed on the two branches of the pancreas and pancreatic tumors respectively, and the model parameters obtained from the pre-training and are used to initialize the model parameters of the dual-branch feature extraction stage. In the second stage, the model parameters of feature extraction will be updated as the model training progresses based on the pre-training weights. The specific weight change formula is as follows:

[0060]

[0061] where is the weight matrix for fine-tuning the pre-training parameters. By adding it to the pre-training weight matrix, the final weight matrix and are obtained. In order to enable the prognosis network to better process the complex and diverse features of the pancreas and pancreatic tumors, we perform an adaptive decomposition on the fine-tuning weight matrix, decomposing it into two low-rank matrices A and B, thus simplifying the process of parameter fine-tuning. The specific implementation formula is as follows:

[0062]

[0063] where , , , d is the rank of the updated parameter matrix, and represent the weight matrices of the pancreas and pancreatic tumors after pre-training updates respectively. During the process of decomposing the fine-tuning weight matrix into and two matrices, the matrix is initialized with a random Gaussian distribution, providing a more balanced starting point for network training.

[0064] S6: Fuse the dual-branch features, as Figure 3As shown, the fusion module under the branch guidance includes two parts: a multi-scale feature extraction module and a multi-view information fusion module.

[0065] Multi-scale feature extraction structure: This module performs multi-scale extraction on the features of the two branches respectively and to obtain multi-scale features using dilated convolutions with different dilation rates to construct four parallel feature groups for obtaining multi-receptive field features. First, we construct the four parallel feature groups, and the specific implementation formula is as follows:

[0066]

[0067] where and represent the convolutional features with a dilation rate of , and represents the Sigmoid function. Next, we aggregate the multi-scale features, and the specific implementation formula is as follows::

[0068]

[0069] where represents the concatenation operation, and and are the output results of the multi-scale feature extraction module.

[0070] Multi-view information fusion module: Through the above multi-scale feature extraction module, we have obtained the multi-scale features of the pancreas and pancreatic tumors respectively and , and based on this, we perform multi-view information fusion. First, we aggregate the same-layer multi-scale features from the pancreas and pancreatic tumors and fuse the two-branch features based on this. Then, through max pooling and average pooling, we can capture the key information in the feature channels from multiple perspectives. Specifically, we use a multi-layer perception mechanism with average pooling and max pooling, and apply the Sigmoid function to calculate the channel attention coefficient K. Then, we multiply the attention coefficient by the features to obtain a reweighted feature vector. Finally, we perform a residual connection on these reweighted features and the multi-scale pancreas and tumor features to obtain the final fused features . The specific implementation formula is as follows:

[0071]

[0072]

[0073]

[0074] where Represents a multi-layer perception mechanism, Represents a max pooling layer, Represents an average pooling layer, Represents a Sigmoid function, Indicates a concatenation operation, Represents the features after concatenating the multi-scale features of the pancreas and pancreatic tumors together.

[0075] S7: Input the features after double-branch fusion into the prognosis network for prognosis survival analysis. The prognosis network contains three fully-connected layers, and DropOut is added after the first two layers to obtain the final prognosis result.

[0076] Through the above mechanism, we obtain the final prognosis survival analysis result of pancreatic ductal carcinoma.

[0077] To prove the effectiveness of the present invention, the present invention also provides comparative experiments and ablation experiments:

[0078] Specifically, the present invention selects CT images from 6 centers, and the detailed information of each center is shown in Table 1:

[0079] Table 1

[0080]

[0081] In the ablation experiment, the experimental results of the present method are compared with the pancreas-tumor parallel branch (PTPB), the pre-trained-fine-tuned adaptive training method (PFAT), and the fusion module under branch guidance (FGFM). The ablation results of the pancreas-tumor parallel branch are shown in Table 2, and the ablation results of the pre-trained-fine-tuned adaptive training method and the fusion module under branch guidance are shown in Table 3. The Kaplan-Meier survival curve results are as Figure 7 shown,

[0082] Table 2

[0083]

[0084] Table 3

[0085]

[0086] The corresponding index is the C-index, which represents the probability that the prediction result is consistent with the actual result. Both the table and the visualization results show the effectiveness of the three modules of the present method.

[0087] In the comparative experiment, the experimental result P of the present method 2- Comparison of the -Net, DeepSurv, DLSP, GuanRank, AdvMIL, pix2surv, and CancerRiskNet methods. The comparison results are shown in Tables 4, 5, 6, 7, and 8, and the Kaplan-Meier survival curve results are as Figure 5 shown, and the visualization results of Grad-CAM are as Figure 6 shown,

[0088] Table 4

[0089]

[0090] Table 5

[0091]

[0092] Table 6

[0093]

[0094] Table 7

[0095]

[0096] Table 8

[0097]

[0098] Among them, to ensure the fairness of the experimental results, the above experiments were all carried out with fine segmentation on the basis of coarse segmentation. Table 4 shows the prognosis comparison results of the internal test set, and Tables 5, 6, 7, and 8 show the prognosis comparison results of the external test set. The present invention can better capture the anatomical characteristics and spatial position relationships of the pancreas and pancreatic tumors, improving the prognosis performance of the present method.

[0099] It should be noted that the above embodiments are not used to limit the protection scope of the present invention. Equivalent transformations or substitutions made on the basis of the above technical solutions all fall within the scope protected by the claims of the present invention.

Claims

1. A method for prognostic survival analysis of pancreatic ductal adenocarcinoma based on a dual-branch adaptive network, characterized in that, It includes the following steps: S1： Preprocessing: Uniformly preprocess the multi-center clinical abdominal CT dataset, which includes: window width and window level adjustment, histogram equalization, Sobel operator, data augmentation; S2: Coarsely segment the preprocessed CT image slices in step S1 to obtain the preliminary position information of the pancreas and pancreatic tumors; S3: According to the coarse segmentation result in step S2, crop the CT image and use a segmentation network to construct a pancreas-pancreatic tumor parallel branch structure, which includes: pancreatic region image, pancreatic region features, pancreatic tumor region image, pancreatic tumor region features, and make full use of the target regions of the pancreas and pancreatic tumors and their neighborhood information; S4: Extract features based on the dual-branch input in step S3, which includes: pancreas encoder and pancreatic tumor encoder; S5: Adopt a two-stage pre-training-fine-tuning adaptive training method for the dual-branch features in step S4. The first stage is the pre-training stage, and the second stage is the fine-tuning adaptive stage. The two stages are serially trained to deeply capture the anatomical features of the pancreas and pancreatic tumors and enhance the learning ability of the prognosis model for complex anatomical characteristics; S6: Fuse the dual-branch features in step S5, and use the fusion module guided by the branches to respectively capture the complex and diverse spatial relative positions of the pancreas and pancreatic tumors from the multi-scale feature extraction module and the multi-view information fusion module; S7: Input the dual-branch fusion features in step S6 into the prognosis network to obtain the final prognosis survival analysis result of pancreatic ductal carcinoma.

2. The method for analyzing the prognostic survival of pancreatic ductal carcinoma based on a dual-branch adaptive network according to claim 1, wherein Uniformly preprocess the multi-center clinical abdominal CT dataset, which includes: (1) Window width and window level adjustment: According to the clinical prior information of the pancreas and pancreatic tumors, set the window width to 200 HU and the window level to 100 HU for the abdominal CT image, and normalize the image to constrain the range of voxel values to [0, 1]; (2) Histogram equalization: Change the gray scale of each pixel in the image by changing the histogram of the image to improve the contrast of pancreatic tumors in the CT image; (3) Sobel operator: Detect the edges of the pancreas and pancreatic tumors to make the contrast between the edges and adipose tissue more obvious; (4) Data augmentation: Rotate the CT image at 90°, 180° and 270° respectively, and perform image scaling to achieve data augmentation.

3. The method for pancreatic ductal adenocarcinoma prognosis survival analysis based on a dual-branch adaptive network according to claim 2, wherein The rough segmentation process in step S2 includes the following steps: based on the location information of pancreatic tumors in clinical prior knowledge, crop the CT slices, and the corresponding coordinates of the cropped area are 100 - 400 in length and 120 - 480 in width, both in pixels. Use the segmentation network to segment the image. The encoder and decoder of the network contain four convolutional layers, which are composed of 2D convolutional layers with 3X3, normalization layers of BatchNormalization, and activation layers with Relu as the activation function. The encoder adds a max pooling layer at the end of each layer, and the decoder adds an upsampling layer at the end of each layer to obtain the rough segmentation result .

4. The method for prognostic survival analysis of pancreatic ductal carcinoma based on a dual-branch adaptive network according to claim 3, wherein Construct a parallel branch structure of pancreas - pancreatic tumor based on the segmentation network. First, crop the CT image centered on the tumor according to the rough segmentation result in step S2. The cropping range is: length: -72~72, width: -56~56, height: -88~88, with the unit of voxel, and the image of the pancreas and its surrounding tissues is obtained after cropping. , then use the segmentation network to segment the image. The encoder and decoder of the network contain four convolutional layers, which are composed of 3D convolutional layers with 5X5, normalization layers of Batch Normalization, and activation layers with Relu as the activation function. The encoder adds a max pooling layer at the end of each layer, and the decoder adds an upsampling layer at the end of each layer to obtain the segmentation results of the pancreas and pancreatic tumor. and , construct the target regions of the pancreas and pancreatic tumor, and use the segmentation result to obtain the CT region image containing only the pancreas and pancreatic tumor. and , in order to obtain the neighborhood information containing the pancreas and pancreatic tumor, use the features of the third layer decoder in the segmentation network as another input. and , in the pancreas branch, construct the pancreatic CT region image and the pancreatic domain features as inputs. In the pancreatic tumor branch, construct the pancreatic tumor CT region image and the pancreatic domain features as inputs.

5. The method for pancreatic ductal adenocarcinoma prognosis survival analysis based on a dual-branch adaptive network according to claim 4, wherein, Extract features from the pancreas-pancreatic tumor parallel branch. First, combine the dual inputs of the pancreas and pancreatic tumors. The specific implementation formula is as follows: Among them and are the final inputs of the combination of CT images and features of the pancreas and pancreatic tumors. C represents the concatenation operation. Then, feature extraction is performed on and respectively. The feature extractor consists of an encoder-decoder structure, including four convolutional layers, which are 3D convolutional layers with a size of 5X5, normalization layers of Batch Normalization, and activation layers with an activation function of Relu. Max pooling layers are added at the end of each layer during encoding, and upsampling layers are added at the end of each layer during decoding. Finally, dual-branch features and are obtained.

6. The method for pancreatic ductal adenocarcinoma prognosis survival analysis based on a dual-branch adaptive network according to claim 5, wherein An adaptive training method of pre-training and fine-tuning is adopted for the dual-branch feature extraction stage. The first stage is the pre-training stage, and the second stage is the fine-tuning and adaptation stage. In the first stage, pre-training is performed on the two branches of the pancreas and pancreatic tumors respectively, and the model parameters obtained from the pre-training are used and to initialize the model parameters of the dual-branch feature extraction stage; in the second stage, the model parameters of the feature extraction are updated as the model training progresses based on the pre-trained weights. The specific weight change formula is as follows: Among them, is the weight matrix for fine-tuning the pre-trained parameters. By adding it to the pre-trained weight matrix, the final weight matrix is obtained. and , the fine-tuning weight matrix is adaptively decomposed into two low-rank matrices A and B. The specific implementation formula is as follows: Among them , , , n and m are the number of rows and columns of the parameter matrix respectively, d is the rank of the parameter matrix, and R represents the set of real numbers and represent the weight matrix of the pancreas and the weight matrix of pancreatic tumors after the pre-training is updated. When decomposing the fine-tuned weight matrix into and two matrices, the matrix is initialized with a random Gaussian distribution, providing a more balanced starting point for the training of the network.

7. The method for prognostic survival analysis of pancreatic ductal carcinoma based on a dual-branch adaptive network according to claim 6, wherein Fuse the dual-branch features. The fusion module guided by the branches includes two parts: the multi-scale feature extraction module and the multi-view information fusion module. Multi-scale feature extraction module: respectively perform multi-scale extraction on the features of the double branches and Perform multi-scale extraction, use dilated convolutions with different dilation rates to construct four parallel feature groups, so as to obtain feature information under different receptive fields. First, construct the four parallel feature groups, and the specific implementation formula is as follows: Among them and represent the convolution features of the pancreas and the tumor at the dilation rate of The following convolution features of the pancreas and the tumor are the features obtained after weighting by the Sigmoid function. Among them and represent the features obtained after weighting the dilation convolution features of the pancreas and the tumor by the Sigmoid function. Among them The dilation rate takes values of 1, 2, 3, and 5 to obtain represent four parallel feature groups of the pancreas represent four parallel feature groups of the tumor represents the Sigmoid function. Secondly, the multi-scale features are aggregated. The specific implementation formula is as follows: Among them represents a concatenation operation, and is the output result of the multi-scale feature extraction module; Multi - perspective Information Fusion Module: Through the above - mentioned multi - scale feature extraction module, multi - scale features of the pancreas and pancreatic tumors are obtained respectively and , based on which multi - perspective information fusion is carried out. First, the multi - scale features of the same layer from the pancreas and pancreatic tumors are aggregated together, and based on this, the fusion of double - branch features is carried out. Then, through max - pooling and average - pooling, the key information in the feature channels can be captured from multiple perspectives. Specifically, a multi - layer perception mechanism is used together with average - pooling and max - pooling, and the Sigmoid function is applied to calculate the channel attention coefficient K. Then, the attention coefficient is multiplied by the features to obtain a re - weighted feature vector. Finally, these re - weighted features are connected with the multi - scale pancreas and tumor features through residual connection to obtain the final fusion features , and the specific implementation formula is as follows: Among them represents a multi-layer perception mechanism, represents a max pooling layer, represents an average pooling layer, represents a Sigmoid function, denotes a concatenation operation, represents the features after concatenating the multi-scale features of the pancreas and pancreatic tumors together.

8. The method for prognostic survival analysis of pancreatic ductal carcinoma based on a dual-branch adaptive network according to claim 1, wherein Input the features after dual-branch fusion into the prognosis network for prognosis survival analysis. The prognosis network includes three fully connected layers, and DropOut is added after the first two layers to obtain the final prognosis result. Analyze the final prognosis result to classify pancreatic ductal carcinoma patients into high-risk groups and low-risk groups according to the CT image.

Images