Rectum cancer T stage prediction method based on MR image multi-task gated attention network

Abstract

The invention provides a rectal cancer T stage prediction method based on an MR image multi-task gated attention network, and the method comprises the following steps: carrying out the preprocessing of an original rectal MR image, and cutting out a 3D image block of a rectal tumor; secondly, constructing a segmentation-classification multi-task gating attention fusion network, and setting a multi-task fusion loss function; then, on the basis of amplifying the training sample library by using a data enhancement method, training the model by using a self-supervision and joint learning method; and finally, outputting the rectal cancer T staging prediction probability by using the trained multi-task gated attention network. The method is high in automation degree, and the precision of rectal cancer T stage risk prediction is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a method for predicting the T stage of rectal cancer based on a multi-task gated attention network of MR images. Background Art

[0002] Colorectal cancer (CRC) is one of the most common malignant tumors in the world, with the highest incidence and mortality rates, of which more than one-third are rectal cancer. At present, due to the rapid urbanization process in my country and the change in residents' eating habits, the incidence of CRC is increasing year by year, and there is a trend of younger people suffering from the disease, which seriously threatens my country's health care and the lives and health of residents. At present, for patients with early rectal cancer (T1-3, N0, M0), early direct surgical resection can achieve a five-year survival rate of more than 90%. On this basis, early identification of T1-T2 rectal cancer can help some T1 patients to obtain the opportunity for endoscopic submucosal dissection to preserve the complete intestine and mesorectum and maximize intestinal function; and early identification of T2-T3 rectal cancer can help screen out some T3 patients with escalated risks, such as mesorectal fascia involvement and extramural vascular invasion, so as to facilitate preoperative intervention for these patients, such as short-term radiotherapy or neoadjuvant therapy before surgery, to ensure negative circumferential margins after radical resection and reduce the risk of postoperative recurrence. Therefore, for patients with rectal cancer who are suitable for direct surgery in the early stage, accurate T staging of the tumor is crucial.

[0003] In clinical practice, magnetic resonance imaging (MRI) has the characteristics of high soft tissue resolution and spatial resolution, especially the imaging advantages of rectal wall, perirectal mesentery and mesofascia, and can effectively display rectal tumors and adjacent tissue structures. It has been recommended by domestic and foreign guidelines as the primary imaging examination for preoperative evaluation of rectal cancer. Even so, some T2 rectal cancers will have inflammatory reactive connective tissue hyperplasia in the perirectal mesentery, resulting in blurred tumor boundaries on MRI images; at the same time, some T1 rectal cancers with larger tumors are also difficult to distinguish from T2 rectal cancers because the rectal submucosal layer and muscle layer are difficult to accurately demarcate on MRI images. In addition, the traditional MRI preoperative evaluation of rectal cancer depends on the personal clinical experience of radiologists, so the diagnostic accuracy and repeatability show individual differences. Studies have shown that even in the context of widespread use of high-resolution MRI, the proportion of over-T staging of rectal cancer still reaches 57%. Therefore, people have proposed and developed radiomics and deep learning (DL) methods, which not only conduct deeper exploration of tumor morphological characteristics, but also combine massive high-order spatiotemporal features to characterize tumor heterogeneity. At present, there are many radiomics-related studies on preoperative T staging of rectal cancer. However, few related studies have been implemented, and they have the common characteristics of poor robustness and low repeatability. DL models usually use end-to-end convolutional neural networks to establish prediction or classification models based on deep learning to explore the image characteristics of tumors. At present, some studies have explored the feasibility of DL technology in assessing the preoperative T staging of rectal cancer, but the sample size of related studies is small, and the repeatability remains to be discussed.

[0004] The multi-task deep learning model can achieve shared representation learning of features of different tasks, better utilize complementary information between different tasks, and improve the overall performance of the model, thereby improving the performance and reliability of the overall model. The present invention proposes a rectal cancer T staging prediction method based on MR image multi-task gated attention network, which can achieve rectal cancer T staging risk stratification and assist clinical diagnosis and treatment decision-making. Summary of the invention

[0005] In order to overcome the shortcomings and deficiencies in the above-mentioned prior art, the present invention provides a method for predicting the T stage of rectal cancer based on a multi-task gated attention network of MR images. The method first pre-processes the original rectal MR images and cuts out the 3D image blocks of the rectal tumor; secondly, constructs a segmentation-classification multi-task gated attention fusion network and sets a multi-task fusion loss function; then, on the basis of amplifying the training sample library using a data enhancement method, the model is trained using a self-supervision and joint learning method; finally, the trained multi-task gated attention network is used to output the prediction probability of the T stage of rectal cancer. The method of the present invention has a high degree of automation and improves the accuracy of the risk prediction of the T stage of rectal cancer. The method of the present invention has a high degree of automation and improves the accuracy of the risk prediction of the T stage of rectal cancer.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a method for predicting T stage of rectal cancer based on MR image multi-task gated attention network, the method comprising the following steps:

[0007] S1: Preprocess the original rectal MR images and crop the 3D image blocks of the rectal tumors;

[0008] S2: Construct a segmentation-classification multi-task gated attention fusion network and set a multi-task fusion loss function;

[0009] S3: Based on the use of data augmentation methods to expand the training sample library, the model is trained using self-supervision and joint learning methods;

[0010] S4: Use the trained multi-task gated attention network to output the predicted probability of rectal cancer T stage.

[0011] Furthermore, the S1 comprises the following steps:

[0012] S11: Input original rectal MR image I MRI ;

[0013] S12: Use the Z-Score standardization method to normalize the grayscale value. If the grayscale value of the original magnetic resonance image is I MRI , then the gray value of the normalized image is I Norm , the calculation formula is as follows: Among them, μ MRI is the mean gray value of the original image, and N is the number of image pixels;

[0014] S13: using a B-spline resampling algorithm to standardize the resolution of the image so that the image resolution reaches a preset resolution;

[0015] S14: Mark the location of the rectal cancer tumor on the MR image, and cut the rectal tumor area into a 3D image block of a fixed size according to the location of the tumor center point.

[0016] Furthermore, the step 2 comprises the following steps:

[0017] S21: Input preprocessed 3D image block I containing the rectal tumor area Crop ;

[0018] S22: Based on the U-Net network architecture, construct the image encoding module 1 and the image segmentation decoding module to realize the construction of the rectal tumor area segmentation network and output the tumor segmentation result Seg;

[0019] S23: Combine the rectal tumor segmentation result Seg with the 3D image block I Crop Perform mask processing to obtain the rectal cancer tumor area mask image I Mask , construct image encoding module 2 to extract I Mask Multi-scale deep image features F Multiscale2 , obtain the multi-scale depth image feature library 2;

[0020] S24: Use the gated fusion network structure to build a gated attention network, and transform the multi-scale deep image features F in the image encoding module 1 in the segmentation network Multiscale1 With F Multiscale2 Fusion, and then input into the fully connected network to obtain the T stage prediction result P TStage ;

[0021] S25: Combine the tumor segmentation and T stage prediction network modules to construct a multi-task gated attention fusion network;

[0022] S26: Constructing a hybrid loss function L for the segmentation-classification multi-task network MultiTask , the calculation formula is as follows:

[0023] L MultiTask =λ1L Seg +λ2L Clf

[0024]

[0025] L ckf =-(1-P TStage ) γ log(P TStage )

[0026] Among them, λ1 and λ2 are weight coefficients, L seg represents the rectal tumor segmentation loss, L clf represents the T-stage classification prediction loss, Mask i As the gold standard for tumor segmentation, Seg i is the prediction result of the model segmentation network, P TStage represents the predicted probability of T stage.

[0027] Further, the S24 comprises the following steps:

[0028] S241: Multi-scale deep image features F Multiscale1 and multi-scale deep image features F Multiscale2 Perform adaptive flat pooling and adaptive maximum pooling operations respectively, and flatten and stretch the pooled features into one dimension;

[0029] S242: Use the linear weighted fusion method to combine the pooled F Multiscale1 and F Multiscale2 Perform fusion to obtain the fused image feature 1F Fusion1 And the fused image feature 2F Fusion2 ;

[0030] S243: Using the linear weighted fusion method, F Fusion1 and F Fusion2 Perform fusion to obtain multi-scale fusion image features;

[0031] S244: Input the multi-scale fusion image features into the fully connected network, and finally obtain the T stage prediction result P TStage .

[0032] Furthermore, S3 comprises the following steps:

[0033] S31: Use 3D image data enhancement methods, such as flipping, rotating, scaling, cropping, and translating, to expand the training set samples;

[0034] S32: Use contrastive learning self-supervised learning method to train the image encoding module in the multi-task model, and the self-supervised learning loss function L unsupervised , the calculation formula is as follows:

[0035]

[0036] Among them, f i and f k represents the encoding of two different samples after the encoder, τ is an adjustable coefficient, and sim represents the similarity calculation;

[0037] S33: Use the pre-trained model of self-supervised learning to further train the multi-task model with a mixed loss function L MultiTask As the loss function, the AdamW adaptive learning rate adjuster is used to train the network, and the optimal network parameters are selected as the final model parameters.

[0038] Based on the above technical solution, compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention provides a method for predicting rectal cancer T stage based on MR image multi-task gated attention network. The method first pre-processes the original rectal MR image and cuts out the 3D image block of the rectal tumor; secondly, constructs a segmentation-classification multi-task gated attention fusion network and sets the multi-task fusion loss function; then, on the basis of amplifying the training sample library by using the data enhancement method, the model is trained by using the self-supervision and joint learning method; finally, the trained multi-task gated attention network is used to output the rectal cancer T stage prediction probability. The method of the present invention has a high degree of automation, which improves the accuracy of rectal cancer T stage risk prediction; the method of the present invention has a high degree of automation, which improves the accuracy of rectal cancer T stage risk prediction, realizes the T stage risk stratification of different patients, and helps to assist clinical stratification management and individualized treatment of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the drawings without paying creative work.

[0041] Figure 1 This is a flow chart of the method for predicting T stage of rectal cancer based on MR image multi-task gated attention network of the present invention;

[0042] Figure 2 Schematic diagram of the segmentation-classification multi-task gated attention fusion network;

[0043] Figure 3 Schematic diagram of gated fusion attention network fusion;

[0044] Figure 4 The experimental results of Example 3 are shown in FIG. 3 , where (a) is the ROC curve diagram and (b) is the confusion matrix. DETAILED DESCRIPTION

[0045] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0046] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0047] Example 1

[0048] Figure 1 Flow chart of the method for predicting rectal cancer T stage based on MR image multi-task gated attention network of the present invention, as shown in Figure 1 As shown, a method for rectal cancer T stage prediction based on MR image multi-task gated attention network includes the following steps:

[0049] S1: Preprocess the original rectal MR images and crop the 3D image blocks of the rectal tumors;

[0050] S2: Construct a segmentation-classification multi-task gated attention fusion network and set a multi-task fusion loss function;

[0051] S3: Based on the use of data augmentation methods to expand the training sample library, the model is trained using self-supervision and joint learning methods;

[0052] S4: Use the trained multi-task gated attention network to output the predicted probability of rectal cancer T stage.

[0053] Example 2

[0054] Example 2 is further described based on Example 1:

[0055] A method for rectal cancer T staging prediction based on MR image multi-task gated attention network, comprising the following steps:

[0056] S1: Preprocess the original rectal MR images and crop the 3D image blocks of the rectal tumors;

[0057] S2: Construct a segmentation-classification multi-task gated attention fusion network and set a multi-task fusion loss function;

[0058] S3: Based on the use of data augmentation methods to expand the training sample library, the model is trained using self-supervision and joint learning methods;

[0059] S4: Use the trained multi-task gated attention network to output the predicted probability of rectal cancer T stage.

[0060] Wherein, S1 includes the following steps:

[0061] S11: Input original rectal MR image I MRI ;

[0062] S12: Use the Z-Score standardization method to normalize the grayscale value. If the grayscale value of the original magnetic resonance image is I MRI , then the gray value of the normalized image is I Norm , the calculation formula is as follows: Among them, μ MRI is the mean gray value of the original image, and N is the number of image pixels;

[0063] S13: using a B-spline resampling algorithm to standardize the resolution of the image so that the image resolution reaches a preset resolution;

[0064] S14: Mark the location of the rectal cancer tumor on the MR image, and cut the rectal tumor area into a 3D image block of a fixed size according to the location of the tumor center point.

[0065] Wherein, S2 includes the following steps:

[0066] S21: Input preprocessed 3D image block I containing the rectal tumor area Crop ;

[0067] S22: Based on the U-Net network architecture, Swin-Transformer is used to construct the image encoding module 1 and the image segmentation decoding module to realize the construction of the rectal tumor area segmentation network and output the tumor segmentation result Seg;

[0068] S23: Combine the rectal tumor segmentation result Seg with the 3D image block I Crop Perform mask processing to obtain the rectal cancer tumor area mask image I Mask , using Swin-Transformer to build image encoding module 2 to extract I Mask Multi-scale deep image features F Multiscale2 , obtain the multi-scale depth image feature library 2;

[0069] S24: Use the gated fusion network structure to build a gated attention network, and transform the multi-scale deep image features F in the image encoding module 1 in the segmentation network Multiscal e1 and F Multiscale2 Fusion, and then input into the fully connected network to obtain the T stage prediction result P TStage ;

[0070] S25: Combine the tumor segmentation and T staging prediction network modules to build a multi-task gated attention fusion network. Figure 2 Schematic diagram of the segmentation-classification multi-task gated attention fusion network of the present invention;

[0071] S26: Constructing a hybrid loss function L for the segmentation-classification multi-task network MultiTask , the calculation formula is as follows:

[0072] L MultiTask =λ1L Seg +λ2L Clf

[0073]

[0074] L ckf =-(1-P TStage ) γ log(P TStage )

[0075] Among them, λ1 and λ2 are weight coefficients, L seg represents the rectal tumor segmentation loss, L clf represents the T-stage classification prediction loss, Mask i As the gold standard for tumor segmentation, Seg i is the prediction result of the model segmentation network, P TStage represents the predicted probability of T stage.

[0076] Further, Figure 3 This is a schematic diagram of the gated fusion attention network fusion, as shown in Figure 3 As shown, the S24 includes the following steps:

[0077] S241: Multi-scale deep image features F Multiscale1 and multi-scale deep image features F Multiscale2 Perform adaptive flat pooling and adaptive maximum pooling operations respectively, and flatten and stretch the pooled features into one dimension;

[0078] S242: Use the linear weighted fusion method to combine the pooled F Multiscal e1 and F Multiscale2 Perform fusion to obtain the fused image feature 1F Fusion1 And the fused image feature 2F Fusion2 ;

[0079] S243: Using the linear weighted fusion method, F Fusion1 and F Fusion2 Perform fusion to obtain multi-scale fusion image features;

[0080] S244: Input the multi-scale fusion image features into the fully connected network, and finally obtain the T stage prediction result P TStage .

[0081] Among them, S3 includes the following steps:

[0082] S31: Use 3D image data enhancement methods, such as flipping, rotating, scaling, cropping, and translating, to expand the training set samples;

[0083] S32: Use contrastive learning self-supervised learning method to train the image encoding module in the multi-task model, and the self-supervised learning loss function L unsupervised , the calculation formula is as follows:

[0084]

[0085] Among them, f i and f k represents the encoding of two different samples after the encoder, τ is an adjustable coefficient, and sim represents the similarity calculation;

[0086] S33: Use the pre-trained model of self-supervised learning to further train the multi-task model with a mixed loss function L MultiTask As the loss function, the AdamW adaptive learning rate adjuster is used to train the network, and the optimal network parameters are selected as the final model parameters.

[0087] Example 3

[0088] The following is an experimental process of the method for predicting rectal cancer T stage based on MR image multi-task gated attention network of the present invention:

[0089] 1. Experimental conditions:

[0090] This experiment collected MR images of 958 patients with rectal cancer from the Affiliated Cancer Hospital of Fudan University, which were divided into training set and validation set 1 according to 8:2; MR images of 94 patients from Nantong Cancer Hospital were collected as validation set 2; MR images of 169 patients from Shanxi Cancer Hospital were collected as validation set 3. All MR images were T2WI sequences, without other primary tumors. In this experiment, patients with T stage 1 and 2 were grouped as one group, and patients with T stage 3 and 4 were grouped as another group for binary classification.

[0091] 2. Experimental results and analysis

[0092] By using T2WI sequence images to build a multi-task gated attention prediction model, Figure 4 To compare the experimental results of the model on different data sets, Figure 4As can be seen in (a), the area under the curve (AUC) is 0.92±0.01 in the training set, 0.88±0.02 in test set 1, 0.79±0.05 in test set 2, and 0.73±0.04 in test set 3. Figure 4 (b) lists the confusion matrix of the model on four data sets. It can be seen that the rectal cancer T staging prediction method based on MR image multi-task gated attention network of the present invention has high accuracy and good model stability.

[0093] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for rectal cancer T stage prediction based on MR image multi-task gated attention network, characterized in that: The method comprises the following steps: S1: Preprocess the original rectal MR images and crop the 3D image blocks of the rectal tumors; S2: Construct a segmentation-classification multi-task gated attention fusion network and set a multi-task fusion loss function; S3: Based on the use of data augmentation methods to expand the training sample library, the model is trained using self-supervision and joint learning methods; S4: Use the trained multi-task gated attention network to output the predicted probability of rectal cancer T stage.

2. The prediction method according to claim 1, characterized in that: The S1 comprises the following steps: S11: Input original rectal MR image I MRI ; S12: Use the Z-Score standardization method to normalize the grayscale value. If the grayscale value of the original magnetic resonance image is I MRI , then the gray value of the normalized image is I Norm , the calculation formula is as follows: Among them, μ MRI is the mean gray value of the original image, and N is the number of image pixels; S13: using a B-spline resampling algorithm to standardize the resolution of the image so that the image resolution reaches a preset resolution; S14: Mark the location of the rectal cancer tumor on the MR image, and cut the rectal tumor area into a 3D image block of a fixed size according to the location of the tumor center point.

3. The prediction method according to claim 1, characterized in that: The S2 comprises the following steps: S21: Input preprocessed 3D image block I containing the rectal tumor area Crop ; S22: Based on the U-Net network architecture, construct the image encoding module 1 and the image segmentation decoding module to realize the construction of the rectal tumor area segmentation network and output the tumor segmentation result Seg; S23: Combine the rectal tumor segmentation result Seg with the 3D image block I Crop Perform mask processing to obtain the rectal cancer tumor area mask image I Mask , construct image encoding module 2 to extract I Mask Multi-scale deep image features F Multiscale2 , obtain the multi-scale depth image feature library 2; S24: Use the gated fusion network structure to build a gated attention network, and transform the multi-scale deep image features F in the image encoding module 1 in the segmentation network Multiscale1 With F Multiscale2 Fusion, and then input into the fully connected network to obtain the T stage prediction result P TStage ; S25: Combine the tumor segmentation and T stage prediction network modules to construct a multi-task gated attention fusion network; S26: Constructing a hybrid loss function L for the segmentation-classification multi-task network MultiTask , the calculation formula is as follows: L MultiTask =λ1L Seg +λ2L Clf L clf =-(1-P TStage ) γ log(P TStage ) Among them, λ1 and λ2 are weight coefficients, L seg represents the rectal tumor segmentation loss, L clf represents the T-stage classification prediction loss, Mask i As the gold standard for tumor segmentation, Seg i is the prediction result of the model segmentation network, P TStage represents the predicted probability of T stage.

4. The prediction method according to claim 3, characterized in that: The S24 comprises the following steps: S241: Multi-scale deep image features F Multiscale1 and multi-scale deep image features F Multiscale2 Perform adaptive flat pooling and adaptive maximum pooling operations respectively, and flatten and stretch the pooled features into one dimension; S242: Use the linear weighted fusion method to combine the pooled F Multiscale1 and F Multiscale2 Perform fusion to obtain the fused image feature 1F Fusion1 And the fused image feature 2F Fusion2 ; S243: Using the linear weighted fusion method, F Fusion1 and F Fusion2 Perform fusion to obtain multi-scale fusion image features; S244: Input the multi-scale fusion image features into the fully connected network, and finally obtain the T stage prediction result P TStage .

5. The prediction method according to claim 1, characterized in that: The S3 comprises the following steps: S31: Use 3D image data enhancement methods, such as flipping, rotating, scaling, cropping, and translating, to expand the training set samples; S32: Use contrastive learning self-supervised learning method to train the image encoding module in the multi-task model, and the self-supervised learning loss function L unsupervised , the calculation formula is as follows: Among them, f i and f k represents the encoding of two different samples after the encoder, τ is an adjustable coefficient, and sim represents the similarity calculation; S33: Use the pre-trained model of self-supervised learning to further train the multi-task model with a mixed loss function L MultiTask As the loss function, the AdamW adaptive learning rate adjuster is used to train the network, and the optimal network parameters are selected as the final model parameters.

Images