Prostate tumor image analysis method based on bimodal multi-task learning model

Abstract

The application discloses a prostate tumor image analysis method based on a bimodal multi-task learning model, and comprises the following steps: acquiring a prostate region image to be segmented, and pre-processing the prostate region image, wherein the prostate region image comprises a T2 weighted image and a diffusion weighted image; a bimodal multi-task learning model is constructed, training data sets are extracted, and the bimodal multi-task learning model is trained, wherein the bimodal multi-task learning model comprises a segmentation part and a grading part, the segmentation part comprises an encoder and a decoder, an SPM module is constructed in the encoder to assist feature extraction, a Transformer unit is constructed in the decoder to enhance global feature representation, a PI-RADS multi-scale grading network is constructed in the grading part to obtain more detailed information, and a PI-RADS grading result is output; and the pre-processed prostate region image is input into the bimodal multi-task learning model to obtain a prostate tumor image segmentation result and a grading result.

Description

Technical Field

[0001] This invention relates to the field of tumor image segmentation technology, and in particular to a method for prostate tumor image analysis based on a bimodal multi-task learning model. Background Technology

[0002] Prostate cancer (PCa) is a common cancer among middle-aged and elderly men, and the mortality rate is increasing year by year. Biparametric magnetic resonance imaging (bpMRI) mainly includes two imaging modalities: T2-weighted imaging (T2WI) and diffusion-weighted imaging (DWI). These two modalities can capture different pathological information and complement each other effectively, playing a crucial role in PCa image analysis. However, most current intelligent diagnostic methods for prostate cancer are based on a single modality or a single task, neglecting the information interaction between the two modalities and the task relevance between multi-task learning. Summary of the Invention

[0003] In view of the above-mentioned prior art, the present invention provides a method for prostate tumor image analysis based on a bimodal multi-task learning model, which mainly solves the technical problems existing in the background art.

[0004] To achieve the above objectives, the technical solution of this invention is implemented as follows: a prostate tumor image analysis method based on a bimodal multi-task learning model, the method comprising the following steps:

[0005] The prostate region image to be segmented is acquired and preprocessed. The prostate region image includes a T2-weighted image and a diffusion-weighted image.

[0006] A bimodal multi-task learning model is constructed, a training dataset is extracted, and the bimodal multi-task learning model is trained. The bimodal multi-task learning model includes a segmentation part and a hierarchical part. The segmentation part includes an encoder and a decoder. An SPM module is constructed in the encoder to assist in feature extraction. A Transformer unit is constructed in the decoder to enhance global feature representation. In the hierarchical part, a PI-RADS multi-scale hierarchical network is constructed to obtain more refined detail information and output PI-RADS hierarchical results.

[0007] The preprocessed prostate region image is input into a bimodal multi-task learning model to obtain prostate tumor image segmentation and grading results.

[0008] Optionally, the prostate region images are preprocessed, specifically including performing affine transformations on the prostate region images and flipping each prostate region image from left to right and from top to bottom, and rotating the images by 90°, 180° and 270°.

[0009] Optionally, after the preprocessed prostate region image is input into the encoder, the encoder first performs an initial convolution calculation on the preprocessed prostate region image with a convolution kernel of 3×3×3 to generate a feature map with 16 channels.

[0010] Optionally, the encoder in the segmentation part is equipped with multiple SPM modules. The T2WI feature map and DWI feature map with 16 channels are first input into the first SPM module. The block fusion layer in the SPM module first performs block fusion on the T2WI feature map and DWI feature map to obtain a concatenated result of 8 feature maps. Then, the concatenated result is linearly transformed on the feature map through a regularization layer and two fully connected layers. Finally, the output of the SPM module is realized through a 1×1×1 convolutional layer. The output of the SPM module is then subjected to a 3×3×3 convolution calculation.

[0011] Optionally, for the output of each SPM module, attention features are output using the VAM module. The VAM module contains two sets of convolutional blocks: the first set is a standard convolutional group of different sizes, and the second set is an dilated convolutional group of different dilation rates.

[0012] Optionally, the outputs of the VAM module and the SPM module are input into the decoder for sequential cascading, and the global feature representation is enhanced by the Transformer unit.

[0013] Optionally, the PI-RADS multi-scale hierarchical network is divided into a high-level feature fusion module and a classifier. The high-level feature fusion module consists of two parts. For the first part, the different features from before and after in the Transformer unit are used as input to obtain a first fusion result. For the second part, the first fusion result is used as input to obtain a second fusion result. The first fusion result and the second fusion result are added together to obtain the high-level fused feature map. High-level fusion feature maps Input the classifier to obtain the probability of PI-RADS automatic classification.

[0014] Optionally, the segmentation part includes two loss functions. The first loss function is the Dice loss function, which is directly aimed at optimizing the segmentation performance evaluation metric.

[0015]

[0016] The second loss function is the Focal loss function:

[0017]

[0018] The total loss function for the segmentation is:

[0019]

[0020] In the formula, p is the predicted probability, g is the ground truth label, n is the number of pixels in the image, and x i For the i-th pixel value, η is the balancing factor of the Focal loss function, λ is the parameter for smoothing the weighting rate, and η is the weighting factor. D η represents the weighting factor of the Dice loss function. F This represents the weighting factor of the Focal loss function.

[0021] Optionally, the loss function in the PI-RADS multi-scale network is the cross-entropy loss function:

[0022]

[0023] In the formula, K represents the number of datasets, and g k p represents the truth label. k This represents the probability output by the classification model.

[0024] Optionally, the overall loss function in the bimodal multi-task learning model is:

[0025]

[0026] In the formula, α seg and α cls This indicates an uncertain weight.

[0027] The beneficial effects of this invention are as follows: The prostate tumor image analysis method based on a bimodal multi-task learning model proposed in this application designs a VAM module in the encoder stage. The VAM contains two sets of convolutional blocks: the first set is a standard convolutional group of different sizes, and the second set is an dilated convolutional group with different dilation rates. The collaboration between standard and dilated convolutions allows different features to be integrated into the visual attention module, ultimately resulting in visual features rich in detail. A SPM feature downsampling module is proposed; the design of SPM provides more detailed semantic information to the network, which is beneficial for information sharing between the prostate and tumor. At the encoder end, a Transformer technique is proposed to obtain more refined global and local detail information. The PI-RADS high-level feature fusion module constructed in the classification task obtains richer global and local feature information through the fusion of the two-part structure. A loss function is also designed to address the class imbalance problem among the prostate, tumor, and background regions. The segmentation results show that the proposed algorithm's segmentation results are highly consistent with expert manual segmentation results, and the segmentation and classification of the prostate and PCa also have higher performance compared to existing methods. The proposed bimodal multitasking model achieved a better ROC curve and a higher AUC value in PI-RADS automatic classification. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the prostate tumor image analysis method based on a bimodal multi-task learning model in an embodiment of this application.

[0029] Figure 2 This is a schematic diagram of the overall structure of the bimodal multitask learning model in the embodiments of this application;

[0030] Figure 3 This is a schematic diagram illustrating the working principle of the SPM module in the embodiments of this application;

[0031] Figure 4 This is a schematic diagram of the overall structure of the VAM module in the embodiments of this application;

[0032] Figure 5 This is a schematic diagram of the overall structure of the PI-RADS high-level feature fusion module in the embodiments of this application;

[0033] Figure 6 This is a schematic diagram showing the performance of PCa in two modalities: T2-weighted image and diffusion-weighted image, as described in the embodiments of this application. Detailed Implementation

[0034] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. In the following description, the expression "some embodiments" refers to a subset of all possible embodiments; however, it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0035] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0036] It should be understood that the present invention can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Furthermore, the terminology used herein is intended only to describe particular embodiments and is not intended to limit the invention. When used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “compose” and / or “comprising,” when used in this specification, identify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups. When used herein, the term “and / or” includes any and all combinations of the associated listed items.

[0037] It should also be noted that when an element is referred to as being "fixed to" another element, it can be directly attached to the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "inner," "outer," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementation.

[0038] To fully understand this invention, a detailed structure will be presented in the following description to illustrate the technical solution proposed by this invention. Optional embodiments of the invention are described in detail below; however, in addition to these detailed descriptions, the invention may have other embodiments.

[0039] Please refer to the attached document. Figure 1 This application provides a method for prostate tumor image analysis based on a bimodal multi-task learning model, the method comprising the following steps:

[0040] S1. Obtain an image of the prostate region to be segmented, and preprocess the prostate region image, which includes a T2-weighted image and a diffusion-weighted image.

[0041] S2. Construct a bimodal multi-task learning model, extract the training dataset, and train the bimodal multi-task learning model. The bimodal multi-task learning model includes a segmentation part and a hierarchical part. The segmentation part includes an encoder and a decoder. An SPM module is constructed in the encoder to assist in feature extraction. A Transformer unit is constructed in the decoder to enhance global feature representation. In the hierarchical part, a PI-RADS multi-scale hierarchical network is constructed to obtain more refined detail information and output PI-RADS hierarchical results.

[0042] S3. Input the preprocessed prostate region image into a dual-modal multi-task learning model to obtain prostate tumor image segmentation and grading results.

[0043] See Figure 2 The bimodal multi-task learning model constructed in this application operates on a 64×64×64 3D block. This block size was chosen to achieve multi-scale feature learning while limiting memory requirements. The input to the encoder stage is first processed by two 3×3×3 convolutional layers, followed by batch normalization (BN) and parameter rectified linear unit (PReLU) activation. A VAM module is designed in the encoder stage to extract features at different scales, which are then applied to the VAM module to obtain feature information with different receptive fields. A shared block fusion mechanism is proposed in the encoder downsampling stage to map T2WI and DWI images to the same spatial region of interest, maintaining spatial and structural consistency. Transformer units are embedded at the bottom to enhance feature representation and further learn global features. A high-level feature fusion (HLFF) mechanism is built before the PI-RADS automatic grading classifier to obtain finer detail information. The decoder stage fuses Transformer features and VAM features to derive segmentation probability maps for the prostate and tumor. For each 3D input block, the segmentation task stage outputs a probability map of the same size in an end-to-end manner. Finally, the aforementioned probability maps are graded through a grading section.

[0044] Furthermore, the prostate region images are preprocessed, specifically including performing affine transformations on the prostate region images and flipping each prostate region image from left to right and from top to bottom, and rotating the images by 90°, 180° and 270°.

[0045] Specifically, affine registration is a commonly used method by physicians for depicting tumor volume in T2WI and DWI. In addition, sample augmentation was performed on all training data, flipping each prostate T2WI and DWI image from left to right and top to bottom; the images were also rotated by 90°, 180°, and 270°; the flipping preserved visual structure. After data augmentation, the sample size was sufficient to train the deep learning model, avoiding overfitting.

[0046] Furthermore, after the preprocessed prostate region image is input into the encoder, the encoder first performs an initial convolution calculation on the preprocessed prostate region image with a convolution kernel of 0.05 to generate a feature map with 16 channels. The encoder in the segmentation part is equipped with multiple SPM modules. The T2WI feature map and DWI feature map with 16 channels are first input into the first SPM module. The block fusion layer in the SPM module first performs block fusion on the T2WI feature map and DWI feature map to obtain a stitched result of stitching 8 feature maps. Then, the stitched result is linearly transformed on the feature map through a regularization layer and two fully connected layers. Finally, the output of the SPM module is realized through a convolutional layer, and the output of the SPM module is further convolutionally calculated.

[0047] The encoder uses a series of convolution and transform operations from the input Global semantic features are extracted from the input (3D spatial resolution W×H×D, number of channels C). First, an initial convolution operation with an input kernel size of 3×3×3 is performed to generate a feature map with 16 channels. To capture spatial and depth feature representations, feature maps are generated using SPM (Simultaneous Permeation Model) and multiple stacked convolutional blocks. Each convolutional block consists of batch normalization, a Leaky ReLU function, and a 3×3×3 convolution. When multimodal medical images are used for segmentation or detection tasks, the most common approach is to use multiple encoder branches to extract features from each modality separately. Using this method, T2WI and DWI do not benefit from each other during feature extraction.

[0048] See Figure 3Building upon this foundation, this application designs an SPM module to assist encoder branching and improve the effectiveness of feature extraction. Using the SPM module maps T2WI and DWI feature maps to the same spatial region of interest, maintaining spatial and structural consistency. Assuming the input to the SPM is a 3D feature map with a spatial resolution of W×H×D and C channels, block fusion divides each (W / 2)×(H / 2)×(D / 2) adjacent pixel into a block, then concatenates pixels at the same location to obtain 8 feature maps. These 8 feature maps are then concatenated, and a regularization layer and two fully connected layers are used to linearly transform the feature maps, changing the number of channels from C to C / 2.

[0049] After block fusion, the feature size is halved and the number of channels is doubled. Furthermore, 1×1×1 convolutions facilitate information exchange between T2WI and DWI channels, improving the model's non-linear representation. The encoder branch has independent encoder blocks and the same SPM used for feature extraction in both T2WI and DWI.

[0050] During training, the weights of the SPM are updated twice within a training iteration (the first input to the SPM is the feature encoding of T2WI, and the second input is the feature encoding of DWI). It is assumed that when the input to the SPM comes from T2WI, the output of the SPM can contain some corresponding DWI information, and vice versa. This also means that the output of each SPM can represent information from both modalities (i.e., T2WI and DWI). Through this module, T2WI can remove false positives from the DWI feature map and also helps detect lesions that are not obvious in the T2WI signal. Conversely, T2WI features help distinguish lesions that are similar to the anatomical features of surrounding organs.

[0051] Secondly, in this application, the following symbols are defined: Represents the sample set S L The tag set. Where x (i) This represents the mode of the input i-th sample, y (i) The truth labels represent prostate and tumor data. This indicates a PI-RADS score of ≤3 or ≥4.

[0052] For the i-th sample, g (i) =0 indicates a PI-RADS score ≤3, g (i) =1 indicates a PI-RADS score ≥4.

[0053] Furthermore, for the output of each SPM module, attention features are output using the VAM module. The VAM module contains two sets of convolutional blocks: the first set is a standard convolutional group of different sizes, and the second set is an dilated convolutional group of different dilation rates.

[0054] See Figure 4 Specifically, the VAM module aims to acquire global and local information through a combination of convolution and dilated convolution. This reflects the visual attention mechanism, which consists of two parts: blurred achromatic peripheral vision and highly sensitive color central vision. This enables humans to quickly shift their gaze from the fovea to a salient region. The entire process involves parallel retinal processing of various input features and the gradual fusion of different features with the participation of the visual attention mechanism. For example, the feature map F at model input scale i... i and When i=5, the feature is directly input as F5, and the output is a feature that is integrated with F. i Attention features of the same size In the designed VAM module, convolutional layers of different scales represent the visual center, and dilated convolutional layers with different dilation rates increase the visual receptive field.

[0055] The VAM comprises two sets of convolutional blocks. The first layer consists of standard convolutional blocks of different sizes, and the second layer consists of dilated convolutional blocks with different dilation rates. In the VAM proposed in this application, the input features are first processed using the convolutional blocks. The kernel sizes are 1×1×1, 3×3×3, 5×5×5, and 7×7×7, and each convolution operation is followed by BM and PReLU. This design enriches the learned representation and reduces computational complexity. The generated feature maps are concatenated and passed through the second set of convolutional blocks. The second set of convolutional blocks is a dilated convolutional block used to expand the receptive field, with dilation rates of 1, 3, 5, and 7. The generated feature maps are fused, and the feature map obtained by using the Sigmoid activation method is multiplied with the input features and intermediate features and summed to obtain the final attention features.

[0056] Furthermore, the outputs of the VAM module and the SPM module are input into the decoder for sequential cascading, and the global feature representation is enhanced by the Transformer unit.

[0057] To reshape the image input to fit the transformer's input, the 2D image is reshaped into a series of flat 2D blocks. For 3D volumetric data, ViT's direct tokenization is extended to 3D data, segmenting the data into 3D blocks. However, embedding a large 3D slice increases the computational overhead of the Transformer. To mitigate this, low-resolution high-level feature maps extracted from the encoder are mapped into the Transformer unit to further learn global feature representations. To ensure a comprehensive representation of each 3D block, the Transformer unit employs a linear projection consisting of 3×3×3 convolutional layers to increase the number of channels in the feature map. Location embeddings are added to the block embeddings to preserve positional information. The embedded features are fed into multiple Transformer layers, each model consisting of a Multi-head Self-Attention Mechanism (MSAM) and a Multi-Layer Perceptron (MLP). MSAM is a crucial component of the Transformer layers, allowing the model to simultaneously focus on representational information from different subspaces at different locations. Specifically, the input of the Transformer unit is divided into multiple independent parts, each part is processed in parallel using self-attention operations, and then the results of these connections are projected using a linear projection layer. Layer normalization would normally be used before the MSAM and MLP blocks, but these blocks are omitted for simplicity. Therefore, the feature map of the Transformer unit can be expressed as follows:

[0058]

[0059] In the above formula, δ represents the number of features, θ and φ represent the parameters for feature extraction based on CNN and Transformer, respectively, and x' represents the input feature map of Transformer.

[0060] See Figure 5 Furthermore, the PI-RADS multi-scale hierarchical network is divided into a high-level feature fusion module and a classifier. The high-level feature fusion module includes two parts. For the first part, the different features in the Transformer unit are used as input to obtain the first fusion result. For the second part, the first fusion result is used as input to the second part to obtain the second fusion result. The first fusion result and the second fusion result are added to obtain the high-level fusion feature map. The high-level fusion feature map is input to the classifier to obtain the probability of PI-RADS automatic hierarchical classification.

[0061] Specifically, in the first part, features between two adjacent scales are concatenated from top to bottom, transforming the high-level semantic information of small-scale features into large-scale features. The second part takes a fused feature map F of the two scales as input. First, the feature maps at different scales obtained by the encoder are resampled to the original input image size using trilinear interpolation. To reduce computational cost, these feature maps are compressed into four channels using 1×1×1 convolutions, and the compression results at different scales are concatenated into a fused feature map F. Then, global average pooling is applied to each feature map. avg and global max pooling max To obtain global information, the results of the two pooling methods are summed and input into the Sigmoid function to obtain the scale coefficient attention vector. This vector is then multiplied by the input feature F, and the resulting feature map undergoes a 3×3×3 convolution and a 1×1×1 convolution before being input into the Sigmoid function. The resulting feature map is multiplied by the feature map after the 3×3×3 convolution and added to the scale coefficient attention vector and the original feature F to obtain the second part of the output feature. These aggregated multi-scale features are then concatenated and fused into the final attention feature map. The input is then fed into three fully connected layers consisting of 512, 32, and 2 neurons respectively. Finally, the Softmax function outputs the probabilities of the PI-RADS auto-grading.

[0062] In the segmentation part of the multimodal, multi-task learning network, to address the class imbalance problem among the prostate region, PCa region, and background region in the segmentation task, a hybrid loss function is designed for the model. This function consists of a weighted sum of two functions. The first loss function is the Dice loss function, which directly optimizes the segmentation performance evaluation metric.

[0063]

[0064] The second loss function is the Focal loss function:

[0065]

[0066] The total loss function for the segmentation is:

[0067]

[0068] In the formula, p is the predicted probability, g is the ground truth label, n is the number of pixels in the image, and x i For the i-th pixel value, η is the balancing factor of the Focal loss function, λ is the parameter for smoothing the weighting rate, and η is the weighting factor. D η represents the weighting factor of the Dice loss function. F This represents the weighting factor of the Focal loss function.

[0069] The loss function in the PI-RADS multi-scale network is the cross-entropy loss function.

[0070]

[0071] In the formula, K represents the number of datasets, and g k p represents the truth label. k This represents the probability output by the classification model.

[0072] The overall loss function in the bimodal multi-task learning model is:

[0073]

[0074] In the formula, α seg and α cls This indicates an uncertain weight.

[0075] The training process is described below. The training dataset used in this application was collected from a municipal people's hospital, including 98 patients with PCa from 2013 to 2016. All patients underwent mpMRI and were diagnosed with suspected cancer. A GE 3.0 T SIGNA HDX MRI scanner was used, with an 8-channel abdominal phased array coil as the receiving coil; a rectal coil was not used. All patients underwent prostate biopsy and were diagnosed with PCa during this period. Pathological diagnosis was performed by a hospital board-certified pathologist according to the Gleason grading system. The experimental data for the initial diagnosis of PCa in the 98 patients showed an image voxel size of 512×512×22 on T2WI, corresponding to an actual scan field of view (FOV) of 240mm×240mm and a thickness of 4mm. The image voxel size for DWI was 256×256×22, ​​corresponding to an actual scan field of view (FOV) of 400mm×400mm and a thickness of 4mm. Note that the dataset used has passed the ethical review of the relevant hospital and obtained informed consent from the patients.

[0076] This dataset was manually annotated by two senior urologists with 10-20 years of experience in prostate MRI examinations, who also annotated the prostate and tumor regions. Prior to annotation, the two experts met in person for image annotation training. All image annotation work was performed independently by the two experts. The quantitative results of the inter-expert and intra-expert consistency assessments for this dataset are summarized in Table 1, where A... i (i = 1, 2) represents the segmentation result of the 1st expert in the i-th iteration, B i (i = 1, 2) represents the segmentation result of the second expert in the i-th iteration. A 1&2 And B 1&2The intersection of the segmentation results of the first and second experts are represented respectively. The inter-expert differences are calculated by considering the intersection between each expert. The comparison results within and between experts show very high CC, indicating a very high linear correlation between different experts and among the same experts. The average Jaccard (>90%) and AAD (AAD<5%) indicate high consistency between and within experts. Finally, the intersection segmentation label of the two experts is selected as the ground truth label for the final experiment. In addition, the two experts respectively perform PI-RADS rating annotation on all images (PI-RADS≤3 and PI-RADS≥4), as shown in Table 1. The intersection classification label of the two experts is obtained by calculating ACC (see Equation (9)) as the best annotation result.

[0077]

[0078] Table 1

[0079]

[0080] Based on the aforementioned dataset, simulations were conducted to verify the implementation of the bimodal multi-task learning model for prostate tumor image segmentation and grading.

[0081] First, the T2WI images of 98 patients initially diagnosed with PCa had a voxel size of 512×512×22, corresponding to an actual FOV of 240mm×240mm and a thickness of 4mm. The DWI images had a voxel size of 256×256×22, ​​corresponding to an actual FOV of 400mm×400mm and a thickness of 4mm. Affine registration is a commonly used method for depicting tumor volume in T2WI and DWI. Furthermore, sample augmentation was performed on all training data, flipping each prostate T2WI and DWI image from left to right and top to bottom; and rotating the images by 90°, 180°, and 270°; the flipping preserved visual structure. After data augmentation, the sample size was sufficient to train a deep learning model, avoiding overfitting.

[0082] Secondly, during the simulation verification process, five performance evaluation metrics were used to quantitatively assess the prostate and PCa segmentation results and measure model performance, including CC, overlap (OL), HD, DSC, and mean intersection over union (MIoU):

[0083]

[0084] TP, TN, FP, and FN represent TP, TN, FP, and FN respectively, p(x i ) represents pixel x iThe predicted probability, and g(x) i ) indicates the corresponding truth label on the same pixel.

[0085] Five performance evaluation metrics were used to quantitatively evaluate the results of the PI-RADS automatic grading of the prostate, including calculating ACC, precision (PRE), recall, specificity (SPE), and F1 score (F1) to assess performance.

[0086]

[0087] In the network structure designed in this application, the loss function has four very important hyperparameters: φ, λ, and η. D and η F To investigate the impact of different parameter settings on network performance, the network was trained using the controlled variable method with different parameter values. As shown in Table 2, firstly, (λ, η) were controlled... D η F The parameters are set to (0.5, 0.5, 0.5). To explore the impact of increasing or decreasing the weights φ on network performance, φ is set to 0.1, 0.2, and 0.3. The optimal parameter ratio is determined based on the evaluation metrics DSC and HD, and then (λ, η) is controlled. D η F To explore the impact of increasing or decreasing the weights λ on network performance, the parameters (0.2, 0.5, 0.5) were set to 0.5, 1, and 2. The optimal parameter ratio was determined based on the evaluation metrics DSC and HD. Then, (λ, φ) was controlled to be the optimal parameter ratio and kept constant. (η) was then adjusted... D η F The parameter ratios are (0.1, 0.9) and (0.9, 0.1), used to increase or decrease the weights of DCE loss or Focal loss, thereby judging network performance. Table 2 shows the values ​​when φ is set to 0.2, λ is set to 1, and η... D and η F When the values ​​are set to (0.9, 0.1) respectively, the proposed method achieves the lowest HD and highest DSC for prostate and tumor segmentation.

[0088] Table 2

[0089]

[0090] To demonstrate the accuracy of the designed network architecture in prostate and PCa region segmentation, VAM and SPM were added to the input baseline (V-Net), and different modules were aggregated on the baseline. Detailed results are shown in Table 3. Adding the proposed VAM (B+VAM) and SPM (B+SPM) to the baseline significantly improved performance across five evaluation metrics. In particular, the VAM module increased the DSC of prostate segmentation by 5.62% (89.38% vs 83.76%) and tumor segmentation by 9.91% (86.25% vs 76.34%) compared to the baseline. Therefore, the detailed information provided by the combination of standard convolution and dilated convolution in the VAM module plays a crucial role in prostate and PCa segmentation. Furthermore, VAM without dilated convolution (B+VAM_w / o_Dc) and only containing the Transformer module (B+Transformer) were added to the baseline for comparison with the segmentation performance of the designed VAM and Transformer modules. As shown in Table 3, the designed module achieved better segmentation performance in the prostate and PCa regions. The method proposed in this application uses three modules (VAM, SPM, and Transformer) to collaboratively complete the segmentation task. As shown in Table 3, the proposed method further improves segmentation performance compared to adding only one module. Compared to the baseline, the proposed method improved the DSC value of prostate MRI segmentation by 8.52% (92.28% vs 83.76%) and the DSC value of PCa bpMRI segmentation by 12.39% (88.73% vs 76.34%). In the PI-RADS automatic scoring classification task, the proposed method also further improved segmentation performance compared to adding only one module.

[0091] Compared to the baseline, the proposed method in this application improves ACC, PRE, REC, SPE, and F1 in PI-RADS automatic grading by 8.82%, 9.38%, 8.01%, 8.12%, and 8.72%, respectively. Therefore, VAM, SPM, and Transformer proposed in this application all play crucial roles in target region segmentation, and none can be omitted. The improved segmentation performance for tumors is greater than that for prostate segmentation, indicating that the proposed method is beneficial for segmenting regions with irregular shapes, blurred boundaries, non-fixed locations, and small target sizes. The CC values ​​of the proposed method and the ground truth labels are both greater than 0.9 (0.9734 and 0.9458), indicating a high correlation and consistency between the automatic segmentation results for prostate and PCa and the results of expert manual segmentation. The decrease in the mean HD value also supports the superior segmentation performance of the proposed method. For PI-RADS automatic grading tasks (≥3 and ≤4), the addition of each module improves classification performance, demonstrating the effectiveness of the proposed method.

[0092] Table 3

[0093]

[0094]

[0095] The proposed method uses a dual-modal fusion network, with images from different modalities as input to the encoder stage. T2WI and DWI MRI images of the prostate and prostate gland (PCa) are used as input to the constructed network encoder. PCa, influenced by the degree and region of cancerous transformation, exhibits varied characteristics, inconsistent size, and blurred boundaries on both T2WI and DWI, with significantly different degrees of manifestation across different modalities. Figure 6 It can be seen that some PCa features are only significant on T2WI, while others are only significant on DWI. Therefore, it is necessary to consider the image features of both modalities simultaneously to better train the network. DWI scans have good spatial intensity resolution for lesions, especially malignant tumor tissues. T2WI images can better reflect the anatomical structure of organs and tissues, accurately locate lesions, and display morphology. To demonstrate that the proposed dual-modal fusion method is superior to the single-modal network, a comparison was made with the dual-modal fusion method by inputting only T2WI and DWI, while keeping other modules unchanged. Most current methods extract feature maps through two independent branches, and the two branches cannot benefit from each other during feature extraction. This application uses two separate encoder blocks at each layer to extract information, and then the outputs of the two encoder blocks share a subsampling layer to obtain the same spatial features. The 1×1×1 convolutional kernel promotes information interaction between the T2WI and DWI channels, enhancing the network's learning ability.

[0096] In summary, the proposed model's segmentation results on a large number of diverse male pelvic bpMRI datasets for prostate and PCa demonstrate a high degree of consistency with expert manual segmentation results. Furthermore, the proposed algorithm achieves higher segmentation performance for prostate and PCa segmentation and classification compared to existing methods. The proposed multimodal multitasking network also achieves a better ROC curve and a higher AUC value in PI-RADS automatic classification.

[0097] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for prostate tumor image analysis based on a bimodal multi-task learning model, characterized in that, The method includes the following steps: The prostate region image to be segmented is acquired and preprocessed. The prostate region image includes a T2-weighted image and a diffusion-weighted image. A bimodal multi-task learning model is constructed, a training dataset is extracted, and the bimodal multi-task learning model is trained. The bimodal multi-task learning model includes a segmentation part and a classification part. The segmentation part includes an encoder and a decoder. An SPM module is constructed in the encoder to assist in feature extraction. A Transformer unit is constructed in the decoder to enhance global feature representation. In the classification part, a PI-RADS multi-scale classification network is constructed to obtain more refined detail information and output PI-RADS classification results. The preprocessed prostate region image is input into a bimodal multi-task learning model to obtain prostate tumor image segmentation and grading results. The encoder in the segmentation section includes multiple SPM modules. T2WI and DWI feature maps with 16 channels are first input into the first SPM module. The block fusion layer in this SPM module first fuses the T2WI and DWI feature maps into blocks, resulting in a concatenated result of eight feature maps. This concatenated result is then linearly transformed through a regularization layer and two fully connected layers. Finally, the result is processed... The convolutional layer implements the output of the SPM module, and further processes the output of the SPM module. Convolution calculation; For the output of each SPM module, attention features are output using the VAM module. The VAM module contains two sets of convolutional blocks: the first set is a standard convolutional group of different sizes, and the second set is an dilated convolutional group of different dilation rates. The outputs of the VAM module and the SPM module are input into the decoder for sequential concatenation, and the global feature representation is enhanced by the Transformer unit. The PI-RADS multi-scale hierarchical network consists of a high-level feature fusion module and a classifier. The high-level feature fusion module comprises two parts. For the first part, different features from before and after the transformation unit are used as input to obtain a first fusion result. For the second part, the first fusion result is used as input to obtain a second fusion result. The first fusion result and the second fusion result are then added to obtain the high-level fused feature map. High-level fusion feature maps Input the classifier to obtain the probability of PI-RADS automatic classification.

2. The method for prostate tumor image analysis based on a bimodal multi-task learning model according to claim 1, characterized in that, The preprocessing of the prostate region images specifically includes performing affine transformations on the prostate region images and flipping each prostate region image from left to right and from top to bottom, and rotating the images by 90°, 180° and 270°.

3. The method for prostate tumor image analysis based on a bimodal multi-task learning model according to claim 1, characterized in that, After the preprocessed prostate region image is input into the encoder, the encoder first performs a convolution kernel on the preprocessed prostate region image. The initial convolution calculation generates a feature map with 16 channels.

4. The method for prostate tumor image analysis based on a bimodal multi-task learning model according to claim 1, characterized in that, The segmentation part includes two loss functions. The first loss function is the Dice loss function, which is directly aimed at optimizing the segmentation performance evaluation metric. The second loss function is the Focal loss function: The total loss function for the segmentation is: In the formula, To predict probabilities, For truth labels, The number of pixels in the image. For the i-th pixel value, The balancing factor for the Focal loss function. To smoothly adjust the weighting ratio parameters, This represents the weighting factor of the Dice loss function. This represents the weighting factor of the Focal loss function.

5. The method for prostate tumor image analysis based on a bimodal multi-task learning model according to claim 4, characterized in that, The loss function in the PI-RADS multi-scale network is the cross-entropy loss function. In the formula, K represents the number of datasets. Indicates the truth label, This represents the probability output by the classification model.

6. The method for prostate tumor image analysis based on a bimodal multi-task learning model according to claim 5, characterized in that, The overall loss function in the bimodal multi-task learning model is: In the formula, and This indicates an uncertain weight.

Images