A method and system for prostate cancer segmentation and recognition based on missing modality cue learning

Abstract

This invention relates to the field of computer-aided diagnostic technology, specifically to a prostate cancer segmentation and identification method and system based on missing modality cue learning. Utilizing multi-parameter magnetic resonance imaging (MRI) sequences, a segmentation network with an attention mechanism achieves precise localization of the prostate region, improving the signal-to-noise ratio of subsequent analysis. A metric learning scheme based on weakly supervised labels enables the model to learn highly discriminative feature representations directly related to pathological grade, resolving the inconsistency between the learning objective and the final task in traditional self-supervised methods. Adaptive multimodal fusion is achieved through cross-attention and self-attention mechanisms, adaptively adjusting the contribution weights of different multi-parameter MRI sequences according to the specific circumstances of each case, realizing intelligent and interpretable fusion of multi-sequence information. This improves the accuracy and reliability of prostate cancer pathological aggressiveness prediction, providing a decision support tool for clinical diagnosis.

Description

Technical Field

[0001] This invention relates to the field of computer-aided diagnostic technology, specifically a method and system for prostate cancer segmentation and recognition based on missing modality cue learning. Background Technology

[0002] Prostate cancer is one of the most common malignant tumors in men. Clinically, tumor invasiveness is usually evaluated by biopsy or Gleason Grade Group (GGG) in postoperative pathology; however, biopsy is invasive and has sampling errors. Multi-parametric magnetic resonance imaging (mp-MRI) typically includes T2-weighted imaging (T2WI), diffusion-weighted imaging (DWI), and apparent diffusion coefficient mapping (ADC), which can provide complementary information such as prostate gland structure, changes in cell density, and restricted water molecule diffusion, and is an important tool for the imaging assessment of prostate cancer.

[0003] Existing deep learning-based methods for prostate MRI lesion segmentation often assume that all MRI sequences are available simultaneously, or that different modalities of images are used as multi-channel inputs followed by unified feature fusion. In real-world clinical settings, functional sequences such as DWI or ADC may be missing due to differences in scanning protocols, motion artifacts, metallic implants, substandard image quality, scanning time limitations, or equipment limitations in primary hospitals. Directly removing missing modalities significantly reduces usable data; filling missing images with all-zero matrices introduces spurious inputs inconsistent with the true MRI distribution; and using generative models to complete images may generate artifacts inconsistent with the true lesion morphology, reducing clinical interpretability.

[0004] Existing incomplete MRI segmentation schemes typically mitigate the impact of missing modalities through modality masking, feature libraries, gating weights, or adaptive fusion. However, these schemes often focus on a single segmentation task, and the deep semantics corresponding to the missing modalities may not be dynamically reconstructed within the individual patient context. For clinical decision support in prostate cancer, simply providing segmentation results is insufficient to meet the combined needs of lesion localization and invasiveness assessment. Therefore, a technical solution is needed that does not require image-level forgery, can perform learnable semantic compensation for missing modalities in the latent space, and simultaneously outputs lesion segmentation and GGG grading results. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a prostate cancer segmentation and identification method and system based on missing modality cue learning, so as to solve the problems in the background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: This invention discloses a prostate cancer segmentation and recognition method based on missing modality cue learning, which is executed based on a pre-built recognition model and includes the following steps: A multi-parameter magnetic resonance imaging (MRI) sequence of a prostate patient is acquired, and the MRI sequence is preprocessed to obtain a preprocessed image sequence of multiple modalities. The MRI sequence includes image sequences of multiple modalities, and the preprocessing includes registration, cropping, and normalization. Each image in the preprocessed image sequence is segmented to obtain a 3D patch; and the 3D patch is encoded, flattened, and its position information is added based on a pre-built modality-independent encoder to obtain a 1D visual feature token set for each modality; Based on a pre-built missing state detector, modal integrity is determined for one-dimensional visual feature token sets of multiple modalities. When there is a modal missingness in the one-dimensional visual feature token sets of multiple modalities, missing information is injected into the position corresponding to the missing modality to obtain a mixed input sequence. Based on a pre-built cross-attention reconstruction and feature aggregation module, adaptive missing semantic reconstruction and feature aggregation are performed on the hybrid input sequence to obtain a feature sequence that integrates the complete modality; The feature sequence is decoded based on a pre-built dual-task decoder to obtain a lesion segmentation mask and an invasiveness classification probability distribution.

[0007] In one embodiment of this application, it further includes: When there is no modality missing in the one-dimensional visual feature token set of multiple modalities, feature aggregation is performed on the one-dimensional visual feature token set of multiple modalities based on the pre-built cross-attention reconstruction and feature aggregation module to obtain a feature sequence that integrates the complete modality.

[0008] In one embodiment of this application, the multiple modalities include T2WI mode, DWI mode, and ADC mode, wherein preprocessing the multi-parameter magnetic resonance imaging sequence to obtain a preprocessed image sequence of multiple modalities includes: The image sequences of the DWI mode and the image sequences of the ADC mode are registered to the DWI mode space, or the image sequences of the ADC mode and the DWI mode are registered to a unified reference space to obtain the registered multi-parameter magnetic resonance imaging sequence. The registered multi-parameter magnetic resonance imaging sequence is subjected to a unified voxel spacing and matrix size, and cropped with the prostate center point as the base point and a preset prostate bounding box or region of interest as the cropping range to obtain a cropped image sequence. The cropped image sequence is subjected to histogram normalization and normalization to obtain a preprocessed image sequence.

[0009] In one embodiment of this application, the location information is a learnable modality-specific location code, and the missing information is a learnable semantic placeholder.

[0010] In one embodiment of this application, adaptive missing semantic reconstruction and feature aggregation are performed on the hybrid input sequence based on a pre-built cross-attention reconstruction and feature aggregation module to obtain a feature sequence that integrates the complete modality, including: The missing modality information in the mixed input sequence is used as the query vector. And use the visual token of the corresponding valid modality as the key vector. Sum value vector ; Based on the query vector The key vector and the value vector Performing multi-head cross-attention yields a reconstructed feature sequence containing the missing modalities, wherein the mathematical expression for the multi-head cross-attention is:

[0011]

[0012] In the formula, Indicates the first The output of each attention head, Indicates the first The query projection matrix of the size Indicates the first The key projection matrix of the head. Indicates the first The projection matrix of the head's value, Indicates the scaling factor. Represents the normalized exponential function, The final output of multi-head cross-attention, This indicates a splicing operation. For the first The output of each attention head, For the first The output of each attention head, To output the projection matrix; The reconstructed feature sequence is subjected to feature aggregation to obtain a feature sequence that integrates the complete modality.

[0013] In one embodiment of this application, feature aggregation is performed on the reconstructed feature sequence to obtain a feature sequence that integrates the complete modality, including: The reconstructed feature sequence is processed by residual connection, layer normalization and feedforward network to obtain the reconstruction prompt token; Self-attention fusion is performed on the reconstructed prompt token and the token of the valid modality to obtain a feature sequence that incorporates the complete modality.

[0014] In one embodiment of this application, the recognition model includes a modality-independent encoder with multiple modalities, a missing state detector, a cross-attention reconstruction and feature aggregation module, and a dual-task decoder; The output of the modal independent encoder with multiple modalities is connected to the input of the missing state detector, the output of the missing state detector is connected to the cross-attention reconstruction and feature aggregation module, and the output of the cross-attention reconstruction and feature aggregation module is connected to the input of the dual-task decoder.

[0015] In one embodiment of this application, the training process for the recognition model, the location information, and the missing information includes: S1, obtain multi-parameter magnetic resonance imaging sequence samples and labels of multi-parameter magnetic resonance imaging sequence samples from prostate patients, and perform random modal sample discarding and preprocessing on the multi-parameter magnetic resonance imaging sequence samples to obtain preprocessed samples, wherein the labels include lesion segmentation mask labels and invasiveness grading probability distribution labels. S2, the preprocessed samples are segmented to obtain three-dimensional tile samples; and the three-dimensional tile samples of each modality are input into the corresponding modality-independent encoder and the prediction results are obtained from the dual-task decoder; S3, calculate the loss between the label and the prediction result based on the pre-built loss function, and adjust the recognition model, the location information and the missing information based on backpropagation of the loss; S4. Repeat steps S2-S3 until training is complete.

[0016] In one embodiment of this application, the mathematical expression of the loss function is:

[0017]

[0018]

[0019]

[0020]

[0021]

[0022] In the formula, Indicates the total loss. The segmentation loss is calculated by comparing the lesion segmentation mask prediction result with the lesion segmentation mask label. The weighting balancing coefficients for the segmentation loss are... The classification loss is calculated by comparing the predicted probability distribution of aggression level with the label of the aggression level probability distribution. The consistency loss is calculated between the feature sequences of randomly discarded modes and the feature sequences after mode reconstruction. The weighting coefficients for consistency loss are... and All are weighting coefficients. To measure the difference between the lesion segmentation mask prediction result and the lesion segmentation mask label, For the binary cross-entropy loss per voxel, For voxel indexing, Voxel representation The real label of the place Representation element The predicted probability at that location. To prevent the coefficient from being divided by zero, The total number of voxels. Indicates the training batch size. For sample index, The total number of invasiveness levels. For hierarchical indexing, Indicates sample The true rating is , The sample represents the model's prediction. The classification is The probability, For the set of discarded modalities, For the discarded modality index, For the feature normalization function, The feature sequence obtained by aggregating the features of the complete input. The feature sequence is obtained by discarding modal reconstruction and feature aggregation.

[0023] This application also provides a prostate cancer segmentation and recognition system based on missing modality cue learning, including: The acquisition module is used to acquire multi-parameter magnetic resonance imaging sequences of prostate patients and preprocess the multi-parameter magnetic resonance imaging sequences to obtain preprocessed image sequences of multiple modalities. The multi-parameter magnetic resonance imaging sequences include image sequences of multiple modalities, and the preprocessing includes registration, cropping, and normalization. The segmentation and encoding module is used to segment each image in the preprocessed image sequence to obtain three-dimensional tiles; and to encode, flatten, and add position information to the three-dimensional tiles based on a pre-built modality-independent encoder to obtain a one-dimensional visual feature token set for each modality. The integrity judgment module is used to perform modal integrity judgment on one-dimensional visual feature token sets of multiple modalities based on a pre-built missing state detector, and inject missing information into the position corresponding to the missing modality when there is a modal missingness in the one-dimensional visual feature token sets of multiple modalities, so as to obtain a mixed input sequence. The reconstruction and aggregation module is used to perform adaptive missing semantic reconstruction and feature aggregation on the hybrid input sequence based on the pre-built cross-attention reconstruction and feature aggregation module, so as to obtain a feature sequence that integrates the complete modality; The decoding and recognition module is used to decode the feature sequence based on a pre-built dual-task decoder to obtain the lesion segmentation mask and the invasiveness classification probability distribution.

[0024] The beneficial effects of this invention are as follows: This invention provides a prostate cancer segmentation and identification method and system based on missing modality cue learning. Utilizing multi-parameter magnetic resonance imaging sequences and a segmentation network with an attention mechanism, this application achieves precise localization of the prostate region, improving the signal-to-noise ratio of subsequent analysis. Employing a metric learning scheme based on weakly supervised labels, the model learns highly discriminative feature representations directly related to pathological grade, solving the problem of inconsistency between the learning objective and the final task in traditional self-supervised methods. Furthermore, by achieving adaptive multimodal fusion through cross-attention and self-attention mechanisms, the contribution weights of different multi-parameter magnetic resonance imaging sequences can be adaptively adjusted according to the specific circumstances of each case, achieving intelligent and interpretable fusion of multi-sequence information. As a complete end-to-end automated process, this invention significantly improves the accuracy and reliability of prostate cancer pathological aggressiveness prediction, providing a powerful decision support tool for clinical diagnosis. Attached Figure Description

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart illustrating the training and online inference phases in one embodiment of this application; Figure 2 This is a schematic diagram of the image processing link in the preprocessing process of one embodiment of this application; Figure 3 This is a schematic diagram of the image segmentation and feature extraction process in one embodiment of this application; Figure 4 This is a network structure diagram of missing module semantic reconstruction and feature aggregation in one embodiment of this application; Figure 5 This is an example of the output result in one embodiment of this application. Detailed Implementation

[0026] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0027] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the layers related to the present invention and are not drawn according to the actual number, shape and size of the layers in the actual implementation. In the actual implementation, the form, number and proportion of each layer can be arbitrarily changed, and the layer layout may also be more complex.

[0028] Numerous details are explored in the following description to provide a more thorough explanation of embodiments of the invention; however, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details.

[0029] This application uses a pre-built recognition model to perform a prostate cancer segmentation and recognition method based on missing modality cue learning. The recognition model in this application includes a modality-independent encoder of multiple modalities connected in sequence, a missing state detector, a cross-attention reconstruction and feature aggregation module, and a dual-task decoder. Based on the sub-network modules of the aforementioned perception model, the prostate cancer segmentation and recognition method based on missing modality cue learning in this application is divided into a training phase and an online inference phase. Figure 1 This is a flowchart illustrating the training and online inference phases in one embodiment of this application, as shown below. Figure 1 As shown, the steps include: S1: Obtain multi-parameter MRI data containing missing sequence scenes and corresponding precise pathological labels.

[0030] S2: Image segmentation and feature tokenization based on modality-independent encoders.

[0031] S3: Dynamic injection of modality prompts based on missing states.

[0032] S4: Dynamic reconstruction of missing semantics and feature aggregation based on cross-attention mechanism.

[0033] S5: End-to-end cue network supervised training for pathological grading (GGG).

[0034] S6: Non-invasive prediction of prostate cancer invasiveness with modality loss robustness.

[0035] The following section will elaborate on the specific implementation methods and principles of the above process: Training Phase 1 (1) Obtain multi-parameter magnetic resonance imaging sequence samples and labels of multi-parameter magnetic resonance imaging sequence samples from prostate patients, and perform random modal sample discarding and preprocessing on the multi-parameter magnetic resonance imaging sequence samples to obtain preprocessed samples, wherein the labels include disease segmentation mask labels and invasiveness grading probability distribution labels. Specifically, the multi-parameter magnetic resonance imaging sequence samples in this application include T2-weighted imaging (T2WI) sequence samples, diffusion-weighted imaging (DWI) sequence samples, and apparent diffusion coefficient map (ADC) sequence samples.

[0036] This application obtains multi-parameter MRI data containing missing sequence sample scenes by discarding image sequence samples from one of the modalities (such as DWI modality).

[0037] The preprocessing process includes: (1-1) Register the image sequence samples of the DWI mode and the image sequence samples of the ADC mode to the DWI mode space, or register the image sequence samples of the ADC mode and the DWI mode to a unified reference space to obtain the registered multi-parameter magnetic resonance imaging sequence samples; (1-2) The registered multi-parameter magnetic resonance imaging sequence samples are uniformly voxel spacing and matrix size, and cropped with the prostate center point as the base point and the preset prostate bounding box or region of interest as the cropping range to obtain cropped image sequence samples. (1-3) Perform histogram standardization and normalization on the cropped image sequence samples to obtain preprocessed image sequence samples.

[0038] In the above preprocessing process, DWI and ADC are first registered to T2WI space, or all sequence samples are registered to a unified reference space; then the voxel spacing and matrix size are unified. For each sequence sample, a center crop is performed with the prostate center point, prostate bounding box, or preset ROI as the center. The cropping size can be set to, for example, H×W×D=160×160×32, or it can be adjusted according to the device protocol.

[0039] Histogram normalization is performed on the cropped image to make the intensity distribution comparable under different scanning devices and parameters; then, the non-background areas are... The voxel intensities within the modulus are normalized using z-score. Let the cropped image for modality m be... Its non-background mean is Standard deviation is The normalized image is then:

[0040] in, To prevent extremely small constants with a denominator of zero, the above processing reduces intensity shifts caused by different hospitals, scanners, and protocols, making subsequent encoders more likely to converge. Figure 2 This is a schematic diagram of the image processing chain in the preprocessing flow of one embodiment of this application. Figure 2 Examples of the processing chain for T2WI, DWI, and ADC after original image, center cropping, histogram normalization, and z-score normalization are given.

[0041] (2) The preprocessed samples are segmented to obtain three-dimensional tile samples; and the three-dimensional tile samples of each modality are input into the corresponding modality independent encoder, and the prediction results are obtained from the dual-task decoder; Figure 3 This is a schematic diagram of the image segmentation and feature extraction process in one embodiment of this application, as shown below. Figure 3 As shown, each effective modal image after preprocessing is segmented into non-overlapping 3D patches. Let the normalized image size be... The image patch size is Then the number of image blocks corresponding to each modality is .

[0042] After segmentation, the tiles are input into the modal independent encoder for each modality and run within the model. The specific process is as follows: (2-1) The three-dimensional tiles are encoded, flattened and position information is added based on the pre-built modal independent encoder to obtain a one-dimensional visual feature token set for each modality; Specifically, modality-independent 3D lightweight CNN encoders E_T2, E_DWI, and E_ADC are set for T2WI, DWI, and ADC, respectively. Alternatively, some weights can be shared in the lower convolutional layers while modality-independent parameters are retained in the higher layers. Local spatial features are extracted from each image patch by the encoder, flattened into a d-dimensional feature vector, and added to the modality-specific location encoding to obtain a visual token sequence sample.

[0043] in, Let represent the visual token corresponding to the i-th image block of modality m. For global hierarchical recognition, a global class token CLS is introduced at the beginning of the sequence samples. For missing modalities, this invention does not input the all-zero matrix into the CNN encoder, but skips the image encoding process for that modality and denotes its visual token sequence samples as an empty set. This is to avoid zero-padding disrupting the true data distribution.

[0044] (2-2) Based on the pre-built missing state detector, modal integrity is judged for one-dimensional visual feature token sets of multiple modalities. When there is a modal missing in the one-dimensional visual feature token set of multiple modalities, the missing prompt information is injected into the position corresponding to the missing modality to obtain mixed input sequence samples. This application determines the presence of T2WI, DWI, and ADC by constructing a missing state detector. The detection results are expressed as follows: ,in This indicates that the mode exists. This indicates a missing modality. For example, when DWI is missing but T2WI and ADC are present, .

[0045] Three sets of learnable modal cues are pre-set in the network parameters. , and Each set of prompts is composed of... It consists of d-dimensional continuous vectors, which can be initialized using random normal initialization or truncated normal initialization. The prompt is not an image pixel, nor a fake MRI image, but a learnable semantic placeholder optimized in a deep feature space.

[0046] When the sequence sample is complete, the input sequence sample can be written as:

[0047] in, Encoding spatial location, Modal encoding. When DWI is missing, P_DWI is injected into the original spatial token position of DWI, forming:

[0048] If two modalities are missing simultaneously, corresponding modality prompts are injected for each. Since the prompts retain the modality index and spatial index of the missing modality in the sequence samples, the Transformer can subsequently distinguish between the different states of "the position is missing but semantic reconstruction is needed" and "the position is invalid".

[0049] (2-3) When there is no modality missing in the one-dimensional visual feature token set of multiple modalities, do not perform dynamic injection of modality prompts based on missing states, and directly use the one-dimensional visual feature token set of multiple modalities to perform subsequent steps; (2-4) Based on the pre-built cross-attention reconstruction and feature aggregation module, adaptive missing semantic reconstruction and feature aggregation are performed on the mixed input sequence samples to obtain feature sequence samples that integrate the complete modality; Figure 4This is a network structure diagram of missing module semantic reconstruction and feature aggregation in one embodiment of this application, such as... Figure 4 As shown, the mixed token sequence samples are input into a cross-attention reconstruction and feature aggregation module consisting of L layers of Transformers. In the case of missing modalities (CLS Tokens, T2WI Tokens, DWI Prompt Tokens (learnable), ADC Tokens), the missing modal cue is used as the query vector Q, and the valid modal visual tokens are used as the key vector K and value vector V, respectively, to perform multi-head cross-attention.

[0050]

[0051] In the formula, Indicates the first The output of each attention head, Indicates the first The query projection matrix of the size Indicates the first The key projection matrix of the head. Indicates the first The projection matrix of the head's value, Indicates the scaling factor. Represents the normalized exponential function, The final output of multi-head cross-attention, This indicates a splicing operation. For the first The output of each attention head, For the first The output of each attention head, To output the projection matrix; For complete sequences, CLS Tokens, T2WI Tokens, DWI Tokens, and ADC Tokens are directly input into the feature aggregation stage without cross-attention reconstruction.

[0052] After cross-attention, an updated token is obtained through residual connections, LayerNorm, and a feedforward network. Then, self-attention fusion is performed on the effective modality token and the reconstructed cue token to output the updated CLS token and spatial token. This structure can adaptively reconstruct missing semantics based on the effective modality context for each patient, rather than using fixed-mean features or simple weight gating.

[0053] (2-5) The feature sequence samples are decoded based on the pre-built dual-task decoder to obtain the prediction results, which include the lesion segmentation mask and the invasiveness grade probability distribution; The dual-task decoder in this application includes a pathological grading branch and a lesion segmentation branch.

[0054] Pathological grading branch extracts CLS tokens, inputs them into a multilayer perceptron, and outputs... ,in This indicates the probability that the patient belongs to the GGGk group.

[0055] The lesion segmentation branch reassembles spatial tokens into a 3D feature map according to the spatial index of image blocks. This map is then upsampled, transposed convolutionally, or a convolutional decoder is used to progressively restore it to the input image size. To preserve fine-grained boundary information, shallow features from the modality encoder are concatenated with the decoded features via skip connections during the decoding process. Finally, a lesion segmentation probability map is output via Sigmoid or Softmax, and a lesion segmentation mask is determined based on a threshold or the maximum class.

[0056] Figure 4 A specific medical imaging scenario is illustrated: the model receives T2WI, ADC, and DWI images of the same patient, overlays a ground truth mask and predicted segmentation results on the T2WI image, and simultaneously outputs probability distributions from GGG1 to GGG5. This output format can simultaneously support lesion localization, volume estimation, and invasiveness grading assistance.

[0057] (3) Calculate the loss between the label and the prediction result based on the pre-constructed loss function, and adjust the recognition model, the location information and the missing information based on the backpropagation of the loss; The mathematical expression for the loss function is:

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] In the formula, Indicates the total loss. The segmentation loss is calculated by comparing the lesion segmentation mask prediction result with the lesion segmentation mask label. The weighting balancing coefficients for the segmentation loss are... The classification loss is calculated by comparing the predicted probability distribution of aggression level with the label of the aggression level probability distribution. The consistency loss is calculated between the feature sequences of randomly discarded modes and the feature sequences after mode reconstruction. The weighting coefficients for consistency loss are... and All are weighting coefficients. To measure the difference between the lesion segmentation mask prediction result and the lesion segmentation mask label, For the binary cross-entropy loss per voxel, For voxel indexing, Voxel representation The real label of the place Representation element The predicted probability at that location. To prevent the coefficient from being divided by zero, The total number of voxels. Indicates the training batch size. For sample index, The total number of invasiveness levels. For hierarchical indexing, Indicates sample The true rating is , The sample represents the model's prediction. The classification is The probability, For the set of discarded modalities, For the discarded modality index, For the feature normalization function, The feature sequence obtained by aggregating the features of the complete input. The feature sequence is obtained by discarding modal reconstruction and feature aggregation.

[0064] (4) Repeat steps S2-S3 until training is complete.

[0065] The training process includes: initializing the modal encoder, Transformer, dual-task decoder, and cue parameters; reading training samples and performing preprocessing; randomly simulating or reading real missing states; performing cue injection and forward propagation; calculating the joint loss; updating all network parameters and cue parameters using the Adam or AdamW optimizer; and repeating the iteration until the validation set loss converges.

[0066] II. Online Reasoning Stage In this application, after model training is completed, the model is deployed locally and online inference is performed. The online inference process is essentially the same as the model training phase, specifically including: (1) Obtain multi-parameter magnetic resonance imaging sequences of prostate patients and preprocess the multi-parameter magnetic resonance imaging sequences to obtain preprocessed image sequences of multiple modalities, wherein the multi-parameter magnetic resonance imaging sequences include image sequences of multiple modalities, and the preprocessing includes registration, cropping and standardization. In one embodiment of this application, the preprocessing process is consistent with the preprocessing principle in the training process, including: (1-1) Register the image sequence of the DWI mode and the image sequence of the ADC mode to the DWI mode space, or register the image sequences of the ADC mode and the DWI mode to a unified reference space to obtain the registered multi-parameter magnetic resonance imaging sequence; (1-2) The registered multi-parameter magnetic resonance imaging sequence is uniformly voxel spacing and matrix size, and cropped with the prostate center point as the base point and the preset prostate bounding box or region of interest as the cropping range to obtain the cropped image sequence. (1-3) Perform histogram standardization and normalization on the cropped image sequence to obtain the preprocessed image sequence.

[0067] (2) Segment each image in the preprocessed image sequence to obtain three-dimensional tiles; and encode, flatten and add position information to the three-dimensional tiles based on the pre-built modality independent encoder to obtain a one-dimensional visual feature token set for each modality; (3) Based on the pre-built missing state detector, the modality integrity of the one-dimensional visual feature token set of multiple modalities is judged. When there is a modality missing in the one-dimensional visual feature token set of multiple modalities, the missing prompt information is injected into the position corresponding to the missing modality to obtain the mixed input sequence. When there is no modality missing in the one-dimensional visual feature token set of multiple modalities, the feature aggregation of the one-dimensional visual feature token set of multiple modalities is performed based on the pre-built cross-attention reconstruction and feature aggregation module (no reconstruction is required) to obtain the feature sequence that integrates the complete modality.

[0068] (4) Based on the pre-built cross-attention reconstruction and feature aggregation module, adaptive missing semantic reconstruction and feature aggregation are performed on the hybrid input sequence to obtain a feature sequence that integrates the complete modality, including: (4-1) Use the missing modality missing information in the mixed input sequence as the query vector. And use the visual token of the corresponding valid modality as the key vector. Sum value vector ; (4-2) Based on the query vector The key vector and the value vector Performing multi-head cross-attention yields a reconstructed feature sequence containing the missing modalities, wherein the mathematical expression for the multi-head cross-attention is:

[0069]

[0070] In the formula, Indicates the first The output of each attention head, Indicates the first The query projection matrix of the size Indicates the first The key projection matrix of the head. Indicates the first The projection matrix of the head's value, Indicates the scaling factor. Represents the normalized exponential function, The final output of multi-head cross-attention, This indicates a splicing operation. For the first The output of each attention head, For the first The output of each attention head, To output the projection matrix; (4-3) Perform residual connection, layer normalization and feedforward network processing on the reconstructed feature sequence to obtain the reconstruction prompt token; (4-4) Perform self-attention fusion on the reconstructed prompt token and the token of the valid modality to obtain a feature sequence that fuses the complete modality.

[0071] (5) The feature sequence is decoded based on the pre-built dual-task decoder to obtain the lesion segmentation mask and the invasiveness classification probability distribution.

[0072] Figure 5 Here is an example of the output result in one embodiment of this application. The result output by the dual-task decoder in this application is as follows: Figure 5 As shown.

[0073] In actual deployment, the system first reads the T2WI, DWI, and ADC files of the patient to be tested, and checks whether the files exist, whether the image quality is acceptable, and the registration status with the reference space. For existing sequences, preprocessing and tokenization are performed in the same manner as in the training phase; for missing or substandard sequences, the system marks them as missing and calls the corresponding modality cue that has been trained.

[0074] The system outputs three types of results after one forward propagation: (1) a three-dimensional segmentation mask and overlay visualization of prostate cancer lesions; (2) the probability distribution of GGG1 to GGG5 and the maximum probability category; and (3) sequence missing status and model confidence information. Doctors can make a comprehensive judgment by combining the lesion location, segmentation volume, DWI / ADC signal performance and GGG prediction results.

[0075] The method of this invention does not require all patients to have complete mp-MRI sequences, thus it can better adapt to situations where real clinical data is incomplete; at the same time, since it does not generate pseudo-MRI sequences at the image level, the interpretability of the results and the data processing chain are clearer.

[0076] This invention discloses a prostate cancer segmentation and identification method based on missing modality cue learning. Utilizing multi-parameter magnetic resonance imaging (MRI) sequences and a segmentation network with an attention mechanism, this method achieves precise localization of the prostate region, improving the signal-to-noise ratio of subsequent analysis. Employing a metric learning scheme based on weakly supervised labels, the model learns highly discriminative feature representations directly related to pathological severity, resolving the inconsistency between the learning objective and the final task in traditional self-supervised methods. Furthermore, by achieving adaptive multimodal fusion through cross-attention and self-attention mechanisms, the contribution weights of different multi-parameter MRI sequences can be adaptively adjusted according to the specific circumstances of each case, achieving intelligent and interpretable fusion of multi-sequence information. As a complete end-to-end automated process, this invention significantly improves the accuracy and reliability of prostate cancer pathological aggressiveness prediction, providing a powerful decision support tool for clinical diagnosis.

[0077] This application also provides a prostate cancer segmentation and recognition system based on missing modality cue learning, including: The acquisition module is used to acquire multi-parameter magnetic resonance imaging sequences of prostate patients and preprocess the multi-parameter magnetic resonance imaging sequences to obtain preprocessed image sequences of multiple modalities. The multi-parameter magnetic resonance imaging sequences include image sequences of multiple modalities, and the preprocessing includes registration, cropping, and normalization. The segmentation and encoding module is used to segment each image in the preprocessed image sequence to obtain three-dimensional tiles; and to encode, flatten, and add position information to the three-dimensional tiles based on a pre-built modality-independent encoder to obtain a one-dimensional visual feature token set for each modality. The integrity judgment module is used to perform modal integrity judgment on one-dimensional visual feature token sets of multiple modalities based on a pre-built missing state detector, and inject missing information into the position corresponding to the missing modality when there is a modal missingness in the one-dimensional visual feature token sets of multiple modalities, so as to obtain a mixed input sequence. The reconstruction and aggregation module is used to perform adaptive missing semantic reconstruction and feature aggregation on the hybrid input sequence based on the pre-built cross-attention reconstruction and feature aggregation module, so as to obtain a feature sequence that integrates the complete modality; The decoding and recognition module is used to decode the feature sequence based on a pre-built dual-task decoder to obtain the lesion segmentation mask and the invasiveness classification probability distribution.

[0078] This invention discloses a prostate cancer segmentation and recognition system based on missing modality cue learning. Utilizing multi-parameter magnetic resonance imaging (MRI) sequences and a segmentation network with an attention mechanism, it achieves precise localization of the prostate region, improving the signal-to-noise ratio of subsequent analysis. Employing a metric learning scheme based on weakly supervised labels, the model learns highly discriminative feature representations directly related to pathological severity, resolving the inconsistency between the learning objective and the final task in traditional self-supervised methods. Furthermore, through cross-attention and self-attention mechanisms, adaptive multimodal fusion is achieved, adaptively adjusting the contribution weights of different multi-parameter MRI sequences according to the specific circumstances of each case, realizing intelligent and interpretable fusion of multi-sequence information. As a complete end-to-end automated process, this invention significantly improves the accuracy and reliability of prostate cancer pathological aggressiveness prediction, providing a powerful decision support tool for clinical diagnosis.

[0079] As will be understood by those skilled in the art, the computer-readable storage medium described in this embodiment allows for the implementation of all or part of the steps in the above method embodiments by computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0080] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic terminal performs the steps of the above method.

[0081] In this embodiment, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0082] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0083] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. The embodiments of the invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims.

[0084] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A prostate cancer segmentation and recognition method based on missing modality cue learning, characterized in that, Execution based on a pre-built recognition model, including the following steps: A multi-parameter magnetic resonance imaging (MRI) sequence of a prostate patient is acquired, and the MRI sequence is preprocessed to obtain a preprocessed image sequence of multiple modalities. The MRI sequence includes image sequences of multiple modalities, and the preprocessing includes registration, cropping, and normalization. Each image in the preprocessed image sequence is segmented to obtain a 3D patch; and the 3D patch is encoded, flattened, and its position information is added based on a pre-built modality-independent encoder to obtain a 1D visual feature token set for each modality; Based on a pre-built missing state detector, modal integrity is determined for one-dimensional visual feature token sets of multiple modalities. When there is a modal missingness in the one-dimensional visual feature token sets of multiple modalities, missing information is injected into the position corresponding to the missing modality to obtain a mixed input sequence. Based on a pre-built cross-attention reconstruction and feature aggregation module, adaptive missing semantic reconstruction and feature aggregation are performed on the hybrid input sequence to obtain a feature sequence that integrates the complete modality; The feature sequence is decoded based on a pre-built dual-task decoder to obtain a lesion segmentation mask and an invasiveness classification probability distribution.

2. The prostate cancer segmentation and recognition method based on missing modality cue learning according to claim 1, characterized in that, Also includes: When there is no modality missing in the one-dimensional visual feature token set of multiple modalities, feature aggregation is performed on the one-dimensional visual feature token set of multiple modalities based on the pre-built cross-attention reconstruction and feature aggregation module to obtain a feature sequence that integrates the complete modality.

3. The prostate cancer segmentation and recognition method based on missing modality cue learning according to claim 1, characterized in that, The multiple modalities include T2WI, DWI, and ADC modes. The multi-parameter magnetic resonance imaging sequence is preprocessed to obtain preprocessed image sequences of multiple modalities, including: The image sequences of the DWI mode and the image sequences of the ADC mode are registered to the DWI mode space, or the image sequences of the ADC mode and the DWI mode are registered to a unified reference space to obtain the registered multi-parameter magnetic resonance imaging sequence. The registered multi-parameter magnetic resonance imaging sequence is subjected to a unified voxel spacing and matrix size, and cropped with the prostate center point as the base point and a preset prostate bounding box or region of interest as the cropping range to obtain a cropped image sequence. The cropped image sequence is subjected to histogram normalization and normalization to obtain a preprocessed image sequence.

4. The prostate cancer segmentation and recognition method based on missing modality cue learning according to claim 1, characterized in that, The location information is a learnable modality-specific location code, and the missing information is a learnable semantic placeholder.

5. The prostate cancer segmentation and recognition method based on missing modality cue learning according to claim 1, characterized in that, Based on a pre-built cross-attention reconstruction and feature aggregation module, adaptive missing semantic reconstruction and feature aggregation are performed on the hybrid input sequence to obtain a feature sequence that integrates the complete modality, including: The missing modality information in the mixed input sequence is used as the query vector. And use the visual token of the corresponding valid modality as the key vector. Sum value vector ; Based on the query vector The key vector and the value vector Performing multi-head cross-attention yields a reconstructed feature sequence containing the missing modalities, wherein the mathematical expression for the multi-head cross-attention is: In the formula, Indicates the first The output of each attention head, Indicates the first The query projection matrix of the size Indicates the first The key projection matrix of the head. Indicates the first The projection matrix of the head's value, Indicates the scaling factor. Represents the normalized exponential function, The final output of multi-head cross-attention, This indicates a splicing operation. For the first The output of each attention head, For the first The output of each attention head, To output the projection matrix; The reconstructed feature sequence is subjected to feature aggregation to obtain a feature sequence that integrates the complete modality.

6. The prostate cancer segmentation and recognition method based on missing modality cue learning according to claim 5, characterized in that, The reconstructed feature sequence is subjected to feature aggregation to obtain a feature sequence that integrates the complete modality, including: The reconstructed feature sequence is processed by residual connection, layer normalization and feedforward network to obtain the reconstruction prompt token; Self-attention fusion is performed on the reconstructed prompt token and the token of the valid modality to obtain a feature sequence that incorporates the complete modality.

7. The prostate cancer segmentation and recognition method based on missing modality cue learning according to claim 1, characterized in that, The recognition model includes a modality-independent encoder with multiple modalities, a missing state detector, a cross-attention reconstruction and feature aggregation module, and a dual-task decoder; The output of the modal independent encoder with multiple modalities is connected to the input of the missing state detector, the output of the missing state detector is connected to the cross-attention reconstruction and feature aggregation module, and the output of the cross-attention reconstruction and feature aggregation module is connected to the input of the dual-task decoder.

8. The prostate cancer segmentation and recognition method based on missing modality cue learning according to claim 7, characterized in that, The training process for the recognition model, the location information, and the missing information includes: S1, obtain multi-parameter magnetic resonance imaging sequence samples and labels of multi-parameter magnetic resonance imaging sequence samples from prostate patients, and perform random modal sample discarding and preprocessing on the multi-parameter magnetic resonance imaging sequence samples to obtain preprocessed samples, wherein the labels include lesion segmentation mask labels and invasiveness grading probability distribution labels. S2, the preprocessed samples are segmented to obtain three-dimensional tile samples; and the three-dimensional tile samples of each modality are input into the corresponding modality-independent encoder and the prediction results are obtained from the dual-task decoder; S3, calculate the loss between the label and the prediction result based on the pre-built loss function, and adjust the recognition model, the location information and the missing information based on backpropagation of the loss; S4. Repeat steps S2-S3 until training is complete.

9. A prostate cancer segmentation and recognition method based on missing modality cue learning according to claim 8, characterized in that, The mathematical expression for the loss function is: In the formula, Indicates the total loss. The segmentation loss is calculated by comparing the lesion segmentation mask prediction result with the lesion segmentation mask label. The weighting balancing coefficients for the segmentation loss are... The classification loss is calculated by comparing the predicted probability distribution of aggression level with the label of the aggression level probability distribution. The consistency loss is calculated between the feature sequences of randomly discarded modes and the feature sequences after mode reconstruction. The weighting coefficients for consistency loss are... and All are weighting coefficients. To measure the difference between the lesion segmentation mask prediction result and the lesion segmentation mask label, For the binary cross-entropy loss per voxel, For voxel indexing, Voxel representation The real label of the place Representation element The predicted probability at that location. To prevent the coefficient from being divided by zero, The total number of voxels. Indicates the training batch size. For sample index, The total number of invasiveness levels. For hierarchical indexing, Indicates sample The true rating is , The sample represents the model's prediction. The classification is The probability, For the set of discarded modalities, For the discarded modality index, For the feature normalization function, The feature sequence obtained by aggregating the features of the complete input. The feature sequence is obtained by discarding modal reconstruction and feature aggregation.

10. A prostate cancer segmentation and recognition system based on missing modality cue learning, characterized in that, include: The acquisition module is used to acquire multi-parameter magnetic resonance imaging sequences of prostate patients and preprocess the multi-parameter magnetic resonance imaging sequences to obtain preprocessed image sequences of multiple modalities. The multi-parameter magnetic resonance imaging sequences include image sequences of multiple modalities, and the preprocessing includes registration, cropping, and normalization. The segmentation and encoding module is used to segment each image in the preprocessed image sequence to obtain three-dimensional tiles; and to encode, flatten, and add position information to the three-dimensional tiles based on a pre-built modality-independent encoder to obtain a one-dimensional visual feature token set for each modality. The integrity judgment module is used to perform modal integrity judgment on one-dimensional visual feature token sets of multiple modalities based on a pre-built missing state detector, and inject missing information into the position corresponding to the missing modality when there is a modal missingness in the one-dimensional visual feature token sets of multiple modalities, so as to obtain a mixed input sequence. The reconstruction and aggregation module is used to perform adaptive missing semantic reconstruction and feature aggregation on the hybrid input sequence based on the pre-built cross-attention reconstruction and feature aggregation module, so as to obtain a feature sequence that integrates the complete modality; The decoding and recognition module is used to decode the feature sequence based on a pre-built dual-task decoder to obtain the lesion segmentation mask and the invasiveness classification probability distribution.

Images